Import from NFS

The import nfs command starts a server-side import process from the network file system ( Network File System, NFS) of the YDB cluster hosts, importing data and schema object information in the format described in the File structure article:

ydb [connection options] import nfs [options]

where [connection options] are database connection options

Unlike the tools restore command, the import nfs command always creates objects entirely, so for it to succeed, none of the imported objects (neither directories nor tables) must exist.

If you need to load additional data into existing tables, use the tools restore command directly on the mounted NFS directory.

Command line parameters

[options] — command parameters:

NFS parameters

The import from NFS command requires specifying a mounted directory (or subdirectory) common to all objects involved in the import. Since the import is performed asynchronously on all YDB hosts, the specified directory must be present on each YDB host and mounted in NFS.

--fs-path PATH: path to the mounted directory (or subdirectory).

Imported database schema objects

--destination-path PATH: Target directory for imported objects; the default value is the database root.

--include PATH: Data schema objects to include in the import. Directories are traversed recursively. To include multiple objects, the parameter can be specified multiple times. If not specified, all exported objects are imported.

--exclude STRING: Pattern ( PCRE) to exclude paths from the import. This parameter can be specified multiple times for different patterns.

Additional parameters

Running the import

Server import operation flow

1. A server asynchronous import operation is created.
2. The server reads object metadata from files (scheme.pb, metadata.json, etc.) from the storage.
3. For each object, a new table is created in the database. Target paths must not exist — import cannot overwrite existing tables.
4. Data is loaded from files in parallel on all cluster nodes.

Launch result

Upon successful execution, the import nfs command outputs summary information about the queued import from NFS operation, in the format specified by the --format option. The actual import is performed asynchronously by the server. The summary information includes the operation ID, which can later be used to check the status and perform actions on the operation:

• In the pretty output mode (default), the operation ID is shown in the id field highlighted with pseudographics:

┌───────────────────────────────────────────┬───────┬─────...
| id                                        | ready | stat...
├───────────────────────────────────────────┼───────┼─────...
| ydb://import/8?id=281474976788395&kind=fs | true  | SUCC...
├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴...
| Items:
...

• In the proto-json-base64 output mode, the identifier is in the "id" attribute:

{"id":"ydb://import/8?id=281474976788395&kind=fs","ready":true, ... }

Import status

Data loading is performed in the background. You can get information about the status and progress of the loading by calling the operation get command, whose parameter must be the quoted operation identifier, for example:

ydb -p quickstart operation get "ydb://import/8?id=281474976788395&kind=fs"

The output format operation get is also set by the --format option.

Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string.

The completion of the upload is tracked by the change of the "progress" attribute:

• In the pretty output mode (default), a successfully completed operation is reflected by the value "Done" in the progress field highlighted with pseudographics:

  ┌───── ... ──┬───────┬─────────┬──────────┬─...
  | id         | ready | status  | progress | ...
  ├──────... ──┼───────┼─────────┼──────────┼─...
  | ydb://...   | true  | SUCCESS | Done     | ...
  ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴...
  ...
  

• In the proto-json-base64 output mode, a completed operation is reflected by the value PROGRESS_DONE of the progress attribute:

  {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... }
  

Completing an import operation

After completing the import, use the operation forget command to remove the import from the list of operations:

ydb -p quickstart operation forget "ydb://import/8?id=281474976788395&kind=fs"

List of import operations

To get a list of import operations, use the operation list import/nfs command:

ydb -p quickstart operation list import/nfs

The output format operation list is also set by the --format option.

Examples

Importing to the database root

Importing the contents of the /mnt/nfs/backups/export1 directory on the file system into the database root:

ydb -p quickstart import nfs \
  --fs-path /mnt/nfs/backups/export1

Importing multiple directories

Importing objects from the dir1 and dir2 directories of the export located in /mnt/nfs/backups/export1 on the file system into the database directories with the same names:

ydb -p quickstart import nfs \
  --fs-path /mnt/nfs/backups/export1 \
  --include dir1 --include dir2

Importing an encrypted export

Importing a single table that was exported along the dir/my_table path into the dir1/dir/my_table path from an encrypted export located in /mnt/nfs/backups/export1 on the file system, using a secret key from the ~/my_secret_key file.

ydb -p quickstart import nfs \
  --fs-path /mnt/nfs/backups/export1 --destination-path dir1 \
  --include dir/my_table \
  --encryption-key-file ~/my_secret_key

Getting operation IDs

To get a list of import operation IDs in a format convenient for processing in bash scripts, you can use the jq utility:

ydb -p quickstart operation list import/nfs --format proto-json-base64 | jq -r ".operations[].id"

You will get output where each new line contains an operation ID, for example:

ydb://import/8?id=281474976789577&kind=fs
ydb://import/8?id=281474976789526&kind=fs
ydb://import/8?id=281474976788779&kind=fs

Using these IDs, you can, for example, run a loop to terminate all current operations:

ydb -p quickstart operation list import/nfs --format proto-json-base64 | jq -r ".operations[].id" | while read line; do ydb -p quickstart operation forget $line;done