Import from NFS

The import nfs command starts a server-side process of importing data and schema object information from the network file system ( Network File System, NFS) of the YDB cluster hosts, in the format described in the File structure article:

ydb [connection options] import nfs [options]

where [connection options] are database connection options

Unlike the tools restore command, the import nfs command always creates objects entirely, so for it to succeed, none of the imported objects (neither directories nor tables) should exist.

If you need to additionally load data into existing tables, use the tools restore command directly on the mounted NFS directory.

Command-line parameters

[options] — command parameters:

NFS parameters

The import from NFS command requires specifying a mounted directory (or subdirectory) common to all objects involved in the import. Since the import is performed asynchronously on all YDB hosts, the specified directory must exist on each YDB host and be mounted in NFS.

--fs-path PATH: path to the mounted directory (or subdirectory).

Imported database schema objects

--destination-path PATH: Target directory for imported objects; the default value is the database root.

--include PATH: Data schema objects to include in the import. Directories are traversed recursively. To include multiple objects, the parameter can be specified multiple times. If not specified, all exported objects are loaded.

--exclude STRING: Pattern ( PCRE) for excluding paths from the import. This parameter can be specified multiple times for different patterns.

Alternative method

For backward compatibility, an alternative way to specify the list of objects is supported:

--item STRING: Description of the object to import. The --item parameter can be specified multiple times if you need to import several objects. If the --item or --include parameters are not specified, all objects present in the specified export will be imported. STRING is specified in the <property>=<value>,... format with the following required properties:

source, src, or s — path in NFS (relative to fs-path) with the imported directory or table.
destination, dst, or d — the path in the database for placing the imported directory or table. The final path element must not exist. All directories along the path will be created if they do not exist.

Some features may be unavailable when using alternative syntax (in particular, encrypted backups or listing of export objects).

Additional parameters

--retries NUM: Number of upload retries the server will attempt. Default value: 10.
--skip-checksum-validation: Skip the validation stage of checksums of the uploaded objects.
--encryption-key-file PATH: Path to the file containing the encryption key (only for encrypted exports). This file is binary and must contain the exact number of bytes corresponding to the key length in the selected encryption algorithm (16 bytes for AES-128-GCM, 32 bytes for AES-256-GCM and ChaCha20-Poly1305). The key can also be passed via the YDB_ENCRYPTION_KEY environment variable, in hexadecimal string representation.
--format STRING: Output result format. Valid values:
- pretty — human-readable format (default).
- proto-json-base64 — Protocol Buffers in JSON format, binary strings encoded in Base64.

Running the import

Progress of server import operation

A server asynchronous import operation is created.
The server reads object metadata from files (scheme.pb, metadata.json, etc.) from the storage.
For each object, a new table is created in the database. Target paths must not exist — import cannot overwrite existing tables.
Data is loaded from files in parallel on all nodes of the cluster.

Launch result

Upon successful execution, the import nfs command outputs summary information about the queued import from NFS operation, in the format specified by the --format option. The actual import is performed asynchronously by the server. The summary information includes the operation ID, which can later be used to check the status and perform actions on the operation:

In the pretty output mode (default), the operation ID is shown in the id field highlighted with pseudographics:

┌───────────────────────────────────────────┬───────┬─────...
| id                                        | ready | stat...
├───────────────────────────────────────────┼───────┼─────...
| ydb://import/8?id=281474976788395&kind=fs | true  | SUCC...
├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴...
| Items:
...

In the proto-json-base64 output mode, the identifier is in the "id" attribute:

{"id":"ydb://import/8?id=281474976788395&kind=fs","ready":true, ... }

Import status

Data loading is performed in the background. You can get information about the status and progress of the loading by calling the operation get command, which must be passed a quoted operation ID, for example:

ydb -p quickstart operation get "ydb://import/8?id=281474976788395&kind=fs"

The output format operation get is also set by the option --format.

Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string.

Completion of the upload is tracked by the change of the "progress" attribute:

In the pretty output mode (default), a successfully completed operation is reflected by the value "Done" in the progress field highlighted with pseudographics:

┌───── ... ──┬───────┬─────────┬──────────┬─...
| id         | ready | status  | progress | ...
├──────... ──┼───────┼─────────┼──────────┼─...
| ydb://...   | true  | SUCCESS | Done     | ...
├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴...
...

In the proto-json-base64 output mode, a completed operation is reflected by the value PROGRESS_DONE of the progress attribute:
```
{"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... }
```

Completing the import operation

After the import is complete, use the operation forget command to remove it from the list of operations:

ydb -p quickstart operation forget "ydb://import/8?id=281474976788395&kind=fs"

List of import operations

To get a list of import operations, use the operation list import/nfs command:

ydb -p quickstart operation list import/nfs

The output format operation list is also set by the option --format.

Examples

Note

The examples use the quickstart profile. To learn more, see Creating a profile to connect to a test database.

Importing to the database root

Importing the contents of the /mnt/nfs/backups/export1 directory on the file system into the database root:

ydb -p quickstart import nfs \
  --fs-path /mnt/nfs/backups/export1

Importing multiple directories

Importing objects from the dir1 and dir2 directories of the export located in /mnt/nfs/backups/export1 on the file system into the identically named database directories:

ydb -p quickstart import nfs \
  --fs-path /mnt/nfs/backups/export1 \
  --include dir1 --include dir2

Importing an encrypted export

Importing a single table that was exported at path dir/my_table into the path dir1/dir/my_table from an encrypted export located in /mnt/nfs/backups/export1 on the file system, using a secret key from the ~/my_secret_key file.

ydb -p quickstart import nfs \
  --fs-path /mnt/nfs/backups/export1 --destination-path dir1 \
  --include dir/my_table \
  --encryption-key-file ~/my_secret_key

Getting operation IDs

To get a list of import operation IDs in a format convenient for processing in bash scripts, you can use the jq utility:

ydb -p quickstart operation list import/nfs --format proto-json-base64 | jq -r ".operations[].id"

You will get output where each new line contains an operation ID, for example:

ydb://import/8?id=281474976789577&kind=fs
ydb://import/8?id=281474976789526&kind=fs
ydb://import/8?id=281474976788779&kind=fs

These IDs can be used, for example, to run a loop to complete all current operations:

ydb -p quickstart operation list import/nfs --format proto-json-base64 | jq -r ".operations[].id" | while read line; do ydb -p quickstart operation forget $line;done

Was the article helpful?

Export to NFS

Importing data from a file to an existing table