Export to NFS

The export nfs command starts a server-side process of exporting data and schema object information to the network file system ( Network File System, NFS) of the YDB cluster hosts, in the format described in the File structure article:

ydb [connection options] export nfs [options]

where [connection options] are database connection options

Warning

Export is available only for the following object types:

Command-line parameters

[options] — command parameters:

NFS parameters

The NFS export command requires specifying a mounted directory (or subdirectory) common to all objects involved in the export. Since the export is performed asynchronously on all YDB hosts, the specified directory must exist on each YDB host and be mounted in NFS.

--fs-path PATH: path to the mounted directory (or subdirectory).

List of exported objects

--root-path PATH: Root directory for exported objects. If not specified, the root directory of the database is used.

--include PATH: Data schema objects to include in the export. Directories are traversed recursively. Paths are specified relative to root-path. This parameter can be specified multiple times to include multiple objects. If not specified, all non-system objects in root-path are exported.

--exclude STRING: Pattern ( PCRE) to exclude paths from the export. Paths are specified relative to root-path. This parameter can be specified multiple times for different patterns.

Alternative method

An alternative way to specify the list of objects is supported:

--item STRING: Description of the export object. The --item parameter can be specified multiple times if you need to export several objects. STRING is specified in the <property>=<value>,... format, with the following required properties:

source, src, or s — path to the exported directory or table; . points to the root directory of the database. When specifying a directory, all non-system objects in it are exported, as well as all non-system subdirectories recursively.
destination, dst, or d — path in NFS (relative to --fs-path).

--exclude STRING: Pattern ( PCRE) to exclude paths from the export. This parameter can be specified multiple times for different patterns.

Warning

Exports made using alternative syntax will not contain a list of objects in the backup, so some features may be unavailable for them (in particular, encrypted backups), and import is only possible using the corresponding alternative import syntax.

Additional parameters

--description STRING: Text description of the operation, saved in the operation history.
--retries NUM: Number of retry attempts for export that the server will make. Default value: 10.
--compression STRING: Compress exported data. With the default compression level for the Zstandard algorithm, data can be compressed by 5-10 times. Data compression uses CPU resources and may affect the speed of other database operations. Valid values:
- zstd — compression using the Zstandard algorithm with the default compression level (3).
- zstd-N — compression using the Zstandard algorithm, N — compression level (1 — 22).
--encryption-algorithm ALGORITHM: Encrypt exported data using the specified algorithm. Supported values: AES-128-GCM, AES-256-GCM, ChaCha20-Poly1305.
--encryption-key-file PATH: Path to the file containing the encryption key (only for encrypted exports). This file is binary and must contain the exact number of bytes corresponding to the key length in the selected encryption algorithm (16 bytes for AES-128-GCM, 32 bytes for AES-256-GCM and ChaCha20-Poly1305). The key can also be passed via the YDB_ENCRYPTION_KEY environment variable, in hexadecimal string representation.
--format STRING: Output result format. Valid values:
- pretty — human-readable format (default).
- proto-json-base64 — Protocol Buffers in JSON format, binary strings encoded in Base64.

Running the export

Server export operation flow

A server-side asynchronous export operation is created.
A service directory export-{id} is created in the root directory of the database, where {id} is the numeric operation ID.
A consistent copy of the tables is created in this directory using the CopyTables mechanism.
The data of each table is written from the copy to the storage in the export file structure format, with parallel writing from different nodes of the cluster.
After successful completion, the service directory export-{id} and the table copies are deleted.

Launch result

On successful execution, the export nfs command outputs summary information about the queued NFS export operation, in the format specified by the --format option. The actual export is performed asynchronously by the server. The summary information includes the operation ID, which can be used later to check the status and perform actions on the operation:

In the pretty output mode (default), the operation ID is shown in the id field highlighted with pseudographics:

┌───────────────────────────────────────────┬───────┬─────...
| id                                        | ready | stat...
├───────────────────────────────────────────┼───────┼─────...
| ydb://export/6?id=281474976788395&kind=fs | true  | SUCC...
├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴...
| Include index data: false
| Items:
...

In the proto-json-base64 output mode, the identifier is in the "id" attribute:

{"id":"ydb://export/6?id=281474976788395&kind=fs","ready":true, ... }

Export status

Data export runs in the background. You can get information about the status and progress of the export by calling the operation get command, which must be passed a quoted operation ID, for example:

ydb -p quickstart operation get "ydb://export/6?id=281474976788395&kind=fs"

The output format operation get is also set by the option --format.

Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string.

The completion of the export is tracked by the change of the "progress" attribute:

In the pretty output mode (default), a successfully completed operation is reflected by the value "Done" in the progress field highlighted with pseudographics:

┌───── ... ──┬───────┬─────────┬──────────┬─...
| id         | ready | status  | progress | ...
├──────... ──┼───────┼─────────┼──────────┼─...
| ydb://...   | true  | SUCCESS | Done     | ...
├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴...
...

In the proto-json-base64 output mode, a completed operation is reflected by the value PROGRESS_DONE of the progress attribute:
```
{"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... }
```

Completing the export operation

After the export is complete, use the operation forget command to mark the export as finished (removing it from the list of operations):

ydb -p quickstart operation forget "ydb://export/6?id=281474976788395&kind=fs"

List of export operations

To get the list of export operations, use the operation list export/nfs command:

ydb -p quickstart operation list export/nfs

The output format operation list is also set by the option --format.

Examples

Note

The examples use the quickstart profile. To learn more, see Creating a profile to connect to a test database.

Exporting a database

Export all non-system database objects to the /mnt/nfs/backups/export1 directory on the file system:

ydb -p quickstart export nfs \
  --fs-path /mnt/nfs/backups/export1

Exporting multiple directories

Export objects from the dir1 and dir2 directories of the database to the /mnt/nfs/backups/export1 directory on the file system:

ydb -p quickstart export nfs \
  --fs-path /mnt/nfs/backups/export1 \
  --include dir1 --include dir2

Or using an alternative method:

ydb -p quickstart export nfs \
  --fs-path /mnt/nfs/backups \
  --item src=dir1,dst=export1/dir1 --item src=dir2,dst=export1/dir2

Export with encryption

Export the entire database with encryption:

Using the AES-128-GCM encryption algorithm
Generating a random key with the openssl utility to the ~/my_secret_key file
Reading the generated key from the ~/my_secret_key file
To the /mnt/nfs/backups/export1 directory on the file system

openssl rand -out ~/my_secret_key 16
ydb -p quickstart export nfs \
  --fs-path /mnt/nfs/backups/export1 \
  --encryption-algorithm AES-128-GCM --encryption-key-file ~/my_secret_key

Export the dir1 directory of the database with encryption:

Using the AES-256-GCM encryption algorithm
Generating a random key with the openssl utility to the YDB_ENCRYPTION_KEY environment variable
Reading the generated key from the YDB_ENCRYPTION_KEY environment variable
To the /mnt/nfs/backups/export1 directory on the file system

export YDB_ENCRYPTION_KEY=$(openssl rand -hex 32)
ydb -p quickstart export nfs \
  --root-path dir1 \
  --fs-path /mnt/nfs/backups/export1 \
  --encryption-algorithm AES-256-GCM

Getting operation IDs

To get a list of export operation IDs in a format convenient for processing in bash scripts, you can use the jq utility:

ydb -p quickstart operation list export/nfs --format proto-json-base64 | jq -r ".operations[].id"

You will get output where each new line contains an operation ID, for example:

ydb://export/6?id=281474976789577&kind=fs
ydb://export/6?id=281474976789526&kind=fs
ydb://export/6?id=281474976788779&kind=fs

These IDs can be used, for example, to run a loop to complete all current operations:

ydb -p quickstart operation list export/nfs --format proto-json-base64 | jq -r ".operations[].id" | while read line; do ydb -p quickstart operation forget $line;done

Was the article helpful?

Importing data from S3

Import from NFS