Export to S3-compatible storage

The export s3 command starts a server-side process of exporting data and schema object information to an S3-compatible storage in the format described in the article File structure:

ydb [connection options] export s3 [options]

where [connection options] are database connection options

Command line parameters

[options] — command parameters:

S3 parameters

The export to S3 command requires specifying S3 connection parameters. Since the export is performed asynchronously by the YDB server, the specified endpoint must be accessible for establishing a connection from the server side.

--destination-prefix PREFIX: Key prefix in the S3 bucket.

List of exported objects

--root-path PATH: Root directory for exported objects. If not specified, the root directory of the database is used.

--include PATH: Data schema objects to include in the export. Directories are traversed recursively. Paths are specified relative to root-path. This parameter can be specified multiple times to include multiple objects. If not specified, all non-system objects in root-path are exported.

--exclude STRING: Pattern ( PCRE) to exclude paths from the export. Paths are specified relative to root-path. This parameter can be specified multiple times for different patterns.

Additional parameters

Running the export

Server export operation flow

1. A server asynchronous export operation is created.
2. In the root directory of the database, a service directory export-{id} is created, where {id} is the numeric operation ID.
3. A consistent copy of the tables is created in this directory using the CopyTables mechanism.
4. The data of each table is written from the copy to the storage in the export file structure format, with parallel writing from different cluster nodes.
5. After successful completion, the service directory export-{id} and the table copies are deleted.

Launch result

Upon successful execution, the export s3 command outputs summary information about the queued export operation to S3, in the format specified by the --format option. The actual export is performed asynchronously by the server. The summary information includes the operation ID, which can be used later to check the status and perform actions on the operation:

• In the pretty output mode (default), the operation ID is shown in the id field highlighted with pseudographics:

┌───────────────────────────────────────────┬───────┬─────...
| id                                        | ready | stat...
├───────────────────────────────────────────┼───────┼─────...
| ydb://export/6?id=281474976788395&kind=s3 | true  | SUCC...
├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴...
| StorageClass: NOT_SET
| Items:
...

• In the proto-json-base64 output mode, the identifier is in the "id" attribute:

{"id":"ydb://export/6?id=281474976788395&kind=s3","ready":true, ... }

Export status

Data export runs in the background. You can get information about the status and progress of the export by calling the operation get command, which must be passed the quoted operation ID, for example:

ydb -p quickstart operation get "ydb://export/6?id=281474976788395&kind=s3"

The output format operation get is also set by the --format option.

Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string.

Completion of the export is tracked by changes in the "progress" attribute:

• In the pretty output mode (default), a successfully completed operation is reflected by the value "Done" in the progress field highlighted with pseudographics:

  ┌───── ... ──┬───────┬─────────┬──────────┬─...
  | id         | ready | status  | progress | ...
  ├──────... ──┼───────┼─────────┼──────────┼─...
  | ydb://...   | true  | SUCCESS | Done     | ...
  ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴...
  ...
  

• In the proto-json-base64 output mode, a completed operation is reflected by the value PROGRESS_DONE of the progress attribute:

  {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... }
  

Completing the export operation

After completing the export, use the operation forget command to mark the export as completed (removed from the list of operations):

ydb -p quickstart operation forget "ydb://export/6?id=281474976788395&kind=s3"

List of export operations

To get the list of export operations, use the operation list export/s3 command:

ydb -p quickstart operation list export/s3

The output format operation list is also set by the --format option.

Examples

Exporting a database

Export all non-system database objects to the export1 directory in the mybucket bucket using S3 authentication parameters from environment variables or the ~/.aws/credentials file:

ydb -p quickstart export s3 \
  --s3-endpoint storage.yandexcloud.net --bucket mybucket \
  --destination-prefix export1

Exporting multiple directories

Export objects from the dir1 and dir2 directories of the database to the export1 directory in the mybucket bucket, using explicitly specified S3 authentication parameters:

ydb -p quickstart export s3 \
  --s3-endpoint storage.yandexcloud.net --bucket mybucket \
  --access-key <access-key> --secret-key <secret-key> \
  --destination-prefix export1 --include dir1 --include dir2

Or using an alternative method:

ydb -p quickstart export s3 \
  --s3-endpoint storage.yandexcloud.net --bucket mybucket \
  --access-key <access-key> --secret-key <secret-key> \
  --item src=dir1,dst=export1/dir1 --item src=dir2,dst=export1/dir2

Export with encryption

Export the entire database with encryption:

• Using the AES-128-GCM encryption algorithm
• With generation of a random key by the openssl utility into the ~/my_secret_key file
• With reading the generated key from the ~/my_secret_key file
• To the path prefix export1 in the S3 bucket mybucket
• Using S3 authentication parameters from environment variables or the ~/.aws/credentials file

openssl rand -out ~/my_secret_key 16
ydb -p quickstart export s3 \
  --s3-endpoint storage.yandexcloud.net --bucket mybucket --destination-prefix export1 \
  --encryption-algorithm AES-128-GCM --encryption-key-file ~/my_secret_key

Exporting the dir1 directory of the database with encryption:

• Using the AES-256-GCM encryption algorithm
• By generating a random key with the openssl utility into the YDB_ENCRYPTION_KEY environment variable
• By reading the generated key from the YDB_ENCRYPTION_KEY environment variable
• Into the path prefix export1 in the S3 bucket mybucket
• Using S3 authentication parameters from environment variables or the ~/.aws/credentials file

export YDB_ENCRYPTION_KEY=$(openssl rand -hex 32)
ydb -p quickstart export s3 \
  --root-path dir1 \
  --s3-endpoint storage.yandexcloud.net --bucket mybucket --destination-prefix export1 \
  --encryption-algorithm AES-256-GCM

Getting operation IDs

To get a list of export operation IDs in a format convenient for processing in bash scripts, you can use the jq utility:

ydb -p quickstart operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id"

You will get output where each new line contains an operation ID, for example:

ydb://export/6?id=281474976789577&kind=s3
ydb://export/6?id=281474976789526&kind=s3
ydb://export/6?id=281474976788779&kind=s3

Using these IDs, you can, for example, run a loop to terminate all current operations:

ydb -p quickstart operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" | while read line; do ydb -p quickstart operation forget $line;done