--- metadata: - name: generator content: Diplodoc Platform v5.50.6 alternate: - https://ydb.tech/docs/en/reference/ydb-cli/export-import/export-s3.md?version=main - https://ydb.tech/docs/ru/reference/ydb-cli/export-import/export-s3.md?version=main sourcePath: en/core/reference/ydb-cli/export-import/export-s3.md --- > **Documentation Index:** Fetch the complete configuration index at https://ydb.tech/docs/en/llms.txt # Export to S3-compatible storage The `export s3` command starts a server-side process of exporting data and schema object information to an S3-compatible storage in the format described in the article [File structure](https://ydb.tech/docs/en/reference/ydb-cli/export-import/file-structure.md?version=main): ```bash ydb [connection options] export s3 [options] ``` where [connection options] are [database connection options](https://ydb.tech/docs/en/reference/ydb-cli/connect.md?version=main#command-line-pars) {% note warning %} Export is available only for objects of the following types: - [directory](https://ydb.tech/docs/en/concepts/datamodel/dir.md?version=main). - [row table](https://ydb.tech/docs/en/concepts/datamodel/table.md?version=main#row-oriented-tables). - [secondary index](https://ydb.tech/docs/en/concepts/glossary.md?version=main#secondary-index). - [vector index](https://ydb.tech/docs/en/concepts/glossary.md?version=main#vector-index). - [full-text index](https://ydb.tech/docs/en/concepts/glossary.md?version=main#fulltext-index). - [topic](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=main) (schema only). - [view](https://ydb.tech/docs/en/concepts/datamodel/view.md?version=main). - [asynchronous replication](https://ydb.tech/docs/en/concepts/async-replication.md?version=main). - [transfer](https://ydb.tech/docs/en/concepts/transfer.md?version=main). - [external data source](https://ydb.tech/docs/en/concepts/datamodel/external_data_source.md?version=main). - [external table](https://ydb.tech/docs/en/concepts/datamodel/external_table.md?version=main). For a simpler export of single row and column tables to an S3-compatible data storage, you can use [external data sources](https://ydb.tech/docs/en/concepts/datamodel/external_data_source.md?version=main). For more details, see the article [Exporting data to S3 object storage](https://ydb.tech/docs/en/concepts/query_execution/federated_query/s3/write_data.md?version=main#export-to-s3). {% endnote %} ## Command line parameters {#pars} `[options]` — command parameters: ### S3 parameters {#s3-params} The export to S3 command requires specifying [S3 connection parameters](https://ydb.tech/docs/en/reference/ydb-cli/export-import/auth-s3.md?version=main). Since the export is performed asynchronously by the YDB server, the specified endpoint must be accessible for establishing a connection from the server side. `--destination-prefix PREFIX`: Key prefix in the S3 bucket. ### List of exported objects {#items} `--root-path PATH`: Root directory for exported objects. If not specified, the root directory of the database is used. `--include PATH`: Data schema objects to include in the export. Directories are traversed recursively. Paths are specified relative to `root-path`. This parameter can be specified multiple times to include multiple objects. If not specified, all non-system objects in `root-path` are exported. `--exclude STRING`: Pattern ( [PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) to exclude paths from the export. Paths are specified relative to `root-path`. This parameter can be specified multiple times for different patterns. {% cut "Alternative method" %} An alternative way to specify the list of objects is supported: `--item STRING`: Description of the export object. The `--item` parameter can be specified multiple times if you need to export several objects. `STRING` is specified in the `=,...` format, with the following required properties: - `source`, `src`, or `s` — path to the exported directory or table; `.` points to the root directory of the database. When specifying a directory, all non-system objects in it are exported, as well as all non-system subdirectories recursively. - `destination`, `dst`, or `d` — path (key prefix) in S3 for placing the exported objects. `--exclude STRING`: Pattern ( [PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) to exclude paths from export. This parameter can be specified multiple times for different patterns. {% note warning %} Exports made using alternative syntax will not contain a list of objects in the backup, so some features may be unavailable for them (in particular, encrypted backups), and import is only possible using the corresponding alternative import syntax. {% endnote %} {% endcut %} ### Additional parameters {#aux} | Parameter | Description | | --- | --- | | `--description STRING` | Text description of the operation, saved in the operation history. | | `--retries NUM` | Number of retry attempts for export that the server will make.
Default value: `10`. | | `--compression STRING` | Compress exported data.
With the default compression level for the [Zstandard](https://en.wikipedia.org/wiki/Zstd) algorithm, data can be compressed by 5-10 times. Data compression uses CPU resources and may affect the speed of other database operations.
Allowed values:
| | `--encryption-algorithm ALGORITHM` | Encrypt exported data using the specified algorithm. Supported values: `AES-128-GCM`, `AES-256-GCM`, `ChaCha20-Poly1305`. | | `--encryption-key-file PATH` | Path to the file containing the encryption key (only for encrypted exports). This file is binary and must contain the exact number of bytes corresponding to the key length in the selected encryption algorithm (16 bytes for `AES-128-GCM`, 32 bytes for `AES-256-GCM` and `ChaCha20-Poly1305`). The key can also be passed via the `YDB_ENCRYPTION_KEY` environment variable, in hexadecimal string representation. | | `--format STRING` | Output format.
Allowed values:
| ## Running the export {#exec} ### Server export operation flow {#server-export-workflow} 1. A server asynchronous export operation is created. 2. In the root directory of the database, a service directory `export-{id}` is created, where `{id}` is the numeric operation ID. 3. A consistent copy of the tables is created in this directory using the `CopyTables` mechanism. 4. The data of each table is written from the copy to the storage in the [export file structure](https://ydb.tech/docs/en/reference/ydb-cli/export-import/file-structure.md?version=main) format, with parallel writing from different cluster nodes. 5. After successful completion, the service directory `export-{id}` and the table copies are deleted. ### Launch result {#result} Upon successful execution, the `export s3` command outputs summary information about the queued export operation to S3, in the format specified by the `--format` option. The actual export is performed asynchronously by the server. The summary information includes the operation ID, which can be used later to check the status and perform actions on the operation: - In the `pretty` output mode (default), the operation ID is shown in the id field highlighted with pseudographics: ```text ┌───────────────────────────────────────────┬───────┬─────... | id | ready | stat... ├───────────────────────────────────────────┼───────┼─────... | ydb://export/6?id=281474976788395&kind=s3 | true | SUCC... ├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... | StorageClass: NOT_SET | Items: ... ``` - In the `proto-json-base64` output mode, the identifier is in the "id" attribute: ```json {"id":"ydb://export/6?id=281474976788395&kind=s3","ready":true, ... } ``` ### Export status {#status} Data export runs in the background. You can get information about the status and progress of the export by calling the `operation get` command, which must be passed the **quoted** operation ID, for example: ```bash ydb -p quickstart operation get "ydb://export/6?id=281474976788395&kind=s3" ``` The output format `operation get` is also set by the `--format` option. Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string. Completion of the export is tracked by changes in the "progress" attribute: - In the `pretty` output mode (default), a successfully completed operation is reflected by the value "Done" in the `progress` field highlighted with pseudographics: ```text ┌───── ... ──┬───────┬─────────┬──────────┬─... | id | ready | status | progress | ... ├──────... ──┼───────┼─────────┼──────────┼─... | ydb://... | true | SUCCESS | Done | ... ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴... ... ``` - In the `proto-json-base64` output mode, a completed operation is reflected by the value `PROGRESS_DONE` of the `progress` attribute: ```json {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... } ``` ### Completing the export operation {#forget} After completing the export, use the `operation forget` command to mark the export as completed (removed from the list of operations): ```bash ydb -p quickstart operation forget "ydb://export/6?id=281474976788395&kind=s3" ``` ### List of export operations {#list} To get the list of export operations, use the `operation list export/s3` command: ```bash ydb -p quickstart operation list export/s3 ``` The output format `operation list` is also set by the `--format` option. ## Examples {#examples} {% note info %} The examples use the `quickstart` profile. To learn more, see [Creating a profile to connect to a test database](https://ydb.tech/docs/en/reference/ydb-cli/profile/create.md?version=main#quickstart). {% endnote %} ### Exporting a database {#example-full-db} Export all non-system database objects to the `export1` directory in the `mybucket` bucket using S3 authentication parameters from environment variables or the `~/.aws/credentials` file: ```bash ydb -p quickstart export s3 \ --s3-endpoint storage.yandexcloud.net --bucket mybucket \ --destination-prefix export1 ``` ### Exporting multiple directories {#example-specific-dirs} Export objects from the `dir1` and `dir2` directories of the database to the `export1` directory in the `mybucket` bucket, using explicitly specified S3 authentication parameters: ```bash ydb -p quickstart export s3 \ --s3-endpoint storage.yandexcloud.net --bucket mybucket \ --access-key --secret-key \ --destination-prefix export1 --include dir1 --include dir2 ``` Or using an alternative method: ```bash ydb -p quickstart export s3 \ --s3-endpoint storage.yandexcloud.net --bucket mybucket \ --access-key --secret-key \ --item src=dir1,dst=export1/dir1 --item src=dir2,dst=export1/dir2 ``` ### Export with encryption {#example-encryption} Export the entire database with encryption: - Using the `AES-128-GCM` encryption algorithm - With generation of a random key by the `openssl` utility into the `~/my_secret_key` file - With reading the generated key from the `~/my_secret_key` file - To the path prefix `export1` in the S3 bucket `mybucket` - Using S3 authentication parameters from environment variables or the `~/.aws/credentials` file ```bash openssl rand -out ~/my_secret_key 16 ydb -p quickstart export s3 \ --s3-endpoint storage.yandexcloud.net --bucket mybucket --destination-prefix export1 \ --encryption-algorithm AES-128-GCM --encryption-key-file ~/my_secret_key ``` Exporting the `dir1` directory of the database with encryption: - Using the `AES-256-GCM` encryption algorithm - By generating a random key with the `openssl` utility into the `YDB_ENCRYPTION_KEY` environment variable - By reading the generated key from the `YDB_ENCRYPTION_KEY` environment variable - Into the path prefix `export1` in the S3 bucket `mybucket` - Using S3 authentication parameters from environment variables or the `~/.aws/credentials` file ```bash export YDB_ENCRYPTION_KEY=$(openssl rand -hex 32) ydb -p quickstart export s3 \ --root-path dir1 \ --s3-endpoint storage.yandexcloud.net --bucket mybucket --destination-prefix export1 \ --encryption-algorithm AES-256-GCM ``` ### Getting operation IDs {#example-list-oneline} To get a list of export operation IDs in a format convenient for processing in bash scripts, you can use the [jq](https://stedolan.github.io/jq/download/) utility: ```bash ydb -p quickstart operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" ``` You will get output where each new line contains an operation ID, for example: ```text ydb://export/6?id=281474976789577&kind=s3 ydb://export/6?id=281474976789526&kind=s3 ydb://export/6?id=281474976788779&kind=s3 ``` Using these IDs, you can, for example, run a loop to terminate all current operations: ```bash ydb -p quickstart operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" | while read line; do ydb -p quickstart operation forget $line;done ```