Connecting to S3-compatible object storages

The commands used to export data from S3-compatible storages, export s3 and import s3, use the same parameters for S3 connection and authentication. To learn how to get these parameters for certain cloud providers, see Getting S3 connection parameters below.

Connecting

To connect with S3, you need to specify an endpoint and bucket:

--s3-endpoint HOST: An S3 endpoint. HOST: A valid host name, such as: storage.yandexcloud.net

--bucket STR: An S3 bucket. STR: A string containing the bucket name

Authentication

Except when you import data from a public bucket, to connect, log in with an account that has write access to the bucket (for export to it) and read access to the bucket (for import from it).

You need two parameters to authenticate with S3:

ID of the access key (access_key_id).
Secret access key (secret_access_key).

The YDB CLI takes values of these parameters from the following sources (listed in descending priority):

The command line.
Environment variables.
The ~/.aws/credentials file.

Command line parameters

--access-key: Access key ID.
--secret-key: Secret access key.
--aws-profile: Profile name from the ~/.aws/credentials file. The default value is default.

Environment variables

If a certain authentication parameter is omitted in the command line, the YDB CLI tries to retrieve it from the following environment variables:

AWS_ACCESS_KEY_ID: Access key ID.
AWS_SECRET_ACCESS_KEY: Secret access key.
AWS_PROFILE: Profile name from the ~/.aws/credentials file.

AWS authentication file

If a certain authentication parameter is omitted in the command line and cannot be retrieved from an environment variable, the YDB CLI tries to get it from the specified profile or the default profile in the ~/.aws/credentials file used for authenticating the AWS CLI. You can create this file with the aws configure AWS CLI command.

Getting the S3 connection parameters

Yandex.Cloud

Below is an example of getting access keys for the Yandex.Cloud Object Storage using the Yandex.Cloud CLI.

Install and set up the Yandex.Cloud CLI.
Use the following command to get the ID of your cloud folder (you'll need to add it to the below commands):
```
yc config list
```
The ID of your cloud folder is in the folder-id: line in the result:
```
folder-id: b2ge70qdcff4bo9q6t19
```
To create a service account, run the command:
```
yc iam service-account create --name s3account
```
You can indicate any account name except s3account, or use your existing account name (be sure to replace it when copying the commands below).
Grant roles to your service account according to your intended S3 access level by running the command:
Read (to import data to the YDB database)

Write (to export data from the YDB database)
```
yc resource-manager folder add-access-binding <folder-id> \
  --role storage.viewer --subject serviceAccount:s3account
```
```
yc resource-manager folder add-access-binding <folder-id> \
  --role storage.editor --subject serviceAccount:s3account
```
Where <folder-id> is the cloud folder ID that you retrieved at step 2.

You can also read a full list of Yandex.Cloud roles.

Get static access keys by running the command:

yc iam access-key create --service-account-name s3account

If successful, the command will return the access_key attributes and the secret value:

access_key:
  id: aje6t3vsbj8lp9r4vk2u
  service_account_id: ajepg0mjt06siuj65usm
  created_at: "2018-11-22T14:37:51Z"
  key_id: 0n8X6WY6S24N7OjXQ0YQ
secret: JyTRFdqw8t1kh2-OJNz4JX5ZTz9Dj1rI9hxtzMP1

In this result:

access_key.key_id is the access key ID
secret is the secret access key