Connecting to S3-compatible object storages
The commands used to export data from S3-compatible storages, export s3
and import s3
, use the same parameters for S3 connection and authentication. To learn how to get these parameters for certain cloud providers, see Getting S3 connection parameters below.
Connecting
To connect with S3, you need to specify an endpoint and bucket:
--s3-endpoint HOST
: An S3 endpoint. HOST
: A valid host name, such as: storage.yandexcloud.net
--bucket STR
: An S3 bucket. STR
: A string containing the bucket name
Authentication
Except when you import data from a public bucket, to connect, log in with an account that has write access to the bucket (for export to it) and read access to the bucket (for import from it).
You need two parameters to authenticate with S3:
- ID of the access key (access_key_id).
- Secret access key (secret_access_key).
The YDB CLI takes values of these parameters from the following sources (listed in descending priority):
- The command line.
- Environment variables.
- The
~/.aws/credentials
file.
Command line parameters
--access-key
: Access key ID.--secret-key
: Secret access key.--aws-profile
: Profile name from the~/.aws/credentials
file. The default value isdefault
.
Environment variables
If a certain authentication parameter is omitted in the command line, the YDB CLI tries to retrieve it from the following environment variables:
AWS_ACCESS_KEY_ID
: Access key ID.AWS_SECRET_ACCESS_KEY
: Secret access key.AWS_PROFILE
: Profile name from the~/.aws/credentials
file.
AWS authentication file
If a certain authentication parameter is omitted in the command line and cannot be retrieved from an environment variable, the YDB CLI tries to get it from the specified profile or the default profile in the ~/.aws/credentials
file used for authenticating the AWS CLI. You can create this file with the aws configure
AWS CLI command.
Getting the S3 connection parameters
Yandex.Cloud
Below is an example of getting access keys for the Yandex.Cloud Object Storage using the Yandex.Cloud CLI.
-
Install and set up the Yandex.Cloud CLI.
-
Use the following command to get the ID of your cloud folder (you'll need to add it to the below commands):
yc config list
The ID of your cloud folder is in the
folder-id:
line in the result:folder-id: b2ge70qdcff4bo9q6t19
-
To create a service account, run the command:
yc iam service-account create --name s3account
You can indicate any account name except
s3account
, or use your existing account name (be sure to replace it when copying the commands below). -
Grant roles to your service account according to your intended S3 access level by running the command:
Read (to import data to the YDB database)Write (to export data from the YDB database)yc resource-manager folder add-access-binding <folder-id> \ --role storage.viewer --subject serviceAccount:s3account
yc resource-manager folder add-access-binding <folder-id> \ --role storage.editor --subject serviceAccount:s3account
Where
<folder-id>
is the cloud folder ID that you retrieved at step 2.You can also read a full list of Yandex.Cloud roles.
-
Get static access keys by running the command:
yc iam access-key create --service-account-name s3account
If successful, the command will return the access_key attributes and the secret value:
access_key: id: aje6t3vsbj8lp9r4vk2u service_account_id: ajepg0mjt06siuj65usm created_at: "2018-11-22T14:37:51Z" key_id: 0n8X6WY6S24N7OjXQ0YQ secret: JyTRFdqw8t1kh2-OJNz4JX5ZTz9Dj1rI9hxtzMP1
In this result:
access_key.key_id
is the access key IDsecret
is the secret access key