Deploying a YDB cluster on virtual or bare metal servers

This document describes the procedure to deploy a multi-tenant YDB cluster on several virtual or bare metal servers.

Before you begin

Prerequisites

Make sure you have SSH access to all servers. This is required to install artifacts and run the YDB binary. Your network configuration must allow TCP connections on the following ports (by default):

• 2135, 2136: GRPC for client-cluster interaction.
• 19001, 19002: Interconnect for intra-cluster node interaction.
• 8765, 8766: The HTTP interface for cluster monitoring.

Review the system requirements and the cluster topology.

Select the servers and disks to be used for storing data:

• Use the block-4-2 fault tolerance model for cluster deployment in one availability zone (AZ). Use at least 8 nodes to be able to withstand the loss of 2 of them.
• Use the mirror-3-dc fault tolerance model for cluster deployment in three availability zones (AZ). To survive the loss of a single AZ and of 1 node in another AZ, use at least 9 nodes. The number of nodes in each AZ should be the same.

Run each static node on a separate server.

For more information about hardware requirements, see System requirements and recommendations.

Create a system user and a group to run YDB

On each server that will be running YDB, execute the commands below:

sudo groupadd ydb
sudo useradd ydb -g ydb

To make sure the YDB server has access to block disk devices, you need to add the process owner to the disk user group:

sudo usermod -aG disk ydb

Prepare and format disks on each server

For YDB to work efficiently, we recommend using physical (not virtual) disks larger than 800 GB as block devices.

The minimum disk size must be at least 80 GB, otherwise the node won't be able to use the block store volume. Correct and uninterrupted operation with minimum-size disks is not guaranteed. We recommend using such disks exclusively for informational purposes.

1. Create a partition on the selected disk:

   Alert

   The following step will delete all partitions on the specified disks. Make sure that you specified the disks that have no other data!

   sudo parted /dev/nvme0n1 mklabel gpt -s
   sudo parted -a optimal /dev/nvme0n1 mkpart primary 0% 100%
   sudo parted /dev/nvme0n1 name 1 ydb_disk_ssd_01
   sudo partx --u /dev/nvme0n1
   

   As a result, a disk labeled /dev/disk/by-partlabel/ydb_disk_ssd_01 will appear on the system.

   If you plan to use more than one disk on each server, replace ydb_disk_ssd_01 with a unique label for each one. You'll need to use these disks later in the configuration files.

2. Download and unpack an archive with the ydbd executable and the libraries required for YDB to run:

   mkdir ydbd-stable-linux-amd64
   curl -L https://binaries.ydb.tech/ydbd-stable-linux-amd64.tar.gz | tar -xz --strip-component=1 -C ydbd-stable-linux-amd64
   

3. Create directories to run:

   sudo mkdir -p /opt/ydb/bin /opt/ydb/cfg /opt/ydb/lib
   sudo chown -R ydb:ydb /opt/ydb
   

4. Copy the binary file, libraries, and configuration file to the appropriate directories:

   sudo cp -i ydbd-stable-linux-amd64/bin/ydbd /opt/ydb/bin/
   sudo cp -i ydbd-stable-linux-amd64/lib/libaio.so /opt/ydb/lib/
   sudo cp -i ydbd-stable-linux-amd64/lib/libiconv.so /opt/ydb/lib/
   sudo cp -i ydbd-stable-linux-amd64/lib/libidn.so /opt/ydb/lib/
   

5. Format the disk with the builtin command below:

   sudo LD_LIBRARY_PATH=/opt/ydb/lib /opt/ydb/bin/ydbd admin bs disk obliterate /dev/disk/by-partlabel/ydb_disk_ssd_01
   

   Perform this operation for each disk that will be used for data storage.

Prepare configuration files

Save the YDB configuration file as /opt/ydb/cfg/config.yaml on each cluster node.

For more detailed information about creating configurations, see Cluster configurations.

Start static nodes

sudo su - ydb
cd /opt/ydb
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd server --log-level 3 --syslog --tcp --yaml-config /opt/ydb/cfg/config.yaml  \
--grpc-port 2135 --ic-port 19001 --mon-port 8765 --node static

[Unit]
Description=YDB storage node
After=network-online.target rc-local.service
Wants=network-online.target
StartLimitInterval=10
StartLimitBurst=15

[Service]
Restart=always
RestartSec=1
User=ydb
PermissionsStartOnly=true
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=ydbd
SyslogFacility=daemon
SyslogLevel=err
Environment=LD_LIBRARY_PATH=/opt/ydb/lib
ExecStart=/opt/ydb/bin/ydbd server --log-level 3 --syslog --tcp --yaml-config  /opt/ydb/cfg/config.yaml --grpc-port 2135 --ic-port 19001 --mon-port 8765 --node static
LimitNOFILE=65536
LimitCORE=0
LimitMEMLOCK=3221225472

[Install]
WantedBy=multi-user.target

sudo systemctl start ydbd-storage

Initialize the cluster

The procedure of cluster initialization depends on whether user authentication mode is enabled in the YDB configuration file.

export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd admin blobstorage config init --yaml-file  /opt/ydb/cfg/config.yaml
echo $?

ydb -e grpc://<node1.ydb.tech>:2135 -d /Root \
    --user root --no-password auth get-token --force >token-file

ydb -e grpcs://<node1.ydb.tech>:2135 -d /Root --ca-file /opt/ydb/certs/ca.crt \
     --user root --no-password auth get-token --force >token-file

export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd -f token-file admin blobstorage config init --yaml-file  /opt/ydb/cfg/config.yaml
echo $?

Create the database

To work with tables, you need to create at least one database and run a process to service this database (a dynamic node):

LD_LIBRARY_PATH=/opt/ydb/lib /opt/ydb/bin/ydbd admin database /Root/testdb create ssd:1

If the user authentication is enabled in the cluster, an authentication token must be passed to the database creation command. The procedure to obtain the token is described above in the cluster initialization section.

The authenticated version of the database creation command:

LD_LIBRARY_PATH=/opt/ydb/lib /opt/ydb/bin/ydbd -f token-file admin database /Root/testdb create ssd:1

The following parameters are used in the command examples shown above:

• /Root - name of the cluster root domain, must match the value of the domains_config.domain.name setting in the configuration file;
• testdb - name of the database being created;
• ssd:1 - name of the storage pool and the "box" number within the pool. Name of the pool typically means the device type used, and should match the value of storage_pool_types.kind setting within the domains_config.domain element of the configuration file.

Start the DB dynamic node

sudo su - ydb
cd /opt/ydb
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config  /opt/ydb/cfg/config.yaml \
--tenant /Root/testdb --node-broker <node1.ydb.tech>:2135 --node-broker <node2.ydb.tech>:2135 --node-broker <node3.ydb.tech>:2135

[Unit]
Description=YDB testdb dynamic node
After=network-online.target rc-local.service
Wants=network-online.target
StartLimitInterval=10
StartLimitBurst=15

[Service]
Restart=always
RestartSec=1
User=ydb
PermissionsStartOnly=true
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=ydbd
SyslogFacility=daemon
SyslogLevel=err
Environment=LD_LIBRARY_PATH=/opt/ydb/lib
ExecStart=/opt/ydb/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config  /opt/ydb/cfg/config.yaml --tenant /Root/testdb --node-broker <node1.ydb.tech>:2135 --node-broker <node2.ydb.tech>:2135 --node-broker <node3.ydb.tech>:2135
LimitNOFILE=65536
LimitCORE=0
LimitMEMLOCK=32212254720

[Install]
WantedBy=multi-user.target

sudo systemctl start ydbd-testdb

Initial setup of user accounts

If authentication mode is enabled in the cluster settings file, then you must perform the initial setup of user accounts before starting to work with the YDB cluster.

A single user account root with an empty password, and a standard set of pre-defined user groups, as specified in the Access management chapter, are created automatically during YDB cluster initialization.

In order to perform the initial setup of user accounts in the newly created YDB cluster use the following steps:

1. Install the YDB CLI as described in documentation.

2. Change the password of the root user account:

   ydb -e grpc://<node.ydb.tech>:2136 -d /Root/testdb --user root --no-password \
       yql -s 'ALTER USER root PASSWORD "passw0rd"'
   

   Replace the passw0rd value in the command above with the password to be used.

3. Create the additional user accounts:

   ydb -e grpc://<node.ydb.tech>:2136 -d /Root/testdb --user root \
       yql -s 'CREATE USER user1 PASSWORD "passw0rd"'
   

4. Configure the permissions by putting the user accounts into the builtin groups:

   ydb -e grpc://<node.ydb.tech>:2136 -d /Root/testdb --user root \
       yql -s 'ALTER GROUP `ADMINS` ADD USER user1'
   

In the command examples listed above <node.ydb.tech> is the FQDN of the server running the dynamic node for the database /Root/testdb.

By default YDB CLI application will query the password of the root user on each command. Password value can optionally be stored in the connection profile, as described in the YDB CLI documentation, to avoid the need to repeatably enter it.

If TLS traffic protection has been enabled, then instead of the grpc protocol in the commands above, its secure flavor grpcs must be used, and CA certificate filename has to be specified in the --ca-file parameter (or stored in the connection profile).

Test the created database

1. Install the YDB CLI as described in documentation.

2. Create a test table named test_table:

   ydb -e grpc://<node.ydb.tech>:2136 -d /Root/testdb scripting yql \
   --script 'CREATE TABLE `testdir/test_table` (id Uint64, title Utf8, PRIMARY KEY (id));'
   

   Where <node.ydb.tech> is the FQDN of the server running the dynamic node that supports the /Root/testdb database.

   The command shown above has to be modified if user authentication or TLS traffic protection modes are enabled in the cluster. For example:

   ydb -e grpcs://<node.ydb.tech>:2136 -d /Root/testdb --ca-file ydb-ca.crt --user root scripting yql \
   --script 'CREATE TABLE `testdir/test_table` (id Uint64, title Utf8, PRIMARY KEY (id));'