Deploying YDB cluster manually
- Getting started
- Create a system user and a group to run YDB
- Install YDB software on each server
- Prepare and format disks on each server
- Prepare configuration files
- Copy the TLS keys and certificates to each server
- Start static nodes
- Initialize a cluster
- Create a database
- Run dynamic nodes
- Initial account setup
- Start using the created database
- Checking access to the built-in web interface
- Installing YDB in the unprotected mode
This document describes how to deploy a multi-tenant YDB cluster on multiple bare-metal or virtual servers.
Getting started
Prerequisites
Review the system requirements and the cluster topology.
Make sure you have SSH access to all servers. This is required to install artifacts and run the YDB executable.
The network configuration must allow TCP connections on the following ports (these are defaults, but you can change them by settings):
- 22: SSH service
- 2135, 2136: GRPC for client-cluster interaction.
- 19001, 19002: Interconnect for intra-cluster node interaction
- 8765, 8766: HTTP interface of YDB Embedded UI.
Distinct ports are necessary for gRPC, Interconnect and HTTP interface of each dynamic node when hosting multiple dynamic nodes on a single server.
Make sure that the system clocks running on all the cluster's servers are synced by ntpd
or chrony
. We recommend using the same time source for all servers in the cluster to maintain consistent leap seconds processing.
If the Linux flavor run on the cluster servers uses syslogd
for logging, set up log file rotation usinglogrotate
or similar tools. YDB services can generate substantial amounts of system logs, particularly when you elevate the logging level for diagnostic purposes. That's why it's important to enable system log file rotation to prevent the /var
file system overflow.
Select the servers and disks to be used for storing data:
- Use the
block-4-2
fault tolerance model for cluster deployment in one availability zone (AZ). Use at least eight servers to safely survive the loss of two servers. - Use the
mirror-3-dc
fault tolerance model for cluster deployment in three availability zones (AZ). To survive the loss of one AZ and one server in another AZ, use at least nine servers. Make sure that the number of servers running in each AZ is the same.
Note
Run each static node (data node) on a separate server. Both static and dynamic nodes can run together on the same server. A server can also run multiple dynamic nodes if it has enough computing power.
For more information about hardware requirements, see YDB system requirements and recommendations.
Preparing TLS keys and certificates
The TLS protocol provides traffic protection and authentication for YDB server nodes. Before you install your cluster, determine which servers it will host, establish the node naming convention, come up with node names, and prepare your TLS keys and certificates.
You can use existing certificates or generate new ones. Prepare the following files with TLS keys and certificates in the PEM format:
ca.crt
: CA-issued certificate used to sign the other TLS certificates (these files are the same on all the cluster nodes).node.key
: Secret TLS keys for each cluster node (one key per cluster server).node.crt
: TLS certificates for each cluster node (each certificate corresponds to a key).web.pem
: Concatenation of the node secret key, node certificate, and the CA certificate needed for the monitoring HTTP interface (a separate file is used for each server in the cluster).
Your organization should define the parameters required for certificate generation in its policy. The following parameters are commonly used for generating certificates and keys for YDB:
- 2048-bit or 4096-bit RSA keys
- Certificate signing algorithm: SHA-256 with RSA encryption
- Validity period of node certificates: at least 1 year
- CA certificate validity period: at least 3 years.
Make sure that the CA certificate is appropriately labeled, with the CA property enabled along with the "Digital Signature, Non Repudiation, Key Encipherment, Certificate Sign" usage types.
For node certificates, it's key that the actual host name (or names) match the values in the "Subject Alternative Name" field. Enable both the regular usage types ("Digital Signature, Key Encipherment") and advanced usage types ("TLS Web Server Authentication, TLS Web Client Authentication") for the certificates. Node certificates must support both server authentication and client authentication (the extendedKeyUsage = serverAuth,clientAuth
option in the OpenSSL settings).
For batch generation or update of YDB cluster certificates by OpenSSL, you can use the sample script from the YDB GitHub repository. Using the script, you can streamline preparation for installation, automatically generating all the key files and certificate files for all your cluster nodes in a single step.
Create a system user and a group to run YDB
On each server that will be running YDB, execute the command below:
sudo groupadd ydb
sudo useradd ydb -g ydb
To ensure that YDB can access block disks, add the user that will run YDB processes, to the disk
group:
sudo usermod -aG disk ydb
Install YDB software on each server
- Download and unpack an archive with the
ydbd
executable and the libraries required for YDB to run:
mkdir ydbd-stable-linux-amd64
curl -L https://binaries.ydb.tech/ydbd-stable-linux-amd64.tar.gz | tar -xz --strip-component=1 -C ydbd-stable-linux-amd64
- Create directories for YDB software:
sudo mkdir -p /opt/ydb /opt/ydb/cfg
- Copy the executable and libraries to the appropriate directories:
sudo cp -iR ydbd-stable-linux-amd64/bin /opt/ydb/
sudo cp -iR ydbd-stable-linux-amd64/lib /opt/ydb/
- Set the owner of files and folders:
sudo chown -R root:bin /opt/ydb
Prepare and format disks on each server
For YDB to work efficiently, we recommend using physical (not virtual) disks larger than 800 GB as block devices.
The minimum disk size is 80 GB, otherwise the YDB node won't be able to use the device. Correct and uninterrupted operation with minimum-size disks is not guaranteed. We recommend using such disks exclusively for informational purposes.
Warning
Configurations with disks less than 800 GB or any types of storage system virtualization cannot be used for production services or system performance testing.
We don't recommend storing YDB data on disks shared with other processes (for example, the operating system).
- Create partitions on the selected disks:
Alert
The next operation will delete all partitions on the specified disk. Make sure that you specified a disk that contains no external data.
DISK=/dev/nvme0n1
sudo parted ${DISK} mklabel gpt -s
sudo parted -a optimal ${DISK} mkpart primary 0% 100%
sudo parted ${DISK} name 1 ydb_disk_ssd_01
sudo partx --u ${DISK}
As a result, a disk labeled /dev/disk/by-partlabel/ydb_disk_ssd_01
will appear on the system.
If you plan to use more than one disk on each server, replace ydb_disk_ssd_01
with a unique label for each one. Disk labels should be unique within each server. They are used in configuration files, see the following guides.
To streamline the next setup step, it makes sense to use the same disk labels on cluster servers having the same disk configuration.
- Format the disk by this command built-in the
ydbd
executable:
sudo LD_LIBRARY_PATH=/opt/ydb/lib /opt/ydb/bin/ydbd admin bs disk obliterate /dev/disk/by-partlabel/ydb_disk_ssd_01
Perform this operation for each disk to be used for YDB data storage.
Prepare configuration files
Prepare a configuration file for YDB:
-
Download a sample config for the appropriate failure model of your cluster:
- block-4-2: For a single-data center cluster.
- mirror-3dc: For a cross-data center cluster consisting of 9 nodes.
- mirror-3dc-3nodes: For a cross-data center cluster consisting of 3 nodes.
-
In the
host_configs
section, specify all disks and their types on each cluster node. Possible disk types:- ROT: Rotational, HDD.
- SSD: SSD or NVMe.
host_configs: - drive: - path: /dev/disk/by-partlabel/ydb_disk_ssd_01 type: SSD host_config_id: 1
-
In the
hosts
section, specify the FQDN of each node, their configuration and location in adata_center
orrack
:hosts: - host: node1.ydb.tech host_config_id: 1 walle_location: body: 1 data_center: 'zone-a' rack: '1' - host: node2.ydb.tech host_config_id: 1 walle_location: body: 2 data_center: 'zone-b' rack: '1' - host: node3.ydb.tech host_config_id: 1 walle_location: body: 3 data_center: 'zone-c' rack: '1'
-
Under
blob_storage_config
, edit the FQDNs of all the nodes accommodating your static storage group:- For the
mirror-3-dc
scheme, specify FQDNs for nine nodes. - For the
block-4-2
scheme, specify FQDNs for eight nodes.
- For the
-
Enable user authentication (optional).
If you plan to use authentication and user access differentiation features in the YDB cluster, add the following parameters to the
domains_config
section:domains_config: security_config: enforce_user_token_requirement: true monitoring_allowed_sids: - "root" - "ADMINS" - "DATABASE-ADMINS" administration_allowed_sids: - "root" - "ADMINS" - "DATABASE-ADMINS" viewer_allowed_sids: - "root" - "ADMINS" - "DATABASE-ADMINS"
In the traffic encryption mode, make sure that the YDB configuration file specifies paths to key files and certificate files under interconnect_config
and grpc_config
:
interconnect_config:
start_tcp: true
encryption_mode: OPTIONAL
path_to_certificate_file: "/opt/ydb/certs/node.crt"
path_to_private_key_file: "/opt/ydb/certs/node.key"
path_to_ca_file: "/opt/ydb/certs/ca.crt"
grpc_config:
cert: "/opt/ydb/certs/node.crt"
key: "/opt/ydb/certs/node.key"
ca: "/opt/ydb/certs/ca.crt"
services_enabled:
- legacy
Save the YDB configuration file as /opt/ydb/cfg/config.yaml
on each cluster node.
For more detailed information about creating the configuration file, see YDB cluster configuration.
Copy the TLS keys and certificates to each server
Make sure to copy the generated TLS keys and certificates to a protected folder on each YDB cluster node. Below are sample commands that create a protected folder and copy files with keys and certificates.
sudo mkdir -p /opt/ydb/certs
sudo cp -v ca.crt /opt/ydb/certs/
sudo cp -v node.crt /opt/ydb/certs/
sudo cp -v node.key /opt/ydb/certs/
sudo cp -v web.pem /opt/ydb/certs/
sudo chown -R ydb:ydb /opt/ydb/certs
sudo chmod 700 /opt/ydb/certs
Start static nodes
Run a YDB data storage service on each static cluster node:
sudo su - ydb
cd /opt/ydb
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd server --log-level 3 --syslog --tcp --yaml-config /opt/ydb/cfg/config.yaml \
--grpcs-port 2135 --ic-port 19001 --mon-port 8765 --mon-cert /opt/ydb/certs/web.pem --node static
On each server that will host a static cluster node, create a systemd /etc/systemd/system/ydbd-storage.service
configuration file by the template below. You can also download the sample file from the repository.
[Unit]
Description=YDB storage node
After=network-online.target rc-local.service
Wants=network-online.target
StartLimitInterval=10
StartLimitBurst=15
[Service]
Restart=always
RestartSec=1
User=ydb
PermissionsStartOnly=true
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=ydbd
SyslogFacility=daemon
SyslogLevel=err
Environment=LD_LIBRARY_PATH=/opt/ydb/lib
ExecStart=/opt/ydb/bin/ydbd server --log-level 3 --syslog --tcp \
--yaml-config /opt/ydb/cfg/config.yaml \
--grpcs-port 2135 --ic-port 19001 --mon-port 8765 \
--mon-cert /opt/ydb/certs/web.pem --node static
LimitNOFILE=65536
LimitCORE=0
LimitMEMLOCK=3221225472
[Install]
WantedBy=multi-user.target
Run the service on each static YDB node:
sudo systemctl start ydbd-storage
Initialize a cluster
The cluster initialization operation sets up static nodes listed in the cluster configuration file, for storing YDB data.
To initialize the cluster, you'll need the ca.crt
file issued by the Certificate Authority. Use its path in the initialization commands. Before running the commands, copy ca.crt
to the server where you will run the commands.
Cluster initialization actions depend on whether the user authentication mode is enabled in the YDB configuration file.
To execute administrative commands (including cluster initialization, database creation, disk management, and others) in a cluster with user authentication mode enabled, you must first get an authentication token using the YDB CLI client version 2.0.0 or higher. You must install the YDB CLI client on any computer with network access to the cluster nodes (for example, on one of the cluster nodes) by following the installation instructions.
When the cluster is first installed, it has a single root
account with a blank password, so the command to get the token is the following:
ydb -e grpcs://<node1.ydb.tech>:2135 -d /Root --ca-file ca.crt \
--user root --no-password auth get-token --force >token-file
You can specify any storage server in the cluster as an endpoint (the -e
or --endpoint
parameter).
If the command above is executed successfully, the authentication token will be written to token-file
. Copy the token file to one of the storage servers in the cluster, then run the following commands on the server:
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd -f token-file --ca-file ca.crt -s grpcs://`hostname -f`:2135 \
admin blobstorage config init --yaml-file /opt/ydb/cfg/config.yaml
echo $?
On one of the storage servers in the cluster, run these commands:
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd --ca-file ca.crt -s grpcs://`hostname -f`:2135 \
admin blobstorage config init --yaml-file /opt/ydb/cfg/config.yaml
echo $?
You will see that the cluster was initialized successfully when the cluster initialization command returns a zero code.
Create a database
To work with row-oriented and column-oriented tables, you need to create at least one database and run a process (or processes) to serve this database (dynamic nodes):
To execute the administrative command for database creation, you will need the ca.crt
certificate file issued by the Certificate Authority (see the above description of cluster initialization).
When creating your database, you set an initial number of storage groups that determine the available input/output throughput and maximum storage. For an existing database, you can increase the number of storage groups when needed.
The database creation procedure depends on whether you enabled user authentication in the YDB configuration file.
Get an authentication token. Use the authentication token file that you obtained when initializing the cluster or generate a new token.
Copy the token file to one of the storage servers in the cluster, then run the following commands on the server:
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd -f token-file --ca-file ca.crt -s grpcs://`hostname -s`:2135 \
admin database /Root/testdb create ssd:1
echo $?
On one of the storage servers in the cluster, run these commands:
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd --ca-file ca.crt -s grpcs://`hostname -s`:2135 \
admin database /Root/testdb create ssd:1
echo $?
You will see that the database was created successfully when the command returns a zero code.
The command example above uses the following parameters:
/Root
: Name of the root domain, must match thedomains_config
.domain
.name
setting in the cluster configuration file.testdb
: Name of the created database.ssd:1
: Name of the storage pool and the number of storage groups allocated. The pool name usually means the type of data storage devices and must match thestorage_pool_types
.kind
setting inside thedomains_config
.domain
element of the configuration file.
Run dynamic nodes
Run the YDB dynamic node for the /Root/testdb
database:
sudo su - ydb
cd /opt/ydb
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd server --grpcs-port 2136 --grpc-ca /opt/ydb/certs/ca.crt \
--ic-port 19002 --ca /opt/ydb/certs/ca.crt \
--mon-port 8766 --mon-cert /opt/ydb/certs/web.pem \
--yaml-config /opt/ydb/cfg/config.yaml --tenant /Root/testdb \
--node-broker grpcs://<ydb1>:2135 \
--node-broker grpcs://<ydb2>:2135 \
--node-broker grpcs://<ydb3>:2135
In the command example above, <ydbN>
is replaced by FQDNs of any three servers running the cluster's static nodes.
Create a systemd configuration file named /etc/systemd/system/ydbd-testdb.service
by the following template: You can also download the sample file from the repository.
[Unit]
Description=YDB testdb dynamic node
After=network-online.target rc-local.service
Wants=network-online.target
StartLimitInterval=10
StartLimitBurst=15
[Service]
Restart=always
RestartSec=1
User=ydb
PermissionsStartOnly=true
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=ydbd
SyslogFacility=daemon
SyslogLevel=err
Environment=LD_LIBRARY_PATH=/opt/ydb/lib
ExecStart=/opt/ydb/bin/ydbd server \
--grpcs-port 2136 --grpc-ca /opt/ydb/certs/ca.crt \
--ic-port 19002 --ca /opt/ydb/certs/ca.crt \
--mon-port 8766 --mon-cert /opt/ydb/certs/web.pem \
--yaml-config /opt/ydb/cfg/config.yaml --tenant /Root/testdb \
--node-broker grpcs://<ydb1>:2135 \
--node-broker grpcs://<ydb2>:2135 \
--node-broker grpcs://<ydb3>:2135
LimitNOFILE=65536
LimitCORE=0
LimitMEMLOCK=32212254720
[Install]
WantedBy=multi-user.target
In the file example above, <ydbN>
is replaced by FQDNs of any three servers running the cluster's static nodes.
Run the YDB dynamic node for the /Root/testdb
database:
sudo systemctl start ydbd-testdb
Run additional dynamic nodes on other servers to ensure database scalability and fault tolerance.
Initial account setup
If authentication mode is enabled in the cluster configuration file, initial account setup must be done before working with the YDB cluster.
The initial installation of the YDB cluster automatically creates a root
account with a blank password, as well as a standard set of user groups described in the Access management section.
To perform initial account setup in the created YDB cluster, run the following operations:
-
Install the YDB CLI as described in the documentation.
-
Set the password for the
root
account:
ydb --ca-file ca.crt -e grpcs://<node.ydb.tech>:2136 -d /Root/testdb --user root --no-password \
yql -s 'ALTER USER root PASSWORD "passw0rd"'
Replace the passw0rd
value with the required password.
- Create additional accounts:
ydb --ca-file ca.crt -e grpcs://<node.ydb.tech>:2136 -d /Root/testdb --user root \
yql -s 'CREATE USER user1 PASSWORD "passw0rd"'
- Set the account rights by including them in the integrated groups:
ydb --ca-file ca.crt -e grpcs://<node.ydb.tech>:2136 -d /Root/testdb --user root \
yql -s 'ALTER GROUP `ADMINS` ADD USER user1'
In the command examples above, <node.ydb.tech>
is the FQDN of the server running any dynamic node that serves the /Root/testdb
database.
When running the account creation and group assignment commands, the YDB CLI client will request the root
user's password. You can avoid multiple password entries by creating a connection profile as described in the YDB CLI documentation.
Start using the created database
-
Install the YDB CLI as described in the documentation.
-
Create a test row (
test_row_table
) or column (test_column_table
) oriented table:
ydb --ca-file ca.crt -e grpcs://<node.ydb.tech>:2136 -d /Root/testdb --user root \
yql -s 'CREATE TABLE `testdir/test_row_table` (id Uint64, title Utf8, PRIMARY KEY (id));'
ydb --ca-file ca.crt -e grpcs://<node.ydb.tech>:2136 -d /Root/testdb --user root \
yql -s 'CREATE TABLE `testdir/test_column_table` (id Uint64, title Utf8, PRIMARY KEY (id)) WITH (STORE = COLUMN);'
Here, <node.ydb.tech>
is the FQDN of the server running the dynamic node that serves the /Root/testdb
database.
Checking access to the built-in web interface
To check access to the YDB built-in web interface, open in the browser the https://<node.ydb.tech>:8765
URL, where <node.ydb.tech>
is the FQDN of the server running any static YDB node.
In the web browser, set as trusted the certificate authority that issued certificates for the YDB cluster. Otherwise, you will see a warning about an untrusted certificate.
If authentication is enabled in the cluster, the web browser should prompt you for a login and password. Enter your credentials, and you'll see the built-in interface welcome page. The user interface and its features are described in Using the embedded web UI.
Note
A common way to provide access to the YDB built-in web interface is to set up a fault-tolerant HTTP balancer running haproxy
, nginx
, or similar software. A detailed description of the HTTP balancer is beyond the scope of the standard YDB installation guide.
Installing YDB in the unprotected mode
Warning
We do not recommend using the unprotected YDB mode for development or production environments.
The above installation procedure assumes that YDB was deployed in the standard protected mode.
The unprotected YDB mode is primarily intended for test scenarios associated with YDB software development and testing. In the unprotected mode:
- Traffic between cluster nodes and between applications and the cluster runs over an unencrypted connection.
- Users are not authenticated (it doesn't make sense to enable authentication when the traffic is unencrypted because the login and password in such a configuration would be transparently transmitted across the network).
When installing YDB to run in the unprotected mode, follow the above procedure, with the following exceptions:
- When preparing for the installation, you do not need to generate TLS certificates and keys and copy the certificates and keys to the cluster nodes.
- In the configuration files, remove the
security_config
subsection underdomains_config
. Remove theinterconnect_config
andgrpc_config
sections entirely. - Use simplified commands to run static and dynamic cluster nodes: omit the options that specify file names for certificates and keys; use the
grpc
protocol instead ofgrpcs
when specifying the connection points. - Skip the step of obtaining an authentication token before cluster initialization and database creation because it's not needed in the unprotected mode.
- Cluster initialization command has the following format:
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd admin blobstorage config init --yaml-file /opt/ydb/cfg/config.yaml
echo $?
- Database creation command has the following format:
export LD_LIBRARY_PATH=/opt/ydb/lib
/opt/ydb/bin/ydbd admin database /Root/testdb create ssd:1
- When accessing your database from the YDB CLI and applications, use grpc instead of grpcs and skip authentication.