Setting up cluster monitoring YDB
This page is part of the Observability overview and describes how to configure metric collection for the YDB cluster in Prometheus and visualization in Grafana. On each node, metrics are also available in the built-in HTTP monitoring interface — the section Accessing metrics via the web interface.
Tip
Before you begin, review the metrics description and Grafana dashboards reference.
Determine how the YDB cluster is deployed:
- using one of the cluster deployment methods (Ansible, Kubernetes, or manually)
- or in the quick start configuration (single-node local cluster).
Setting up monitoring with Prometheus and Grafana
Preparing for installation
The full deployment cycle of Prometheus and Grafana is not covered in this section — refer to the documentation of the selected products.
Options for preparing metric collection configuration
Configuration of metric collection from YDB nodes can be prepared using one of the methods below. In all cases, you will need the prometheus_ydb.yml file, which describes the collection of YDB metrics.
The list of storage nodes and dynamic nodes of the database is specified in the ydbd-storage.yml and ydbd-database.yml files, respectively. The paths to these files are set in prometheus_ydb.yml (for more details, see the Running Prometheus with a prepared configuration section).
The recommended way to get a consistent set of files is to generate a metric collection configuration using the generate_promconf playbook.
Go to the working directory of the Ansible playbook for your cluster:
cd <path_to_ansible_project>
Run the configuration generation playbook for Prometheus:
ansible-playbook ydb_platform.ydb.generate_promconf
The playbook will create a directory promconf with the following contents:
prometheus_ydb.yml— Prometheus configuration file.ydbd-storage.yml: list of cluster storage nodes.ydbd-database.yml— list of dynamic database nodes.ca.crt: certificate used when deploying the cluster.grafana-dashboards— directory with Grafana dashboard templates. Templates are loaded from the GitHub repository.
Check the contents of the generated files ydbd-storage.yml and ydbd-database.yml. The list of nodes and ports must match the actual cluster topology, including the nodes displayed in the built-in cluster UI.
Example of checking the contents of the configuration directory:
cd promconf
ls -la
Example output:
-rw-rw-r-- 1 1818 ca.crt
drwxrwxr-x 2 4096 grafana-dashboards
-rw-rw-r-- 1 17532 prometheus_ydb.yml
-rw-rw-r-- 1 165 ydbd-database.yml
-rw-rw-r-- 1 164 ydbd-storage.yml
Checking node availability — via HTTPS; see Metrics in Prometheus format.
Copy files from the ydb/deploy/prometheus directory of the YDB repository.
Fill in the targets sections in ydbd-storage.yml and ydbd-database.yml: specify the hosts and monitoring ports (--mon-port) of all storage nodes and dynamic database nodes from which to collect metrics (to determine the port, see How to determine the monitoring port).
In prometheus_ydb.yml, for metric collection tasks, set scheme: http and disable or remove the tls_config parameters.
Checking node availability — via HTTP; see Metrics in Prometheus format.
Copy the files from the ydb/deploy/prometheus directory of the YDB repository.
Fill in the targets sections in ydbd-storage.yml and ydbd-database.yml: specify the hosts and monitoring ports (--mon-port) of all storage nodes and dynamic database nodes from which metrics should be collected (to determine the port, see How to determine the monitoring port).
In prometheus_ydb.yml, for metric collection tasks, set scheme: https and configure tls_config. Specify the path to the certificate authority certificate (CA) that signed the cluster's TLS certificates:
scheme: https
tls_config:
ca_file: '<ydb-ca-file>'
Make sure that all paths in tls_config point to existing files and that the user under which Prometheus is running has read permissions for them.
If your configuration uses a client certificate and key, add them to tls_config:
tls_config:
ca_file: '<ydb-ca-file>'
cert_file: '<ydb-client-cert-file>'
key_file: '<ydb-client-key-file>'
Check availability nodes — over HTTPS (curl with https://, see Metrics in Prometheus format). Specify in curl flag --cacert with the same path as in ca_file.
If YDB is running locally in a single-node configuration and monitoring listens on one port (often 8765), in both files — ydbd-storage.yml and ydbd-database.yml — in the targets section, specify the same address, for example localhost:8765 or <hostname>:8765.
Regardless of the chosen method for preparing the configuration, the steps for starting and checking Prometheus below are the same.
Starting Prometheus with the prepared configuration
Place the edited files in any convenient directory on the machine where Prometheus is running (next to the binary or separately, for example /etc/prometheus). Place the files prometheus_ydb.yml, ydbd-storage.yml, and ydbd-database.yml in the same folder.
In prometheus_ydb.yml, in each job of the scrape_configs section, the file_sd_configs parameter specifies which files to take the target list from — ydbd-storage.yml and ydbd-database.yml. By default, the paths in file_sd_configs are relative: Prometheus looks for these files relative to the process's working directory at startup. If you specify absolute paths, you can start Prometheus from any working directory.
If Prometheus is started only for YDB, in the --config.file parameter specify the full or relative path to prometheus_ydb.yml. Before starting, navigate to the configuration directory and start the process from there:
cd <path_to_config_dir>
prometheus --config.file=prometheus_ydb.yml
If Prometheus is already used for other systems, do not replace the main config with the prometheus_ydb.yml file in --config.file. Add to the existing configuration file (usually prometheus.yml) the jobs from the scrape_configs section in prometheus_ydb.yml — all entries with job_name starting with ydb/.
Make sure that ydbd-storage.yml and ydbd-database.yml are accessible via the paths in file_sd_configs of the transferred jobs (see the paragraph about file_sd_configs above).
Transfer the global: section from prometheus_ydb.yml only if necessary and check the values against those already set in your config. Start Prometheus with your config, for example:
prometheus --config.file=<your_prometheus.yml>
After changes, check the configuration:
promtool check config <your_prometheus.yml>
Check that Prometheus is running and responding:
curl "http://localhost:9090/-/healthy"
In the Prometheus web interface (usually port 9090), open Status → Targets and make sure the metric scrape groups are in UP state (successful collection). The exception is the group related to topics: if there are no topics in the database, it may display a 204 No content response. This is not a sign of a configuration error.
Configuring Grafana
Connecting Prometheus as a data source
- Open the Grafana web interface.
- Go to Connections → Data sources → Add data source.
- Select the Prometheus type.
- In the Name field, specify an arbitrary data source name, for example
ydb. - In the Prometheus server URL field, specify the URL of the Prometheus instance that already has metric collection configured from the YDB cluster (for example,
http://localhost:9090if Grafana and Prometheus are on the same machine and Prometheus listens on the default port). - If necessary, fill in the authentication, TLS, and timeout fields according to your installation's policy.
- Click Save & test at the bottom of the screen. If configured correctly, a message about a successful request to the Prometheus API will be displayed (for example,
Successfully queried the Prometheus API).
Additionally, see the Prometheus instructions for creating a data source in Grafana.
Importing dashboards
Ready-made YDB dashboards are located in the repository. If you used an Ansible playbook to generate the Prometheus configuration, the dashboard templates will be placed in the grafana-dashboards subdirectory. Import the JSON files into Grafana via the web interface or provisioning.
The composition of panels and recommendations for using dashboards are provided in the Grafana dashboards reference.
Result
Example dashboard in Grafana
After importing, open the YDB Essential Metrics dashboard. At the top, select the Prometheus data source (as in connecting) and the database name.
After configuration, make sure that the YDB targets in Prometheus are in UP state (see Starting Prometheus with the prepared configuration), and that metrics are displayed on the YDB Essential Metrics dashboard.
Accessing metrics via the web interface
In addition to collection in Prometheus, each cluster node provides a built-in HTTP interface for viewing metrics in a browser. The interface shows the current metric values at the time of the request, without history; continuous collection and storage are configured in Prometheus (see above). The interface listens on port --mon-port (default 8765) on the node host.
The main page is http://<ydb-server-address>:<ydb-port>/counters/: it displays a list of metric groups (subsystems). The names of groups and metrics are in the metric description.
where:
<ydb-server-address>— address of the YDB server.<ydb-port>— the node monitoring port, the--mon-portparameter at startup. Default value:8765. To determine the port on a specific host, see below.
When TLS is enabled on the node, use the https:// scheme in the URL.
How to determine the monitoring port (--mon-port)
If the port is unknown, determine it on the host where the YDB node is running (via SSH or local console). If the cluster has multiple servers, run the command on each of them if necessary.
ps aux | grep ydbd
The output may contain several ydbd processes with different --mon-port values (for example, a static and a dynamic node on the same server). Add to monitoring all ports of the nodes whose metrics you need to collect. The screenshot highlights individual values only as an example — refer to the actual command output on your hosts.
Metric groups on the main page
The main page lists metric groups by subsystem — auth, compile, grpc, kqp, pdisks, vdisks, and others. Each group is a link to a page with metrics of that subsystem.
Viewing metrics of a group (subsystem)
To open the metrics of a single group (subsystem), go to the URL:
http://<ydb-server-address>:<ydb-port>/counters/counters=<servicename>/
<servicename>— the group (subsystem) name. Available groups are displayed on the main page of the monitoring web interface (see Metric groups on the main page).
Use http or https according to the cluster TLS configuration (as in Metrics in Prometheus format).
For example, server resource utilization metrics are in the utils group:
http://<ydb-server-address>:<ydb-port>/counters/counters=utils
Metrics in Prometheus format
The same node provides metrics in Prometheus format — at a URL with the /prometheus suffix. These are the addresses that Prometheus scrapes (the metrics_path parameter in scrape_configs).
http://<ydb-server-address>:<ydb-port>/counters/counters=<servicename>/prometheus
Check endpoint availability:
curl "http://<ydb-server-address>:<ydb-port>/counters/counters=<servicename>/prometheus"
https://<ydb-server-address>:<ydb-port>/counters/counters=<servicename>/prometheus
Check endpoint availability:
curl --cacert <path-to-ca.crt> "https://<ydb-server-address>:<ydb-port>/counters/counters=<servicename>/prometheus"
Specify in --cacert the same path as in ca_file in tls_config (see configuration preparation). With a self-signed certificate, you can add the -k flag for diagnostics.
Relationship with Prometheus configuration
In the template prometheus_ydb.yml, hosts and ports match ydbd-storage.yml and ydbd-database.yml. For each metric group in scrape_configs, a metrics_path of the form /counters/counters=<servicename>/prometheus is set — the same subsystems as in the list on the main page of the web interface.
Other systems that support the Prometheus format (Zabbix, Amazon CloudWatch, etc.) connect to the same URLs.