Metrics reference

This document provides a reference for the main YDB system metrics used for monitoring cluster state, diagnosing performance, and analyzing load.

How to view current metric values in the built-in web interface and configure their collection in Prometheus and Grafana is described in the Configuring monitoring of the YDB cluster section.

Resource usage metrics

Metric name Type, units	Description Labels
resources.storage.used_bytes IGAUGE, bytes	Size of user and service data stored in the distributed network storage. resources.storage.used_bytes = resources.storage.table.used_bytes + resources.storage.topic.used_bytes.
resources.storage.table.used_bytes IGAUGE, bytes	Size of user and service data stored by tables in the distributed network storage. Service data includes data of primary, secondary indexes, vector indexes, full-text indexes, local Bloom indexes, and local min_max index.
resources.storage.topic.used_bytes IGAUGE, bytes	Size of the distributed network storage used by topics. Equals the sum of topic.storage_bytes values of all topics.
resources.storage.limit_bytes IGAUGE, bytes	Limit on the size of user and service data that the database can store in the distributed network storage.

Common gRPC API metrics

Metric name Type, units	Description Labels
api.grpc.request.bytes RATE, bytes	Size of requests received by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table or data_streams. - method – name of the gRPC API service method, for example ExecuteDataQuery (for the table service), or PutRecord, GetRecords (for the data_streams service).
api.grpc.request.dropped_count RATE, units	Number of requests whose processing was terminated at the transport (gRPC) level due to an error. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.request.inflight_count IGAUGE, units	Number of requests being processed simultaneously by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.request.inflight_bytes IGAUGE, bytes	Size of requests being processed simultaneously by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.response.bytes RATE, bytes	Size of responses sent by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.response.count RATE, units	Number of responses sent by the database in a certain time period. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery. - status – request execution status; for more details on statuses, see the Error handling section.
api.grpc.response.dropped_count RATE, units	Number of responses whose sending was terminated at the transport (gRPC) level due to an error. Labels: - api_service – name of the gRPC API service, for example table. - method – name of the gRPC API service method, for example ExecuteDataQuery.
api.grpc.response.issues RATE, units	Number of errors of a certain type that occurred during query execution in a certain time period. Labels: - issue_type – error type, the only value is optimistic_locks_invalidation; for more details on lock invalidation, see the Transactions and queries to YDB section.

gRPC API metrics for topics

Metric name Type, units	Description Labels
grpc.topic.stream_read.commits RATE, units	Number of commits of the Ydb::TopicService::StreamRead method. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.bytes RATE, units	Number of bytes read by the Ydb::TopicService::StreamRead method. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.messages RATE, units	Number of messages read by the Ydb::TopicService::StreamRead method. Labels: - topic – topic name. - consumer – consumer name.
grpc.topic.stream_read.partition_session.errors RATE, units	Number of errors when working with a partition. Labels: - topic – topic name. - consumer – reader name.
grpc.topic.stream_read.partition_session.started RATE, count	Number of sessions started per unit of time. Labels: - topic – topic name. - consumer – reader name.
grpc.topic.stream_read.partition_session.stopped RATE, count	Number of sessions stopped per unit of time. Labels: - topic – topic name. - consumer – reader name.
grpc.topic.stream_read.partition_session.starting_count RATE, count	Number of sessions being started (that is, the client received a command to start a session, but the client has not started the session yet). Labels: - topic – topic name. - consumer – reader name.
grpc.topic.stream_read.partition_session.stopping_count RATE, count	Number of sessions being stopped. Labels: - topic – topic name. - consumer – reader name.
grpc.topic.stream_read.partition_session.count RATE, count	Number of partition_session. Labels: - topic – topic name. - consumer – reader name.
grpc.topic.stream_write.bytes RATE, bytes	Number of bytes written by the Ydb::TopicService::StreamWrite method. Labels: - topic – topic name.
grpc.topic.stream_write.uncommitted_bytes RATE, bytes	Number of bytes written by the Ydb::TopicService::StreamWrite method within uncommitted transactions. Labels: - topic – topic name.
grpc.topic.stream_write.errors RATE, count	Number of errors when calling the Ydb::TopicService::StreamWrite method. Labels: - topic – topic name.
grpc.topic.stream_write.messages RATE, count	Number of messages written by the Ydb::TopicService::StreamWrite method. Labels: - topic – topic name.
grpc.topic.stream_write.uncommitted_messages RATE, count	Number of messages written by the Ydb::TopicService::StreamWrite method within uncommitted transactions. Labels: - topic – topic name.
grpc.topic.stream_write.partition_throttled_milliseconds HIST_RATE, count	Histogram counter. Intervals are set in milliseconds. Shows the number of messages that waited on the quota. Labels: - topic – topic name.
grpc.topic.stream_write.sessions_active_count GAUGE, count	Number of open write sessions. Labels: - topic – topic name.
grpc.topic.stream_write.sessions_created RATE, count	Number of created write sessions. Labels: - topic – topic name.

HTTP API metrics

Metric name Type, units	Description Labels
api.http.data_streams.request.count RATE, count	Number of requests over HTTP. Labels: - method – HTTP API service method name, e.g., PutRecord, GetRecords. - topic – topic name.
api.http.data_streams.request.bytes RATE, bytes	Total size of requests over HTTP. Labels: - method – HTTP API service method name, in this case only PutRecord. - topic – topic name.
api.http.data_streams.response.count RATE, count	Number of responses over HTTP. Labels: - method – HTTP API service method name, e.g., PutRecord, GetRecords. - topic – topic name. - code – HTTP response code.
api.http.data_streams.response.bytes RATE, bytes	Total size of responses over HTTP. Labels: - method – HTTP API service method name, in this case only GetRecords. - topic – topic name.
api.http.data_streams.response.duration_milliseconds HIST_RATE, count	Histogram counter. Intervals are set in milliseconds. Shows the number of responses whose execution time falls within a certain interval. Labels: - method – HTTP API service method name. - topic – topic name.
api.http.data_streams.get_records.messages RATE, count	Number of messages read by the GetRecords method. Labels: - topic – topic name.
api.http.data_streams.put_record.messages RATE, count	Number of messages sent by the PutRecord method. Labels: - topic – topic name.
api.http.data_streams.put_records.failed_messages RATE, count	Number of messages sent by the PutRecords method that were not written. Labels: - topic – topic name.
api.http.data_streams.put_records.successful_messages RATE, count	Number of messages sent by the PutRecords method that were successfully written. Labels: - topic – topic name.
api.http.data_streams.put_records.total_messages RATE, count	Number of messages sent by the PutRecords method. Labels: - topic – topic name.

Kafka API metrics

Metric name Type, units	Description Labels
api.kafka.request.count RATE, count	Number of requests over the Kafka protocol per unit time. Labels: - method – Kafka API service method name, e.g., PRODUCE, SASL_HANDSHAKE.
api.kafka.request.bytes RATE, bytes	Total size of requests over the Kafka protocol per unit time. Labels: - method – Kafka API service method name, e.g., PRODUCE, SASL_HANDSHAKE.
api.kafka.response.count RATE, count	Number of responses via the Kafka protocol per unit time. Labels: - method – name of the Kafka API service method, e.g., PRODUCE, SASL_HANDSHAKE. - error_code – Kafka response code.
api.kafka.response.bytes RATE, bytes	Total size of responses via the Kafka protocol per unit time. Labels: - method – name of the Kafka API service method, e.g., PRODUCE, SASL_HANDSHAKE.
api.kafka.response.duration_milliseconds HIST_RATE, units	Histogram counter. Defines a set of intervals in milliseconds and for each of them shows the number of requests with execution time falling within that interval. Labels: - method – Kafka API service method name.
api.kafka.produce.failed_messages RATE, units	Number of messages per unit time sent by the PRODUCE method that were not written. Labels: - topic – topic name.
api.kafka.produce.successful_messages RATE, units	Number of messages per unit time sent by the PRODUCE method that were successfully written. Labels: - topic – topic name.
api.kafka.produce.total_messages RATE, units	Number of messages per unit time sent by the PRODUCE method. Labels: - topic – topic name.

Session metrics

Metric name Type, units	Description Labels
table.session.active_count IGAUGE, units	Number of sessions currently open by clients.
table.session.closed_by_idle_count RATE, units	Number of sessions closed by the database server in a certain period of time due to exceeding the time allocated for an inactive session's lifetime.

Transaction processing metrics

Transaction execution duration can be analyzed using a histogram counter. Intervals are set in milliseconds. The chart shows the number of transactions whose duration falls within a certain time interval.

Metric name Type, units	Description Labels
table.transaction.total_duration_milliseconds HIST_RATE, units	Number of transactions with a specific execution duration on the server and client. Execution duration is the time from the explicit or implicit start of a transaction to the commit or rollback. It includes the transaction processing time on the server and the time on the client between sending different requests within the same transaction. Labels: - tx_kind – transaction type, possible values read_only, read_write, write_only, pure.
table.transaction.server_duration_milliseconds HIST_RATE, units	Number of transactions with a specific execution duration on the server. Execution duration is the time of executing queries within a transaction on the server. It does not include the waiting time on the client between sending individual requests in the same transaction. Labels: - tx_kind – transaction type, possible values read_only, read_write, write_only, pure.
table.transaction.client_duration_milliseconds HIST_RATE, units	Number of transactions with a specific execution duration on the client. Execution duration is the waiting time on the client between sending individual requests in the same transaction. It does not include the execution time of queries on the server. Labels: - tx_kind – transaction type, possible values read_only, read_write, write_only, pure.

Query processing metrics

Metric name Type, units	Description Labels
table.query.request.bytes RATE, bytes	Size of YQL query text and parameter values for queries that arrived at the database in a certain period of time.
table.query.request.parameters_bytes RATE, bytes	Size of parameters for queries that arrived at the database in a certain period of time.
table.query.response.bytes RATE, bytes	Size of responses sent by the database in a certain period of time.
table.query.compilation.latency_milliseconds HIST_RATE, units	Histogram counter. Intervals are set in milliseconds. Shows the number of successfully completed compilation requests whose duration falls within a certain time interval.
table.query.compilation.active_count IGAUGE, units	Number of compilations currently in progress.
table.query.compilation.count RATE, units	Number of compilations that completed successfully in a certain period of time.
table.query.compilation.errors RATE, units	Number of compilations that failed in a certain period of time.
table.query.compilation.cache_hits RATE, units	Number of queries in a certain period of time that did not require compiling the query because a previously created plan was in the prepared query cache.
table.query.compilation.cache_misses RATE, count	The number of queries in a given period of time that required compiling a query.
table.query.execution.latency_milliseconds HIST_RATE, count	Histogram counter. Intervals are specified in milliseconds. Shows the number of queries whose execution time falls within a certain interval.

Row table partition metrics

Metric name Type, units of measurement	Description Labels
table.datashard.row_count GAUGE, count	Number of rows in all row tables of the database.
table.datashard.size_bytes GAUGE, bytes	Size of data in all row tables of the database.
table.datashard.used_core_percents HIST_GAUGE, %	Histogram counter. Intervals are specified in percentages. Shows the number of partitions of row tables that use computing resources in a proportion that falls within a certain interval.
table.datashard.read.rows RATE, count	Number of rows read by all partitions of all row tables in the database during a given period of time.
table.datashard.read.bytes RATE, bytes	Size of data read by all partitions of all row tables in the database during a given period of time.
table.datashard.write.rows RATE, count	Number of rows written by all partitions of all row tables in the database during a given period of time.
table.datashard.write.bytes RATE, bytes	Size of data written by all partitions of all row tables in the database during a given period of time.
table.datashard.scan.rows RATE, count	Number of rows read via gRPC API calls StreamExecuteScanQuery or StreamReadTable by all partitions of all row tables in the database over a certain time period.
table.datashard.scan.bytes RATE, bytes	Size of data read via gRPC API call StreamExecuteScanQuery or StreamReadTable by all partitions of all row tables in the database over a certain time period.
table.datashard.bulk_upsert.rows RATE, count	Number of rows added via gRPC API call BulkUpsert to all partitions of all row tables in the database over a certain time period.
table.datashard.bulk_upsert.bytes RATE, bytes	Size of data added via gRPC API call BulkUpsert to all partitions of all row tables in the database over a certain time period.
table.datashard.erase.rows RATE, count	Number of rows deleted from all row tables in the database during a given period of time.
table.datashard.erase.bytes RATE, bytes	Size of data deleted from all row tables in the database during a given period of time.
table.datashard.cache_hit.bytes RATE, bytes	Total volume of row table data successfully retrieved from memory (cache). A larger volume of data retrieved from the cache indicates efficient cache usage without accessing the distributed storage.
table.datashard.cache_miss.bytes RATE, bytes	Total volume of row table data that was requested but not found in memory (cache) and was read from distributed storage. Indicates potential areas for cache optimization.

Columnar table partition metrics

Metric name Type, units of measurement	Description Labels
table.columnshard.write.rows RATE, count	Number of rows written by all partitions of all columnar tables in the database during a given period of time.
table.columnshard.write.bytes RATE, bytes	Size of data written by all partitions of all columnar tables in the database during a given period of time.
table.columnshard.scan.rows RATE, count	Number of rows read via gRPC API calls StreamExecuteScanQuery or StreamReadTable by all partitions of all columnar tables in the database over a certain time period.
table.columnshard.scan.bytes RATE, bytes	Size of data read via gRPC API call StreamExecuteScanQuery or StreamReadTable by all partitions of all columnar tables in the database over a certain time period.
table.columnshard.bulk_upsert.rows RATE, count	Number of rows added via gRPC API call BulkUpsert to all partitions of all columnar tables in the database over a certain time period.
table.columnshard.bulk_upsert.bytes RATE, bytes	Size of data added via gRPC API call BulkUpsert to all partitions of all columnar tables in the database over a certain time period.

Resource usage metrics (Dedicated mode only)

Metric name Type units of measurement	Description Labels
resources.cpu.used_core_percents RATE, %	CPU usage. A value of 100 means that one of the cores is 100% utilized. The value can be greater than 100 for configurations with more than 1 core. Labels: - pool – compute pool, possible values user, system, batch, io, ic.
resources.cpu.limit_core_percents IGAUGE, %	CPU available to the database as a percentage. For example, for a database of three nodes with 4 cores in pool=user on each node, the value of this sensor will be 1200. Labels: - pool – compute pool, possible values user, system, batch, io, ic.
resources.memory.used_bytes IGAUGE, bytes	RAM used by database nodes.
resources.memory.limit_bytes IGAUGE, bytes	RAM available to database nodes.

Query processing metrics (Dedicated mode only)

Metric name Type units of measurement	Description Labels
table.query.compilation.cache_evictions RATE, count	Number of queries evicted from the prepared query cache during a given period of time.
table.query.compilation.cache_size_bytes IGAUGE, bytes	Size of the prepared query cache.
table.query.compilation.cached_query_count IGAUGE, count	Number of queries in the prepared query cache.

Topic metrics

Metric name Type, units of measurement	Description Labels
topic.producers_count GAUGE, count	Number of unique topic sources. Labels: - topic – topic name.
topic.storage_bytes GAUGE, bytes	Topic size in bytes. Labels: - topic – topic name.
topic.read.bytes RATE, bytes	Number of bytes read from the topic. Labels: - topic – topic name. - consumer – reader name.
topic.read.messages RATE, count	Number of messages read from the topic. Labels: - topic – topic name. - consumer – reader name.
topic.read.lag_messages RATE, count	Total number of messages not yet read by the given reader across the topic. Labels: - topic – topic name. - consumer – reader name.
topic.read.lag_milliseconds HIST_RATE, count	Histogram counter. Intervals are specified in milliseconds. Shows the number of messages for which the difference between the read time and the message creation time falls within a given interval. Labels: - topic – topic name. - consumer – reader name.
topic.write.bytes RATE, bytes	Size of written data. Labels: - topic – topic name.
topic.write.uncommited_bytes RATE, bytes	Size of data written as part of not yet completed transactions. Labels: - topic – topic name.
topic.write.uncompressed_bytes RATE, bytes	Size of decompressed written data. Labels: - topic – topic name.
topic.write.messages RATE, count	Number of written messages. Labels: - topic – topic name.
topic.write.uncommitted_messages RATE, count	Number of messages written as part of not yet completed transactions. Labels: - topic – topic name.
topic.write.message_size_bytes HIST_RATE, count	Histogram counter. Intervals are specified in bytes. Shows the number of messages whose size matches the interval boundaries. Labels: - topic – topic name.
topic.write.lag_milliseconds HIST_RATE, count	Histogram counter. Intervals are specified in milliseconds. Shows the number of messages for which the difference between the write time and the message creation time falls within a given interval. Labels: - topic – topic name.

Aggregated topic partition metrics

The following table lists aggregated partition metrics for a topic. Maximum and minimum values are calculated across all partitions of the specified topic.

Metric name Type, units	Description Labels
topic.partition.init_duration_milliseconds_max GAUGE, milliseconds	Maximum partition initialization delay. Labels: - topic – topic name.
topic.partition.producers_count_max GAUGE, count	Maximum number of sources in a partition. Labels: - topic – topic name.
topic.partition.storage_bytes_max GAUGE, bytes	Maximum partition size in bytes. Labels: - topic – topic name.
topic.partition.uptime_milliseconds_min GAUGE, count	Minimum partition uptime after restart. Normally during a rolling restart topic.partition.uptime_milliseconds_min is close to 0, after the rolling restart ends, the value of topic.partition.uptime_milliseconds_min should increase to infinity. Labels: - topic – topic name.
topic.partition.total_count GAUGE, count	Total number of partitions in the topic. Labels: - topic – topic name.
topic.partition.alive_count GAUGE, count	Number of partitions reporting their metrics. Labels: - topic – topic name.
topic.partition.committed_end_to_end_lag_milliseconds_max GAUGE, milliseconds	Maximum (across all partitions) difference between the current time and the creation time of the last committed message. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.committed_lag_messages_max GAUGE, units	Maximum (across all partitions) difference between the last partition offset and the committed partition offset. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.committed_read_lag_milliseconds_max GAUGE, milliseconds	Maximum (across all partitions) difference between the current time and the write time of the last committed message. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.end_to_end_lag_milliseconds_max GAUGE, milliseconds	Difference between the current time and the minimum creation time among all messages read in the last minute across all partitions. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.lag_messages_max GAUGE, units	Maximum difference (across all partitions) between the last offset in the partition and the last read offset. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.read.idle_milliseconds_max GAUGE, milliseconds	Maximum idle time (how long the partition has not been read from) across all partitions. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.read.lag_milliseconds_max GAUGE, milliseconds	Difference between the current time and the minimum write time among all messages read in the last minute across all partitions. Labels: - topic – topic name. - consumer – consumer name.
topic.partition.write.lag_milliseconds_max GAUGE, milliseconds	Maximum difference between the write time and the creation time among all messages written in the last minute. Labels: - topic – topic name.
topic.partition.write.speed_limit_bytes_per_second GAUGE, bytes per second	Write quota in bytes per second per partition. Labels: - topic – topic name.
topic.partition.write.throttled_nanoseconds_max GAUGE, nanoseconds	Maximum write throttling time (waiting on quota) across all partitions. In the limit, if topic.partition.write.throttled_nanoseconds_max = 10^9, it means that the entire second was spent waiting on quota. Labels: - topic – topic name.
topic.partition.write.bytes_per_day_max GAUGE, bytes	Maximum number of bytes written in the last 24 hours across all partitions. Labels: - topic – topic name.
topic.partition.write.bytes_per_hour_max GAUGE, bytes	Maximum number of bytes written in the last hour across all partitions. Labels: - topic – topic name.
topic.partition.write.bytes_per_minute_max GAUGE, bytes	Maximum number of bytes written in the last minute across all partitions. Labels: - topic – topic name.
topic.partition.write.idle_milliseconds_max GAUGE, milliseconds	Maximum partition write idle time. Labels: - topic – topic name.

Resource pool metrics

Metric name Type, units	Description Labels
kqp.workload_manager.CpuQuotaManager.AverageLoadPercentage RATE, units	Average database load, DATABASE_LOAD_CPU_THRESHOLD operates based on this metric.
kqp.workload_manager.InFlightLimit GAUGE, units	Limit on the number of concurrently running queries.
kqp.workload_manager.GlobalInFly GAUGE, units	Current number of concurrently running queries. Displayed only for pools with CONCURRENT_QUERY_LIMIT or DATABASE_LOAD_CPU_THRESHOLD enabled.
kqp.workload_manager.QueueSizeLimit GAUGE, units	Size of the queue of queries waiting to be executed.
kqp.workload_manager.GlobalDelayedRequests GAUGE, units	Number of queries waiting in the execution queue. Displayed only for pools with CONCURRENT_QUERY_LIMIT or DATABASE_LOAD_CPU_THRESHOLD enabled.

See also

• Grafana dashboards for monitoring YDB metrics