Glossary YDB

This article provides an overview of the terms and definitions used in YDB and its documentation. It begins with key terms that are useful to get familiar with early in your work with YDB, and the rest of the article contains more advanced terms that may be useful later.

Key terminology

This section describes terms that are useful to anyone working with YDB, regardless of their role or usage scenario.

Cluster

A cluster YDB is a set of interconnected nodes YDB that exchange data to execute user queries and reliably store data. These nodes form one of the supported cluster topologies, which directly affects its reliability and performance characteristics.

Clusters YDB are multi-tenant and can contain several isolated databases.

Database

As in most database management systems, a database in YDB is a logical container for other entities, such as tables. However, in YDB, the namespace within databases is hierarchical, like in virtual file systems, and thus directories allow for a more structured organization of entities.

Another important characteristic of YDB databases is that they are usually allocated dedicated computing resources. As a result, creating a database requires additional actions by DevOps engineers.

Node

YDB A node is a server process that runs an executable file called ydbd. Multiple nodes YDB can run on a single physical server or virtual machine, which is common practice. Thus, in the context of YDB, nodes are not synonymous with hosts.

Since YDB uses a storage and compute separation approach, ydbd has several operation modes that define the node type. The available node types are described below.

Database node

Database nodes (also known as tenant nodes or compute nodes) process user queries addressed to a specific logical database. Their state is only in RAM and can be restored from distributed storage. The set of database nodes of a given cluster YDB can be considered the compute layer of that cluster. Thus, adding database nodes and allocating additional resources (CPU and RAM) to them are the main ways to increase the compute resources of a database.

The main role of database nodes is to run various tablets and actors, as well as to receive incoming requests over the network.

Storage node

Storage nodes are stateful nodes responsible for long-term storage of data fragments. The set of storage nodes of a given cluster YDB is called distributed storage and can be considered as the storage layer of that cluster. Thus, adding additional storage nodes and their disks is the primary way to increase the storage capacity and I/O throughput of the cluster.

Hybrid node

Hybrid node is a process that simultaneously performs both roles of a database node and a storage node. Hybrid nodes are often used for development purposes. For example, you can run a container with a full-featured YDB containing only one ydbd process in hybrid mode. They are rarely used in production environments.

Static node

Static nodes are configured manually during initial cluster initialization or reconfiguration. Typically, they serve as storage nodes, but it is technically possible to configure them as database nodes as well.

Dynamic node

Dynamic nodes are added to and removed from the cluster on the fly. They can only serve as database nodes.

Distributed storage

Distributed storage, Blob storage, or BlobStorage is a distributed fault-tolerant data storage layer in YDB. It has a specialized API designed for storing immutable data fragments of a tablet.

Many terms related to the implementation of distributed storage are discussed below.

Storage group

A storage group is a place for reliable data storage, similar to RAID but using disks from multiple servers. Depending on the chosen cluster topology, storage groups use different algorithms to ensure high availability, similar to standard RAID levels.

Distributed storage typically manages a large number of relatively small storage groups. Each group can be assigned to a specific database to increase the disk space capacity and I/O throughput available to that database.

Static and dynamic storage groups are physical, meaning their data is placed directly on VDisks.

Static group

A static group is a special storage group created during the initial cluster deployment. Its main role is to store data of system tablets, which can be considered as cluster-level metadata.

A static group may require special attention during major cluster maintenance, such as decommissioning an availability zone.

Dynamic group

Ordinary storage groups that are not static are called dynamic groups. They are called dynamic because they can be created and deleted on the fly while the cluster is running.

Virtual storage group

A virtual storage group is an entity that is not actually a storage group but appears as one from the outside (provides a similar external interface). It can store its data in other storage groups or in S3.

Storage pool

A storage pool is a set of data storage devices with similar characteristics. Each storage pool is assigned a unique name within the YDB cluster. Technically, each storage pool consists of multiple physical disks ( PDisk). Each storage group is created in a specific storage pool, which determines the performance characteristics of the storage group through the selection of appropriate storage devices. Typically, separate storage pools are created for devices of different types (e.g., NVMe, SSD, and HDD) or for specific models of these devices that have different capacity and access speed.

Actor

The actor model is one of the fundamental approaches to concurrency used in YDB. In this model, actors are lightweight user-space processes that can have and modify their private state but can only influence each other indirectly through message passing. YDB has its own implementation of this model, which is described below.

In YDB, actors with reliably persisted state are called tablets.

Tablet

A tablet is one of the core building blocks and abstractions of YDB. It represents an entity responsible for a relatively small segment of user or system data. Typically, a tablet manages up to several gigabytes of data, but some types of tablets can handle larger volumes.

For example, a row-based user table is managed by one or more tablets of type DataShard, with each tablet responsible for a continuous range of primary keys and their corresponding data.

End users sending queries to a YDB cluster for execution do not need to know the details of tablets, their types, or how they work, but this knowledge can be useful, for example, for performance optimization.

Technically, tablets are actors with state reliably stored in distributed storage. This state allows the tablet to continue operating on another database node if the previous one fails or becomes overloaded.

Tablet implementation details and related terms, as well as main tablet types, are discussed below.

Transactions

YDB implements transactions at two main levels:

• Local database and the rest of the tablet infrastructure allow tablets to manipulate their state using local transactions with serializable isolation level. Technically, they are not local to a single node, since this state is stored remotely in distributed storage.
• In the context of YDB, the term distributed transactions usually refers to transactions that span multiple tablets. For example, transactions between tables or even rows of a single table are often distributed.
• Single-shard transactions cover a single tablet and execute faster. For example, transactions between rows of a single table partition are often single-shard.

These mechanisms allow YDB to provide strong consistency.

The implementation of distributed transactions is discussed in a separate article DataShard: distributed transactions, and below is a list of several related terms.

Sessions

Logical connections to the database that store the context needed for executing queries and managing transactions. Sessions are described in more detail in the section Sessions.

Client-side timeout

Client-side timeout is a time limit that an application or YDB SDK waits for a database operation to complete (for example, executing a query or receiving a response to a gRPC call). When this time expires, the client usually aborts the wait: closes the connection or data stream, receives an error from the transport or SDK — before the server has had a chance to return an explicit response (see codes of YDB server responses).

If the client-side timeout is shorter than the query execution time on the YDB side, then due to the specifics of query processing in the cluster, a query interrupted on the client may continue to execute on the server for some time. If this situation occurs on a large scale, the server becomes overloaded with queries for which the client is not waiting for a response. Therefore, frequent retries of the same query immediately after a timeout can exacerbate the overload. For more details, see the articles Retry cascade effect and Overloaded errors; retry policies in the SDK are described in the section Handling errors.

Transaction retry

Transaction retry — a client practice of re-executing a transaction from the beginning upon a retryable error (for example, a temporary network failure or optimistic locking conflict). In YDB, retries should be performed at the transaction level, not at the level of individual queries within it. Built-in retry policies in the YDB SDK and integrations (for example, spring-ydb-retry) implement this approach. For more details, see Handling errors.

Exponential backoff

Exponential backoff (also known as backoff) — a pause strategy between transaction retry attempts: the wait interval increases exponentially with each attempt, usually with an upper limit. YDB SDKs often use two levels of backoff — fast and slow — depending on the error type. For more details, see Handling retryable errors.

Jitter

Jitter is a small random variation added to the delays of exponential backoff. It helps avoid simultaneous retries by many clients after a common failure (a "retry storm") and distributes the load more evenly.

Idempotency

Idempotency is a property of an operation: repeated execution has the same effect as a single execution (for example, UPSERT with a deterministic primary key or read operations). Transaction retries are safe only for idempotent operations or for retry errors where the server guarantees that the transaction was not committed. SDK and client libraries YDB can extend the set of retryable status codes if the calling code marks the operation as idempotent.

Transaction interceptor

Transaction interceptor is a Spring Framework component that wraps methods annotated with @Transactional and manages transaction boundaries. Modules like spring-ydb-retry replace the standard Spring interceptor, adding transaction retry logic around transactional methods.

Implicit transactions

Implicit transaction is a query execution mode where the transaction mode is not specified. In this case, YDB independently determines whether to wrap them in a transaction. This mode is described in more detail in Implicit Transactions.

Multi-version concurrency control

Multi-version concurrency control, also known as MVCC, is a method used by YDB to allow multiple concurrent transactions to access the database without interfering with each other. It is described in more detail in a separate article Multi-Version Concurrency Control (MVCC).

Streaming queries

A type of query designed for stream processing of an unbounded data stream. Unlike regular queries, streaming queries have no limits on execution duration, automatically restart on errors, and periodically save their state as checkpoints for fault tolerance. Watermarks are used to track processing progress based on event time.

Streaming queries are described in more detail in a separate article Streaming queries.

Streaming query checkpoints

The periodically saved state of a streaming query, necessary for automatically restoring its operation after failures in a distributed system. For more details on checkpoints, see the article Checkpoints.

Streaming query watermarks

A monotonically increasing lower bound on the event times in a streaming query that may still arrive in the stream. When the watermark reaches value X, the system declares that all events with time less than X have been received with high probability. For more details on watermarks, see the article Watermarks.

Topology

YDB supports several topologies of a cluster (or topology), described in more detail in a separate article Cluster topology YDB. Below are explanations of several related terms.

Availability zones and regions

An availability zone is a data center or its isolated segment with minimal physical distance between nodes and minimal risk of failure simultaneously with other availability zones. Thus, availability zones should not share common infrastructure such as power, cooling, or external network connections.

A region is a large geographic area containing multiple availability zones. The distance between availability zones in a region should be about 500 km or less. YDB writes data to each availability zone in the region synchronously, ensuring reasonable latency and uninterrupted operation in the event of a failure of one of the availability zones.

Rack

Rack or server rack is equipment used to organize the placement of multiple servers. Servers in the same rack are more likely to become unavailable simultaneously due to rack-level issues related to power, cooling, etc. YDB can take into account information about which server is in which rack when placing each data fragment in environments based on physical servers.

Pile

Pile is a set of nodes that can fail or be shut down simultaneously while maintaining the operability of other parts of the cluster (pile). A pile can remain operational when other cluster nodes are shut down. Piles are used in bridge mode to split the cluster into several parts between which synchronous replication is performed. A pile can consist of nodes from one or more regions.

Bridge mode

Bridge mode is a special cluster topology in which data is stored with synchronous replication between several piles. The features of this mode are described in Bridge mode and in Bridge cluster operation mode.

Table

A table is a structured piece of information organized into rows and columns. Each row represents a single record or item, and each column is a specific attribute or field with a defined data type.

There are two main approaches to representing tabular data in memory or on disks: row-oriented (row by row) and column-oriented (column by column). The chosen approach greatly affects the performance characteristics of operations on this data: the former is more suitable for transactional workloads (OLTP), and the latter for analytical workloads (OLAP). YDB supports both approaches.

Row-oriented table

Row-oriented tables store data for all or most columns of each row physically close together. They are described in more detail in Row-Oriented Tables.

Column-oriented table

Column-oriented tables or columnar tables store data for each column separately. They are optimized for building aggregates over a small number of columns, but are less suitable for accessing specific rows, as rows need to be reconstructed from their cells on the fly. They are described in more detail in Column-Oriented Tables.

Primary key

Primary key is an ordered list of columns whose values uniquely identify a row. It is used to create the table's primary index. It is set by the YDB user when creating a table and significantly affects the performance of operations on that table.

Guidance on choosing primary keys is provided in Choosing a primary key.

Primary index

Primary index or primary key index is the main data structure used to find rows in a table. It is created based on the selected primary key and determines the physical order of rows in the table; thus, each table can have only one primary index. The primary index is unique.

Secondary index

Secondary index is an additional data structure used to find rows in a table, typically when this cannot be done efficiently using the primary index. Unlike the primary index, secondary indexes are managed independently of the table's main data. Thus, a table can have multiple secondary indexes for different scenarios. YDB capabilities regarding secondary indexes are described in a separate article Secondary indexes. A secondary index can be either unique or non-unique.

Special types of secondary indexes are distinguished separately: vector index, full-text index, and JSON index.

Vector index

A vector index is an additional data structure used to speed up the vector search problem when there is a large amount of data and exact vector search without an index does not work satisfactorily.
The capabilities of YDB for approximate nearest neighbor search (ANN search) using vector indexes are described in a separate article Vector Indexes.

Vector index is a specialized type of secondary index designed for similarity search, unlike traditional secondary indexes, which are optimized for equality or range searches.

Full-text index

A full-text index is an additional data structure used to speed up text search across a table column (by words and phrases, and, when using N-grams, by substrings).

The full-text search capabilities and index parameters are described in the articles Fulltext Indexes and Fulltext search.

JSON index

JSON index is an additional data structure used to speed up predicates with the JSON_EXISTS and JSON_VALUE functions on a column of type Json or JsonDocument. Unlike traditional secondary indexes optimized for equality or range searches on individual table columns, the JSON index works with arbitrary JsonPath paths within a JSON document.

A JSON index, like a full-text index, is built on top of an inverted index, but uses its own JSON document tokenizer. JSON search capabilities are described in the articles JSON indexes and Searching JSON document contents.

Local index

A local index is an auxiliary structure that is stored together with the table data (unlike a global secondary index, which materializes a separate index table). A local index is used when reading the main table on the storage side. For more information, see local indexes.

Bloom filter

Bloom filter — probabilistic data structure that allows you to quickly check whether an element belongs to a set. False positives are possible, but false negatives are not.

Local Bloom index

A Local Bloom index is a special case of a local index: a probabilistic filter based on column values using a Bloom filter, which speeds up selective queries by skipping data fragments where the searched value is guaranteed to be absent. For more information, see Bloom indexes, local indexes.

Local min_max index

A local min_max index is a special case of a local index: a range filter that stores the minimum and maximum value of one column for each data fragment and skips fragments whose range cannot satisfy the query predicate. For more details: min_max index, local indexes.

Column family

Column family or column group is a feature that allows storing subsets of columns of a row table separately in a separate family or group. The main use case is storing some columns on other disk types (moving less important columns to HDD) or with different compression settings. If the workload requires many column families, consider using column tables.

Column encoding

Column encoding is a mechanism for optimizing data storage in table columns that reduces disk space usage and speeds up some operations.

Time to Live

Time to live or TTL is a mechanism for automatically deleting old rows from a table asynchronously in the background. It is described in a separate article Time to Live (TTL) and Eviction to External Storage.

Representation

A view is a way to save a query and access its results as if they were a real table. The view itself does not store data, except for the query text. The query stored in the view is executed on each SELECT from it, generating the returned result. Any changes to the tables referenced by the view are immediately reflected in the results of reading from it.

Views can be user-defined or system.

User-defined views

User views are created by the user using the CREATE VIEW command. They are described in more detail in View.

System views

System views are special views automatically created by the system for monitoring the state of a database and cluster. They are located in the special directory .sys, which is in the root folder of each database. System views for databases are described in System database views; system views for the cluster, as well as access control issues, are described in Cluster System Views.

Topic

A message queue is used for reliable asynchronous communication between different systems by passing messages. YDB provides infrastructure that ensures "exactly once" semantics in such communications. Using it, you can guarantee that no messages are lost and no random duplicates occur.

Topic is a named entity in a message queue, designed for interaction between writers and readers.

Several terms related to topics are given below. How topics work in YDB is explained in more detail in a separate article Topic.

Partition

For horizontal scaling, topics are divided into separate elements called partitions. Thus, partitions are the unit of parallelism within a topic. Messages within each partition are ordered.

However, subsets of data managed by a single data shard or column shard may also be called partitions.

Offset

Offset is a sequence number that identifies a message within a partition.

Writer

A producer is an entity that writes new messages to a topic.

Reader

A consumer is an entity that reads messages from a topic.

Change data capture

Change data capture or CDC is a mechanism that allows subscribing to a change stream on a specific table. Technically, it is implemented on top of topics. It is described in more detail in a separate article Change Data Capture (CDC).

Change stream

Change stream is an ordered list of changes to a table, placed in a topic.

Backup collection

Backup collection is a schema object that organizes full and incremental backups for selected row tables. Collections provide point-in-time recovery, maintaining backup chains and ensuring consistent recovery of multiple tables. A table can belong to only one backup collection at a time.

For more information, see Backup collection.

Backup

Backup is a copy of data at a specific point in time that can be used for data recovery. In the context of backup collections, there are two types:

• Full backup: A complete snapshot of all data in the collection. Serves as the basis for backup chains and can be restored independently.
• Incremental backup: Captures only changes (inserts, updates, deletes) since the previous backup. Requires the entire chain of backups for recovery.

Backup chain

Backup chain is an ordered sequence of backups, starting with a full backup followed by zero or more incremental backups. Each incremental backup depends on all previous backups in the chain. Deleting any backup in the chain makes subsequent incremental backups unrecoverable.

Async replication instance

Async replication instance is a named entity that stores the settings of asynchronous replication (connection settings, list of replicated objects, etc.). It can also be used to obtain information about the state of asynchronous replication: initial scan progress, lag, errors, etc.

Replicated object

A replicated object is an object (for example, a table) for which asynchronous replication is configured.

Replica object

A replica object is a "mirror copy" of the replicated object, automatically created by the asynchronous replication instance. Typically, it is read-only.

Transfer instance

Transfer instance is a named entity that stores the settings of a transfer, including connection settings and data transformation rules. It can also be used to obtain information about the transfer state, such as errors.

Coordination node

Coordination node is a schema object that allows client applications to create semaphores for coordinating their actions. Coordination nodes are used to implement distributed locks, service discovery, leader election, and other scenarios. For more details, see coordination nodes.

Semaphore

Semaphore is an object inside a coordination node that provides a synchronization mechanism for distributed applications. Semaphores can be permanent or temporary and support create, acquire, release, and monitor operations. For more details, see semaphores in YDB.

Resource pool

Resource pool is a schema object that describes the limits imposed on resources (CPU, RAM, etc.) available for executing queries in this resource pool. A query is always executed in some resource pool. By default, all queries are executed in a resource pool named default, which imposes no restrictions. For more details on using resource pools, see the article Workload Manager — resource consumption management.

Resource pool classifier

Resource pool classifier is an object designed to manage the distribution of queries among resource pools. It describes the rules by which a resource pool is selected for each query. These classifiers are global for the entire database and apply to all queries entering it. For more details on their usage, see the article Workload Manager — resource consumption management.

YQL

YQL (YDB Query Language) is a high-level language for working with the system. It is a dialect of ANSI SQL. There are many materials dedicated to YQL, including a tutorial, reference guide, and recipes.

Federated queries

Federated queries are functionality that allows you to run queries against data stored in systems external to the YDB cluster.

Below are explanations of several terms related to federated queries. How federated queries work in YDB is explained in more detail in a separate article Federated queries.

External data source

External data source or external connection is metadata that describes how to connect to a supported external system to execute federated queries.

External table

External table is metadata that describes a specific data set that can be retrieved from an external data source.

Secret

Secret is confidential metadata that requires special handling. For example, secrets can be used in definitions of external data sources and represent entities such as passwords and tokens.

Authentication token

Auth token is a token used for authentication in YDB.

YDB supports different authentication methods and various token types.

mTLS

mTLS (mutual TLS) is a TLS mode in which not only does the client verify the server certificate, but the server also requests and verifies the client's client certificate when establishing a connection.

Client certificate

Client certificate — a digital certificate issued and used by a client — an application, user, or node YDB — for device authentication at the TLS connection stage and for client certificate authentication at the request level.

Cluster schema

The YDB cluster schema is the hierarchical namespace of the YDB cluster. The top-level element of this namespace is the cluster schema root. The child elements of the cluster schema root are databases. Inside each database, you can create an arbitrary hierarchy of objects (tables, topics, etc.) using nested directories.

Database schema

A database schema is a subset of the cluster's hierarchical namespace that belongs to the database.

Database root

A database root is the path to the database in the cluster schema.

Schema root

The cluster schema root is the root element of the YDB namespace, whose child elements are databases.

Schema object

A database schema consists of schema objects, which can be databases, tables (including external tables), topics, folders, etc.

For organizational convenience, schema objects form a hierarchy using folders.

Folder

As in file systems, a folder or directory is a container for schema objects.

Folders can contain subfolders, and such nesting can be of arbitrary depth.

Access object

An access object in authorization is an entity for which access rights and restrictions are configured. In YDB, access objects are schema objects.

Each schema object has an owner and an access control list on that object, granted to users and groups (access subjects).

Access subject

An access subject is an entity that can access access objects and perform certain actions in the system.

Obtaining access during these requests and actions depends on the configured access control lists and the subject's access level.

An access subject can be a user or a group.

Access right

An access right is an entity that reflects permission for an access subject to perform a specific set of operations in a cluster or database on a specific access object.

Access right inheritance

Access right inheritance is a mechanism where access rights granted on parent access objects are inherited by child objects in the hierarchical database structure. This ensures that permissions granted at a higher level of the hierarchy apply to all lower levels, unless they are explicitly overridden.

Access control list

An access control list or ACL is a list of all rights granted to access subjects (users and groups) on a specific access object.

Access level

An access level provides an access subject with additional capabilities when working with schema objects, as well as the ability to perform operations on the cluster as a whole. YDB uses hierarchical access levels:

• Database.
• Viewer.
• Monitoring.
• Administration.

The access level for a subject is configured using access level lists.

Access level list

An access level list or permission list is a list of SIDs of access subjects that are allowed a specific access level.

In YDB, there are several such lists that define who has which access levels.

For details on access control lists, their hierarchy, and how they work, see the Access Control Lists section of the authorization documentation.

Owner

Owner is an access subject (user or group) that has full rights to a specific access object.

User

User is a person who uses YDB to perform a specific function.

In YDB, there are different types of users depending on the method of creation:

• Local users in YDB databases.
• External users from third-party directories.

A user is identified by an SID.

Local user

A user whose account is created directly in YDB using the YQL command CREATE USER or during initial security configuration.

External user

A YDB user whose account is created in a third-party directory, for example, an LDAP directory or IAM system.

Group

Group or access group is a named set of users and other groups with equal capabilities for their members.

A group is identified by an SID.

Role

A role is a named set of access rights used to assign to users or groups of users.

Roles in YDB are implemented using groups that are created during the initial cluster deployment and are assigned a specific access list on the cluster schema root. For more information about roles, see Initial cluster security configuration.

SID

SID or security identifier is a string of the form <name> or <name>@<auth-domain> that identifies an access subject. It is used in authentication, authorization, access lists, and access control lists.

An SID identifies an individual user or user group.

The optional suffix @<auth-domain> identifies the source of the access subject, i.e., the external directory or system from which it was obtained. For example, users or groups from an LDAP directory may have the suffix @ldap. The absence of a suffix means that the user or group is created and exists directly in YDB.

Query optimizer

Query optimizer is a set of YDB components responsible for converting the logical representation of a query into a specific physically executable plan to obtain the requested result. The main goal of the optimizer is to select, among all possible query execution plans, one that is sufficiently efficient in terms of predicted execution time and cluster resource consumption. It is described in more detail in a separate article Query Optimization in YDB.

Compilation cache

Compilation cache or compile cache is a cache of compiled queries on each node of the cluster. It is used to avoid recompilation: if the query text is already in the node's cache, no additional compilation is performed. For more details, see the Query compilation cache section.

Advanced terminology

This section explains terms that are useful for YDB contributors and users who want to understand more deeply what happens inside the system.

Actor implementation

Actor system

Actor system is a C++ library with an implementation of the actor model for YDB needs.

Actor service

Actor service is an actor that has a well-known name and typically runs as a single instance on a node.

ActorId

ActorId is a unique identifier of an actor or tablet in a cluster.

Actor system interconnect

Actor system interconnect, interconnect is the internal network layer of a cluster. All actors communicate with each other in the system through the interconnect.

Local

Local is an actor service running on each node. It directly manages tablets on its node and interacts with Hive. It registers with Hive and receives commands to start tablets.

Tablet implementation

A tablet is an actor with persistent state. It includes a set of data that the tablet is responsible for, and a state machine through which the tablet's data (or state) is modified. A tablet is a fault-tolerant entity because its data is stored in distributed storage, which survives disk and node failures. A tablet automatically restarts on another node if the previous one fails or becomes overloaded. Data in a tablet is modified sequentially, as the system infrastructure guarantees that there is no more than one tablet leader through which tablet data changes are performed.

A tablet solves the same problem as the Paxos and Raft algorithms in other systems, namely the problem of distributed consensus. From a technical standpoint, a tablet implementation can be described as a replicated state machine (RSM) on top of a shared log, since the tablet state is fully described by an ordered log of commands stored in a distributed and fault-tolerant storage.

During execution, the tablet state machine is managed by three components:

1. The common tablet part ensures log consistency and recovery in case of failures.
2. An executor is an abstraction of the local database, namely the data structures and code that organize work with data stored by the tablet.
3. An actor with user code that implements the specific logic of a particular tablet type.

In YDB, there are several types of specialized tablets that store various data for different tasks. Many YDB features, such as tables and topics, are implemented as different types of tablets. Thus, reusing the tablet infrastructure is one of the key means of extensibility of YDB as a platform.

Typically, a YDB cluster runs orders of magnitude more tablets than the processes or threads that other systems would use for a cluster of similar size. In a YDB cluster, hundreds of thousands and millions of tablets can easily run simultaneously.

Since a tablet stores its state in distributed storage, it can be (re)started on any node of the cluster. Tablets are identified by a TabletID, a 64-bit number assigned when the tablet is created.

Tablet leader

Tablet leader is the current active leader of a given tablet. The tablet leader accepts commands, assigns them an order, and confirms them to the outside world. It is guaranteed that at any moment there is at most one leader for each tablet.

Tablet candidate

A tablet candidate is one of the election participants that wants to become the leader of a given tablet. If the candidate wins the election, it becomes the tablet leader.

Tablet replica

A tablet follower (also known as a hot standby) is a copy of the tablet leader that applies the log of commands accepted by the leader (with some delay). A tablet can have zero or more replicas. Replicas perform two main functions:

• In case the leader terminates or fails, replicas are preferred candidates for the new leader, as they can become the leader much faster than other candidates because they have applied most of the log.
• Replicas can respond to read-only requests if the client explicitly opts into an optional relaxed transaction mode that allows stale reads.

Tablet generation

Tablet generation is a number that identifies the reincarnation of the tablet leader. It changes only when a new leader is selected and always increases.

Tablet local database

Tablet local database or local database is a set of data structures and associated code that manage the state of a tablet and the data it stores. Logically, the state of the local database is represented by a set of tables, very similar to relational tables. Modifications to the state of the local database are performed by local tablet transactions created by the tablet's user actor.

Each table of the local database is stored as an LSM tree.

Log-structured merge-tree

Log-structured merge-tree is a data structure designed to optimize write and read performance in storage systems. It is used in YDB to store tables of the local database and data of VDisks.

MemTable

All data written to the tables of the local database is initially stored in an in-memory data structure called MemTable. When the MemTable reaches a specified size, it is flushed to disk as an immutable data structure SST.

Sorted string table

Sorted string table or SST is an immutable data structure that stores table rows sorted by key, facilitating efficient key lookup and range scans. Each SST consists of a continuous series of small data pages, typically about 7 KiB each, which further optimizes reading data from disk. An SST is usually part of an LSM tree.

Tablet pipe

tablet pipe or TabletPipe is a virtual connection that can be established with a tablet. It includes finding the tablet leader by TabletID. This is the recommended way to work with a tablet. The term open a pipe to a tablet describes the process of resolving (finding) a tablet in the cluster and establishing a virtual communication channel with it.

TabletID

TabletID is a unique identifier of a tablet within a cluster.

Bootstrapper

Bootstrapper is the main mechanism for starting tablets, used for system tablets (e.g., Hive, DS controller, root SchemeShard). Hive initializes the remaining tablets.

Shared cache

Shared cache is an actor that stores data pages recently read from distributed storage. Caching these pages reduces the number of disk I/O operations and speeds up data retrieval, improving overall system performance.

Memory controller

Memory controller is an actor that manages memory limits YDB.

Spilling

Spilling is a memory management mechanism in YDB that temporarily offloads intermediate query data to external storage when such data exceeds the available RAM of a node. In YDB, disk is currently used for spilling.

For more details about spilling, see Spilling.

Tablet types

Tablets can be considered as a framework for building reliable components operating in a distributed system. Many components of YDB — both system and those working with user data — are implemented using this framework; the main ones are listed below.

SchemeShard

SchemeShard or Scheme shard is a system tablet that stores the database schema, including metadata of user tables, topics, etc.

In addition, there is a root SchemeShard that stores information about databases created in the cluster.

DataShard

DataShard or Data shard is a tablet that manages a segment of a row-based user table. A logical user table is divided into segments by continuous ranges of the table's primary key. Each such range is managed by a separate DataShard tablet. The range itself is also called a partition. The DataShard tablet stores data row by row, which is efficient for OLTP workloads.

ColumnShard

ColumnShard or Column shard is a tablet that stores a data segment of a column-based user table.

KeyValue Tablet

KeyValue or KV Tablet is a tablet that implements a simple key → value mapping, where keys and values are strings. It also has several specific features, such as locks.

PersQueue Tablet

PersQueue or persistent queue tablet is a tablet that implements the concept of a topic. Each topic consists of one or more partitions, and each partition is managed by a separate instance of the PQ tablet.

TxAllocator

TxAllocator or transaction allocator is a system tablet that allocates unique transaction identifiers ( TxID) in the cluster. Typically, there are several such tablets in the cluster, from which a transaction proxy pre-allocates and caches ranges for local issuance within a single process.

Coordinator

coordinator is a system tablet that ensures global ordering of transactions. The coordinator's task is to assign a logical time PlanStep to each transaction planned through this coordinator. Each transaction is assigned exactly one coordinator, selected by hashing its TxId.

Mediator

Mediator is a system tablet that distributes transactions planned by coordinators among transaction participants. Mediators ensure the advancement of global time. Each transaction participant is associated with exactly one mediator. Mediators avoid the need for a full set of connections between all coordinators and all participants of all transactions.

Hive

Hive is a system tablet responsible for starting and managing other tablets. Its responsibilities include moving tablets between nodes in case of failure or overload of a node. More information about Hive can be found in a separate article.

CMS

CMS or cluster management system is a system tablet responsible for managing information about the current state of the cluster YDB. This information is used to perform gradual cluster restarts without affecting user workloads, maintenance, cluster reconfiguration, etc.

NodeBroker

NodeBroker is a system tablet that is responsible for registering dynamic nodes in the cluster.

BSController

BSController or blob storage controller manages the dynamic configuration of the distributed storage, including information about PDisk, VDisk, and storage groups. It interacts with node warden to start various components of the distributed storage. It interacts with Hive to allocate channels to tablets.

Console

Console is a system tablet responsible for storing dynamic configuration and delivering it to cluster nodes.

Kesus

Kesus is a tablet that implements a coordination node.

SysViewProcessor

SysViewProcessor is a tablet that stores data of some system views.

SequenceShard

SequenceShard is a tablet that serves Sequence objects, which are used to implement serial data types.

ReplicationController

ReplicationController is a tablet responsible for the process of asynchronous replication.

StatisticsAggregator

StatisticsAggregator is a tablet responsible for collecting statistics used in cost-based optimization.

Slot

Slot in YDB can be used in two contexts:

• Slot is a portion of server resources allocated to run one node YDB. The typical slot size is 10 CPU cores and 50 GB of RAM. Slots are used when the cluster YDB is deployed on servers or virtual machines with sufficient resources to host multiple slots.
• VDisk slot or VSlot is a share of PDisk that can be allocated to one of the VDisk.

State storage

State storage or StateStorage is a distributed service that stores information about tablets, namely:

• The current leader of the tablet or its absence.
• Tablet replicas.
• Tablet generation and step (generation:step).

State storage is used as a service for tablet name resolution, i.e., to obtain ActorId from TabletID. StateStorage is also used in the tablet leader election process.

The information in the state storage is volatile. Thus, it is lost on power failure or process restart. Despite the name, this service is not a permanent long-term storage. It contains only information that is easy to recover and that does not need to be durable. However, the state storage stores information on multiple nodes to minimize the impact of node failures. This service can also be used to gather a quorum, which is used for selecting tablet leaders.

Due to its nature, the state storage service operates on a best-effort basis. For example, the absence of multiple tablet leaders is guaranteed through the leader election protocol on distributed storage, not on state storage.

For more details on the structure of StateStorage and related subsystems, see the section Metadata distribution services.

Board

Board is a distributed service designed to store metadata as key-value pairs. It is used, among other things, to store information about endpoints.

For more details on the structure of Board and related subsystems, see the section Metadata distribution services.

SchemeBoard

SchemeBoard is a distributed service designed to store metadata as key-value pairs. It is used, among other things, to store information about schemas.

For more details on the structure of SchemeBoard and related subsystems, see the section Metadata distribution services.

Compaction

Compaction is an internal background process of rebuilding the LSM tree data. Data in VDisk and local databases is organized as LSM trees. Therefore, a distinction is made between VDisk compaction and tablet compaction. The compaction process is usually quite resource-intensive, so measures are taken to minimize the associated overhead, for example, by limiting the number of concurrently running compactions.

gRPC proxy

gRPC proxy is a proxy system for external user requests. Client requests enter the system via the gRPC protocol, then the proxy component translates them into internal calls to execute these requests, transmitted over interconnect. This proxy provides an interface for both request-response and bidirectional streaming.

Distributed configuration

Distributed configuration or DistConf is an internal configuration mechanism of the cluster that provides startup and configuration of static nodes, automatic management of the static storage group and State Storage. Distributed configuration starts before any tablets, storage groups, and State Storage.

For more details on the distributed configuration architecture, see Internals of the V2 configuration mechanism.

Distributed storage implementation

Distributed storage is a distributed fault-tolerant data storage layer that stores binary records called LogoBlob, addressed using a specific type of identifier called LogoBlobID. Thus, distributed storage is a key-value store that maps a LogoBlobID to a string of up to 10 MB. Distributed storage consists of multiple storage groups, each of which is an independent data repository.

Distributed storage stores immutable data, with each immutable data block identified by a specific LogoBlobID key. The distributed storage API is very specific, intended only for use by tablets to store their data and change logs. Thus, it is not intended for general-purpose data storage. Data in distributed storage is deleted using special barrier commands. Due to the absence of mutations in its interface, distributed storage can be implemented without implementing distributed consensus. Distributed storage is just one of the components that tablets use to implement distributed consensus.

LogoBlob

LogoBlob is a set of binary immutable data identified by a LogoBlobID and stored in distributed storage. The data block size is limited at the VDisk level and above in the stack. Currently, the maximum data block size that a VDisk can handle is 10 MB.

LogoBlobID

LogoBlobID is the identifier of a LogoBlob in distributed storage. It has a structure of the form [TabletID, Generation, Step, Channel, Cookie, BlobSize, PartID]. The main elements of LogoBlobID are:

• TabletID is the ID of the tablet that owns the LogoBlob.
• Generation is the generation of the tablet in which the data block was written.
• Channel is the channel of the tablet on which the LogoBlob is written.
• Step is an incremental counter, usually within the tablet generation.
• Cookie is a unique identifier of a data block within a single Step. The cookie is typically used when writing multiple data blocks to one Step.
• BlobSize is the size of the LogoBlob.
• PartID is the identifier of a data block part. It is important when the original LogoBlob is split into parts using erasure coding, and the parts are written to the corresponding VDisk and storage groups.

Replication

Replication is a process that ensures there are enough copies (replicas) of data to maintain the desired availability characteristics of a YDB cluster. It is typically used in geo-distributed clusters YDB.

Error correction coding

Erasure coding is a data encoding method where the original data is supplemented with redundancy and split into multiple fragments, enabling recovery of the original data if one or more fragments are lost. It is widely used in YDB clusters with a single availability zone, as opposed to replication with 3 replicas. For example, the most popular erasure coding scheme 4+2 provides the same reliability as three replicas, with a space overhead of 1.5 compared to 3.

PDisk

PDisk or physical disk is a component that controls a physical disk drive (block device). In other words, PDisk is a subsystem that implements an abstraction similar to a specialized file system on top of block devices (or files emulating a block device for testing purposes). PDisk provides data integrity control (including erasure coding of sector groups to recover data on individual damaged sectors, integrity control using checksums), transparent encryption of all data on the disk, and transactional guarantees for disk operations (write confirmation strictly after fsync).

PDisk contains a scheduler that ensures sharing of device bandwidth among multiple clients ( VDisk). PDisk divides the block device into blocks called slots (about 128 megabytes in size; smaller blocks are also allowed). At any given time, no more than one VDisk can own each slot. PDisk also maintains a recovery log shared by PDisk service records and all VDisks.

VDisk

VDisk or virtual disk is a component that implements data storage of distributed storage LogoBlob on PDisk. VDisk stores all its data on PDisk. One VDisk corresponds to one PDisk, but typically several VDisks are associated with one PDisk. Unlike PDisk, which hides blocks and logs behind it, VDisk provides an interface at the LogoBlob and LogoBlobID level, for example writing a LogoBlob, reading LogoBlobID data, and deleting a set of LogoBlobs using a special command. VDisk is a member of a storage group. VDisk itself is local, but many VDisks in a given group provide reliable data storage. VDisks in a group synchronize data with each other and replicate data in case of losses. The set of VDisks in a storage group forms a distributed RAID.

Yard

Yard is the name of the PDisk API. It allows VDisk to read and write data to blocks and logs, reserve blocks, delete blocks, and transactionally acquire and release ownership of blocks. In some contexts, Yard can be considered a synonym for PDisk.

Skeleton

Skeleton is an actor that provides an interface to VDisk.

SkeletonFront

SkeletonFront is a proxy actor for Skeleton that controls the flow of messages coming into Skeleton.

Proxy

Distributed storage proxy, DS-proxy, or BS-proxy acts as a client library for performing operations with distributed storage. The users of DS-proxy are tablets that write to and read from distributed storage. DS-proxy hides the distributed nature of distributed storage from the user. The task of DS-proxy is to write to a quorum of VDisk, perform retries when necessary, and control the write/read flow to prevent VDisk overload.

Technically, DS-proxy is implemented as an actor service launched by node warden on each node for each storage group, handling all requests to the group (write, read, and delete of LogoBlob, group locking). When writing data, DS-proxy performs error-correcting coding of the data, splitting the LogoBlob into parts that are then sent to the corresponding VDisks. DS-proxy performs the reverse process when reading, receiving parts from VDisks and reconstructing the LogoBlob from them.

Node warden

Node warden or BS_NODE is an actor service on each cluster node that launches PDisks, VDisks, and DS proxies of static storage groups when the node starts. It also interacts with the DS controller to launch PDisk, VDisk, and DS proxies of dynamic groups. The DS proxy of dynamic groups is launched on demand: node warden processes "undelivered" messages to DS proxies, launches the corresponding DS proxies, and receives group configuration from the DS controller.

Failure realm

A fail realm is a set of failure domains that can fail simultaneously due to a common cause. A correlated failure of two VDisks in the same fail realm is more likely than a failure of two VDisks from different fail realms.

An example of a fail realm is a set of equipment located in a single data center or availability zone that can fail entirely due to a natural disaster, large-scale power outage, or other similar event.

Failure domain

A fail domain is a set of equipment that can fail simultaneously. A correlated failure of two VDisks in the same fail domain is more likely than a failure of two VDisks from different fail domains. In the case of different fail domains, the probability of simultaneous failure also depends on whether the domains in question belong to the same fail realm or different ones.

An example of a fail domain is a set of disks connected to a single server, since all disks of a particular server may become unavailable if the server's power supply or network controller fails. Typically, all servers located in a single server rack are considered to belong to a common fail domain, because power or network issues at the rack level cause all equipment in it to become unavailable. Thus, a typical fail domain corresponds to a server rack (if the cluster is configured with rack-aware topology) or a single server.

Failures at the failure domain level are automatically handled by YDB without stopping the cluster.

Distributed storage channel

A distributed storage channel, DS channel, or channel is a logical connection between a tablet and a storage group. A tablet can write data to different channels, and each channel maps to a specific storage group. Having multiple channels allows a tablet to:

• Write more data than a single storage group can contain.
• Store different LogoBlobs in different storage groups, with different properties, such as erasure coding or on different media (HDD, SSD, NVMe).

Distributed transaction implementation

Below are explained terms related to the implementation of distributed transactions. The implementation itself is described in a separate article DataShard: distributed transactions.

Deterministic transactions

Distributed transactions in YDB are inspired by the research paper Building Deterministic Transaction Processing Systems without Deterministic Thread Scheduling by Alexander Thomson and Daniel J. Abadi from Yale University. The paper introduced the concept of deterministic transaction processing, which allows efficient processing of distributed transactions. The original paper imposed restrictions on the types of operations that could be performed in this way. Since these restrictions hindered real user scenarios, YDB developed algorithms to overcome these restrictions, using deterministic transactions as stages of user transaction execution with additional orchestration and locking.

Optimistic locking

As in many other database management systems, YDB queries can place locks on specific data fragments, such as table rows, to ensure that concurrent changes do not bring them into an inconsistent state. However, YDB checks these locks not at the beginning of transactions, but when attempting to commit them. The first approach is called pessimistic locking (for example, used in PostgreSQL), and the second is called optimistic locking (used in YDB).

Transaction lock invalidation

Transaction Lock Invalidation (TLI) is normal behavior YDB when parallel transactions conflict within optimistic locks. If one transaction (the violator) writes data and thereby breaks the locks of another transaction (the victim), YDB detects this when the victim commits and rolls it back with error transaction locks invalidated. For more details on TLI diagnostics, see Transaction lock invalidation.

Preparation phase

Preparation phase is a transaction phase during which the transaction body is registered on all participating shards.

Execution phase

Execution phase is a transaction phase during which the scheduled transaction is executed and a response is generated.

In some cases, instead of preparation and execution, the transaction is executed immediately and a response is generated. For example, this happens for transactions affecting only one shard or for consistent reads from a data snapshot (snapshot).

Dirty operations

In the case of read-only transactions, similar to "read uncommitted" in other database management systems, it may be necessary to read data that has not yet been committed to disk. This is called dirty operations.

Read-write set

Read-write set or RW-set is a set of data that will participate in the execution of a distributed transaction. It combines the data of the read set, which will be read, and the write set, for which modifications will be performed.

Read set

Read set or ReadSet data is what participating shards send during transaction execution. In the case of data transactions, it may contain information about the state of optimistic locks, the shard's readiness to commit, or a decision to abort the transaction.

Transaction proxies

Transaction proxy or TX_PROXY is a service that orchestrates the execution of many distributed transactions: sequential phases, phase execution, scheduling, and result aggregation. In the case of direct orchestration by other actors (for example, QP data transactions), it is used for caching and allocating unique TxIDs.

Transaction flags

Transaction flags or TxFlags is a bitmask of flags that modify the execution of a transaction in some way.

Transaction ID

TxID is a unique identifier assigned to each transaction when it is accepted YDB.

Transaction order ID

Transaction order id is a unique identifier assigned to each transaction during scheduling. It consists of PlanStep and Transaction ID.

Plan step

PlanStep or Step is the logical time at which a set of transactions is scheduled to execute.

Mediator time

During distributed transaction execution, mediator time is the logical time up to which (inclusive) a participant shard must know the entire execution plan. It is used to advance time when there are no transactions on a specific shard, to determine whether it can read from a snapshot.

MiniKQL

MiniKQL is a language that allows expressing a single deterministic transaction in the system. It is a functional, strongly typed language. Conceptually, the language describes a graph of reading from the database, performing computations on the read data, and writing results to the database and/or to a special document representing the query result (for display to the user). A MiniKQL transaction must explicitly specify its read set (data to be read) and assume deterministic branching (for example, no randomness).

MiniKQL is a low-level language. End users of the system only see queries in YQL, which relies on MiniKQL in its implementation.

Query Processor

Query Processor or QP (formerly KQP) is a component YDB responsible for orchestrating the execution of user queries and generating the final response.

Global schema

Global scheme, global schema, or database schema is the schema of all data stored in a database. It consists of tables and other entities, such as topics. Metadata about these entities is called the global schema. The term is used in contrast to local schema, which refers to the data schema inside a tablet. YDB users never see the local schema and work only with the global schema.

KiKiMR

KiKiMR is the former name of YDB, used before it became an open-source product. It may still be encountered in source code, old articles, videos, etc.