YDB Architecture Overview

Introduction

YDB is a horizontally scalable, distributed, and fault-tolerant database system designed as a versatile platform for high performance — for example, a typical cluster node can process tens of thousands of queries per second. The system supports geographically distributed (cross-datacenter) configurations, ranging from small clusters with a few nodes to large-scale deployments of thousands of servers capable of efficiently handling hundreds of petabytes of data.

Key Features and Capabilities of YDB

• Horizontal scaling and automatic sharding: data and workload are dynamically distributed across available hardware resources as data volume or query intensity grows.
• Fault tolerance: automatic recovery from failures of nodes, racks, or availability zones.
• Data high availability and durability: ensured through automatic synchronous data replication within the cluster.
• Strong consistency and ACID transactions: the system provides distributed transactions with serializable isolation. Consistency and isolation levels can be relaxed when higher performance is required.
• YQL: a SQL dialect optimized for large-scale data and complex processing scenarios.
• Relational data model: supports both row-oriented and column-oriented tables, enabling efficient handling of both transactional (OLTP) and analytical (OLAP) workloads within a single system.
• Hierarchical namespace: tables, topics, and other objects are organized in a hierarchical namespace, similar to a filesystem.
• Asynchronous replication: near real-time data synchronization between YDB databases — both within a single cluster and across different clusters.
• Streaming data processing and distribution:
  • Topics: storage and streaming delivery of unstructured messages to multiple subscribers. Supports the Kafka protocol.
  • Change Data Capture (CDC): built-in stream of table data changes published as a topic.
  • Transfers: automated data delivery from topics to tables.
• Federated queries: execute queries against external data sources (e.g., S3) as part of YQL queries, without prior data import to YDB storage.
• Vector indexes: support for storing and searching vector embeddings — ideal for semantic search, similarity matching, and ML use cases.
• Observability: built-in metrics, logs, and dashboards.
• Security and audit: data encryption (at-rest and in-transit), operation auditing, and support for authentication and authorization — see Security.
• Tools, integrations, and APIs: YDB CLI for running queries, administration, and debugging. SDKs for C++, C#, Go, Java, Node.js, PHP, Python, and Rust. Integrations with various third-party systems. Learn more in Integrations YDB and Languages and APIs.
• Open architecture: source code is available under the Apache License 2.0. The system uses the open gRPC protocol, enabling client implementations in any programming language.

Key Use Cases

YDB is a versatile platform suitable for a wide range of scenarios requiring scalability, reliability, and flexibility. Typical use cases include:

• In distributed systems requiring strong consistency or support for multi-row and multi-table transactions. YDB combines NoSQL-like scalability with the consistency and integrity guarantees of relational databases.
• Systems that store and process very large datasets and require nearly unlimited horizontal scaling (production clusters with thousands of nodes, handling millions of RPS and petabytes of data).
• High-load systems relying on manual sharding of relational databases. YDB simplifies architecture by automatically handling the sharding logic, re-sharding, query routing, and cross-shard transactions out of the box.
• New product development with uncertain load patterns or expected scale beyond the limits of traditional relational database management systems (RDBMS).
• Projects requiring a flexible platform capable of handling diverse workloads and use cases — including transactional, streaming, and analytical.

How It Works?

Fully explaining how YDB works in detail takes quite a while. Below you can review several key highlights and then continue exploring the documentation to learn more.

YDB Architecture

YDB clusters typically run on commodity hardware with a shared-nothing architecture. From a bird's eye view, YDB exhibits a layered architecture. The compute and storage layers are disaggregated; they can either run on separate sets of nodes or be co-located.

One of the key building blocks of YDB's compute layer is called a tablet. Tablets are stateful logical components implementing various aspects of YDB.

The next level of detail of the overall YDB architecture is explained in the General YDB schema article.

Hierarchy

From the user's perspective, everything inside YDB is organized in a hierarchical structure using directories. It can have arbitrary depth depending on how you choose to organize your data and projects. Even though YDB does not have a fixed hierarchy depth like in other SQL implementations, it will still feel familiar as this is exactly how any virtual filesystem looks.

Table

YDB provides users with a well-known abstraction — tables. In YDB, there are two main types of tables:

• Row-oriented tables are designed for OLTP workloads.
• Column-oriented tables are designed for OLAP workloads.

Logically, from the user's perspective, both types of tables look the same. The main difference between row-oriented and column-oriented tables lies in how the data is physically stored. In row-oriented tables, the values of all columns in each row are stored together. In contrast, in column-oriented tables, each column is stored separately, meaning that cells from different rows are stored next to each other within the same column.

Regardless of the type, each table must have a primary key. Column-oriented tables can only have NOT NULL columns in primary keys. Table data is physically sorted by the primary key.

Partitioning works differently in row-oriented and column-oriented tables:

• Row-oriented tables are automatically partitioned by primary key ranges, depending on the data volume.
• Column-oriented tables are partitioned by the hash of the partitioning columns.

Each partition of a table is processed by a specific tablet, called a data shard for row-oriented tables and a column shard for column-oriented tables.

Split by Load

Data shards will automatically split into more as the load increases. They automatically merge back to the appropriate number when the peak load subsides.

Split by Size

Data shards will also automatically split when the data size increases. They automatically merge back if enough data is deleted.

Automatic Balancing

YDB evenly distributes tablets among available nodes. It moves heavily loaded tablets from overloaded nodes. CPU, memory, and network metrics are tracked to facilitate this.

Distributed Storage Internals

YDB doesn't rely on any third-party filesystem. It stores data by directly working with disk drives as block devices. All major disk kinds are supported: NVMe, SSD, or HDD. The PDisk component is responsible for working with a specific block device. The abstraction layer above PDisk is called VDisk. There is a special component called DSProxy between a tablet and VDisk. DSProxy analyzes disk availability and characteristics and chooses which disks will handle a request and which won't.

Distributed Storage Proxy (DSProxy)

A common fault-tolerant setup of YDB spans three datacenters or availability zones (AZ). When YDB writes data to three AZs, it doesn't send requests to obviously bad disks and continues to operate without interruption even if one AZ and a disk in another AZ are lost.

What's Next?

If you are interested in more specifics about various aspects of YDB, check out neighboring articles in this documentation section. If you are ready to jump into more practical content, you can continue to the quick start or YQL tutorials.