Spilling
Spilling in General
Spilling is a memory management mechanism that temporarily offloads intermediate data arising from computations and exceeding available node RAM capacity to external storage. In YDB, disk storage is currently used for spilling. Spilling enables execution of user queries that require processing large data volumes exceeding available node memory.
In data processing systems, including YDB, spilling is essential for:
- processing queries with large data volumes when intermediate results don't fit in RAM
- executing complex analytical operations (aggregations, table joins) over large datasets
- optimizing query performance through intermediate materialization of part of the data in external memory, which in certain scenarios can accelerate overall execution time
Spilling operates based on the memory hierarchy principle:
- Random Access Memory (RAM) — fast but limited.
- External storage — slower but more capacious.
When memory usage approaches the limit, the system:
- serializes part of the data
- saves it to external storage
- frees the corresponding memory
- continues processing the query using data remaining in memory
- loads data back into memory, when necessary, to continue computations
Spilling in YDB
YDB implements the spilling mechanism through the Spilling Service, an actor service that provides temporary storage for data blobs. Spilling is only performed on database nodes. Detailed technical information about it is available in Spilling Service.
Types of Spilling in YDB
YDB implements two primary types of spilling that operate at different levels of the computational process:
These types work independently and can activate simultaneously within a single query, providing comprehensive memory management.
Computation Spilling
YDB compute cores automatically offload intermediate data to disk when executing operations that require significant memory. This type of spilling is implemented at the level of individual computational operations and activates when memory limits are reached.
Main usage scenarios:
- Aggregations — when grouping large data volumes, the system offloads intermediate hash tables to disk.
- Join operations — when joining large tables, the Grace Hash Join algorithm is used with data partitioning and offloading to disk.
Operation Mechanism
Compute nodes contain specialized objects for monitoring memory usage. When the data volume approaches the set limit:
- The system switches to spilling mode.
- Data is serialized and divided into blocks (buckets).
- Part of the blocks is transferred to the Spilling Service for disk storage.
- Metadata about the data location is kept in memory.
- The system continues processing the data remaining in memory, which frees additional space.
- When necessary, data is loaded back and processed.
Transport Spilling
This type of spilling operates at the level of data transfer between different query execution stages. The system automatically buffers and offloads data when transfer buffers overflow. This helps avoid blocking the execution of data-generating operations, even when receiving operations are not ready to accept data.
Operation Mechanism
The data transfer system continuously monitors its state:
- Buffering: Incoming data accumulates in the transfer system's internal buffers
- Fill control: The system tracks buffer fill levels
- Automatic spilling: When limits are reached, data is automatically serialized and transferred to the Spilling Service
- Continued operation: The transfer system continues accepting new data after freeing memory space
- Recovery: When the next stage is ready, data is read from external storage and passed further
Interaction with Memory Controller
When executing queries, YDB tries to stay within the memory limit set by the memory controller. To continue fitting within this limit as intermediate computations grow, spilling is used. For more details, see the Memory Management section.