table_service_config

The table_service_config section contains configuration parameters for the table service, including spilling settings.

resource_manager

The resource_manager subsection specifies resource management parameters for the table service.

Vector index level table cache

The vector index level table (indexImplLevelTable) vector_kmeans_tree stores centroids that YDB accesses at each step of descending the cluster tree during vector search. To avoid re-reading these centroids from the distributed storage on every query, YDB can cache them in memory on each database node.

The cache is created separately on each node, uses LRU eviction, and gradually grows its size up to the specified limit.

table_service_config:
  resource_manager:
    kqp_level_cache_max_size_bytes: 0
    kqp_level_cache_increase_batch_size_bytes: 33554432

resource_manager.kqp_level_cache_max_size_bytes

Type: uint64
Default: 0 (cache disabled)
Description: Maximum size of the vector index level table cache in bytes per database node. Setting a non-zero value enables the cache.

resource_manager.kqp_level_cache_increase_batch_size_bytes

Type: uint64
Default: 33554432 (32 MiB)
Description: Step size for increasing the level table cache. The cache grows in chunks of this size up to the value of kqp_level_cache_max_size_bytes and decreases by the same chunks when the limit is lowered.

Query execution memory limits

The resource_manager subsection sets the spilling threshold relative to the query memory pool on the node. The size of this pool is controlled by the query_execution_limit_percent / query_execution_limit_bytes parameters in memory_controller_config.

table_service_config:
  resource_manager:
    spilling_percent: 80

resource_manager.spilling_percent

Type: double
Default: 80
Description: The fill threshold of the query memory pool at which YDB starts considering spilling the preferred memory management method. The basis for calculation is the Query Processor pool, whose size is set by query_execution_limit_percent / query_execution_limit_bytes.

When the total memory consumption by queries on a node exceeds spilling_percent percent of the available pool, computational operations that support spilling (Grace Hash Join, aggregations, etc.) receive a signal to offload intermediate data to disk instead of further increasing RAM consumption.

For example, with the default value of 80, spilling is activated when the query pool is about 80% full.

The threshold applies to the total query memory pool on the node (size — see query_execution_limit_percent / query_execution_limit_bytes).

Interaction with other parameters

spilling_percent determines when the query pool on a node is so full that further growth in RAM should give way to spilling. activities_limit_percent limits the memory of activities overall and indirectly affects the available RAM, but does not replace the pool relative to which spilling_percent is calculated.

Recommendations

• Decrease spilling_percent (for example, to 70) if you need to offload heavy queries to disk earlier and reduce the risk of exhausting the memory pool.
• Increase spilling_percent (for example, to 90) if disk spilling too often degrades performance and the node has enough RAM.
• Coordinate spilling_percent with query_execution_limit_percent / query_execution_limit_bytes: when increasing the query pool limit, you can raise the spilling threshold if the node has enough RAM.

spilling_service_config

Spilling is a memory management mechanism in YDB that temporarily saves data to disk when RAM is insufficient.

Enabling

Spilling is enabled by default. The following parameter controls enabling and disabling the spilling service.

local_file_config.enable

Location: table_service_config.spilling_service_config.local_file_config.enable
Type: boolean
Default: true
Description: Enables or disables the spilling service. When disabled (false), spilling does not function, which may lead to errors when processing large amounts of data.

Possible errors

• Spilling Service not started / Service not started — attempt to use spilling when the Spilling Service is disabled. See Spilling Service Not Started

table_service_config:
  spilling_service_config:
    local_file_config:
      enable: true

Main configuration parameters

table_service_config:
  spilling_service_config:
    local_file_config:
      root: ""
      max_total_size: 21474836480

Directory configuration

local_file_config.root

Type: string
Default: "" (temporary directory)
Description: File directory for storing spilling files. The directory must already exist and be readable and writable by the ydbd process: YDB does not create it, see Requirements for the root directory. If the value is empty, the OS temporary directory is used, see Default directory.

For each ydbd process, a separate directory is created inside root:

spilling-tmp-<node_id>-<spilling_service_id>-<username>

Where:

• node_id: node identifier
• spilling_service_id: unique instance identifier (a GUID) created once during Spilling Service initialization when the ydbd process starts
• username: OS username under which the ydbd process runs

Spilling files are stored inside each such directory.

Example of a full path to a spilling directory:

/tmp/spilling-tmp-1-1144b692-f1e5d361-f3960fe8-f607582e-kikimr

Where:

• /tmp: value of the root parameter (or the system temporary directory if root is not set)
• 1: node identifier
• 1144b692-f1e5d361-f3960fe8-f607582e: Spilling Service instance identifier
• kikimr: OS username

Requirements for the root directory

YDB does not create the root directory: it must exist before the nodes start. The cluster administrator or the deployment system has to create it in advance. The ydbd process only creates its own subdirectory inside root.

All OS users that run database nodes need read and write access to root: a node creates its own subdirectory there, writes spilling files into it, and reads them back. If static and dynamic nodes run under different OS users and share the same root, each of these users needs such access, for example, mode 1777, where the sticky bit prevents users from deleting other users' subdirectories.

Default directory

If root is not set, the OS temporary directory of the ydbd process that runs the node is used:

• if the TMPDIR environment variable is set for the process, its value is used
• otherwise, /tmp is used

The requirements described above apply to this directory as well, but /tmp usually already exists with mode 1777 and needs no extra setup.

Nevertheless, it is recommended to set root explicitly, to a dedicated directory and preferably on a separate disk.

Directory cleanup at startup

Each process start writes to its own directory with a new spilling_service_id, and spilling files are only needed while the process is running, so directories of previous runs are deleted. The cleanup runs asynchronously when the Spilling Service starts.

Directories whose node_id and username match the current process and whose spilling_service_id differs from the one of the current run are deleted. This way, disk space is reclaimed on the next node start after an abnormal process termination, such as SIGKILL or a power outage.

The following are not deleted:

• root itself: YDB neither creates nor deletes it, it only writes inside
• other nodes' directories (spilling-tmp-<other_node_id>-...): those nodes might be running right now
• directories with the same node_id but a different username: another ydbd process might be running under that OS user
• any foreign files and directories inside root: the name has to match the format completely, including a spilling_service_id that is a GUID

Possible errors

• Permission denied: insufficient directory access permissions. See Permission Denied

local_file_config.max_total_size

Type: uint64 \nDefault: 21474836480 (20 GiB) \nDescription: Maximum total size of all spilling files on each node. If the limit is exceeded, spilling operations fail with an error. The total spilling limit across the entire cluster equals the sum of max_total_size values from all nodes.

Recommendations

• Set the value based on available disk space

Possible errors

• Total size limit exceeded: X/YMb: maximum total size of spilling files exceeded. See Total Size Limit Exceeded

Memory management

Relationship with memory_controller_config

Spilling activation is closely related to the memory controller settings. Detailed configuration of memory_controller_config is described in a separate article.

The key parameter for spilling is activities_limit_percent, which determines the amount of memory allocated for query processing activities. This parameter affects the memory available for user queries and, consequently, the frequency of spilling activation.

Spilling threshold in resource_manager

The actual threshold at which computational operations switch to spilling is controlled by the resource_manager.spilling_percent parameter. It determines at what fill level of the Query Processor memory pool the intermediate query data starts being offloaded to disk. The pool size is set in memory_controller_config. For more details, see the Query execution memory limits section.

Impact on spilling

• When activities_limit_percent increases, more memory is available for queries → spilling is activated less often
• When activities_limit_percent decreases, less memory is available for queries → spilling is activated more often
• When spilling_percent decreases, spilling is triggered at a lower fill level of the query pool
• When spilling_percent increases, tasks take longer to increase RAM consumption before switching to spilling

File system requirements

File descriptors

Configuration examples

High-load system

For maximum performance in high-load systems, it is recommended to increase the spilling size:

table_service_config:
  spilling_service_config:
    local_file_config:
      root: ""
      max_total_size: 107374182400   # 100 GiB

Limited resources

For systems with limited resources, it is recommended to use conservative settings:

table_service_config:
  spilling_service_config:
    local_file_config:
      root: ""
      max_total_size: 5368709120     # 5 GiB

Full example

table_service_config:
  spilling_service_config:
    local_file_config:
      enable: true
      root: "/var/spilling"
      max_total_size: 53687091200    # 50 GiB

See also

• Spilling concept
• Spilling Service architecture
• Spilling troubleshooting
• Memory controller configuration
• Monitoring YDB
• Performance diagnostics