Hive

Hive is a tablet responsible for managing other tablets, including selecting nodes for them to run on and deciding when to rebalance tablets.

The creation and deletion of tablets is initiated by the SchemeShard tablet. When a tablet is created, Hive assigns it a unique TabletId, fills in TabletStorageInfo, chooses the most suitable node, and sends a command to start the tablet on that node. In some abnormal situations, a tablet may interrupt its operation, in which case the node on which it was running sends a message to Hive. Hive also assumes that if the connection with a certain node is lost, the tablets running on it have stopped. In such cases, Hive restarts the tablets on other nodes, incrementing the generation.

A YDB cluster runs multiple Hives:

• A single root Hive responsible for the system tablets of all databases in the cluster. All nodes in the cluster are registered in the root Hive.
• A database Hive (one per database) is responsible for the tablets servicing the user load of a specific database. Only the compute nodes of the database are registered in that database's Hive.

When a node registers, it informs Hive of the types of tablets and the number of tablets that can be run on it.

Resource usage metrics

Hive evaluates resource usage to evenly distribute tablets across nodes. For each tablet, the usage of four types of resources is tracked:

1. CPU — processor consumption, calculated as the number of microseconds spent on tablet work in the last second. A value of one second corresponds to 100% load of a single core.
2. Memory — the amount of RAM consumed by the tablet.
3. Network — the amount of traffic generated by the tablet.
4. Counter — a fake resource for even distribution of tablets. If a tablet has a nonzero consumption of any other resource, its Counter value is 0; otherwise, it is 1. This way, Counter is used for any tablets for which there is no data on real consumption, as well as for tablets where real consumption tracking is disabled. By default, this applies only to column-oriented tables.

Additionally, to determine overloaded nodes, YDB uses memory consumption and processor resources in the actor system thread pools on each node. These values are converted into relative values (a number from 0 to 1). The maximum of these relative values is used as the node's overall resource consumption value — Node usage. Hive also applies aggregation over a window to all metrics to account for load spikes.

Resource usage information is used for choosing a node for a tablet. For example, if Hive has information only on the CPU consumption of a tablet, it tries to find a node with the lowest CPU load. If information on multiple resources is available, the highest of the resource usage values is used.

Autobalancing

At certain moments, Hive may start an auto-balancing process that moves tablets between nodes to improve load distribution. The situations when autobalancing occurs are listed below. The auto-balancer works iteratively, making decisions about moving tablets one at a time. It selects the most loaded node, chooses a tablet that runs on this node using weighted randomness, and chooses a more suitable node for it. This process is repeated until balance is restored. The way node load is determined depends on the type of balancing. For example, in case of CPU consumption imbalance, CPU usage is used. For uneven distribution of column-oriented tables, the number of tablets is used instead.

Resource usage imbalance

To quantify the balance of resource usage, Hive uses the Scatter metric, which is calculated separately for each resource using the following formula:

$$\mathrm{Scatter} = \frac{\mathrm{MaxUsage} - \mathrm{MinUsage}}{\mathrm{MaxUsage}}$$

$\mathrm{MaxUsage}$ and $\mathrm{MinUsage}$ are, respectively, the maximum and minimum usage of a resource across all nodes. To normalize consumption on each node, the number of available resources on the node is used, which may vary between nodes. Under low loads, this metric may fluctuate significantly. To avoid this, when calculating $\mathrm{Scatter}$, it is assumed that resource usage cannot be lower than 30%. The balancer is triggered if $Scatter$ exceeds a threshold.

The maximum value of $Scatter$ across different resources is available as the sensor Hive/MAX(BalanceScatter) in the tablets subgroup.

Node overload

Overloaded nodes can negatively affect YDB performance. CPU overload raises latencies, and consuming all memory can cause the node to crash with an out-of-memory error. The balancer is triggered when the load of one node exceeds 90% while the load on another node falls below 70%.

The maximum resource usage on a node is reported by the Hive/MAX(BalanceUsageMax) sensor in the tablets subgroup.

Even distribution for a specific object

For tablets that use the Counter resource, the evenness of tablet distribution for each object (each table) is tracked in the form of the ObjectImbalance metric, similar to the $Scatter$ described above. Restarting nodes may break the balance in tablet distribution and trigger balancing.

The maximum value of ObjectImbalance across different objects is reported by the Hive/MAX(BalanceObjectImbalance) sensor in the tablets subgroup.