Overview of testing with load actors

Testing system performance is an important stage of adding changes to the YDB core. With load testing, you can:

  • Determine the performance metrics and compare them to the values before the changes.
  • Test how the system runs under high or peak loads.

The task to load a high-performance distributed system is non-trivial. A software developer often has to run multiple client instances to achieve the required server load. YDB implements a simple and convenient load generation mechanism: load actors. Actors are created and run directly on a cluster, which allows avoiding the use of additional resources for starting client instances. Load actors can be run on arbitrary cluster nodes: one, all, or only selected ones. You can create any number of actors on any node.

With load actors, you can test both the entire system and its individual components:

PDisk
PDisk
VDisk
VDisk
Device
Device
Key-value
Tablet
Key-value...
Distributed Storage Proxy
Distributed Storage Proxy
Data Shard
Tablet
Data Shard...
Query Processor
Query Processor
VDisk
VDisk
PDisk
PDisk
Distributed Storage Group
Distributed Storage Group
KqpLoad
KqpLoad
KeyValueLoad
KeyValueLoad
StorageLoad
StorageLoad
VDiskLoad
VDiskLoad
PDiskWriteLoad, PDiskReadLoad, PDiskLogLoad
PDiskWriteLoad, PDis...
Load actors
Load actors
YDB components
YDB components
Device
Device
Text is not SVG - cannot display

For example, you can generate a load on Distributed Storage without using tablet and Query Processor layers. This lets you test different system layers separately and find bottlenecks in an efficient way. By combining a variety of actor types, you can run different types of load.

Note

This feature is under development; the source code is available in the main branch of the YDB repository. To learn how to build YDB from the source code, see the instructions.

Actor types

Type Description
KqpLoad Generates a load on the Query Processor layer and loads all cluster components.
KeyValueLoad Loads a key-value tablet.
StorageLoad Loads Distributed Storage without using tablet and Query Processor layers.
VDiskLoad Tests the performance of writes to the VDisk.
PDiskWriteLoad Tests the performance of writes to the PDisk.
PDiskReadLoad Tests the performance of reads from the PDisk.
PDiskLogLoad Tests if cuts from the middle of the PDisk log are correct.
MemoryLoad Allocates memory, useful when testing the logic.
Stop Stops either all or only the specified actors.

Running a load

You can run load using the following tools:

  • Cluster Embedded UI: Allows you to create, based on a configuration, and start a load actor either on the current node or all tenant nodes at once.
  • The ydbd utility: Allows you to send the actor configuration to any cluster node specifying the nodes to create and run the actor on.

The use case described below shows how to create and run the KqpLoad actor. The actor accesses the /slice/db database as a key-value store using 64 threads with a 30-second load. Before the test, the actor creates the necessary tables and deletes them once the test is completed. When being created, the actor is automatically assigned a tag. The same tag will be assigned to the test result.

  1. Open the page for managing load actors on the desired node (for example, http://<address>:8765/actors/load, where address is the address of the cluster node to run the load on).

  2. Paste the actor configuration into the input/output field:

    KqpLoad: {
        DurationSeconds: 30
        WindowDuration: 1
        WorkingDir: "/slice/db"
        NumOfSessions: 64
        UniformPartitionsCount: 1000
        DeleteTableOnFinish: 1
        WorkloadType: 0
        Kv: {
            InitRowCount: 1000
            PartitionsByLoad: true
            MaxFirstKey: 18446744073709551615
            StringLen: 8
            ColumnsCnt: 2
            RowsCnt: 1
        }
    }
    
  3. To create and run the actor, click:

    • Start new load on current node: Runs the load on the current node.
    • Start new load on all tenant nodes: Runs the load on all the tenant nodes at once.

You'll see the following message in the input/output field:

{"status":"OK","tag":1}
  • status: Load run status.
  • tag: Tag assigned to the load.
  1. Create an actor configuration file:

    NodeId: 1
    Event: {
        KqpLoad: {
            DurationSeconds: 30
            WindowDuration: 1
            WorkingDir: "/slice/db"
            NumOfSessions: 64
            UniformPartitionsCount: 1000
            DeleteTableOnFinish: 1
            WorkloadType: 0
            Kv: {
                InitRowCount: 1000
                PartitionsByLoad: true
                MaxFirstKey: 18446744073709551615
                StringLen: 8
                ColumnsCnt: 2
                RowsCnt: 1
            }
        }
    }
    
    • NodeId: ID of the node to start the actor on. To specify multiple nodes, list them in separate lines:

      NodeId: 1
      NodeId: 2
      ...
      NodeId: N
      Event: {
      ...
      
    • Event: Actor configuration.

  2. Start the actor:

    ydbd load-test --server <endpoint> --protobuf "$(cat <proto_file>)"
    
    • endpoint: Node gRPC endpoint (for example, grpc://<address>:<port>, where address is the node address and port is the node gRPC port).
    • proto_file: Path to the actor configuration file.

Viewing test results

You can view the test results using the Embedded UI. For a description of output parameters, see the documentation of the respective actor.

  1. Open the page for managing load actors on the desired node (for example, http://<address>:<port>/actors/load, where address is the node address and port is the HTTP port used for monitoring the node under load).

  2. Click Results.

    This shows the results of completed tests. Find the results with the appropriate tag.

    load-actors-finished-tests