Transfer load

Starts the load in the form of transactions YDB involving topics and tables simultaneously. The data is read from the topic and written to the table. To simulate a real load, you can set various input parameters: the number of messages, the size of messages, the target write speed, the number of consumers and producers, the number of partitions. During operation, the console displays the results: the number of written messages, the speed of writing messages, etc.

Note

The examples use the quickstart profile. To learn more, see Creating a profile to connect to a test database.

Initializing the test environment

Before starting the load, it is necessary to initialize the test environment. You can use the command ydb workload transfer topic-to-table init to do this. It will create a topic and a table with the necessary parameters.

Command syntax:

ydb [global options...] workload transfer topic-to-table init [options...]

View the command description:

ydb workload transfer topic-to-table init --help

Parameters of the subcommand:

Parameter name Parameter description Default value
--topic Topic name transfer-topic
--consumer-prefix Prefix of the consumers name workload-consumer
--table Table name transfer-table
--consumers Number of topic consumers 1
--topic-partitions Number of topic partitions 128
--table-partitions Number of table partitions 128

After executing the init subcommand, a table, topic and consumers will be created. Reader names are created by the rule ${CONSUMER_PREFIX}-${INDEX}. The value of ${INDEX} is an integer from 0 to the value of the parameter --consumers minus 1.

For example, the command ydb --profile quickstart workload transfer topic-to-table init --consumers 2 --topic-partitions 143 --table-partitions 237 will create a topic transfer-topic with 2 consumers, 143 partitions, and a table transfer-table with 237 partitions. The consumer names are workload-consumer-0 and workload-consumer-1.

Running a load test

The test simulates the load from an application that receives messages from a topic, processes them and writes the processing results to a database table.

During the operation of the program, two types of work streams are simulated:

  • Input stream: messages are written to the topic in the non-transaction mode. The user can control the writing speed, the message size, the number of producers.
  • Processing flow: messages are read from the topic and written to the table using the YDB transaction.

The following actions are performed in the processing flow within a single transaction:

  • messages from the topic are being read until the --commit-period period has expired;
  • one UPSERT command and a COMMIT command are executed on the table to commit the transaction after the period expires.

Command syntax:

ydb [global options...] workload transfer topic-to-table run [options...]

View the command description:

ydb workload transfer topic-to-table run --help

Parameters of the subcommand:

Parameter name Parameter Description Default value
--seconds, -s Duration of the test in seconds 60
--window, -w Duration of the statistics collection window in seconds 1
--quiet, -q Output only the final test result 0
--print-timestamp Print the time together with the statistics of each time window 0
--percentile Percentile in statistics output 50
--warmup The warm-up time of the test in seconds. No statistics are calculated during this time 5
--topic Topic name transfer-topic
--consumer-prefix Prefix of the consumers name workload-consumer
--table Table name transfer-table
--producer-threads, -p Number of producer threads 1
--consumer-threads, -t Number of consumer threads 1
--consumers, -c Number of consumers 1
--message-size, -m Message size in bytes. It is possible to specify in KB, MB, GB by adding suffixes K, M, G respectively 10240
--message-rate Target total write speed. In messages per second. Excludes the use of the --byte-rate parameter 0
--byte-rate Target total write speed. In bytes per second. Excludes the use of the --message-rate parameter. It is possible to specify in KB/s, MB/s, GB/s by adding suffixes K, M, G respectively 0
--commit-period The period between COMMIT calls. In seconds 10
--commit-messages The period between COMMIT calls. In number of messages 1000000
--only-topic-in-tx Only topic partitions are forced to participate in transactions. Excludes the use of the --only-table-in-tx parameter 0
--only-table-in-tx Only table shards are forced to participate in transactions. Excludes the use of the --only-topic-in-tx parameter 0

For example, the command ydb --profile quickstart workload transfer topic-to-table run will run a test lasting 60 seconds. The data for the first 5 seconds will not be taken into account in the work statistics. Example of console output:

Window  Write speed     Write time      Inflight        Read speed      Topic time      Select time     Upsert time     Commit time
#       msg/s   MB/s    percentile,ms   percentile,msg  msg/s   MB/s    percentile,ms   percentile,ms   percentile,ms   percentile,ms
1       0       0       0               0               0       0       0               0               0               0
2       0       0       0               0               0       0       0               0               0               0
3       0       0       0               0               0       0       0               0               0               0
4       0       0       0               0               0       0       0               0               0               0
5       0       0       0               0               0       0       0               0               0               0
6       103     1       1023            83              103     1       1025            0               0               0
7       103     1       999             78              103     1       1001            0               0               0
8       103     1       1003            93              103     1       1002            0               0               0
9       103     1       1003            88              103     1       1003            0               0               0
10      103     1       999             79              103     1       999             0               0               0
11      103     1       1119            89              0       0       0               0               0               0
12      103     1       1023            90              206     2       1028            90              223             695
13      103     1       975             84              103     1       976             0               0               0
14      103     1       1003            91              103     1       1006            0               0               0
15      103     1       1003            93              103     1       1005            0               0               0
16      103     1       1103            89              103     1       1100            0               0               0
17      103     1       1063            89              103     1       1061            0               0               0
...
  • Window — the serial number of the time window for collecting statistics.
  • Write speed — the speed of writing messages by producers. In messages per second and in megabytes per second.
  • Write time — the specified percentile of the message writing time in ms.
  • Inflight — the maximum number of messages waiting for confirmation for all batches.
  • Lag — the specified percentile of maximum number of messages waiting to be read in the statistics collection window. Messages for all batches are taken into account.
  • Lag time — the specified percentile of message delay time in ms.
  • Read speed — the speed of reading messages by consumers. In messages per second and in megabytes per second.
  • Select time, Upsert time, Commit time — the specified percentile of the execution time of Select, Insert, Commit operations in ms.

Removing the test environment

After the test is completed, you can delete the test environment.

Command syntax:

ydb [global options...] workload transfer topic-to-table clean [options...]

View the command description:

ydb workload transfer topic-to-table clean --help

Parameters of the subcommand:

Parameter Name Parameter Description Default value
--topic Topic name transfer-topic
--table Table name transfer-table

For example, the command ydb --profile quickstart workload transfer topic-to-table clean will delete the topic transfer-topic, its consumers and the table transfer-table.