Transfer load
Starts the load in the form of transactions YDB involving topics and tables simultaneously. The data is read from the topic and written to the table. To simulate a real load, you can set various input parameters: the number of messages, the size of messages, the target write speed, the number of consumers and producers, the number of partitions. During operation, the console displays the results: the number of written messages, the speed of writing messages, etc.
Note
The examples use the quickstart profile. To learn more, see Creating a profile to connect to a test database.
Initializing the test environment
Before starting the load, it is necessary to initialize the test environment. You can use the command ydb workload transfer topic-to-table init to do this. It will create a topic and a table with the necessary parameters.
Command syntax:
ydb [global options...] workload transfer topic-to-table init [options...]
global options— global parameters.options- parameters of the subcommand.
View the command description:
ydb workload transfer topic-to-table init --help
Parameters of the subcommand:
| Parameter name | Parameter description | Default value |
|---|---|---|
--topic |
Topic name | transfer-topic |
--consumer-prefix |
Prefix of the consumers name | workload-consumer |
--table |
Table name | transfer-table |
--consumers |
Number of topic consumers | 1 |
--topic-partitions |
Number of topic partitions | 128 |
--table-partitions |
Number of table partitions | 128 |
After executing the init subcommand, a table, topic and consumers will be created. Reader names are created by the rule ${CONSUMER_PREFIX}-${INDEX}. The value of ${INDEX} is an integer from 0 to the value of the parameter --consumers minus 1.
For example, the command ydb --profile quickstart workload transfer topic-to-table init --consumers 2 --topic-partitions 143 --table-partitions 237 will create a topic transfer-topic with 2 consumers, 143 partitions, and a table transfer-table with 237 partitions. The consumer names are workload-consumer-0 and workload-consumer-1.
Running a load test
The test simulates the load from an application that receives messages from a topic, processes them and writes the processing results to a database table.
During the operation of the program, two types of work streams are simulated:
- Input stream: messages are written to the topic in the non-transaction mode. The user can control the writing speed, the message size, the number of producers.
- Processing flow: messages are read from the topic and written to the table using the YDB transaction.
The following actions are performed in the processing flow within a single transaction:
- messages from the topic are being read until the
--commit-periodperiod has expired; - one
UPSERTcommand and aCOMMITcommand are executed on the table to commit the transaction after the period expires.
Command syntax:
ydb [global options...] workload transfer topic-to-table run [options...]
global options— global parameters.options- parameters of the subcommand.
View the command description:
ydb workload transfer topic-to-table run --help
Parameters of the subcommand:
| Parameter name | Parameter Description | Default value |
|---|---|---|
--seconds, -s |
Duration of the test in seconds | 60 |
--window, -w |
Duration of the statistics collection window in seconds | 1 |
--quiet, -q |
Output only the final test result | 0 |
--print-timestamp |
Print the time together with the statistics of each time window | 0 |
--percentile |
Percentile in statistics output | 50 |
--warmup |
The warm-up time of the test in seconds. No statistics are calculated during this time | 5 |
--topic |
Topic name | transfer-topic |
--consumer-prefix |
Prefix of the consumers name | workload-consumer |
--table |
Table name | transfer-table |
--producer-threads, -p |
Number of producer threads | 1 |
--consumer-threads, -t |
Number of consumer threads | 1 |
--consumers, -c |
Number of consumers | 1 |
--message-size, -m |
Message size in bytes. It is possible to specify in KB, MB, GB by adding suffixes K, M, G respectively |
10240 |
--message-rate |
Target total write speed. In messages per second. Excludes the use of the --byte-rate parameter |
0 |
--byte-rate |
Target total write speed. In bytes per second. Excludes the use of the --message-rate parameter. It is possible to specify in KB/s, MB/s, GB/s by adding suffixes K, M, G respectively |
0 |
--tx-commit-interval |
The period between transaction COMMIT calls. In milliseconds |
1000 |
--tx-commit-messages |
The period between transaction COMMIT calls. In number of messages |
1000000 |
--only-topic-in-tx |
Only topic partitions are forced to participate in transactions. Excludes the use of the --only-table-in-tx parameter |
0 |
--only-table-in-tx |
Only table shards are forced to participate in transactions. Excludes the use of the --only-topic-in-tx parameter |
0 |
For example, the command ydb --profile quickstart workload transfer topic-to-table run will run a test lasting 60 seconds. The data for the first 5 seconds will not be taken into account in the work statistics. Example of console output:
Window Write speed Write time Inflight Read speed Topic time Select time Upsert time Commit time
# msg/s MB/s percentile,ms percentile,msg msg/s MB/s percentile,ms percentile,ms percentile,ms percentile,ms
1 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0 0
6 103 1 1023 83 103 1 1025 0 0 0
7 103 1 999 78 103 1 1001 0 0 0
8 103 1 1003 93 103 1 1002 0 0 0
9 103 1 1003 88 103 1 1003 0 0 0
10 103 1 999 79 103 1 999 0 0 0
11 103 1 1119 89 0 0 0 0 0 0
12 103 1 1023 90 206 2 1028 90 223 695
13 103 1 975 84 103 1 976 0 0 0
14 103 1 1003 91 103 1 1006 0 0 0
15 103 1 1003 93 103 1 1005 0 0 0
16 103 1 1103 89 103 1 1100 0 0 0
17 103 1 1063 89 103 1 1061 0 0 0
...
Window— the serial number of the time window for collecting statistics.Write speed— the speed of writing messages by producers. In messages per second and in megabytes per second.Write time— the specified percentile of the message writing time in ms.Inflight— the maximum number of messages waiting for confirmation for all batches.Lag— the specified percentile of maximum number of messages waiting to be read in the statistics collection window. Messages for all batches are taken into account.Lag time— the specified percentile of message delay time in ms.Read speed— the speed of reading messages by consumers. In messages per second and in megabytes per second.Select time,Upsert time,Commit time— the specified percentile of the execution time of Select, Insert, Commit operations in ms.
Removing the test environment
After the test is completed, you can delete the test environment.
Command syntax:
ydb [global options...] workload transfer topic-to-table clean [options...]
global options— global parameters.options- parameters of the subcommand.
View the command description:
ydb workload transfer topic-to-table clean --help
Parameters of the subcommand:
| Parameter Name | Parameter Description | Default value |
|---|---|---|
--topic |
Topic name | transfer-topic |
--table |
Table name | transfer-table |
For example, the command ydb --profile quickstart workload transfer topic-to-table clean will delete the topic transfer-topic, its consumers and the table transfer-table.