Message pipeline processing
The use of the topic read
and topic write
commands with standard I/O devices and support for reading messages in streaming mode lets you build full-featured integration scenarios with message transfer across topics and their conversion. This section describes a number of these scenarios.
Note
The examples use the db1
profile. To learn more, see Creating a connection profile.
-
Transferring a single message from
topic1
indb1
totopic2
indb2
, waiting for it to appear in the source topicydb -p db1 topic read topic1 -c c1 -w | ydb -p db2 topic write topic2
-
Transferring all one-line messages that appear in
topic1
indb1
totopic2
indb2
in background mode. You can use this scenario if it's guaranteed that there are no0x0A
bytes (newline) in source messages.ydb -p db1 topic read topic1 -c c1 --format newline-delimited -w | \ ydb -p db2 topic write topic2 --format newline-delimited
-
Transferring an exact binary copy of all messages that appear in
topic1
indb1
totopic2
indb2
in background mode with base64-encoding of messages in the transfer stream.ydb -p db1 topic read topic1 -c c1 --format newline-delimited -w --transform base64 | \ ydb -p db2 topic write topic2 --format newline-delimited --transform base64
-
Transferring a limited batch of one-line messages filtered by the
ERROR
substringydb -p db1 topic read topic1 -c c1 --format newline-delimited | \ grep ERROR | \ ydb -p db2 topic write topic2 --format newline-delimited
-
Writing YQL query results as messages to
topic1
ydb -p db1 yql -s "select * from series" --format json-unicode | \ ydb -p db1 topic write topic1 --format newline-delimited