Local and external topics in streaming queries

Streaming queries read events from topics and can write results back to topics. The source and destination of messages can be either a topic in the same database where the query is executed, or a topic in another database YDB.

All use cases of streaming queries work the same for local{#local-topics} and external{#external-topics} topics. The same query can simultaneously read a local topic, write to an external one, and vice versa.

Local topics

Local topics: topics created in the same database YDB as the streaming query.

In the query text, they are referred to by a short name — just like a table in the current database:

SELECT * FROM input_topic WITH (FORMAT = json_each_row, SCHEMA = (...));

INSERT INTO output_topic SELECT ...;

External topics

External topics — topics located in another database YDB.

Access to them from a streaming query is performed only through a pre-created external data source with the YDB source type. Creating the object is the CREATE EXTERNAL DATA SOURCE command; if authentication is required, secrets are used.

After creating a source, for example named ext_source, referring to topic input_topic in the external database is written as follows:

SELECT * FROM ext_source.input_topic WITH (FORMAT = json_each_row, SCHEMA = (...));

The name ext_source in the documentation is conditional — in your database, the source may be named differently; it is important that it matches in CREATE EXTERNAL DATA SOURCE and in the prefix before the topic name.

See also

Previous
Streaming queries
Next
Typical Patterns