---
metadata:
  - name: generator
    content: Diplodoc Platform v5.52.0
alternate:
  - https://ydb.tech/docs/en/concepts/streaming-query/streaming-query.md?version=v26.1
  - https://ydb.tech/docs/ru/concepts/streaming-query/streaming-query.md?version=v26.1
  - href: en/concepts/streaming-query/streaming-query.md
    type: text/markdown
    title: Markdown version
  - href: ../../llms.txt
    type: text/markdown
    title: llms.txt
sourcePath: en/core/concepts/streaming-query/streaming-query.md
---
> **Documentation Index:** Fetch the complete configuration index at https://ydb.tech/docs/en/llms.txt

# Streaming queries

A streaming query is a type of query designed for continuous processing of an unbounded data stream ( [stream processing](https://en.wikipedia.org/wiki/Stream_processing)). Unlike regular queries, a streaming query does not terminate after returning a result but runs continuously, processing data as it arrives.

Stream data processing is a widely used approach implemented in systems such as [Apache Flink](https://flink.apache.org/), [Apache Kafka Streams](https://kafka.apache.org/documentation/streams/), [Amazon Kinesis Data Analytics](https://aws.amazon.com/kinesis/data-analytics/). Typical use cases include monitoring and alerting, metric aggregation over time windows, real-time event transformation and filtering, enriching events with reference data, and pattern detection in event sequences.

YDB implements stream processing as part of a unified data platform. In addition to typical streaming query use cases, integration into the YDB common platform allows you to read data from [topics](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1) YDB, process change streams from [row tables](https://ydb.tech/docs/en/concepts/datamodel/table.md?version=v26.1#row-oriented-tables) via [CDC](https://ydb.tech/docs/en/concepts/cdc.md?version=v26.1), and write processing results to output topics or directly to tables. This enables building data processing pipelines within YDB with minimal latency.

## Differences from regular queries {#differences}

Regular queries work with data already stored in tables. A query executes, returns a result, and terminates. A streaming query is created and runs indefinitely until explicitly canceled by the user. Data continuously arrives in a topic, flows through the query, and is written to a sink — another topic or table.

| Characteristic | Regular queries | Streaming queries |
| --- | --- | --- |
| Data | Finite set in tables | Infinite stream of events |
| Lifetime | Terminates after processing | Runs continuously |
| Result | Available after completion | Updated as data arrives |
| Recovery on failures | Manual restart | Automatic recovery from [checkpoint](../../dev/streaming-query/checkpoints.md) |

## Data sources and sinks {#data-flow}

Streaming queries read data from [topics](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1) YDB and write results to [topics](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1) or [tables](https://ydb.tech/docs/en/concepts/datamodel/table.md?version=v26.1) YDB. Data can arrive in a topic from external systems (for example, an application writes events via the SDK), but the streaming query itself only works with YDB entities. Direct reading from external systems, such as Apache Kafka, or writing to tables in other databases, such as PostgreSQL, is not supported.

### Sources {#sources}

**Topics** are the main source of streaming data. The query reads messages from one or more [topics](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1) and processes them as they arrive. Data can be written to a topic by:

* **External applications** — via the [YDB SDK](https://ydb.tech/docs/en/reference/ydb-sdk/index.md?version=v26.1) or [Kafka API](https://ydb.tech/docs/en/reference/kafka-api/index.md?version=v26.1). For example, a service sends telemetry events or logs to a YDB topic, and the streaming query processes them.
* **CDC (Change Data Capture)** — change streams from tables, implemented via built-in [topics](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1). They allow reacting to inserts, updates, and deletes of records in real time. For more information, see [Change Data Capture (CDC)](https://ydb.tech/docs/en/concepts/cdc.md?version=v26.1).

### Sinks {#sinks}

**Topics** — for passing results to other systems or subsequent processing stages.

**Tables** — for materializing results. Data is saved via [UPSERT](https://ydb.tech/docs/en/yql/reference/syntax/upsert_into.md?version=v26.1) and is available for regular SQL queries.

## Guarantees {#guarantees}

Under normal operation, streaming queries provide an [at-least-once](https://en.wikipedia.org/wiki/Reliable_messaging#At-least-once_delivery) data delivery guarantee — on failures, the query automatically recovers from a [checkpoint](https://ydb.tech/docs/en/dev/streaming-query/checkpoints.md?version=v26.1) and resumes reading from saved offsets.

However, it is important to consider the limitations of the current implementation:

- **Offset reset on query recreation:** changing the query text via [DROP](https://ydb.tech/docs/en/yql/reference/syntax/drop-streaming-query.md?version=v26.1) + [CREATE](https://ydb.tech/docs/en/yql/reference/syntax/create-streaming-query.md?version=v26.1) deletes the checkpoint. The new query starts reading from the end of the topic, skipping events that arrived between the deletion of the old version and the start of the new one.
- **Incomplete aggregates due to lack of watermarks:** due to the absence of a [watermarks](https://en.wikipedia.org/wiki/Watermark_(data_synchronization)) mechanism, time windows are closed based on system time (wall-clock). Events that arrive late (e.g., due to slow partitions or network lag) are not included in the aggregates of already closed windows.

Detailed description of guarantees, anomalies, and ways to minimize them — in the [Data delivery guarantees](https://ydb.tech/docs/en/dev/streaming-query/guarantees.md?version=v26.1) section.

{% note info %}

We are constantly working on developing streaming processing mechanisms. In future versions, the guarantees provided will be improved.

{% endnote %}

## Limitations {#limitations}

{% note warning %}

- The query must contain at least one read from a topic, as streaming processing requires a continuous input data stream.
- `JOIN` of two streams is not supported (temporary architectural limitation).

{% endnote %}

The following are also not supported in the current version:

- The [important reader](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1#important-consumer) flag for consumers used by streaming queries.
- [Autopartitioning](https://ydb.tech/docs/en/concepts/datamodel/topic.md?version=v26.1#autopartitioning) (split/merge of partitions) of topics used by streaming queries. When the number of partitions in a topic from which a running streaming query reads increases, new partitions will not be processed.
- Changing the text of a running query. To change a query, you need to recreate it — delete it using [DROP STREAMING QUERY](https://ydb.tech/docs/en/yql/reference/syntax/drop-streaming-query.md?version=v26.1) and create it again using [CREATE STREAMING QUERY](https://ydb.tech/docs/en/yql/reference/syntax/create-streaming-query.md?version=v26.1).

## Management {#control}

Streaming queries are created, modified, and deleted using YQL commands:

- [CREATE STREAMING QUERY](https://ydb.tech/docs/en/yql/reference/syntax/create-streaming-query.md?version=v26.1) — creation.
- [ALTER STREAMING QUERY](https://ydb.tech/docs/en/yql/reference/syntax/alter-streaming-query.md?version=v26.1) — modification and state management.
- [DROP STREAMING QUERY](https://ydb.tech/docs/en/yql/reference/syntax/drop-streaming-query.md?version=v26.1) — deletion.

Query state is available in the system table [`.sys/streaming_queries`](https://ydb.tech/docs/en/dev/system-views.md?version=v26.1#streaming).

## Query language {#syntax}

Streaming queries are written in [YQL](https://ydb.tech/docs/en/yql/reference/index.md?version=v26.1) and support familiar SQL constructs: [SELECT](https://ydb.tech/docs/en/yql/reference/syntax/select/index.md?version=v26.1), [WHERE](https://ydb.tech/docs/en/yql/reference/syntax/select/where.md?version=v26.1), [GROUP BY](https://ydb.tech/docs/en/yql/reference/syntax/select/group-by.md?version=v26.1), [JOIN](https://ydb.tech/docs/en/yql/reference/syntax/select/join.md?version=v26.1). For working with time windows, [GROUP BY HOP](https://ydb.tech/docs/en/yql/reference/syntax/select/group-by.md?version=v26.1#group-by-hop) is used.

A single streaming query can read multiple input topics, use the [UNION ALL](https://ydb.tech/docs/en/yql/reference/syntax/select/union.md?version=v26.1#union-all) construct to combine data streams, and write the result to multiple output topics and/or tables (see more details in the articles [Write formats](https://ydb.tech/docs/en/dev/streaming-query/streaming-query-formats.md?version=v26.1#write_formats) and [Writing to tables](https://ydb.tech/docs/en/dev/streaming-query/table-writing.md?version=v26.1)).

## See also

- [Data delivery guarantees](https://ydb.tech/docs/en/dev/streaming-query/guarantees.md?version=v26.1) — guarantees, anomalies in window aggregation, and recommendations.
- [Quickstart: reading and writing topics](https://ydb.tech/docs/en/recipes/streaming_queries/topics.md?version=v26.1) — step-by-step guide.
- [Topic read and write formats](https://ydb.tech/docs/en/dev/streaming-query/streaming-query-formats.md?version=v26.1) — data formats.