SQS API

YDB supports working with topics via the SQS protocol.

Work with a single topic can be done simultaneously via multiple protocols. For example, writing can be done using the Topic API, and reading using the SQS API, and vice versa.

When creating a topic with the CreateQueue SQS API command, the topic is created with auto-partitioning enabled: with one partition and the automatic ability to increase up to 10 active partitions. Auto-partitioning parameters can be changed via YQL or YDB CLI.

Reading via SQS API

For reading via SQS API, a shared (common) reader is used, which must be created on the topic before starting reading via the SQS protocol. If the topic is created with the CreateQueue SQS API command, a shared (common) reader named ydb-sqs-consumer is automatically created.

Messages written via the Topic API can be compressed with gzip, lzop, or zstd algorithms. When reading via the SQS protocol, the server does not decompress them but transmits them as base64: the reader must perform base64 decoding and then decompression.

Messages written via the Topic API without compression may contain binary data. When reading via the SQS protocol, the server also encodes them in base64 — the reader must perform base64 decoding.

To allow the reader to determine the compression algorithm, the attribute BodyEncoding is provided with each message. If the attribute BodyEncoding is missing, decompressing the message is not required. Possible attribute values: gzip, lzop, zstd, or base64.

Example of a message whose content is compressed:

{
    "Messages": [
        {
            "MessageId": "D523647F-5E89-560B-B1D0-152201831603",
            "ReceiptHandle": "CAAQAA==",
            "MD5OfBody": "d620a162c499920254054f78eac4feed",
            "Body": "KLUv/QBYaQAAeWRiIHdyaXRlZCAyCg==",
            "Attributes": {
                "SentTimestamp": "1780660726888",
                "BodyEncoding": "zstd"
            }
        }
    ]
}

Writing via SQS API

When writing to a topic via the SQS protocol, messages are evenly distributed across partitions. It is guaranteed that all messages with the same MessageGroupId will end up in the same partition.

For writing via the SQS protocol, message deduplication is supported by DeduplicationMessageId, and if DeduplicationMessageId is not provided — by message content. Deduplication by content can be enabled with the CreateQueue and SetQueueAttributes commands by specifying the ContentBasedDeduplication parameter.

Deduplication by content is implemented with a 5-minute window: a message with a duplicate DeduplicationMessageId can be written again after 5 minutes or more.

There is a limit on the number of messages that can be written to a topic partition with deduplication by content enabled — 500 messages per second. If you need to write more messages to the topic, you should increase the number of partitions. The limit for a topic is calculated as 500 messages/sec/partition × number of partitions. For example, to write 10 thousand messages per second, create a topic with 20 partitions.

Documentation sections

• Authentication
• Examples
• Creating a queue using YQL