Asynchronous Replication

Asynchronous replication allows for synchronizing data between YDB databases in near real time. It can also be used for data migration between databases with minimal downtime for applications interacting with these databases. Such databases can be located in the same YDB cluster as well as in different clusters.

Overview

Asynchronous replication is based on Change Data Capture and operates on logical data. The following diagram illustrates the replication process:

As shown in the diagram above, asynchronous replication involves two databases:

1. Source. A database with replicated objects.
2. Target. A database where an asynchronous replication instance and replica objects will be created.

Asynchronous replication consists of the following stages:

• Initialization
• Initial table scan
• Change data replication

Initialization

Initialization of asynchronous replication includes the following steps:

• Creating an asynchronous replication instance on the target database using the CREATE ASYNC REPLICATION YQL expression.
• Establishing a connection with the source database. The target database connects to the source using the connection parameters specified during the creation of the asynchronous replication instance.

• The following objects are created for replicated objects on the source:
  • changefeeds on the source
  • replica objects on the target

Initial Table Scan

During the initial table scan the source data is exported to changefeeds. The target runs consumers that read the source data from the changefeeds and write it to replicas.

You can get the progress of the initial table scan from the description of the asynchronous replication instance.

Synchronization Using Backup and Restore

With INITIAL_SCAN = FALSE, the initial data is transferred using server-side backup to S3 and restore from S3 operations. This method uses the backup mechanism to transfer large datasets.

Replication is created in the PAUSED state and prepares changefeeds without an initial scan. After all changefeeds are ready, the user creates a backup of the source tables and restores them on the target. Changes on the source accumulate in the changefeeds during the transfer. The target tables are not attached to replication at this stage.

After the restore is complete, the user switches replication to STANDBY using ALTER ASYNC REPLICATION. The target checks the restored table schemas, attaches the tables as replica objects, and starts applying accumulated changes. Attached tables become read-only.

To ensure data completeness, changefeeds must be ready before the backup is created, and accumulated records must remain available until their processing is acknowledged. The availability period is set with the AVAILABILITY_PERIOD parameter. For the procedure and checks, see the guide to initial synchronization using backup and restore.

Choosing an Initial Synchronization Method

Both methods require reading the source data and writing it to the target. Their additional costs differ:

Cost	Initial scan	Backup and restore
Data processing	Each source row is serialized into a CDC message, written to a changefeed, read, and applied on the target	Data is exported and imported by server-side operations; only changes pass through CDC
Temporary storage	Initial data and changes are stored in changefeeds according to the availability period	A consistent copy of the tables in the source and a backup in S3 are required; changefeeds store changes according to the availability period
Network transfer	Initial data is transferred from the source changefeeds to the target	Data is transferred from the source to S3 and then from S3 to the target; export compression can reduce the transfer volume
Setup and operation	Synchronization starts automatically when replication is created	Requires S3 storage, separate backup and restore operations, checking their completion, and starting replication

An initial scan is simpler to set up and suits small datasets. For large datasets, backup and restore can reduce the cost of processing the initial data through CDC, but require additional storage and S3 operations. Actual costs and duration depend on data volume, the rate of changes, cluster resources, compression, and the terms of S3 usage.

Change Data Replication

After the initial table scan is completed, the consumers read the change data and write it to replicas.

Each change data block has its creation time (). Consumers track the reception time of the change data (). Thus, you can use the following formula to calculate the replication lag:

You can also get the replication lag from the description of the asynchronous replication instance with the --stats option. If a consumer stops, loses its session, or does not report fresh statistics for 60 seconds, the lag value becomes unavailable. A missing value means that the lag is unknown, not zero.

Storing Changes in the Source

The AVAILABILITY_PERIOD parameter determines how long changefeeds retain messages whose processing has not yet been acknowledged by replication. It is set when replication is created and cannot be changed using ALTER ASYNC REPLICATION.

The default value is 'UNLIMITED': unacknowledged messages are retained until their processing is acknowledged. To limit the availability period, specify a positive interval, for example, Interval('PT48H').

Regardless of the selected availability period, messages in the topic are deleted almost immediately after the consumer acknowledges their processing, freeing up space in the source and reducing the cost of storing changes that have already been processed. The AVAILABILITY_PERIOD parameter protects only unacknowledged messages from deletion.

Condition	Unlimited: 'UNLIMITED'	Finite period: for example, Interval('PT48H')
Extended downtime or lag	Unacknowledged messages are retained regardless of their age	Unacknowledged messages may be deleted after the specified period expires
Storage usage in the source	Accumulated messages can keep growing in volume as long as replication does not acknowledge processing	The age of messages retained for replication is limited by the specified period
Initial transfer using a backup	No age limit for accumulated changes; enough storage space must be provided	Export, import, and processing of accumulated changes must fit within the selected period
What to monitor	Free space in the source and replication lag	Replication lag and the time remaining before the availability period expires

The availability period applies to the age of messages, not the duration of the current downtime. The PAUSED state does not suspend this period. If required messages have already been deleted, initial synchronization must be repeated to restore data completeness.

With 'UNLIMITED', replication uses an important consumer. With a finite period, it uses a consumer with a specified message availability period. This allows unacknowledged messages to be retained longer than the topic message retention period.

Restrictions

• The set of replicated objects is immutable and is generated when YDB creates an asynchronous replication instance.

• YDB supports the following types of replicated objects:

  • row-based tables, including those with synchronous secondary indexes (unique indexes are supported; vector indexes are excluded)

  • directories

    YDB will replicate all row-based tables that are located in the given directories and subdirectories at the time the asynchronous replication instance is created.

• During asynchronous replication, you cannot add or delete columns in the source tables.

• During asynchronous replication, replicas are available only for reading.

Error Handling During Asynchronous Replication

Possible errors during asynchronous replication can be grouped into the following classes:

• Temporary failures, such as transport errors, system overload, etc. Requests will be resent until they are processed successfully.
• Critical errors, such as access violation errors, schema errors, etc. Replication will be aborted, and the description of the asynchronous replication instance will include the text of the error.

For more information about error classes and how to address them, refer to Error Handling.

Consistency Levels of Replicated Data

Row-Level Data Consistency

The Change Data Capture used for asynchronous replication guarantees that changes to the same primary key are delivered in the exact order in which they occurred at the source. This ensures row-level data consistency.

Data written to replica objects in this mode becomes available for reading immediately.

Global Data Consistency

When transactions are performed at the source, several rows in a table or even rows in different table partitions may be atomically modified. Global data consistency at the target implies preserving atomicity: changes "appear" (become available for reading) in a consistent manner.

To ensure global data consistency, changefeeds must be created with barriers enabled. Data is buffered directly in the partitions of replica objects between barriers. When the next set of barriers from all topics is received, the changes are committed and become available for reading. As a result, the data at the target is consistent as of the timestamp specified in the barrier. If we assume that barriers are created every 10 seconds at the source (the default value), then under normal conditions the target also commits and publishes the set of changes from the source every 10 seconds.

Change Commit Interval

By default changes are committed no more than once every 10 seconds. You can override the change commit interval by specifying the COMMIT_INTERVAL option when creating an asynchronous replication instance.

Asynchronous Replication Completion

Completion of asynchronous replication might be an end goal of data migration from one database to another. In this case the client stops writing data to the source, waits for the zero replication lag, and completes replication. After the replication process is completed, replicas become available both for reading and writing. Then you can switch the load from the source database to the target database and complete data migration.

Before completing replication, get an up-to-date lag value using scheme describe --stats. If the value is unavailable, wait for reporting to resume: a missing value or a previously displayed zero does not confirm that all changes have been transferred.

To complete asynchronous replication, use the ALTER ASYNC REPLICATION YQL expression.

Dropping an Asynchronous Replication Instance

When you drop an asynchronous replication instance:

• Changefeeds are deleted in the source tables.
• The source tables are unlocked, and you can add and delete columns again.
• Optionally, all replicas are deleted.
• Asynchronous replication instance is deleted.

To drop an asynchronous replication instance, use the DROP ASYNC REPLICATION YQL expression.