Asynchronous replication

Asynchronous replication allows for synchronizing data between YDB databases in near real time. It can also be used for data migration between databases with minimal downtime for applications interacting with these databases. Such databases can be located in the same YDB cluster as well as in different clusters.

Overview

Asynchronous replication is based on Change Data Capture and operates on logical data. The following diagram illustrates the replication process:

As shown in the diagram above, asynchronous replication involves two databases:

1. Source. A database with replicated objects.
2. Target. A database where an asynchronous replication instance and replica objects will be created.

Asynchronous replication consists of the following stages:

• Initialization
• Initial table scan
• Change data replication

Initialization

Initialization of asynchronous replication includes the following steps:

• Creating an asynchronous replication instance on the target database using the CREATE ASYNC REPLICATION YQL expression.
• Establishing a connection with the source database. The target database connects to the source using the connection parameters specified during the creation of the asynchronous replication instance.

• The following objects are created for replicated objects on the source:
  • changefeeds on the source
  • replica objects on the target

Initial table scan

During the initial table scan the source data is exported to changefeeds. The target runs consumers that read the source data from the changefeeds and write it to replicas.

You can get the progress of the initial table scan from the description of the asynchronous replication instance.

Change data replication

After the initial table scan is completed, the consumers read the change data and write it to replicas.

Each change data block has its creation time ($created\_at$). Consumers track the reception time of the change data ($received\_at$). Thus, you can use the following formula to calculate the replication lag:

$$replication\_lag = received\_at - created\_at$$

You can also get the replication lag from the description of the asynchronous replication instance.

Restrictions

• The set of replicated objects is immutable and is generated when YDB creates an asynchronous replication instance.

• YDB supports the following types of replicated objects:

  • row-based tables

  • directories

    YDB will replicate all row-based tables that are located in the given directories and subdirectories at the time the asynchronous replication instance is created.

• During asynchronous replication, you cannot add or delete columns in the source tables.

• During asynchronous replication, replicas are available only for reading.

Error handling during asynchronous replication

Possible errors during asynchronous replication can be grouped into the following classes:

• Temporary failures, such as transport errors, system overload, etc. Requests will be resent until they are processed successfully.
• Critical errors, such as access violation errors, schema errors, etc. Replication will be aborted, and the description of the asynchronous replication instance will include the text of the error.

For more information about error classes and how to address them, refer to Error handling.

Asynchronous replication completion

Completion of asynchronous replication might be an end goal of data migration from one database to another. In this case the client stops writing data to the source, waits for the zero replication lag, and completes replication. After the replication process is completed, replicas become available both for reading and writing. Then you can switch the load from the source database to the target database and complete data migration.

To complete asynchronous replication, use the ALTER ASYNC REPLICATION YQL expression.

Dropping an asynchronous replication instance

When you drop an asynchronous replication instance:

• Changefeeds are deleted in the source tables.
• The source tables are unlocked, and you can add and delete columns again.
• Optionally, all replicas are deleted.
• Asynchronous replication instance is deleted.

To drop an asynchronous replication instance, use the DROP ASYNC REPLICATION YQL expression.