Backup concepts

YDB ensures data safety during hardware failures through replication and fault tolerance. However, replication does not protect against logical errors: an accidental DROP TABLE, an erroneous mass UPDATE, or DELETE will be replicated to all replicas. To protect against such scenarios, backup is required — a separate copy of data that can be restored from.

Full backup

A full backup is a snapshot of table data at a specific point in time. YDB provides several ways to create full backups, from simple to more functional.

Copying tables within a cluster

The simplest way is to create a copy of a table (or several tables) within the same cluster using the ydb tools copy command. The copy is created atomically from a consistent snapshot and uses a copy-on-write mechanism, so the operation is fast.

Suitable for:

• Quickly creating a safety copy before a risky operation.
• Cloning data for testing.

Dump to file system

The ydb tools dump and ydb tools restore commands allow you to dump data to a local file system and restore it back.

Suitable for:

• Local development and testing.
• Small databases.
• Creating a copy on a separate medium.

Export to S3-compatible storage

The ydb export s3 and ydb import s3 commands allow you to export and import data to an external S3-compatible storage.

Suitable for:

• Disaster recovery (data is stored outside the cluster).
• Data migration between clusters.
• Long-term archiving.

Incremental backup

When working with large tables, repeatedly creating full backups can be too expensive. Incremental backup solves this problem: after an initial full copy, each subsequent increment captures only the changes (inserts, updates, deletes) that occurred since the previous backup.

Incremental backups are organized into a chain:

Full backup → Incremental₁ → Incremental₂ → ... → Incrementalₙ

To restore, the entire chain is needed: first the full copy is applied, then all increments sequentially. Restoration is performed to the state at the time of the last increment in the chain.

Incremental backup is implemented using backup collections.

Backup of system tablets

The system tablet backup mechanism provides incremental copying of cluster metadata — such as Hive, BSController, and SchemeShard — to the local file system of cluster hosts.

This mechanism is used to restore cluster metadata when restoring from database backups is technically possible but not suitable in terms of time or effort. A typical scenario is when the total volume of databases in the cluster is large due to their number, the size of individual databases, or a combination of both; a full import/restore of all data to a new cluster in such a case leads to prolonged downtime. In this scenario, you can restore the system tablets and bring the cluster back to a working state without performing a mass restore of user data on the new cluster.

If the volume of databases allows standard restoration, use export/import or dump/restore first. System tablet backup should be used as a special mechanism for situations where you need to restore exactly the cluster metadata and reduce the volume of restoration operations.

How it works

Backup consists of two components:

• State snapshot — on each start, the tablet scans all its tables and writes its full state to a backup, including the data schema. Scanning is performed based on a state snapshot and does not block tablet operation.
• Change log — on each data or schema change, the tablet asynchronously writes the change to the log in parallel with writing to the distributed storage. When the log size exceeds the snapshot size, the tablet automatically takes a new snapshot.

Backups are created locally on the host where the tablet is currently running. Therefore, the most up-to-date copy is on the host where the tablet was running just before the failure.

The number of stored backups on a host is limited in the configuration. After a successful snapshot, the oldest copy is automatically deleted when the limit is exceeded. Incomplete copies (without a fully written snapshot) are deleted when a new backup is created.

Comparison of approaches

See also

• Backup and restore — practical guide

• Backup collections — architecture and limitations

• Recipes for backing up system tablets — enabling and restoring

• Backup and restore — practical guide

• export/import reference — CLI commands

• YQL reference:

  • CREATE BACKUP COLLECTION
  • BACKUP
  • RESTORE
  • DROP BACKUP COLLECTION