Recovering system tablets

Note

For conceptual information about the mechanism, see the System tablet backup section.

Warning

Recovering system tablets is a critical operation that may result in data loss. Perform it only if you have a clear understanding of the problem and after consulting with the operations team. Before starting, make sure you have reviewed all the steps.

Step 1. Put the tablet into Recovery mode

The tablet to be recovered must be put into Recovery mode.

Warning

If recovery is performed after a complete loss of the static group, all system tablets must be put into Recovery mode simultaneously with recreating the static group on new hosts. If you recreate the static group without putting the tablets into Recovery mode, they will automatically start on top of an empty static group, leading to incorrect cluster operation.

  1. Determine the ID of the system tablet to be recovered. The tablet ID can be found in the Tablets section of the YDB UI.

  2. Save the current cluster configuration to a file config.yaml.

  3. Determine the list of hosts where the system tablet to be recovered can run based on the cluster configuration. This list will be used in subsequent steps for restarting nodes and updating the configuration.

    For this, it is convenient to use a script that takes the tablet ID and the cluster configuration file as input, which stores the list of nodes where system tablets can run and the mapping from nodes to hosts. The script combines this information and outputs the list of hosts where the given system tablet can run, in a format suitable for pssh.

    Create an isolated environment with PyYAML:

    python3 -m venv ./recovery && ./recovery/bin/pip install pyyaml

    Create the script file:

    cat > find-tablet-hosts.py <<'EOF'
#!/usr/bin/env python3
import argparse
import yaml


def find_nodes(cfg, tid):
    bootstrap = (cfg.get("bootstrap_config") or {}).get("tablet") or []
    for tablet in bootstrap:
        if str(tablet.get("info", {}).get("tablet_id")) == tid:
            return tablet.get("node") or []

    for tablets_by_type in (cfg.get("system_tablets") or {}).values():
        for tablet in tablets_by_type or []:
            if str(tablet.get("info", {}).get("tablet_id")) == tid:
                return tablet.get("node") or []

    return []


parser = argparse.ArgumentParser()
parser.add_argument("--config-path", required=True, help="Path to cluster configuration file")
parser.add_argument("--tablet-id", required=True, help="System tablet id")
args = parser.parse_args()

with open(args.config_path) as f:
    cfg = yaml.safe_load(f) or {}

cfg = cfg.get("config", cfg)

hosts_by_id = {h["node_id"]: h["host"] for h in cfg.get("hosts") or []}

node_ids = find_nodes(cfg, args.tablet_id)
if not node_ids:
    node_ids = list(hosts_by_id)

for node_id in sorted(set(node_ids)):
    host = hosts_by_id.get(node_id)
    if host is not None:
        print(host)
EOF

    Run the script, substituting the path to your configuration and the tablet ID:

    Example:

    ./recovery/bin/python find-tablet-hosts.py --config-path config.yaml --tablet-id <tablet-id> > hosts.txt

  4. Modify the saved configuration by adding boot_type: RECOVERY to the startup configuration section of the tablet being recovered and reduce the list of nodes where the tablet being recovered can run (the node array) to a single node.
    Example for tablet Hive with ID 72057594037968897:

    • When using bootstrap_config:

          bootstrap_config:
        tablet:
        - type: FLAT_HIVE
            node:
            - 1
            info:
                tablet_id: '72057594037968897'
                channels:
                - channel: 0
                history:
                - from_generation: 0
                    group_id: 0
                channel_erasure_name: mirror-3-dc
                - channel: 1
                history:
                - from_generation: 0
                    group_id: 0
                channel_erasure_name: mirror-3-dc
                - channel: 2
                history:
                - from_generation: 0
                    group_id: 0
                channel_erasure_name: mirror-3-dc
            boot_type: RECOVERY

    • When using system_tablets:

      flat_hive:
- info:
    tablet_id: 72057594037968897
  node:
  - 9
  boot_type: RECOVERY

  5. Update the configuration in the cluster.

    • When using configuration V1, update the static configuration on all hosts obtained in step 3 using the command:

      Copy the updated configuration to a temporary file on each host:

      pscp -h hosts.txt config.yaml /tmp/config.yaml.new

      Make a backup of the current configuration, replace it with the new version, and protect it from accidental changes:

      pssh -h hosts.txt 'sudo cp /opt/ydb/cfg/config.yaml /opt/ydb/cfg/config.yaml.bak && sudo mv /tmp/config.yaml.new /opt/ydb/cfg/config.yaml && sudo chattr +i /opt/ydb/cfg/config.yaml'

    • When using configuration V2, follow the instructions.

  6. Restart all nodes where the tablet being recovered can run. If any node is unavailable and cannot be restarted, isolate it from the cluster over the network — for example, using a firewall.

    Example:

    pssh -h hosts.txt 'sudo systemctl restart ydbd-storage; sleep 10'

    Warning

    Make sure that all nodes where the tablet being recovered can run are restarted with the updated configuration or isolated from the cluster over the network before starting recovery. During recovery, old data is erased and replaced with data from the backup. If at that moment the tablet starts in normal mode on a node with the old configuration, it may start working with partially recovered data.

  7. Make sure that:

    • There are no issues with the tablet in HealthCheck.
    • The tablet is not restarting.
    • The recovery form is available in the tablet's App in the YDB UI.

Step 2. Find the backup files

  1. On each host obtained in step 3, check for the presence of backups. The path to backups is determined by the path parameter in the system_tablet_backup_config configuration section.

    The name of each backup contains key information: backup_<timestamp>_g<generation>_s<step>, where:

    • timestamp — backup creation time;
    • generation — tablet generation, increases with each tablet restart;
    • step — tablet step within a generation, increases with each change in tablet state.

  2. Select the most recent backup suitable for recovery.

    Select the backup with the maximum generation, and if generations are equal, with the maximum step.

    Make sure the backup is fully written. The selected backup must contain the snapshot directory, not snapshot.tmp. The presence of snapshot.tmp means that the snapshot write was not completed and the backup is not suitable for recovery. In this case, select the previous most recent backup.

    ls /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/snapshot/

    manifest.json
schema.json
Tablet.json
TabletFollowerGroup.json
...

    Check that the backup files contain data:

    ls -lh /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/snapshot

    total 128K
-rw-r--r-- 1 ydb disk  591 May 27 11:34 manifest.json
-rw-r--r-- 1 ydb disk  12K May 27 11:34 schema.json
-rw-r--r-- 1 ydb disk  47K May 27 11:34 Tablet.json
-rw-r--r-- 1 ydb disk  41K May 27 11:34 TabletFollowerGroup.json
...

    Calculate the checksum of the changelog.json file and verify it against the checksum recorded in changelog.json.sha256:

    sha256sum /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/changelog.json

    ea4bdc2f7afaf7b6d35adbf13b5360e4e4a19046f742effdf1b4f0bb9c449185

    cat /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/changelog.json.sha256

    ea4bdc2f7afaf7b6d35adbf13b5360e4e4a19046f742effdf1b4f0bb9c449185

    If the checksums differ, two situations are possible:

    • The last entry in changelog.json was not fully written. In this case, the checksum stored in changelog.json.sha256 is found in one of the entries in changelog.json in the prev_sha256 field. To recover, you need to edit the files: remove/complete the incompletely written entries in changelog.json and update the checksum in changelog.json.sha256.
    • Data corruption has occurred, recovery is impossible.

Step 3. Transfer the backup files

  1. Determine which host the tablet is running on in Recovery mode. To do this, open the YDB UI and find the host where the tablet is running.

  2. If the backup files are on a different host, copy them to the host with the tablet in Recovery mode using scp, rsync, or any other available tool:

    Copy the backup from the backup directory to your home directory:

    cp -R /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222 ~/backup_20251007T193502_g214_s1222

    Copy the backup to the target host:

    scp -r ~/backup_20251007T193502_g214_s1222 target-host:~/backup_20251007T193502_g214_s1222

    If direct copying between hosts is not available, copy through an intermediate host:

    
scp -r backup-host:~/backup_20251007T193502_g214_s1222 backup_20251007T193502_g214_s1222

scp -r backup_20251007T193502_g214_s1222 target-host:~/backup_20251007T193502_g214_s1222

    After copying the backup to the target host, grant access to it for the ydb process:

    chown -R ydb ~/backup_20260527104029Z_g1911_s

Step 4. Perform the recovery

  1. Open the App of the tablet being restored in the YDB UI.

  2. In the recovery form, specify the full path to the directory with the backup files, for example:

    /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222

  3. If necessary, set the flags:

    • Dry Run — performs a trial recovery without making changes to the storage. Allows you to verify the correctness of the backup. After the trial recovery completes, you must restart the tablet.
    • Skip Checksum Validation — skips the checksum verification of the backup files.

    Warning

    Always perform a Dry Run before recovery to verify the correctness of the backup.

    Warning

    Skip checksum verification only if you have manually edited the backup files. In other cases, it is recommended to keep verification enabled to ensure the integrity of the restored data.

  4. Click the Start Restore button. Before starting the recovery, the system will ask for confirmation, as the operation overwrites the existing tablet data. Confirm the action to start. After the recovery starts, the form becomes unavailable — a restart is not possible until the tablet is restarted.

    Note

    You can also start the recovery using curl by sending a POST request to the tablet's App page with the restoreBackup parameter:

    curl -X POST "http://<host>:<mon_port>/tablets/app?TabletID=72057594037968897&restoreBackup=/tablet/hive/72057594037968897/backup_20251007T193502_g214_s1222"

    Where <host> is the cluster node address, <mon_port> is the monitoring port of this node.

  5. Monitor the recovery progress. The page does not update automatically — to get the current status, refresh the page manually. The recovery duration depends on the backup size.

    Note

    The recovery is performed server-side and does not depend on the browser. You can close the page or browser — this will not interrupt the recovery process. When you reopen the page, the current status will be displayed.

    The current operation status is displayed below the form. Possible statuses:

    • Restoring from '<path>' — recovery is in progress, data from the backup is being read and written to the storage. Additionally, a progress bar with the operation completion percentage is displayed.
    • Restore from '<path>' completed successfully — recovery completed successfully. You can proceed to the next step.
    • Restore from '<path>' completed, but changelog is not fully restored — the main tablet data has been restored, but the tail of the change log in the backup is damaged, and some recent changes are lost. Examine which data was lost and, if necessary, restore it manually, or proceed to the next step.
    • Restore from '<path>' failed: <error description> — recovery completed with an error. Examine the error description and retry if necessary.

    To forcibly interrupt the recovery, restart the tablet.

  6. Wait for the recovery to complete successfully. If the recovery form becomes available again (the Start Restore button is active, the status is reset), this means the tablet was restarted and the recovery was interrupted. In this case, start the recovery again from step 2.

Step 5. Return the tablet to normal operation mode

After successful recovery:

  1. Restore the configuration to its original state by removing boot_type: RECOVERY from the startup configuration section of the tablet being recovered and restoring the original list of nodes where the tablet being recovered can run.

    • When using configuration V1, update the static configuration on all hosts obtained in step 3 using the command:

      Remove the protection against accidental changes and replace the new configuration with the original version:

      pssh -h hosts.txt 'sudo chattr -i /opt/ydb/cfg/config.yaml && sudo mv /opt/ydb/cfg/config.yaml.bak /opt/ydb/cfg/config.yaml'

    • When using configuration V2, follow the instructions.

  2. Restart all nodes where the tablet being recovered can run. If any nodes were isolated from the cluster over the network in previous steps, remove the network isolation.

    Example:

    pssh -h hosts.txt 'sudo systemctl restart ydbd-storage; sleep 10'

  3. Make sure that:

    • There are no issues with the tablet in HealthCheck.
    • The tablet does not restart.
    • The recovery form is absent in the tablet's App in YDB UI.

  4. Restart all cluster nodes one by one to synchronize the state of internal in-memory caches with the tablet state. After restarting each node, wait for it to return to a healthy state and make sure there are no issues in HealthCheck; only then proceed to the next node.

Previous
Enable and configure
Next
Backup to NFS