Recovering system tablets

Step 1. Put the tablet into Recovery mode

The tablet to be recovered must be put into Recovery mode.

1. Determine the ID of the system tablet to be recovered. The tablet ID can be found in the Tablets section of the YDB UI.

2. Save the current cluster configuration to a file config.yaml.

   • When using configuration V1, save the static configuration.
   • When using configuration V2, follow the instructions.

3. Determine the list of hosts where the system tablet to be recovered can run based on the cluster configuration. This list will be used in subsequent steps for restarting nodes and updating the configuration.

   For this, it is convenient to use a script that takes the tablet ID and the cluster configuration file as input, which stores the list of nodes where system tablets can run and the mapping from nodes to hosts. The script combines this information and outputs the list of hosts where the given system tablet can run, in a format suitable for pssh.

   Create an isolated environment with PyYAML:

   python3 -m venv ./recovery && ./recovery/bin/pip install pyyaml
   

   Create the script file:

   cat > find-tablet-hosts.py <<'EOF'
   #!/usr/bin/env python3
   import argparse
   import yaml
   
   
   def find_nodes(cfg, tid):
       bootstrap = (cfg.get("bootstrap_config") or {}).get("tablet") or []
       for tablet in bootstrap:
           if str(tablet.get("info", {}).get("tablet_id")) == tid:
               return tablet.get("node") or []
   
       for tablets_by_type in (cfg.get("system_tablets") or {}).values():
           for tablet in tablets_by_type or []:
               if str(tablet.get("info", {}).get("tablet_id")) == tid:
                   return tablet.get("node") or []
   
       return []
   
   
   parser = argparse.ArgumentParser()
   parser.add_argument("--config-path", required=True, help="Path to cluster configuration file")
   parser.add_argument("--tablet-id", required=True, help="System tablet id")
   args = parser.parse_args()
   
   with open(args.config_path) as f:
       cfg = yaml.safe_load(f) or {}
   
   cfg = cfg.get("config", cfg)
   
   hosts_by_id = {h["node_id"]: h["host"] for h in cfg.get("hosts") or []}
   
   node_ids = find_nodes(cfg, args.tablet_id)
   if not node_ids:
       node_ids = list(hosts_by_id)
   
   for node_id in sorted(set(node_ids)):
       host = hosts_by_id.get(node_id)
       if host is not None:
           print(host)
   EOF
   

   Run the script, substituting the path to your configuration and the tablet ID:

   Example:

   ./recovery/bin/python find-tablet-hosts.py --config-path config.yaml --tablet-id <tablet-id> > hosts.txt
   

4. Modify the saved configuration by adding boot_type: RECOVERY to the startup configuration section of the tablet being recovered and reduce the list of nodes where the tablet being recovered can run (the node array) to a single node.
   Example for tablet Hive with ID 72057594037968897:

   • When using bootstrap_config:

         bootstrap_config:
             tablet:
             - type: FLAT_HIVE
                 node:
                 - 1
                 info:
                     tablet_id: '72057594037968897'
                     channels:
                     - channel: 0
                     history:
                     - from_generation: 0
                         group_id: 0
                     channel_erasure_name: mirror-3-dc
                     - channel: 1
                     history:
                     - from_generation: 0
                         group_id: 0
                     channel_erasure_name: mirror-3-dc
                     - channel: 2
                     history:
                     - from_generation: 0
                         group_id: 0
                     channel_erasure_name: mirror-3-dc
                 boot_type: RECOVERY
     

   • When using system_tablets:

     flat_hive:
     - info:
         tablet_id: 72057594037968897
       node:
       - 9
       boot_type: RECOVERY
     

5. Update the configuration in the cluster.

   • When using configuration V1, update the static configuration on all hosts obtained in step 3 using the command:

     Copy the updated configuration to a temporary file on each host:

     pscp -h hosts.txt config.yaml /tmp/config.yaml.new
     

     Make a backup of the current configuration, replace it with the new version, and protect it from accidental changes:

     pssh -h hosts.txt 'sudo cp /opt/ydb/cfg/config.yaml /opt/ydb/cfg/config.yaml.bak && sudo mv /tmp/config.yaml.new /opt/ydb/cfg/config.yaml && sudo chattr +i /opt/ydb/cfg/config.yaml'
     

   • When using configuration V2, follow the instructions.

6. Restart all nodes where the tablet being recovered can run. If any node is unavailable and cannot be restarted, isolate it from the cluster over the network — for example, using a firewall.

   Example:

   pssh -h hosts.txt 'sudo systemctl restart ydbd-storage; sleep 10'
   

   Warning

   Make sure that all nodes where the tablet being recovered can run are restarted with the updated configuration or isolated from the cluster over the network before starting recovery. During recovery, old data is erased and replaced with data from the backup. If at that moment the tablet starts in normal mode on a node with the old configuration, it may start working with partially recovered data.

7. Make sure that:

   • There are no issues with the tablet in HealthCheck.
   • The tablet is not restarting.
   • The recovery form is available in the tablet's App in the YDB UI.

Step 2. Find the backup files

1. On each host obtained in step 3, check for the presence of backups. The path to backups is determined by the path parameter in the system_tablet_backup_config configuration section.

   The name of each backup contains key information: backup_<timestamp>_g<generation>_s<step>, where:

   • timestamp — backup creation time;
   • generation — tablet generation, increases with each tablet restart;
   • step — tablet step within a generation, increases with each change in tablet state.

2. Select the most recent backup suitable for recovery.

   Select the backup with the maximum generation, and if generations are equal, with the maximum step.

   Make sure the backup is fully written. The selected backup must contain the snapshot directory, not snapshot.tmp. The presence of snapshot.tmp means that the snapshot write was not completed and the backup is not suitable for recovery. In this case, select the previous most recent backup.

   ls /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/snapshot/
   

   manifest.json
   schema.json
   Tablet.json
   TabletFollowerGroup.json
   ...
   

   Check that the backup files contain data:

   ls -lh /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/snapshot
   

   total 128K
   -rw-r--r-- 1 ydb disk  591 May 27 11:34 manifest.json
   -rw-r--r-- 1 ydb disk  12K May 27 11:34 schema.json
   -rw-r--r-- 1 ydb disk  47K May 27 11:34 Tablet.json
   -rw-r--r-- 1 ydb disk  41K May 27 11:34 TabletFollowerGroup.json
   ...
   

   Calculate the checksum of the changelog.json file and verify it against the checksum recorded in changelog.json.sha256:

   sha256sum /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/changelog.json
   

   ea4bdc2f7afaf7b6d35adbf13b5360e4e4a19046f742effdf1b4f0bb9c449185
   

   cat /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222/changelog.json.sha256
   

   ea4bdc2f7afaf7b6d35adbf13b5360e4e4a19046f742effdf1b4f0bb9c449185
   

   If the checksums differ, two situations are possible:

   • The last entry in changelog.json was not fully written. In this case, the checksum stored in changelog.json.sha256 is found in one of the entries in changelog.json in the prev_sha256 field. To recover, you need to edit the files: remove/complete the incompletely written entries in changelog.json and update the checksum in changelog.json.sha256.
   • Data corruption has occurred, recovery is impossible.

Step 3. Transfer the backup files

1. Determine which host the tablet is running on in Recovery mode. To do this, open the YDB UI and find the host where the tablet is running.

2. If the backup files are on a different host, copy them to the host with the tablet in Recovery mode using scp, rsync, or any other available tool:

   Copy the backup from the backup directory to your home directory:

   cp -R /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222 ~/backup_20251007T193502_g214_s1222
   

   Copy the backup to the target host:

   scp -r ~/backup_20251007T193502_g214_s1222 target-host:~/backup_20251007T193502_g214_s1222
   

   If direct copying between hosts is not available, copy through an intermediate host:

   
   scp -r backup-host:~/backup_20251007T193502_g214_s1222 backup_20251007T193502_g214_s1222
   
   scp -r backup_20251007T193502_g214_s1222 target-host:~/backup_20251007T193502_g214_s1222
   

   After copying the backup to the target host, grant access to it for the ydb process:

   chown -R ydb ~/backup_20260527104029Z_g1911_s
   

Step 4. Perform the recovery

1. Open the App of the tablet being restored in the YDB UI.

2. In the recovery form, specify the full path to the directory with the backup files, for example:

   /path/to/backup/directory/hive/72057594037968897/backup_20251007T193502_g214_s1222
   

3. If necessary, set the flags:

   • Dry Run — performs a trial recovery without making changes to the storage. Allows you to verify the correctness of the backup. After the trial recovery completes, you must restart the tablet.
   • Skip Checksum Validation — skips the checksum verification of the backup files.

   Warning

   Always perform a Dry Run before recovery to verify the correctness of the backup.

   Warning

   Skip checksum verification only if you have manually edited the backup files. In other cases, it is recommended to keep verification enabled to ensure the integrity of the restored data.

4. Click the Start Restore button. Before starting the recovery, the system will ask for confirmation, as the operation overwrites the existing tablet data. Confirm the action to start. After the recovery starts, the form becomes unavailable — a restart is not possible until the tablet is restarted.

   Note

   You can also start the recovery using curl by sending a POST request to the tablet's App page with the restoreBackup parameter:

   curl -X POST "http://<host>:<mon_port>/tablets/app?TabletID=72057594037968897&restoreBackup=/tablet/hive/72057594037968897/backup_20251007T193502_g214_s1222"
   

   Where <host> is the cluster node address, <mon_port> is the monitoring port of this node.

5. Monitor the recovery progress. The page does not update automatically — to get the current status, refresh the page manually. The recovery duration depends on the backup size.

   Note

   The recovery is performed server-side and does not depend on the browser. You can close the page or browser — this will not interrupt the recovery process. When you reopen the page, the current status will be displayed.

   The current operation status is displayed below the form. Possible statuses:

   • Restoring from '<path>' — recovery is in progress, data from the backup is being read and written to the storage. Additionally, a progress bar with the operation completion percentage is displayed.
   • Restore from '<path>' completed successfully — recovery completed successfully. You can proceed to the next step.
   • Restore from '<path>' completed, but changelog is not fully restored — the main tablet data has been restored, but the tail of the change log in the backup is damaged, and some recent changes are lost. Examine which data was lost and, if necessary, restore it manually, or proceed to the next step.
   • Restore from '<path>' failed: <error description> — recovery completed with an error. Examine the error description and retry if necessary.

   To forcibly interrupt the recovery, restart the tablet.

6. Wait for the recovery to complete successfully. If the recovery form becomes available again (the Start Restore button is active, the status is reset), this means the tablet was restarted and the recovery was interrupted. In this case, start the recovery again from step 2.

Step 5. Return the tablet to normal operation mode

After successful recovery:

1. Restore the configuration to its original state by removing boot_type: RECOVERY from the startup configuration section of the tablet being recovered and restoring the original list of nodes where the tablet being recovered can run.

   • When using configuration V1, update the static configuration on all hosts obtained in step 3 using the command:

     Remove the protection against accidental changes and replace the new configuration with the original version:

     pssh -h hosts.txt 'sudo chattr -i /opt/ydb/cfg/config.yaml && sudo mv /opt/ydb/cfg/config.yaml.bak /opt/ydb/cfg/config.yaml'
     

   • When using configuration V2, follow the instructions.

2. Restart all nodes where the tablet being recovered can run. If any nodes were isolated from the cluster over the network in previous steps, remove the network isolation.

   Example:

   pssh -h hosts.txt 'sudo systemctl restart ydbd-storage; sleep 10'
   

3. Make sure that:

   • There are no issues with the tablet in HealthCheck.
   • The tablet does not restart.
   • The recovery form is absent in the tablet's App in YDB UI.

4. Restart all cluster nodes one by one to synchronize the state of internal in-memory caches with the tablet state. After restarting each node, wait for it to return to a healthy state and make sure there are no issues in HealthCheck; only then proceed to the next node.