Recovering system tablets

Note

For conceptual information about the mechanism, see the System tablet backup section.

Warning

Recovering system tablets is a critical operation that may result in data loss. Perform it only if you have a clear understanding of the problem and after consulting with the operations team. Before starting, make sure you have reviewed all the steps.

Step 1. Put the tablet into Recovery mode

The tablet to be recovered must be put into Recovery mode. In this mode, the tablet starts and is accessible via the Embedded UI, but does not work normally and does not read data from the distributed storage, allowing recovery operations to be performed. Other tablets will continue to operate normally, allowing the cluster to keep functioning, but some control-plane operations may be unavailable.

Warning

If recovery is performed after a complete loss of the static group, all system tablets must be put into Recovery mode simultaneously with recreating the static group on new hosts. If you recreate the static group without putting the tablets into Recovery mode, they will automatically start on top of an empty static group, leading to incorrect cluster operation.

  1. Determine the ID of the system tablet to be recovered. The tablet ID can be found in the Tablets section of the Embedded UI.

  2. Determine the list of nodes where the system tablet to be recovered can run. This list is located in the bootstrap_config section of the corresponding tablet in the cluster configuration. If the bootstrap_config section is missing from the configuration, use the list of all static nodes of the cluster specified in the hosts section of the cluster configuration.

  3. Modify the configuration by adding boot_mode: RECOVERY to the bootstrap_config section of the tablet being recovered.

    • When using configuration V1, you need to modify the static configuration on all nodes where the tablet being recovered can run.
    • When using configuration V2, follow the instructions.
    • Example for tablet Hive with ID 72057594037968897:
        bootstrap_config:
        tablet:
        - type: FLAT_HIVE
          node:
          - 1
          - 2
          - 3
          info:
              tablet_id: '72057594037968897'
              channels:
              - channel: 0
              history:
              - from_generation: 0
                  group_id: 0
              channel_erasure_name: mirror-3-dc
              - channel: 1
              history:
              - from_generation: 0
                  group_id: 0
              channel_erasure_name: mirror-3-dc
              - channel: 2
              history:
              - from_generation: 0
                  group_id: 0
              channel_erasure_name: mirror-3-dc
          boot_mode: RECOVERY

  4. Restart all nodes where the tablet being recovered can run. If any node is unavailable and cannot be restarted, isolate it from the cluster over the network — for example, using a firewall.

    Warning

    Make sure that all nodes where the tablet being recovered can run are restarted with the updated configuration or isolated from the cluster over the network before starting recovery. During recovery, old data is erased and replaced with data from the backup. If at that moment the tablet starts in normal mode on a node with the old configuration, it may start working with partially recovered data.

  5. Make sure that:

    • There are no issues with the tablet in HealthCheck.
    • The tablet is not restarting.
    • The recovery form is available in the tablet's App in the Embedded UI.

Step 2. Find the backup files

  1. Determine which hosts to search. First, check the hosts where the tablet was running before the failure. These hosts can be identified using logs or a monitoring system.

    If you cannot determine specific hosts, check all hosts where the tablet could have been running. This list is located in the bootstrap_config section of the corresponding tablet in the cluster configuration. If the bootstrap_config section is missing from the configuration, use the list of all static nodes of the cluster specified in the hosts section of the cluster configuration.

  2. Find the directory with backups. On each candidate host, check for the presence of backups. The path to backups is determined by the path parameter in the system_tablet_backup_config configuration:

    ls /path/to/backup/directory/<tablet_type>/<tablet_id>/

    Example for tablet Hive with ID 72057594037968897:

    ls /tablet/hive/72057594037968897/

    backup_20251007T181003_g213_s1001
backup_20251007T191002_g214_s1040
backup_20251007T193502_g214_s1222

  3. Select the most recent backup. The name of each backup contains key information: backup_<timestamp>_g<generation>_s<step>, where:

    • timestamp — backup creation time.
    • — tablet generation, increases with each tablet restart.
    • step — tablet step within a generation, increases with each change in tablet state.

    Select the backup with the maximum generation, and if generations are equal, with the maximum step. If backups are found on multiple hosts, compare them and select the most recent one.

  4. Make sure the backup is fully written. The selected backup must contain the snapshot directory, not snapshot.tmp. The presence of snapshot.tmp means that the snapshot write was not completed and the backup is not suitable for recovery. In this case, select the previous most recent backup.

    ls /tablet/hive/72057594037968897/backup_20251007T193502_g214_s1222/snapshot/

    manifest.json
schema.json
Tablet.json
TabletFollowerGroup.json
...

Step 3. Transfer the backup files

  1. Determine which host the tablet is running on in Recovery mode. To do this, open the Embedded UI and find the node where the tablet is running.

  2. If the backup files are on a different host, copy them to the host with the tablet in Recovery mode using scp, rsync, or any other available tool:

    scp -r /tablet/hive/72057594037968897/backup_20251007T193502_g214_s1222 \
    user@target-host:~/backup_20251007T193502_g214_s1222

  3. Make sure the files are readable by the YDB process on the target host.

Step 4. Perform the recovery

  1. Open the App of the tablet being restored in the Embedded UI.

  2. In the recovery form, specify the full path to the directory with the backup files, for example:

    /tablet/hive/72057594037968897/backup_20251007T193502_g214_s1222

  3. If necessary, set the flags:

    • Dry Run — performs a trial recovery without making changes to the storage. Allows you to verify the correctness of the backup.
    • Skip Checksum Validation — skips the checksum verification of the backup files.

    Warning

    Always perform a Dry Run before the first recovery to verify the correctness of the backup.

    Warning

    Skip checksum verification only if you have manually edited the backup files. In other cases, it is recommended to keep verification enabled to ensure the integrity of the restored data.

  4. Click the Start Restore button. Before starting the recovery, the system will ask for confirmation, as the operation overwrites the existing tablet data. Confirm the action to start. After the recovery starts, the form becomes unavailable — a restart is not possible until the tablet is restarted.

    Note

    You can also start the recovery using curl by sending a POST request to the tablet's App page with the restoreBackup parameter:

    curl -X POST "http://<host>:<mon_port>/tablets/app?TabletID=72057594037968897&restoreBackup=/tablet/hive/72057594037968897/backup_20251007T193502_g214_s1222"

    Where <host> is the cluster node address, <mon_port> is the monitoring port of this node.

  5. Monitor the recovery progress. The page does not update automatically — to get the current status, refresh the page manually. The recovery duration depends on the backup size.

    Note

    The recovery is performed server-side and does not depend on the browser. You can close the page or browser — this will not interrupt the recovery process. When you reopen the page, the current status will be displayed.

    The current operation status is displayed below the form. Possible statuses:

    • Restoring from '<path>' — recovery is in progress, data from the backup is being read and written to the storage. Additionally, a progress bar with the operation completion percentage is displayed.
    • Restore from '<path>' completed successfully — recovery completed successfully. You can proceed to the next step.
    • Restore from '<path>' completed, but changelog is not fully restored — the main tablet data has been restored, but the tail of the change log in the backup is damaged, and some recent changes are lost. Examine which data was lost and, if necessary, restore it manually, or proceed to the next step.
    • Restore from '<path>' failed: <error description> — recovery completed with an error. Examine the error description and retry if necessary.

    To forcibly interrupt the recovery, restart the tablet.

  6. Wait for the recovery to complete successfully. If the recovery form becomes available again (the Start Restore button is active, the status is reset), this means the tablet was restarted and the recovery was interrupted. In this case, start the recovery again from step 2.

Step 5. Return the tablet to normal operation mode

After successful recovery:

  1. Determine the list of nodes on which the system tablet being recovered can run. This list is located in the bootstrap_config section of the corresponding tablet in the cluster configuration. If the bootstrap_config section is absent from the configuration, use the list of all static nodes of the cluster specified in the hosts section of the cluster configuration.

  2. Modify the configuration by removing boot_mode: RECOVERY from the bootstrap_config section of the tablet being recovered.

    • When using configuration V1, you need to change the static configuration on all nodes on which the tablet being recovered can run.
    • When using configuration V2, follow the instructions.

  3. Restart all nodes on which the tablet being recovered can run. If any nodes were isolated from the cluster over the network in previous steps, remove the network isolation.

  4. Make sure that:

    • There are no issues with the tablet in HealthCheck.
    • The tablet does not restart.
    • The recovery form is absent in the tablet's App in Embedded UI.
Previous
Enable and configure
Next
Overview