Self Heal State Storage

During cluster operation, nodes running YDB may fail entirely.

Self Heal State Storage ensures the operability of the metadata distribution subsystem, Board, and SchemeBoard of the cluster if it is impossible to quickly restore failed nodes, and automatically increases the number of replicas of these subsystems when new nodes are added to the cluster.

Self Heal State Storage provides:

• Detection of faulty YDB cluster nodes.
• Relocation of replicas of StateStorage, Board, and SchemeBoard to other nodes or addition of new replicas.

The Self Heal State Storage component is part of the cluster management system CMS Sentinel.

Enabling and disabling Self Heal State Storage

You can enable and disable Self Heal State Storage by changing the configuration.
For the mechanism to work, both CMS Sentinel and distributed configuration must be activated.

1. Get the current cluster configuration using the ydb admin cluster config fetch command:

   ydb [global options...] admin cluster config fetch > config.yaml
   

2. Modify the configuration file config.yaml by changing the value of parameter state_storage_self_heal_config.enable to true or false:

   config:
       self_management_config:
           enabled: true # Enabling distributed configuration
       cms_config:
           sentinel_config:
               enable: true # Enabling Sentinel
               state_storage_self_heal_config:
                   enable: true # Enabling self heal state storage
   

   For the mechanism to work, both CMS Sentinel and distributed configuration must be activated. Make sure they are enabled.
   For more information about migrating to configuration V2 and enabling distributed configuration.
   When the state_storage_self_heal_config.enable parameter is set to true, the mechanism for maintaining the operability and fault tolerance of StateStorage, Board, SchemeBoard is enabled.

3. Update the cluster configuration taking into account the changes made using ydb admin cluster config replace:

   ydb [global options...] admin cluster config replace -f config.yaml
   

Managing automatic configuration changes

In addition to the general enable/disable of Self Heal State Storage (parameter state_storage_self_heal_config.enable, see above), in the self_management_config section of the config.yaml configuration file, you can individually manage automatic configuration changes for each of the metadata distribution subsystems, as well as limit the set of nodes to which Self Heal can move replicas.

config:
    self_management_config:
        enabled: true
        automatic_state_storage_management: true
        automatic_state_storage_board_management: true
        automatic_scheme_board_management: true
        state_storage_self_heal_allowed_nodes: [1, 2, 3, 4, 5, 6, 7, 8]
        state_storage_board_self_heal_allowed_nodes: [1, 2, 3, 4, 5, 6, 7, 8]
        scheme_board_self_heal_allowed_nodes: [1, 2, 3, 4, 5, 6, 7, 8]

Additional Self Heal State Storage parameters

In the cms_config.sentinel_config.state_storage_self_heal_config section of the config.yaml configuration file, you can configure additional parameters of the Self Heal State Storage mechanism that affect how quickly the mechanism responds to changes and how many replicas of the metadata distribution subsystems are created. In the example below, all parameters are shown with default values:

config:
    cms_config:
        sentinel_config:
            enable: true
            state_storage_self_heal_config:
                enable: true
                wait_for_config_step: 60000000
                relax_time: 600000000
                pileup_replicas: false
                override_replicas_in_ring_count: 0
                override_rings_count: 0
                replicas_specific_volume: 200

Checking the result

You can check that the changes have been applied in the CMS section of the cluster Embedded UI (available on port 8765): go to the Sentinel tab to view the status of Sentinel and Self Heal State Storage.