Versions Compared

Old Version 1

changes.mady.by.user Unknown User (rpuch)

Saved on

compared with

New Version 2

changes.mady.by.user Unknown User (rpuch)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. User sees in the monitoring that the CMG majority is lost; they try to bring the lost nodes to life, this does not work
  2. User issues a ‘cluster reset’ command passing new CMG nodes (these must be in the physical topology) (‘ignite cluster reset –cmg-nodes=<new-cmg-nodes>’)
  3. The command gets sent to the first node from new-cmg-nodes
  4. The node that got the repair command is the Repair Conductor
  5. The general procedure for CMG recreation is invoked with new CMG nodes given in the command (see below)

After this sequence is executed, the nodes of the cluster that got the ClusterResetMessage will have switched to the new CMG.

...

  1. The Conductor sends ClusterResetMessage to all the nodes that are currently in the physical topology (including itself, with one difference - if MG is to be repaired, the message to self includes additional attributes: conductor=true, nodes=<set-of-nodes-to-which-the-message-gets-sent>); the message contains:
    1. new CMG nodes
    2. MG nodes (taken from the old CMG state)
    3. cluster name (taken from the old CMG state)
    4. new random cluster ID
    5. mgReplicationFactor, if provided
  2. Upon receiving a ClusterResetMessage, a node does the following:
    1. Stores it to the Vault
    2. Responds with OK
    3. Restarts itself (Conductor does this only after it gets OK from all other nodes to which it sent the message, or a timeout passes)
  3. If a node sees a ClusterResetMessage in the Vault on startup, it uses clusterId from it for handshakes
  4. During node startup, when going to start the CMGManager, if a ClusterResetMessage is in the Vault, the node does the following:
    1. If cluster ID in the local CMG data is different from the cluster ID in the message:
      1. destroys the CMG Raft group and removes local CMG data
      2. Initializes a new CMG Raft node using the new CMG nodes (from the message) as voting set
      3. Submits a CmgInitCommand (having CMG nodes, MG nodes, cluster name, new cluster ID) to the new CMG
      1. If cluster ID in the local CMG data is different from the cluster ID in the message:
      2. Removes the ClusterResetMessage from the Vault
      3. If MG repair is requested in the message, and the MG is not yet available, carries out MG reparation logic (see below); otherwise, proceeds with normal startup
    3. LogicalTopology events are accompanied with context (bearing clusterId in it), so that LogicalTopology listeners are able to react to a non-monotonous change of Logical Topology version to 1 (they can distinguish between version 1 created for old cluster ID and version 1 created for the new one)

    ...

    1. User gets a notification about the absence of a majority of the MG
    2. User tries to restart Ignite nodes hosting MG nodes (or just their RAFT nodes inside Ignite nodes)
    3. If this does not work, the user issues a ‘cluster reset {–cmg-nodes=<new-cmg-nodes>|--node=<existing-node>} --metastorage-replication-factor=N’ command first making sure that every possible node starts and joins. (If the user specifies the --node argument instead of --cmg-nodes, the command will take current CMG voting members set from the CMG leader (via --node); if the CMG is not available, the command will fail)
    4. The general procedure for CMG recreation is invoked (see above); Metastorage-specific steps get executed after the CMG is recreated (see below)

    If the Repair Conductor crashes after it removes the ClusterResetMessage from the Vault, but before it finishes the MG repair, the procedure has to be repeated again. (Failover is absent on purpose to avoid simultaneous repair attempts and to make sure each attempt starts with a node restart)

    ...

    These are the steps that get executed after the general CMG recreation procedure ends when metastorage-specific attributes are present in a ClusterResetMessage. The node has found ClusterResetMessage in its Vault on startup, took part in CMG recreation, removed the message from the Vault, started joining the cluster (passed the basic validation).

    1. The node pauses its startup routine before starting the Metastorage
    2. The Repair Conductor waits till all nodes listed in the message appear in the ‘basically validated’ node set (or higher - that is, become fully validated or appear in the logical topology)
    3. The Conductor broadcasts a MetastorageRepairStartMessage to all nodes that are currently in the ‘basically validated’ node set (or higher)
    4. Upon receiving a MetastorageRepairStartMessage, an Ignite node does the following:
      1. Starts the Metastorage Raft group
      2. Obtains Raft index and term of the Metastorage and sends them in a response to the Conductor
      3. Waits for the Metastorage Raft group majority to get established, then proceeds with the normal startup
    5. After receiving responses from all the nodes (if a node leaves the ‘basically validated’ node set (or a higher set), the procedure ends with an error and ‘cluster reset’ has to be repeated), the Conductor chooses new MG nodes from those that successfully returned responses [the set of nodes that returned successful responses is U] (see details below). This new node set must include at least one of the nodes with the highest index+term among U.
    6. The Conductor submits CmgChangeMgNodesCommand (containing new MG nodes) to the CMG
    7. Execution of CmgChangeMgNodesCommand on a node changes the metastorageNodes in the cluster state
    8. The Conductor then chooses one of the new MG nodes having the highest index+term as the new leader and sends it a BecomeMetastorageLeaderMessage
    9. When a node receives a BecomeMetastorageLeaderMessage, it does resetPeers on the Metastorage Raft group by passing only self as the voting set (and no learners). After it becomes a leader, it returns a successful result to the Conductor. This step is the one that breaks the safety of the Raft protocol.
    10. When a node becomes an MG leader because it executed BecomeMetastorageLeaderMessage, it does not start issuing idle safe time commands and managing learners yet
    11. After getting a successful response to BecomeMetastorageLeaderMessage, the Conductor forms a new Raft configuration including all new MG nodes as the voting set and other nodes from U as learners. It then does changePeers via the new leader to switch to this configuration. After the leader elected on the previous step (the sole leader) gets reelected (or ceases to be a leader), the prohibition for it to send idle safe time commands and manage learners is lifted

    ...

    This is analogous to the same action for CMG reparation, see MIgrating nodes still using the old CMG to the new CMG.

    After such a node (which did not take part in the Metastorage reparation) is migrated to the new cluster, it could potentially cause troubles. Following sections are about preventing them.

    ...

    1. All 3 nodes get segmented (we get 3 network segments, one per node)
    2. User is in the B segment. They see that cmg.available metric drops to 0 on B (the only observable node)
    3. User issues ignite recovery cluster reset --cluster-nodes=B command
    4. B gets ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321). It saves it to the Vault and restarts
    5. During the restart, it starts using new name/ID (Galileo/54321) for network handshakes
    6. Network partition disappears, A and C get restarted, but they cannot connect to B. They form a cluster AC (as A still hosts old CMG)
    7. B finds the message in the Vault, clears the CMG locally and does reinit: the new cluster is formed. Now it removes the message from the Vault.
    8. User issues ignite recovery cluster migrate –old-cluster-node=A –new-cluster-node=B command
    9. As A and C see each other in the physical topology, both A and C get ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321), they save these to their Vaults and restart
    10. Upon restart, A and C find the message in the Vault, clear their local CMG, connect to the new CMG (from the message) and remove the message. Now they are also in the new cluster

    ...

    1. All 3 nodes get segmented (we get 3 network segments, one per node)
    2. User is in the B segment. They see that cmg.available and mg.available metrics drop to 0 on B (the only observable node)
    3. User issues ignite recovery cluster reset --cluster-nodes=B –metastorage-replication-factor=1 command
    4. B gets ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321, mgReplicationFactor=1, conductor=true, nodes=[B]). It saves it to the Vault and restarts
    5. During the restart, it starts using new clusterId (54321) for network handshakes
    6. Network partition disappears, A and C get restarted, but they cannot connect to B. They form a cluster AC (as A still hosts old MG and CMG) and start writing to the Metastorage with term=6 (and indexes=101+)
    7. B finds the message in the Vault, clears the CMG locally and does reinit: the new cluster is formed. Now it removes the message from the Vault.
    8. B finds that the MG on it has <index, term> equal to <100, 5>. It chooses itself as the new MG voting set and does resetPeers. Now we have a second, independent, MG.
    9. User issues ignite recovery cluster migrate –old-cluster-node=A –new-cluster-node=B command
    10. As A and C see each other in the physical topology, both A and C get ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321), they save these to their Vaults and restart
    11. Upon restart, A and C find the message in the Vault, clear their local CMG, connect to the new CMG (from the message) and remove the message.
    12. A and C try to validate their Metastorage. It has diverged, so none of them can join, both switch to the ‘zombie’ state

    ...

    1. Nodes A and B are destroyed
    2. User sees that cmg.available and mg.available metrics drop to 0 on CDE
    3. User issues ignite recovery cluster reset --node=C –metastorage-replication-factor=3 command
    4. CDE get ResetClusterMessage(cmgNodes=[CDE], clusterName=Galileo, clusterId=54321, mgReplicationFactor=3, conductor=true, nodes=[CDE]). They save it to the Vault and restart
    5. During the restart, CDE start using new clusterId (54321) for network handshakes
    6. CDE find the message in the Vault, clear the CMG locally and do reinit: the new cluster is formed. Now CDE remove the message from the Vault.
    7. C is the repair conductor. It initiates MG repair. CDE find that the MG on them have <index, term> equal to <100, 5>, <102, 5> and <102, 5>, correspondingly, and return this information to C.
    8. The conductor chooses CDE as the new voting members
    9. D is appointed to be a leader; then C does changePeers and expands the MG configuration to CDE. Now, there is a fully functional MG, the nodes finish their startup procedure.

    ...

    Sometimes, the user knows that a node is going to fail. In such situations, an API for manual CMG/MG reconfiguration using normal Raft protocol could save the day. This could look like a ‘cmg/metastorage reconfigure’ command that would invoke changePeers on the corresponding group. This is out of scope of this documentdesign.

    Another way is to automatically reconfigure a system group if it loses a minority for some time (analogously to what is done for partitions). This is also out of scope of this documentdesign.

    Handling other disastrous events

    ...

    • /management/v1/recovery/cluster/reset
    • /management/v1/recovery/cluster/migrate
    • /management/v1/recovery/cmg/state/local
    • /management/v1/recovery/cmg/state/global
    • /management/v1/recovery/cmg/restart
    • /management/v1/recovery/cmg/truncate-log-suffix
    • /management/v1/recovery/metastorage/state/local
    • /management/v1/recovery/metastorage/state/global
    • /management/v1/recovery/metastorage/restart
    • /management/v1/recovery/metastorage/truncate-log-suffix

    CLI

    • ignite recovery cluster reset [--node <nodeName> | --cmg-nodes <nodeNames>] [--metastorage-replication-factor=N]
    • ignite recovery cluster migrate --old-cluster-node <nodeName> --new-cluster-node <nodeName>
    • ignite recovery cluster restart cmg|metastorage [--nodes <nodeNames>]
    • ignite recovery cluster states cmg|metastorage [--local [--nodes <nodeNames>] | --global]
    • Ignite recovery cluster truncate cmg|metastorage --index <index>

    ...

    schemaSync.waits is a reservoir (a histogram?) of times of all waits (in milliseconds) caused by schema synchronization (might indicate that something is wrong with the Metastorage)

    Risks and Assumptions

    // Describe project risks, such as API or binary compatibility issues, major protocol changes, etc.

    Discussion Links

    // Links to discussions on the devlist, if applicableLogical topology versions are not monotonous anymore as during a cluster reset version number falls to 1. This has to be handled carefully in rebalancing code.

    Reference Links

    ...

    1. IEP-77: Node Join Protocol and Initialization for Ignite 3
    2. IEP-126: Table/zone disaster recovery

    Tickets

    // Links or report with relevant JIRA tickets.TODO