...

Terminology

...

1. It must be possible to repair CMG and/or MG if they lose majority
2. Old majority returning back online must not cause troubles
3. Best-effort consistency guarantees for MG: allow a node to join the cluster if only minor inconsistencies (like divergence in SafeTime) are detected. If major inconsistencies are there (like diverged Catalog), the node will only be allowed to play a ‘zombie’ role (so that the user is able to copy partitions’ data from it).

Design

Problem of nodes which are offline during a Raft group

...

repair

We can forcefully reset the voting set of a group (breaking the safety of the Raft protocol). This effectively means that we create a new majority that is independent from the old one. Having two independent majorities might cause the following problems if nodes that didn’t take part in the reparation repair come back online:

1. If the returning nodes were included in the voting set of the group before the majority loss, then they can form the old majority again. The new leader would append the new configuration (containing the old voting members) to all nodes that it knows about (for MG, this is all nodes in the cluster due to learners). This might cause the leadership of the group to be hijacked by the old nodes that saw no repair, which might result in inconsistent data in the group state machines
2. It might happen that the nodes that come back online had applied some command X before going offline such that no node (which took part in the reparationrepair) saw that command, leading to the same inconsistent data problem. This could happen like this:
1. ABC is the voting member set, N is a learner
2. AB and N get X, it gets applied at AB and N
3. AB get destroyed, N goes down
4. CDE are appointed as the new voting member set
5. N starts; only this node has X, and X is applied

...

All data stored in CMG either can be safely lost or can be restored from other sources.

First, the user should try to restart CMG nodes to restore the majority. If it does not work, they’ll have to forcefully assign a new majority using the ‘recovery cluster reset’ command (issued manually via CLI/REST).

...

1. User sees in the monitoring that the CMG majority is lost; they try to bring the lost nodes to life, this does not work
2. User issues a ‘cluster reset’ command passing new CMG nodes (these must be in the physical topology) (‘ignite recovery cluster reset –cmg–cluster-management-nodesgroup=<new-cmg-nodes>’)
3. The command gets sent to the first node from new-cmg-nodes
4. The node that got the repair command is the Repair Conductor
5. The general procedure for CMG recreation is invoked with new CMG nodes given in the command (see below)

...

1. The Conductor sends ClusterResetMessage to all the nodes that are currently in the physical topology (including itself, with one difference - if MG is to be repaired, the message to self includes additional attributes: conductor=true, nodes=<set-of-nodes-to-which-the-message-gets-sent>); the message contains:
1. new CMG nodes
2. MG nodes (taken from the old CMG state)
3. cluster name (taken from the old CMG state)
4. new random cluster ID
5. mgReplicationFactor, if provided
2. Upon receiving a ClusterResetMessage, a node does the following:
1. Stores it to the Vault
2. Responds with OK
3. Restarts itself (Conductor does this only after it gets OK from all other nodes to which it sent the message, or a timeout passes)
3. If a node sees a ClusterResetMessage in the Vault on startup, it uses clusterId from it for handshakes
4. During node startup, when going to start the CMGManager, if a ClusterResetMessage is in the Vault, the node does the following:
   1. destroys the CMG Raft group and removes local CMG data
   2. Initializes a new CMG Raft node using the new CMG nodes (from the message) as voting set
   3. Submits a CmgInitCommand (having CMG nodes, MG nodes, cluster name, new cluster ID) to the new CMG
1. If cluster ID in the local CMG data is different from the cluster ID in the message:
2. Removes the ClusterResetMessage from the Vault
3. If MG repair is requested in the message, and the MG is not yet available, carries out MG reparation repair logic (see below); otherwise, proceeds with normal startup
5. LogicalTopology events are accompanied with context (bearing clusterId in it), so that LogicalTopology listeners are able to react to a non-monotonous change of Logical Topology version to 1 (they can distinguish between version 1 created for old cluster ID and version 1 created for the new one)

...

The user issues the ‘cluster migrate’ command via the CLI specifying --old-cluster-node url and --new-cluster-nodeurl.

1. The CLI obtains new cluster state from --new-cluster-nodeurl
2. It then invokes the /recovery/cluster/migrate REST endpoint on the node at --old-cluster-node url sending it the new cluster state
3. The node receiving the command sends ClusterResetMessage (having CMG nodes, MG nodes, cluster name and cluster ID from new cluster node) to every node that it sees in its physical topology (including itself)
4. This makes all the nodes still in the old cluster to switch to the new cluster

...

1. Recreate CMG to make sure the old MG majority cannot intervene
2. Collect information about MG Raft indexes+terms and choose the freshest nodes
3. Reset MG using the freshest nodes as the new voting set
4. Migrate to the new cluster any nodes of the old cluster that were down/segmented during the reparationrepair

Forceful MG repair

1. User gets a notification about the absence of a majority of the MG
2. User tries to restart Ignite nodes hosting MG nodes (or just their RAFT nodes inside Ignite nodes)
3. If this does not work, the user issues a ‘cluster ‘recovery cluster reset {–cmg-nodes--cluster-management-group=<new-cmg-nodes>|--node=<existing-node>} --metastorage-replication-factor=N’ command first making sure that every possible node starts and joins. (If the user specifies the --node argument instead of --cluster-cmgmanagement-nodesgroup, the command will take current CMG voting members set from the CMG leader (via --node); if the CMG is not available, the command will fail)
4. The general procedure for CMG recreation is invoked (see above); Metastorage-specific steps get executed after the CMG is recreated (see below)

...

This is analogous to the same action for CMG reparationrepair, see MIgrating nodes still using the old CMG to the new CMG.

After such a node (which did not take part in the Metastorage reparationrepair) is migrated to the new cluster, it could potentially cause troubles. Following sections are about preventing them.

...

If a network partition happens and the majority of MG remains functional in one segment, while the user only observes another segment where they do a forceful repair, there is a possibility for the MG to diverge. To reduce the probability of such an event, we could add TopologyValidators similar to Apache Ignite 2: if a validator thinks that the current topology became invalid, it switches the Metastorage to read-only mode. For example, a validator might react to a sudden drop of the number of online nodes.

Consequences of MG

...

repair

There are two possibilities:

1. All nodes that are not destroyed returned to the cluster or were cleaned, but the Metastorage did not diverge. In the worst case, we lost some Metastorage updates cluster-wide, but each of them was only known to the nodes that were destroyed or cleaned. Not a single node in the repaired cluster has ever seen any of these updates, including updates to the Catalog. Not a single node in the repaired cluster has any tuple of a version that was lost. So the cluster was just cleanly pushed back in time with regards to the Metastorage.
2. Some nodes that did not take part in the reparation repair were tried to be returned to the cluster, but the Metastorage on them was found to have diverged wrt the new cluster, so they are not let to finish the join procedure. They can only be ‘zombies’. The nodes that took part in the reparation repair demonstrate the same property as in item 1 (above).

...

1. All 3 nodes get segmented (we get 3 network segments, one per node)
2. User is in the B segment. They see that cmg.available metric drops to 0 on B (the only observable node)
3. User issues ignite recovery cluster reset --cluster-nodesmanagement-group=B command
4. B gets ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321). It saves it to the Vault and restarts
5. During the restart, it starts using new name/ID (Galileo/54321) for network handshakes
6. Network partition disappears, A and C get restarted, but they cannot connect to B. They form a cluster AC (as A still hosts old CMG)
7. B finds the message in the Vault, clears the CMG locally and does reinit: the new cluster is formed. Now it removes the message from the Vault.
8. User issues ignite recovery cluster migrate –old-cluster-nodeurl=A –new-cluster-nodeurl=B command
9. As A and C see each other in the physical topology, both A and C get ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321), they save these to their Vaults and restart
10. Upon restart, A and C find the message in the Vault, clear their local CMG, connect to the new CMG (from the message) and remove the message. Now they are also in the new cluster

...

1. All 3 nodes get segmented (we get 3 network segments, one per node)
2. User is in the B segment. They see that cmg.available and mg.available metrics drop to 0 on B (the only observable node)
3. User issues ignite recovery cluster reset --cluster-management-nodesgroup=B –metastorage-replication-factor=1 command
4. B gets ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321, mgReplicationFactor=1, conductor=true, nodes=[B]). It saves it to the Vault and restarts
5. During the restart, it starts using new clusterId (54321) for network handshakes
6. Network partition disappears, A and C get restarted, but they cannot connect to B. They form a cluster AC (as A still hosts old MG and CMG) and start writing to the Metastorage with term=6 (and indexes=101+)
7. B finds the message in the Vault, clears the CMG locally and does reinit: the new cluster is formed. Now it removes the message from the Vault.
8. B finds that the MG on it has <index, term> equal to <100, 5>. It chooses itself as the new MG voting set and does resetPeers. Now we have a second, independent, MG.
9. User issues ignite recovery cluster migrate –old-cluster-nodeurl=A –new-cluster-nodeurl=B command
10. As A and C see each other in the physical topology, both A and C get ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321), they save these to their Vaults and restart
11. Upon restart, A and C find the message in the Vault, clear their local CMG, connect to the new CMG (from the message) and remove the message.
12. A and C try to validate their Metastorage. It has diverged, so none of them can join, both switch to the ‘zombie’ state

...

• ignite recovery cluster reset [--node <nodeName> | --cluster-cmgmanagement-nodes <nodeNames>group <nodeNames>] [--metastorage-replication-factor=N]
• ignite recovery cluster migrate --old-cluster-node url <nodeName> --new-cluster-node url <nodeName>
• ignite recovery cluster restart cmg|metastorage [--nodes <nodeNames>]
• ignite recovery cluster states cmg|metastorage [--local [--nodes <nodeNames>] | --global]
• Ignite recovery cluster truncate cmg|metastorage --index <index>

...