Page History

...

The user issues the ‘cluster migrate’ command via the CLI specifying --old-cluster-node url and --new-cluster-nodeurl.

1. The CLI obtains new cluster state from --new-cluster-nodeurl
2. It then invokes the /recovery/cluster/migrate REST endpoint on the node at --old-cluster-node url sending it the new cluster state
3. The node receiving the command sends ClusterResetMessage (having CMG nodes, MG nodes, cluster name and cluster ID from new cluster node) to every node that it sees in its physical topology (including itself)
4. This makes all the nodes still in the old cluster to switch to the new cluster

...

1. User gets a notification about the absence of a majority of the MG
2. User tries to restart Ignite nodes hosting MG nodes (or just their RAFT nodes inside Ignite nodes)
3. If this does not work, the user issues a ‘cluster reset {–cmg-nodes=<new-cmg-nodes>|--node=<existing-node>} --metastorage-replication-factor=N’ command first making sure that every possible node starts and joins. (If the user specifies the --node argument instead of --cmgcluster-management-nodesgroup, the command will take current CMG voting members set from the CMG leader (via --node); if the CMG is not available, the command will fail)
4. The general procedure for CMG recreation is invoked (see above); Metastorage-specific steps get executed after the CMG is recreated (see below)

...

1. All 3 nodes get segmented (we get 3 network segments, one per node)
2. User is in the B segment. They see that cmg.available metric drops to 0 on B (the only observable node)
3. User issues ignite recovery cluster reset --cluster-management-nodesgroup=B command
4. B gets ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321). It saves it to the Vault and restarts
5. During the restart, it starts using new name/ID (Galileo/54321) for network handshakes
6. Network partition disappears, A and C get restarted, but they cannot connect to B. They form a cluster AC (as A still hosts old CMG)
7. B finds the message in the Vault, clears the CMG locally and does reinit: the new cluster is formed. Now it removes the message from the Vault.
8. User issues ignite recovery cluster migrate –old-cluster-nodeurl=A –new-cluster-nodeurl=B command
9. As A and C see each other in the physical topology, both A and C get ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321), they save these to their Vaults and restart
10. Upon restart, A and C find the message in the Vault, clear their local CMG, connect to the new CMG (from the message) and remove the message. Now they are also in the new cluster

...

1. All 3 nodes get segmented (we get 3 network segments, one per node)
2. User is in the B segment. They see that cmg.available and mg.available metrics drop to 0 on B (the only observable node)
3. User issues ignite recovery cluster reset --cluster-nodesmanagement-group=B –metastorage-replication-factor=1 command
4. B gets ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321, mgReplicationFactor=1, conductor=true, nodes=[B]). It saves it to the Vault and restarts
5. During the restart, it starts using new clusterId (54321) for network handshakes
6. Network partition disappears, A and C get restarted, but they cannot connect to B. They form a cluster AC (as A still hosts old MG and CMG) and start writing to the Metastorage with term=6 (and indexes=101+)
7. B finds the message in the Vault, clears the CMG locally and does reinit: the new cluster is formed. Now it removes the message from the Vault.
8. B finds that the MG on it has <index, term> equal to <100, 5>. It chooses itself as the new MG voting set and does resetPeers. Now we have a second, independent, MG.
9. User issues ignite recovery cluster migrate –old-cluster-nodeurl=A –new-cluster-nodeurl=B command
10. As A and C see each other in the physical topology, both A and C get ResetClusterMessage(cmgNodes=[B], clusterName=Galileo, clusterId=54321), they save these to their Vaults and restart
11. Upon restart, A and C find the message in the Vault, clear their local CMG, connect to the new CMG (from the message) and remove the message.
12. A and C try to validate their Metastorage. It has diverged, so none of them can join, both switch to the ‘zombie’ state

...

• ignite recovery cluster reset [--node <nodeName> | --cluster-cmgmanagement-nodes <nodeNames>group <nodeNames>] [--metastorage-replication-factor=N]
• ignite recovery cluster migrate --old-cluster-node url <nodeName> --new-cluster-node url <nodeName>
• ignite recovery cluster restart cmg|metastorage [--nodes <nodeNames>]
• ignite recovery cluster states cmg|metastorage [--local [--nodes <nodeNames>] | --global]
• Ignite recovery cluster truncate cmg|metastorage --index <index>

...