...

This document uses a notation of "initialized" and "empty" nodes. An initialized node is a node that has received the "init" message command sometime in its lifetime and therefore possesses the Cluster Tag and the Meta Storage Topology version. An empty node is a node that has never received the "init" command and therefore does not possess the aforementioned properties.

...

A cluster tag is a string that uniquely identifies a cluster. It is generated once per cluster and is distributed across the nodes during the "init" phasenode join process. The purpose of a cluster tag is to understand whether a joining node used to be a member of another cluster, in which case its Meta Storage Topology Version is not comparable and the joining node should be rejected. Together In other words, Cluster Tag, together with the Meta Storage Topology Version, it creates a partial ordering that allows to compare different configuration versionsMeta Storage configurations.

A cluster tag should consist of two parts:

1. Human-readable part: a string property that is set by the system administrator. Its purpose is to make the debugging and error reporting easier.
2. Unique part: a generated unique string (e.g. a UUID). Its purpose is to ensure that cluster tags Cluster Tags are different between different clusters.

...

TODO: describe coordinator re-election.

Initial cluster setup

This section describes the process of assembling a new Ignite cluster from a set of nodes.

1. Initial set of nodes is configured, including the following properties:
   1. List of all nodes in the initial cluster setup (provided by the IP Finder).
2. A Join Coordinator is elected (see "Join Coordinator election");
3. The Join Coordinator generates a Cluster Tag , (if it doesnhasn't already have it in its Vault (e.g. an existing cluster is being restartedbeen generated);
4. All other nodes connect to the Coordinator and provide the following information:
   1. Ignite product version;
   2. Cluster Tag, if any (if a node has obtained it at any time during its life);
   3. Meta Storage Topology version, if any (if a node has obtained it at any time during its life).
5. All of the aforementioned parameters get compared with the information , stored on the Coordinator, and if all of the parameters are the same, the joining node is allowed into the cluster. Otherwise, the joining node is rejected.
6. Join Coordinator adds the new node to the list of validated nodes. This can be implemented using different approaches, for example it is possible to piggyback on the current Cluster Membership implementation (ScaleCube) and to modify its transport protocol so that only valid nodes will be visible to others (similar to the implementation of the Handshake Protocol). Another way is to explicitly store and maintain a list of valid nodes in the Meta Storage.
7. If the joining node is allowed to enter the topology, it receives the following parameters from the Coordinator:
   1. Cluster Tag;Meta Storage Topology version (if any, see "Cluster initialization").

DISCUSSION NEEDED: What to do if the Coordinator dies during any step of the setup process.

Cluster initialization

This section describes the next step of the Ignite cluster lifecycle: moving the nodes from the "zombie" state into the "active" state.

1. After the cluster has been established, it remains in the "zombie" state, until the "init" command arrives;
2. "Init" command is sent by the administrator either directly to the Join Coordinator, or to any other node, in which case it should be redirected to the Join Coordinator;
3. The "init" command should specify the following information:
   
   1. Human-readable part of the Cluster Tag;
   2. List of addresses of the nodes that should host the Meta Storage Raft group (a.k.a. Meta Storage Configuration).
4. The Join Coordinator completes the creation of the Cluster Tag by generating the unique part and generates the initial Meta Storage Topology Version property (if it hasn't already been generated);
5. The Join Coordinator atomically broadcasts the Meta Storage Configuration to all valid nodes in the topology. If this step is successful, then Meta Storage is considered to be initialized and available;
6. The Join Coordinator persists the following information into the Meta Storage (therefore propagating it to all nodes):
   1. Cluster Tag;
   2. List of addresses of all nodes that have passed the initial validation;
   3. Meta Storage Configuration Topology Version.

DISCUSSION NEEDED: What to do if the Coordinator dies during any step of the initialization process.

...

1. The new node connects to a random node. If this node is not the Join Coordinator, the random node redirects the new node to it.
2. The new node provides its Ignite product version and is validated on the Coordinator (see "Initial cluster setup").
3. If the validation is successful, the new node is added to the list of validated nodes and receives the list of Meta Storage Configuration hosts as the response.

In case of errors or a timeout, the whole process should be repeated by the joining node.

...

DISCUSSION NEEDED: What to do if the joining node's Meta Storage Topology Version is greater than the Version stored on the Coordinator.?

Changes in API

TODO

Risks and Assumptions

...