ID | IEP-77 |
Author | Aleksandr Polovtsev |
Sponsor | |
Created |
|
Status | DRAFT |
Ignite clusters are required to have a dynamically changing topology due to the nature of distributed systems - nodes can fail and need to be restarted at any time, so new nodes can be introduced or removed from the cluster. Users expect that topology changes do not break the consistency of the cluster and it remains operational.
This document describes the process of creating a cluster of Ignite nodes and adding new nodes to it. It describes the following concepts:
DISCUSSION NEEDED: are there any properties that need to be retrieved from the Meta Storage?
The "init" command moves the cluster from the "zombie" state into the "active" state. It is supposed to have the following characteristics (note that the "init" command has not been specified at the moment of writing and is out of scope of this document, so all statements are approximate and can change in the future):
This document uses a notation of "initialized" and "empty" nodes. An initialized node is a node that has received the "init" message sometime in its lifetime and therefore possesses the Cluster Tag and the Meta Storage Topology version. An empty node is a node that has never received the "init" command and does not possess the aforementioned properties.
Meta Storage Topology Version is a property that should be used to compute the most "recent" state of a given Meta Storage configuration. At the moment of writing, Meta Storage configuration consists of a list of cluster node names that host the Meta Storage Raft group. A possible implementation can be a monotonically increasing counter, which is increased each time this list is updated.
The node join process is proposed to be made centralized: a single node is granted the role of the Join Coordinator which is responsible of orchestrating the cluster lifecycle and validating new nodes.
A cluster tag is a string that uniquely identifies a cluster. It is generated once per cluster and is distributed across the nodes during the "init" phase. The purpose of a cluster tag is to understand whether a joining node used to be a member of another cluster, in which case its Meta Storage Topology Version is not comparable and the joining node should be rejected. Together with the Meta Storage Topology Version, it creates a partial ordering that allows to compare different configuration versions.
A cluster tag should consist of two parts:
DISCUSSION NEEDED: human-readable part can also be generated automatically, similar to Ignite 2.
NOTE: this section is not described as thorough as it needs to be in order to save some time and finalize the election protocol after a discussion.
Before the nodes can start joining a cluster, a node should be elected as the Join Coordinator. For the sake of simplicity, the following algorithm is proposed, which can later be replaced with something more sophisticated:
DISCUSSION NEEDED: What to do when constructing a cluster from some amount of stopped nodes with different Meta Storage configuration? Should it be overridden by the "init" command?
DISCUSSION NEEDED: What if we are restarting a cluster and also introducing a new node? What if it is elected as the coordinator?
DISCUSSION NEEDED: Can we actually use Raft and use its leader as the Join Coordinator? Can we use Raft's log to store the Meta Storage Topology Version? This way, it will be possible
TODO: describe coordinator re-election.
DISCUSSION NEEDED: What to do if the Coordinator dies during any step of the setup process.
DISCUSSION NEEDED: What to do if the Coordinator dies during any step of the initialization process.
This section describes a scenario when a new node wants to join an initialized cluster. Depending on the node configuration, there exist multiple possible scenarios:
If an empty node tries to join a cluster the following process is proposed:
In case of errors or a timeout, the whole process should be repeated by the joining node.
If an initialized node tries to join a cluster the following process is proposed:
DISCUSSION NEEDED: What to do if the joining node's Meta Storage Topology Version is greater than the Version stored on the Coordinator.
Current TopologyService
will be renamed to NetworkTopologyService
. It is proposed to extend this service to add validation handlers that will validate the joining nodes on the network level.
/** * Class for working with the cluster topology on the network level. */ public interface NetworkTopologyService { /** * This topology member. */ ClusterNode localMember(); /** * All topology members. */ Collection<ClusterNode> allMembers(); /** * Handlers for topology events (join, leave). */ void addEventHandler(TopologyEventHandler handler); /** * Returns a member by a network address */ @Nullable ClusterNode getByAddress(NetworkAddress addr); /** * Handlers for validating a joining node. */ void addValidationHandler(TopologyValidationHandler handler); }
The new service will have the same API, but will work on top of the Meta Storage, and will provide methods to work with the list of validated nodes. In addition to that, it will perform the validation of incoming nodes against the Meta Storage, based on the registered validation handlers.
/** * Class for working with the cluster topology on the Meta Storage level. Only fully validated nodes are allowed to be present in such topology. */ public interface TopologyService { /** * This topology member. */ ClusterNode localMember(); /** * All topology members. */ Collection<ClusterNode> allMembers(); /** * Handlers for topology events (join, leave). */ void addEventHandler(TopologyEventHandler handler); /** * Returns a member by a network address */ @Nullable ClusterNode getByAddress(NetworkAddress addr); /** * Handlers for validating a joining node. */ void addValidationHandler(TopologyValidationHandler handler); }
TopologyService
will depend on the MessagingService
(to respond and listen to validation requests) and on the MetaStorageManager
(for interacting with the Meta Storage).
// Links to discussions on the devlist, if applicable.