ID | IEP-4 | ||||||||
Author | |||||||||
Sponsor | Alexey Goncharuk | ||||||||
Created | 2017-10-16 | ||||||||
Status |
|
Table of Contents |
---|
When the persistent storage was introduced for Apache Ignite, cluster membership mechanism was not changed, which means that every node startup or shutdown is treated as a cluster member change. This leads to several performance, usability and consistency issues:
Other persistence-enabled clusters, such as Cassandra, require a manual acknowledgment when adding or removing nodes. If a node fails, it is considered temporarily offline, and no data movement starts until an administrator confirms that the node is offline permanently.
To resolve the issues described above, we introduce a concept of affinity baseline topology: a target set of nodes intended to keep data for persistence-enabled caches. We will also attach a list of hashes to each baseline topology generated on branching points that will allow us to keep track of activation history and prevent data divergence in the cluster. A branching point is a cluster action which may affect data integrity of the cluster. One of the examples of branching points is cluster activation.
Baseline topology and branching history must be stored in a reliable fail-safe metadata storage which should be available for reading and updating upon node join. This is needed to detect branch divergence and prevent stale nodes from joining the cluster. Each baseline topology change and branching action is saved in the metadata storage.
Affinity for persistence-enabled caches is calculated using the baseline topology and then offline nodes are subtracted from the mapping. When a new node joins the cluster or a node goes offline, the baseline topology does not change, only affinity mapping is corrected with regard to offline nodes. Since affinity must be calculated for offline nodes, the cluster must be able to create 'phantom' topology nodes and pass them to an affinity function. Since affinity function may use arbitrary node attributes, we will introduce an interface that declares which node attributes are used in affinity calculation. Required node attributes will be stored in the metadata storage.
Baseline topology for a newly created cluster (or an old cluster with old persistence files) is created during the first cluster activation. The following baseline topology changes should be either confirmed manually or approved automatically via a user-defined pluggable interface. When a cluster with existing baseline topology is started, it waits for all nodes in the baseline topology to go online. Once all the nodes are started, the cluster can perform auto-activation.
Functionality that is not related to data affinity (compute grid, services) is not affected by baseline topology.
It's necessary to add an ability to manage baseline topology to both command-line utility set (visorcmd, control.sh script) and consider adding it to the WebConsole.
Additional information in the metadata storage increases disk space consumption. Since we store additional information about caches, we introduce another point of configuration validation. Metadata storage introduction may require additional changes to PageMemory because metadata storage may be required during node join (currently, the memory recovery happens after discovery start).
// Links to various reference documents, if applicable.
// Links or report with relevant JIRA tickets.