Page History

...

• SlotPartitionTable: a consistent-slot-hashing-based Partition Table, which divides the whole data into a fixed number of slots (10000 slots by default), distributes the slots evenly to all DataGroups, hashes the partition key to one of the slots, and the data holder will be the holder of the slot. Upon the DataGroups are updated, the slots are re-distributed and the changes are recorded for data migration.

Heartbeat ThreadHeartbeatThread

Heartbeat Thread is an important component of a Raft Member. As the character of a Raft Member is always one of LEADER, FOLLOWER, or ELECTOR(CANDIDATE), it utilizes Heartbeat Thread in three ways: as a LEADER, Heartbeat Thread sends heartbeats to followers; as a FOLLOWER, Heartbeat Thread periodically checks whether the leader has timed out; as a CANDIDATE, it starts elections and send election requests to other nodes until one of the nodes becomes a valid leader.

...

1. Start-up script `start-node.sh` creates ClusterMain, the program entry point of a cluster node;
2. ClusterMain creates a MetaClusterServer, which receives MetaGroup RPCs from internal_meta_port;
3. MetaClusterServer checks its configuration with other nodes, ensuring that at least over half of the nodes have consistent configuration;
4. MetaClusterServer initializes the underlying IoTDB;
5. MetaClusterServer creates a MetaMember, which is initialized as an ELECTOR, handles MetaGroup RPCs and manages a partition table;
6. MetaMember tries to load the partition table from the local storage if it exists;
7. MetaMember creates its MetaHeartbeatThread;
8. MetaHeartbeatThread sends election requests to other cluster nodes;
9. The quorum of the MetaGroup agree with the election and send responses to MetaClusterServer;
10. MetaClusterServer lets MetaMember handle these responses;
11. MetaMember gathers the responses and confirms that it has become a LEADER, then create a partition table if there is none;
12. MetaMember creates DataClusterServer, which receives DataGroup RPCs from internal_metadata_port;
13. DataClusterServer creates DataMembers depending on the partition table and the number of replications k; k DataMembers will be created, each for a DataGroup the node is in;
14. DataMembers establish their own DataHeartbeatThreads, and by following similar procedures, they become FOLLOWERs or LEADERs;
15. MetaMember creates ClientServer, which receives requests from clients, so by now the node is ready to serve.

Image Modified

Figure.2 Start-up of the MetaGroup leader

Fig.3 demonstrates the start-up of a follower node. Followers and leaders are the same for the first few steps, and the divergence point is that followers fail to gather enough votes for themselves before they receives heartbeat from a legal leader.

1. Start-up script `start-node.sh` creates ClusterMain, the program entry point of a cluster node;
2. ClusterMain creates a MetaClusterServer, which receives MetaGroup RPCs from internal_meta_port;
3. MetaClusterServer checks its configuration with other nodes, ensuring that at least over half of the nodes have consistent configuration;
4. MetaClusterServer initializes the underlying IoTDB;
5. MetaClusterServer creates a MetaMember, which is initialized as an ELECTOR, handles MetaGroup RPCs and manages a partition table;
6. MetaMember tries to load the partition table from the local storage if it exists;
7. MetaMember creates its MetaHeartbeatThread;
8. The leader of MetaGroup sends a heartbeat to MetaClusterServer;
9. MetaClusterServer lets MetaMember handle the heartbeat;
10. MetaMember becomes a LEADER, then update its partition table from the hearbeat if the heartbeat provides a newer heartbeat;
11. MetaMember creates DataClusterServer, which receives DataGroup RPCs from internal_data_port;
12. DataClusterServer creates DataMembers depending on the partition table and the number of replications k; k DataMembers will be created, each for a DataGroup the node is in;
13. DataMembers establish their own DataHeartbeatThreads, and by following similar procedures, they become FOLLOWERs or LEADERs;
14. MetaMember creates ClientServer, which receives requests from clients, so by now the node is ready to serve.

Image Added

Figure.3 Start-up of the MetaGroup leader

How cluster metadata related operations are executed is explained in Figure.4. Notice that when we say metadata in the context of the cluster module, we mean cluster configurations, storage groups, and authentication info, which may be different from concepts of the stand-alone IoTDB. As the flow is simpler compared with start-up, leader side and follower side are put together for the convenience of comparison.