Page History

...

Detecting and removing failed nodes from topology

Detecting failed nodes is a responsibility of each server node in cluster. However detecting failed servers and clients works slightly different and will be considered separately.

Server nodes failures - general case

As servers are organized into ring each node can easily detect failure of its next neighbor node when sending of next discovery message fails.
However process of removing failed node from the ring (there is a special structure called "ring" on each server node which reflects topology view of current node, see ServerImpl#ring field) is not that simple and has to be managed by coordinator. It contains two steps.

Step one

On the first step server detected that it next has failed adds it to the special local map called failedNodes. Using this map it will filter failed nodes from all servers presented in the ring until the second step is completed for each failed node.

Along with adding next node to failedNodes map server picks up from the ring next server right after the failed and tries to establish connection with it. In case of success found server becomes new next, otherwise the process repeats from the beginning: adding node to failedNodes map, picking up next to failed and so on.

When ring is restored and next alive server node found, current node adds info about all nodes from failedNodes map to any discovery message it is about to send and sends it (see how this field is handled in ServerImpl$RingMessageWorker#sendMessageAcrossRing method).Any disco message can inform about failed nodes in topology (see TcpDiscoveryAbstractMessage#failedNodes collection and logic around) and all nodes receiving any disco message start processing it from updating their local failedNodes maps with info from that disco message (see ServerImpl$RingMessageWorker#processMessage where method processMessageFailedNodes is called almost at the beginning).

Also server detected failed node is responsible for creating TcpDiscoveryNodeFailedMessage - special discovery message that starts second step of removing failed node.

Step two

Second step starts on coordinator.

When coordinator receives TcpDiscoveryNodeFailedMessage it verifies it and sends it across the ring. All servers (including coordinator) update their local topology versions, finally remove failed node[s] from the ring structure and failedNodes map (see ServerImpl$RingMessageWorker#processNodeFailedMessage).

Client nodes failures

Section patiently waits for someone to contribute content

...