GMSHealthMonitor Message Sequence Diagram

GMSHealthMonitor makes sure that each member in the distributed is alive and communicating to this member. To make sure that we create the ring of members based on current view. On this ring, each member make sure that the next member in ring (its neighbor) is communicating with it. For that we record last message timestamp from its neighbor. And if it sees its neighbor has not communicated in last period(member-timeout) then we check whether its neighbor is still alive or not. Based on that we informed probable coordinators to remove its neighbor from the view.

HeartbeatMessage

Each Member periodically sends HeartbeatMesage (UDP) to all the other members in the distributed system, including the coordinator. Upon receiving the HearbeatMessage, the receiving member updates its record associated with the sender of the timestamp when the message is received. The receiver does not reply to such HeartbeatMessages.

HeartbeatRequestMessage

For each member, another monitoring thread is checking the timestamp of its neighbor's HeartbeatMessage. If it has not received the HeartbeatMessage from its neighbor for more than 5 seconds (the default for membership timeout). It will start sending HeartbeatRequestMessage to its neighbor to check whether its neighbor is still alive. If its neighbor is still alive, it will respond with HeartbeatMessage. Upon receiving the response from its neighbor, the member will update its record of timestamp accordingly. If no response for HeartbeatRequestMessage is received from its neighbor, it will check the timestamp record again. It is possible that the neighbor has sent another HeartbeatMessage during the waiting period.

SuspectMembersMessage and Final Check

If there is still no response from its neighbor, and no update on the timestamp, the member will initiate a suspicion on its neighbor. Basically, the member will then send SuspectMemberMessage which includes a list of SuspectRequests to the coordinators. Upon receiving of SuspectMemberMessage, the coordinator then send HeartbeatMessage to the member. It will also start final check on the suspect member. To do final check, the coordinator sends HeartbeatRequestMessage to the suspect member. At the same time, it also start TCP final check, which initiate a TCP connection to the suspect member and exchange the messages if the suspect member is still alive. If final check failed, the coordinator then asks GMSJoinLeave to remove the suspect member from the system.