THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
GMSHealthMonitor makes sure that each member in the distributed system is alive and communicating to this member. To make sure that, we create the ring of members based on current view. On this ring, each member makes sure that the next member in the ring (its neighbor) is communicating with it. For that we record last message timestamp from its neighbor. And if it sees its neighbor has not communicated in last period(member-timeout) then we check whether its neighbor is still alive or not. Based on that we informed probable coordinators to remove its neighbor from the view, if its neighbor is not alive.
HeartbeatMessage
Each Member periodically sends HeartbeatMesage (UDP) to the coordinator and two other members in the distributed system. The two members selected are most likely monitoring the sender of the HeartbeatMessage. Upon receiving the HearbeatMessage, the receiving member updates its record of receiving timestamp associated with the sender. The requestID of HeartbeatMessage is set to -1 so that the receiver does not look for a thread waiting for the message. HeartbeatMessages are themselves used as replies to HeartbeatRequestMessages(see next section).
HeartbeatRequestMessage
For each member, another monitoring thread is checking the timestamp of its neighbor's HeartbeatMessage or any other messages received from its neighbor. Note that any message received from another member counts as a heartbeat. See GMSHealthMonitor.contactedBy(). If the member has not received the HeartbeatMessage from its neighbor for more than 5 seconds (the default for membership timeout). It will start sending HeartbeatRequestMessage to its neighbor to check whether its neighbor is still alive. If its neighbor is still alive, it will respond with HeartbeatMessage with requestID set to the same as HeartbeatRequestMessage. Upon receiving the response from its neighbor, the member will update its record of timestamp accordingly. If no response for HeartbeatRequestMessage is received from its neighbor, it will check its timestamp record again. It is possible that the neighbor has sent another HeartbeatMessage during the waiting period.
SuspectMembersMessage and Final Check
If there is still no response from its neighbor, and no update on the timestamp, the member will initiate a suspicion on its neighbor. Basically, the member will then send SuspectMemberMessage which includes a list of SuspectRequests to a list of recipients. Depending on the size of the view, the list of recipients may contain all the members in the view, if the view size is less than or equal to 4. If the view size is larger than 4, the list of recipients may have up to 7 members, which include 5 members preferred to be coordinator, the sender itself and a random member. There is only one coordinator. How the recipient reacts to the SuspectMemberMessage depends on whether it is the coordinator or not. If it is the coordinator, it will start final check on the suspect member. To do final check, the coordinator sends HeartbeatRequestMessage to the suspect member, expecting a response from suspect member. At the same time, the coordinator also starts TCP final check, which initiates a TCP connection to the suspect member and exchanges the messages if the suspect member is still alive.
If final check failed, the coordinator then asks GMSJoinLeave to remove the suspect member from the system.
If the recipient of SuspectMembersMessage is not the coordinator, it checks to see if it should become the coordinator and initiate a final check. If it is not the coordinator and should not become the coordinator, it records the SuspectRequest for subsequent use.