Geode's dynamic membership system is based on "membership views", an idea taken from JGroups.  If you understand the components of a membership view and how these views are used it will help you in understanding what's going on in a cluster when nodes are started, are shut down or are kicked out.

A membership view is composed of four main things:

  1.  the membership ID of the coordinator
  2. the view identifier
  3. an ordered list of the IDs of the members in the cluster
  4. designation of which member is the "lead member"

A view also tracks other things like the members that departed between the previous view and the current view, failure detection ports for each member, and other metadata.

The String representation of a view looks like this:

View[creatorID|viewID] members: [oldestMemberID, nextOldestMemberID, etc. ]

For example,

View[10.118.32.95(gemfire4_host1:14476:locator)<ec><v0>:1024|4] members: [10.118.32.95(gemfire4_host1:14476:locator)<ec><v0>:1024, 10.118.26.119(gemfire6_host2:7064:locator)<ec><v1>:1024, 10.118.26.119(gemfire3_host2:7051)<v2>:1025{lead}, 10.118.26.119(gemfire3_host2:7045)<v3>:1026, 10.118.32.95(gemfire1_host1:14440)<v4>:1025, 10.118.32.95(gemfire1_host1:14434)<v4>:1026]

Coordinator: 10.118.32.95(gemfire4_host1:14476:locator)<ec><v0>:1024

View identifier: 4

Members: 10.118.32.95(gemfire4_host1:14476:locator)<ec><v0>:1024, 10.118.26.119(gemfire6_host2:7064:locator)<ec><v1>:1024, 10.118.26.119(gemfire3_host2:7051)<v2>:1025{lead}, 10.118.26.119(gemfire3_host2:7045)<v3>:1026, 10.118.32.95(gemfire1_host1:14440)<v4>:1025, 10.118.32.95(gemfire1_host1:14434)<v4>:1026

Lead member: 10.118.26.119(gemfire3_host2_7051:7051)<v2>:1025{lead}

Who creates a membership view?

Membership views are created by a membership coordinator.  Typically this is the oldest Locator in a cluster but it may be a server if locators are not running.  A member of the cluster elects itself coordinator if it detects that the current coordinator has departed and that it is now the oldest member of the cluster eligible to be coordinator.  When a member decides it should be coordinator it will announce it in its log file like this:

This member is becoming the membership coordinator with address membershipID

This will be followed by preparation of a new membership view in a new View Creator thread that it starts.  View preparation involves sending the new membership view to all members of the cluster, sending removal messages to members deemed to have crashed and sending join responses to new members.  Other members of the cluster are required to acknowledge the prepared view.  If one or more members don't respond they are examined and possibly removed from membership.  In this case the new view is abandoned and a new view is prepared that includes removal of these non-responsive members.  This is how we can quickly detect a network partition.

Once the view is prepared we send another message to have the view installed.  Membership listeners are then notified of new members and crashed or shut-down members.

Other log messages associated with views that can be useful are:

preparing new view View.toString - logged by coordinator when sending the view

sending new view View.toString - logged by coordinator when installing the view

received new view: View.toString - logged by each member when it receives the view installation message

View Identifiers

View identifiers are integers and they increase in value by one unless there has been a change in coordinator.  In that case they increase by 5 plus some random number between 0 and 10.

Lead Member

The oldest non-locator member in the view is typically the lead member.  A lead member gets extra weight in quorum calculations (see below).

Membership Identifiers

A membership Identifier uniquely identifies each member of the cluster.  A new identifier is generated for a locator or server when it joins the cluster - identifiers are never reused.

A membership id has these primary components:

  • InetAddress of the member
  • Name of the member
  • Process ID of the member (if available)
  • Type of member (e.g., locator or server)
  • Whether the member is preferred to be the membership coordinator ("<ec>")
  • View identifier in which the member joined ("<v1>")
  • The membership port of the member (currently a JGroups UDP port number)

The string representation of an ID looks like this:

InetAddress(name:PID:type)<ec><v123>: port

For example,

10.118.32.95(gemfire4_host1:14476:locator)<ec><v0>:1024

InetAddress: 10.118.32.95 - this will be a host name in some circumstances

Name: gemfire4_host1

Process ID: 14476

Type of member: locator

Preferred for Coordinator: <ec> - yes, it's preferred for coordinator

View identifier: <v0> - it was in the initial membership view

Membership port: 1024

Membership Weights and Quorum

Each member has a quorum weight.  This is used by members to determine if there has been a possible network partition.  In case of a partition a sub-group with 51% or more of the weight of the last installed membership view will continue to run.  Other sub-groups will shut down.

Locators are given a weight of 3.

Servers are given a weight of 10.

The lead member is given an additional weight of 5.

When there has been a loss of weight in a new view the coordinator will log its quorum calculations.  For example,

View Creator is processing 4 requests for the next membership view

10.118.26.119(gemfire6_host2:7064:locator)<ec><v1>:1024 had a weight of 3

10.118.26.119(gemfire3_host2:7051)<v2>:1025 had a weight of 15

10.118.26.119(gemfire3_host2:7045)<v3>:1026 had a weight of 10

total weight lost in this view change is 28 of 51. Quorum has been lost!


Other Uses of Membership Views

Here are a few of the uses of the Membership View:

Distributed Lock Service

The dlock service uses the membership view to designate an Elder member.  This member is tasked with managing recovery of granted lock state if a lock grantor crashes.

Region Creation

When a cache Region is created the membership view is used to communicate the fact to all other members and find out who already has the same Region.  This information is used to form a DistributionAdvisor for the Region.  The DistributionAdvisor knows who should receive different kinds of messages and also tracks the addition/removal of nodes as the cluster evolves.  It also invokes application Listener objects to notify them of these events.

Message Distribution

When the peer-to-peer messaging system is told to send a message to another member of the cluster the messaging system will only give up trying to send the message if the destination is no longer in the membership view.

When waiting for replies we also do not give up waiting on a member unless it is no longer in the membership view.

Function Execution

The function execution service indirectly uses the membership view to know who is in the cluster when it needs to execute a function on all servers.


  • No labels