Geode JMX MBeans

Geode uses Java MBeans, specifically MXBeans, to expose management controls and monitoring features.

To provide cluster level management and monitoring, Geode uses a federated Open MBean strategy. The Java classes interact with a single MBeanServer that aggregates MBeans from other local and remote members. This strategy provides a consolidated, single-agent view of the distributed system.

The implementation is based on JMX  (industry-standard and friendly to generic JMX clients), this allows use of any JMX compliant tools (third party or custom build) to manage and monitor Geode cluster. For example, JConsole.

Create and Register a New Geode JMX MBean

The steps to create and register a new Geode JMX MBean are:

1. Define the MXBean interface
2. Implement the MXBean interface with MBean class
3. Implement the MBeanBridge
4. Handle MBean Lifecycle
   1. Modify ManagementListener and ManagementAdapter
   2. Define an ObjectName for the MBean
   3. Register the MBean with the MBeanServer
   4. Federate the MBean
   5. Aggregate the MBean

Define the MXBean interface

This interface provides the bean's attributes and operations. An example is RegionMXBean.

The RegionMXBean interface is partially defined like:

public interface RegionMXBean {
  public String getName();
  public String getFullPath();
  public RegionAttributesData listRegionAttributes();
  public PartitionAttributesData listPartitionAttributes();
  ...
}

Methods starting with 'get' become JMX attributes (e.g. getName will become JMX attribute Name); other methods become JMX operations (e.g. listPartitionAttributes).

Implement the MXBean interface with MBean class

The MBean provides the behavior for the MXBean. An example is RegionMBean.

public class RegionMBean extends NotificationBroadcasterSupport implements RegionMXBean {
  private RegionMBeanBridge bridge;
  public RegionMBean(RegionMBeanBridge bridge) {
    this.bridge = bridge;
  }
  public String getName() {
    return bridge.getName();
  }
  public String getFullPath() {
    return bridge.getFullPath();
  }
  public RegionAttributesData listRegionAttributes() {
    return bridge.listRegionAttributes();
  }
  public PartitionAttributesData listPartitionAttributes() {
    return bridge.listPartitionAttributes();
  ...
}

Implement the MBeanBridge

As you can see from the RegionMBean code sample above, the MBean behavior is normally delegated to an object the acts as a bridge between the management layer and the cache layer called a Bridge. An example is RegionMBeanBridge.

The MBeanBridge does the actual work of providing the data for the MBean. There are several ways it does this, including:

• Accessing data directly from the object being monitored
• Accessing raw statistic values
• Computing rates
• Computing average latencies
• Providing data beans

Accessing Data Directly from the Object Being Monitored

One way the MBeanBridge provides data for an MBean is by accessing data directly from the object being monitored. An example from RegionMBeanBridge is:

public String getFullPath() {   
  return region.getFullPath();
}

Accessing Raw Statistic Values

Another way that the MBeanBridge provides data for an MBean is by accessing statistic values directly. An example from DiskStoreMBeanBridge is:

The constructor initializes disStoreStats field:

public DiskStoreMBeanBridge(DiskStore ds) {
	...
	this.diskStoreStats = diskStore.getStats();
	...
}

The getTotalBackupCompleted method accesses the backupsCompleted statistic directly from the statistics:

public int getTotalBackupCompleted(){
	return getDiskStoreStatistic(StatsKey.BACKUPS_COMPLETED).intValue();
}

public Number getDiskStoreStatistic(String statName) {
	if(diskStoreStats != null){
		return diskStoreStats.getStats().get(statName);  
	}
	return 0;
}

Computing Rates

Another data type provided by the MBeanBridge is the rate of change for a particular statistic. This is done using a MBeanStatsMonitor and a StatsRate object.

The MBeanStatsMonitor is a StatisticsListener that tracks specific statistics and updates their values as they change. The MBeanStatsMonitor is registered with a ValueMonitor (in the statistics system) and is notified via the handleNotification callback whenever the statistics it is monitoring change.

The example below from GatewaySenderMBeanBridge shows how to instantiate a MBeanStatsMonitor and add statistics to it to be monitored.

private MBeanStatsMonitor monitor;

public GatewaySenderMBeanBridge(GatewaySender sender) {
  this.monitor = new MBeanStatsMonitor(ManagementStrings.GATEWAY_SENDER_MONITOR.toLocalizedString());
  GatewaySenderStats stats = abstractSender.getStatistics();
  addGatewaySenderStats(stats);
}

public void addGatewaySenderStats(GatewaySenderStats gatewaySenderStats) {
  monitor.addStatisticsToMonitor(gatewaySenderStats.getStats());
}

Once this is done, all the statistics in the GatewaySenderStats are monitored by the MBeanStatsMonitor. At this point, rate of change data can be returned by the MBeanBridge using a StatsRate. An example from GatewaySenderMBeanBridge is batchesDispatchedRate:

private StatsRate batchesDispatchedRate;

batchesDispatchedRate = new StatsRate(StatsKey.GATEWAYSENDER_BATCHES_DISTRIBUTED, StatType.INT_TYPE, monitor);

A StatsRate is created on a numeric count statistic. In this case, its batchesDistributed.

To retrieve that rate, the StatsRate getRate method is invoked like:

public float getBatchesDispatchedRate() {
  return batchesDispatchedRate.getRate();
}

The getRate method returns the difference between the current and previous value for the statistic.

Computing Average Latencies

Once the MBeanStatsMonitor is initialized like above, average latency data can also be provided by the MBeanBridge using a StatsAverageLatency. An example from GatewaySenderMBeanBridge is batchDistributionAvgLatency:

private StatsAverageLatency batchDistributionAvgLatency;

batchDistributionAvgLatency = new StatsAverageLatency(
  StatsKey.GATEWAYSENDER_BATCHES_DISTRIBUTED, 
  StatType.INT_TYPE, 
  StatsKey.GATEWAYSENDER_BATCHES_DISTRIBUTE_TIME, 
  monitor);

A StatsAverageLatency is created on a numeric count statistic and a time for that statistic. In this case, its batchesDistributed and batchDistributionTime. Note: average latencies are maintained only if enable-time-statistics=true.

To retrieve that latency, the StatsAverageLatency getAverageLatency method is invoked like:

public long getAverageDistributionTimePerBatch() {
  return batchDistributionAvgLatency.getAverageLatency();
}

The getAverageLatency method returns the time statistic divided by the numeric count statistic.

Providing Data Beans

Another data type provided by the MBeanBridge is a data beans containing several bits of primitive data. An example is JVMMetrics provided by MemberMBeanBridge fetchJVMMetrics:

public JVMMetrics fetchJVMMetrics() {
	long gcCount = getGCStatistic(StatsKey.VM_GC_STATS_COLLECTIONS).longValue();
	long gcTimeMillis = getGCStatistic(StatsKey.VM_GC_STATS_COLLECTION_TIME).longValue();
	long initMemory = memoryMXBean.getHeapMemoryUsage().getInit();
	long committedMemory = memoryMXBean.getHeapMemoryUsage().getCommitted();
	long usedMemory = getVMStatistic(StatsKey.VM_USED_MEMORY).longValue();
	long maxMemory = memoryMXBean.getHeapMemoryUsage().getMax();
	int totalThreads = getVMStatistic(StatsKey.VM_STATS_NUM_THREADS).intValue();
	return new JVMMetrics(gcCount, gcTimeMillis, initMemory, committedMemory,
	  usedMemory, maxMemory, totalThreads);
}

The JVMMetrics class is defined like:

public class JVMMetrics {
  private long gcCount;
  private long gcTimeMillis;
  private long initMemory;
  private long committedMemory;
  private long usedMemory;
  private long maxMemory;
  private int totalThreads;
  @ConstructorProperties( { "gcCount", "gcTimeMillis", "initMemory",
      "committedMemory", "usedMemory", "maxMemory", "totalThreads"
  })
  public JVMMetrics(long gcCount, long gcTimeMillis, long initMemory,
    long committedMemory, long usedMemory, long maxMemory, int totalThreads) {
    this.gcCount = gcCount;
    this.gcTimeMillis = gcTimeMillis;
    this.initMemory = initMemory;
    this.committedMemory = committedMemory;
    this.usedMemory = usedMemory;
    this.maxMemory = maxMemory;
    this.totalThreads = totalThreads;
  }
}

Note: The JVMMetrics (and other data beans) are returned to the JMX client as CompositeData objects.

Handle MBean Lifecycle

Modify ManagementListener and ManagementAdapter

When a Cache is created, it creates a ManagementListener that listens for ResourceEvents (in the GemFireCacheImpl constructor). Each ResourceEvent triggers an MBean to be instantiated, registered and federated. For example, when a Region is created, the REGION_CREATE ResourceEvent is sent like: 

public Region createVMRegion(String name, RegionAttributes p_attrs, InternalRegionArguments internalRegionArgs) {
  ...
  system.handleResourceEvent(ResourceEvent.REGION_CREATE, rgn);
}

In the ManagementListener, the REGION_CREATE ResourceEvent causes the RegionMBean to be instantiated and registered like:

In ManagementListener:
 
public void handleEvent(ResourceEvent event, Object resource) {
 ...
 case REGION_CREATE:
   adapter.handleRegionCreation(createdRegion);
   break;
 ...
}
 
In ManagementAdapter:
 
public <K, V> void handleRegionCreation(Region<K, V> region) {
 ...
 RegionMBeanBridge<K, V> bridge = RegionMBeanBridge.getInstance(region);
 RegionMXBean regionMBean = new RegionMBean<K, V>(bridge);
 ObjectName regionMBeanName = MBeanJMXAdapter.getRegionMBeanName(cacheImpl
     .getDistributedSystem().getDistributedMember(), region.getFullPath());
 ObjectName changedMBeanName = service.registerInternalMBean(regionMBean, regionMBeanName);
 service.federate(changedMBeanName, RegionMXBean.class, true);
 ...
}

Define an ObjectName for the MBean

Each MBean requires a unique name called an ObjectName. The ObjectName contains a domain and a set of key/value pairs. The Geode MBeans use the GemFire domain and most define properties for service, name, type and member.

The MBeanJMXAdapter getRegionMBeanName method defines the RegionMBean ObjectName like:

return getObjectName((MessageFormat.format(OBJECTNAME__REGION_MXBEAN,
 new Object[] { makeCompliantRegionPath(regionPath), getMemberNameOrId(member) })));

An example from the above method is:

GemFire:service=Region,name=/data,type=Member,member=192.168.2.13(67581)<ec><v1>-1025)

Register the MBean with the MBeanServer

Registering the MBean with the MBeanServer causes the MBean to be available to JMX clients of the member.

Each MBean is registered in the member by the SystemManagementService registerInternalMBean method right after the MBean is instantiated like:

service.registerInternalMBean(regionMBean, regionMBeanName);

Federate the MBean

Federating the MBean causes the MBean to be registered in the JMX manager's MBeanServer which in turn causes the MBean to be available to JMX clients of the JMX manager. Federation is the act of gathering the current values for the attributes of all the MBeans and sending them to the JMX manager. When an MBean is federated, a FederationComponent created for it which contains the MBean's current attribute values. The FederationComponent is instantiated by the SystemManagementService federate method right after the MBean is registered like:

service.federate(changedMBeanName, RegionMXBean.class, true);

This causes a FederationComponent to be instantiated for the MBean and that FederationComponent to be added to the LocalManager’s map of FederationComponents.

Once the new MBean has been federated, its attributes will automatically be sent to and updated in the JMX manager.

The ManagementTask periodically sends the latest state of each MBean to the JMX manager using its FederationComponent. It iterates through the LocalManager’s map of FederationComponents, and for each one, it:

• causes the FederationComponent to refresh its state

• adds the FederationComponent to the map of all FederationComponents for the member (if its state has changed or it is a new component)

• sends the map to the JMX manager using putAll on the monitoring region for the member

Each FederationComponent refreshes its state via the refreshObjectState method. This method iterates and invokes the FederationComponent's getters (which are the MBean interface getters). If there are no getter methods, there is no state to update.

When the putAll event is received by the JMX manager, its ManagementCacheListener is invoked.

When the FederationComponent is initially created, the ManagementCacheListener afterCreate method is invoked. This causes a proxy representation of the MBean to be registered in the MBeanServer using MBeanJMXAdapter registerMBeanProxy.

Some examples are:

• GemFire:service=CacheService,name=LuceneService,type=Member,member=192.168.2.13(69570)<ec><v1>-1025

• GemFire:type=Member,member=192.168.2.13(69570)<ec><v1>-1025

• GemFire:service=LockService,name=gatewayEventIdIndexMetaData_lockService,type=Member,member=192.168.2.13(69570)<ec><v1>-1025

• GemFire:service=DiskStore,name=DEFAULT,type=Member,member=192.168.2.13(69570)<ec><v1>-1025

• GemFire:service=CacheServer,port=58176,type=Member,member=192.168.2.13(69570)<ec><v1>-1025

• GemFire:service=AsyncEventQueue,queue=full_index#_data,type=Member,member=192.168.2.13(69570)<ec><v1>-1025

• GemFire:service=Region,name=/data,type=Member,member=192.168.2.13(69570)<ec><v1>-1025

What is registered for each MBean by the registerMBeanProxy method is a MBeanProxyInvocationHandler. This object handles requests from JMX clients. When a request is received, it is handled in one of two ways. All requests for attributes are handled by the MBeanProxyInvocationHandler delegateToObjectState method. This method gets the FederationComponent from the local monitoring region and retrieves the value of the requested attribute. All requests for operations are handled by the MBeanProxyInvocationHandler delegateToFunctionService method. This method executes the ManagementFunction to forward the operation to the appropriate member. The ManagementFunction executes in the member and uses the MBeanServer to invoke the operation on the MBean.

Aggregation

 Depending on the MBean, an AggregateHandler is invoked to aggregate the data in the JMX manager.

 When the ManagementCacheListener afterUpdate method is invoked, it retrieves the proxy and updates the state of the MBean.

 More details to come.