Geode uses Java MBeans, specifically MXBeans, to expose management controls and monitoring features.
To provide cluster level management and monitoring, Geode uses a federated Open MBean strategy. The Java classes interact with a single MBeanServer that aggregates MBeans from other local and remote members. This strategy provides a consolidated, single-agent view of the distributed system.
The implementation is based on JMX (industry-standard and friendly to generic JMX clients), this allows use of any JMX compliant tools (third party or custom build) to manage and monitor Geode cluster. For example, JConsole.
Create and Register a New Geode JMX MBean
...
- Define the MXBean interface
- Implement the MXBean interface with MBean class
- Implement the MBeanBridge
- Handle MBean Lifecycle
- Modify ManagementListener and ManagementAdapter
- Define an ObjectName for the
- MBean
- Register the MBean with
- the MBeanServer
- Federate the MBean
- Aggregate the MBean
Define the MXBean interface
...
- Accessing data directly from the object being monitored
- Accessing raw statistic values
- Computing rates
- Computing average latencies
- Providing data bean objectsbeans
Accessing Data Directly from the Object Being Monitored
...
The getAverageLatency method returns the time statistic divided by the numeric count statistic.
Providing Data
...
Beans
Another data type provided by the MBeanBridge is a data beans containing several bits of primitive data. An example is JVMMetrics provided by MemberMBeanBridge fetchJVMMetrics
More to come.
Define an ObjectName for the MBean
Each MBean requires a unique name called an ObjectName. The ObjectName contains a domain and a set of key/value pairs. The Geode MBeans use the GemFire domain and most define properties for service, name, type and member.
The MBeanJMXAdapter getRegionMBeanName method defines the RegionMBean ObjectName like:
Code Block | ||||
---|---|---|---|---|
| ||||
return getObjectName((MessageFormat.format(OBJECTNAME__REGION_MXBEAN,
new Object[] { makeCompliantRegionPath(regionPath), getMemberNameOrId(member) }))); |
An example from the above method is:
GemFire:service=Region,name=/data,type=Member,member=192.168.2.13(67581)<ec><v1>-1025)
Register the MBean with the MBeanServer at that ObjectName
Registering the MBean with the MBeanServer causes the MBean to be available to JMX clients of the member.
| |
public JVMMetrics fetchJVMMetrics() {
long gcCount = getGCStatistic(StatsKey.VM_GC_STATS_COLLECTIONS).longValue();
long gcTimeMillis = getGCStatistic(StatsKey.VM_GC_STATS_COLLECTION_TIME).longValue();
long initMemory = memoryMXBean.getHeapMemoryUsage().getInit();
long committedMemory = memoryMXBean.getHeapMemoryUsage().getCommitted();
long usedMemory = getVMStatistic(StatsKey.VM_USED_MEMORY).longValue();
long maxMemory = memoryMXBean.getHeapMemoryUsage().getMax();
int totalThreads = getVMStatistic(StatsKey.VM_STATS_NUM_THREADS).intValue();
return new JVMMetrics(gcCount, gcTimeMillis, initMemory, committedMemory,
usedMemory, maxMemory, totalThreads);
} |
The JVMMetrics class is defined Each MBean is registered in the member by the SystemManagementService registerInternalMBean method right after the MBean is instantiated like:
Code Block | ||||
---|---|---|---|---|
| ||||
service.registerInternalMBean(regionMBean, regionMBeanName); |
Federate the MBean
Federating the MBean causes the MBean to be registered in the JMX manager's MBeanServer which in turn causes the MBean to be available to JMX clients of the JMX manager.
ManagementListener
| |
public class JVMMetrics {
private long gcCount;
private long gcTimeMillis;
private long initMemory;
private long committedMemory;
private long usedMemory;
private long maxMemory;
private int totalThreads;
@ConstructorProperties( { "gcCount", "gcTimeMillis", "initMemory",
"committedMemory", "usedMemory", "maxMemory", "totalThreads"
})
public JVMMetrics(long gcCount, long gcTimeMillis, long initMemory,
long committedMemory, long usedMemory, long maxMemory, int totalThreads) {
this.gcCount = gcCount;
this.gcTimeMillis = gcTimeMillis;
this.initMemory = initMemory;
this.committedMemory = committedMemory;
this.usedMemory = usedMemory;
this.maxMemory = maxMemory;
this.totalThreads = totalThreads;
}
} |
Note: The JVMMetrics (and other data beans) are returned to the JMX client as CompositeData objects.
Handle MBean Lifecycle
Modify ManagementListener and ManagementAdapter
When a Cache is created, it creates a ManagementListener that listens for ResourceEvents (in the GemFireCacheImpl constructor). Each ResourceEvent triggers When a Cache is created, it creates a ManagementListener that listens for ResourceEvents (in the GemFireCacheImpl constructor). Each ResourceEvent triggers an MBean to be instantiated, registered and federated. For example, when a a Region is is created, the the REGION_CREATE ResourceEvent is is sent like:
Code Block | ||||
---|---|---|---|---|
| ||||
public Region createVMRegion(String name, RegionAttributes p_attrs, InternalRegionArguments internalRegionArgs) { ... system.handleResourceEvent(ResourceEvent.REGION_CREATE, rgn); } |
In the the ManagementListener, the the REGION_CREATE ResourceEvent causes the causes the RegionMBean to to be instantiated and registered like:
Code Block | ||||
---|---|---|---|---|
| ||||
In ManagementListener: public void handleEvent(ResourceEvent event, Object resource) { ... case REGION_CREATE: adapter.handleRegionCreation(createdRegion); break; ... } In ManagementAdapter: public <K, V> void handleRegionCreation(Region<K, V> region) { ... RegionMBeanBridge<K, V> bridge = RegionMBeanBridge.getInstance(region); RegionMXBean regionMBean = new RegionMBean<K, V>(bridge); ObjectName regionMBeanName = MBeanJMXAdapter.getRegionMBeanName(cacheImpl .getDistributedSystem().getDistributedMember(), region.getFullPath()); ObjectName changedMBeanName = service.registerInternalMBean(regionMBean, regionMBeanName); service.federate(changedMBeanName, RegionMXBean.class, true); ... } |
Federation
Define an ObjectName for the MBean
Each MBean requires a unique name called an ObjectName. The ObjectName contains a domain and a set of key/value pairs. The Geode MBeans use the GemFire domain and most define properties for service, name, type and member.
The MBeanJMXAdapter getRegionMBeanName method defines the RegionMBean ObjectName like:
Code Block | ||||
---|---|---|---|---|
| ||||
return getObjectName((MessageFormat.format(OBJECTNAME__REGION_MXBEAN,
new Object[] { makeCompliantRegionPath(regionPath), getMemberNameOrId(member) }))); |
An example from the above method is:
GemFire:service=Region,name=/data,type=Member,member=192.168.2.13(67581)<ec><v1>-1025)
Register the MBean with the MBeanServer
Registering the MBean with the MBeanServer causes the MBean to be available to JMX clients of the member.
Each MBean is registered in the member by the SystemManagementService registerInternalMBean method right after the MBean is instantiated like:
Code Block | ||||
---|---|---|---|---|
| ||||
service.registerInternalMBean(regionMBean, regionMBeanName); |
Federate the MBean
Federating the MBean causes the MBean to be registered in the JMX manager's MBeanServer which in turn causes the MBean to be available to JMX clients of the JMX manager. Federation is the act of gathering the current values for the attributes of all the MBeans and sending them to the JMX manager. When an MBean is federated, a FederationComponent created for it which contains the MBean's current attribute values. The FederationComponent is instantiated by the SystemManagementService federate method right after the MBean is registered like:
...
When the FederationComponent is initially created, the ManagementCacheListener afterCreate method is invoked. This causes a proxy representation of the MBean to be registered in the MBeanServer using MBeanJMXAdapter registerMBeanProxy.
Some examples are:
GemFire:service=CacheService,name=LuceneService,type=Member,member=192.168.2.13(69570)<ec><v1>-1025
GemFire:type=Member,member=192.168.2.13(69570)<ec><v1>-1025
GemFire:service=LockService,name=gatewayEventIdIndexMetaData_lockService,type=Member,member=192.168.2.13(69570)<ec><v1>-1025
GemFire:service=DiskStore,name=DEFAULT,type=Member,member=192.168.2.13(69570)<ec><v1>-1025
GemFire:service=CacheServer,port=58176,type=Member,member=192.168.2.13(69570)<ec><v1>-1025
GemFire:service=AsyncEventQueue,queue=full_index#_data,type=Member,member=192.168.2.13(69570)<ec><v1>-1025
GemFire:service=Region,name=/data,type=Member,member=192.168.2.13(69570)<ec><v1>-1025
...
What is registered for each ObjectName by the registerMBeanProxy method is a each MBean by the registerMBeanProxy method is a MBeanProxyInvocationHandler. This object handles requests from JMX clients. MBean state is retrieved When a request is received, it is handled in one of two ways, either by operation or by attribute.
All operations are forwarded to the appropriate member using the MBeanProxyInvocationHandler delegateToFunctionService method. This method executes the ManagementFunction in the member. The ManagementFunction uses the local MBeanServer to invoke the operation on the MBean represented by the ObjectName.
All attributes are retrieved in the JMX manager using the MBeanProxyInvocationHandler delegateToObjectState method. This method gets the FederationComponent from the local monitoring region and retrieves value of the requested attribute.
Aggregation
Depending on the MBean, an AggregateHandler is invoked to aggregate the data.
. All requests for attributes are handled by the MBeanProxyInvocationHandler delegateToObjectState method. This method gets the FederationComponent from the local monitoring region and retrieves the value of the requested attribute. All requests for operations are handled by the MBeanProxyInvocationHandler delegateToFunctionService method. This method executes the ManagementFunction to forward the operation to the appropriate member. The ManagementFunction executes in the member and uses the MBeanServer to invoke the operation on the MBean.
Aggregation
Depending on the MBean, an AggregateHandler is invoked to aggregate the data in the JMX manager.
When the ManagementCacheListener afterUpdate method When the ManagementCacheListener afterUpdate method is invoked, it retrieves the proxy and updates the state of the MBean.
More details to come on aggregation.