1 Introduction
Federation of MBeanServers means that Member-1 has a master MBeanServer which provides a proxied view of all the MBeans that Member-1 hosts. Federation will also involve cascading of operations and notifications across master and managed members.
This specification defines Federation as service for gemfire mbeans and some platform mbeans as well as any MBean as desired by the User.
Requirements
The Mbean federation strategy focus on the below mentioned requirements.
- It should be scalable.
- Performance impact of federation should be minimal. Push-based model is advisable as opposed to standard way of proxy like RMI Pull-method based proxies.
- User should be able to monitor the Distributed System from any third party tool compliant with JMX. User should be able to reach all mbeans from managing node as well as managed nodes
- Management API should provide mechanism for filtering out mbeans in-order to limit impact on scalability in large-systems.
Assumptions
- Strict consistency across managing member and managed members is not desired.
- Strict real time data is not mandated, its alright as the system would give a close enough picture.
Scope
This spec only points out federation strategy for Mbeans. The Mbean modeling and details about the Mbeans are mentioned in related document section.
Related Document
- https://cwiki.apache.org/confluence/display/GEODE/Proposed+GemFire+MBeans
- https://cwiki.apache.org/confluence/display/GEODE/MBeanServer+Federation+Strategy
- https://cwiki.apache.org/confluence/download/attachments/61309759/JMXFederation-POC.pdf
Terminology
Managed Node : Each member of Distributed system is a Managed node. It can be monitored independently as well as from a single agent view.
Managing Node : The member which will provide the single agent view to monitor and manage all the members. There can multiple managing nodes in the system
Hidden Region : This term is used for internal meta-data region that will be used for replication of MBean states.
2 Overview Of GemFire Federation
GemFire Services For Management & Monitoring
GemFire will provide following services for Management and Monitoring.
- Single Agent View of Distributed System MBeans
- Management Resources : Connectors and predefined MBeans.
- Inbuilt Federation Mechanism
Single-Agent View
With a single-agent view, the application or manager interacts with one MBeanServer. Application and manager development is much easier because they don't have to find the right MBeanServer to make a request on an MBean. That MBeanServer is responsible for aggregating all of the MBeans in the MBeanServers with which it is federated. This is very convenient because the application or manager can interact with a local MBeanServer that has services and adapters to interact with other local or remote MBeanServers. The location of the federated MBeanServers does not change how the application or manager is programmed to interact with its MBeanServer. For the sake of this discussion, we will refer to the single-agent MBeanServer as Managing Node. The MBeanServers that the master interacts with (and the application does not) will be referred to as Managed Node.
GemFire achieves Single-Agent View by creating MBean proxies at Managing Node side and keeping their state in sync with their Managed node counterpart.
Management Resources
Managed Node
Any manageable node which is not managing other members is managed node.
Managed node has following resources allocated so that it can answer jmx queries both locally and remotely.
- An RMI Connector which will allow JMX Client to connect directly to the node to and get access to all local MBeans.
- Local MBeans : representing the local monitored components on this node.Built in platform MBeans
- MemberMBean
- RegionMBean
- ClientServiceMBean
- LockServiceMBean
- DiskStoreMBean
- GatewaySenderMBean
- GatewayReceiverMBean
- LocatorMBean
- Management MBean with Scope=LOCAL
- Built in platform MBeans
- Custom MBeans
Managing Node
Managing node has following extra resources allocated so that it can answer jmx queries
- An RMI Connector which will allow JMX Client to connect to and get access to all MBeans of the distributed system
- Local MBeans : representing the local monitored components on this node. Same as any other managed node
- Aggregate MBeans :Management MBean with Scope=ALL which allows various DS-wide operations.
- DistributedSystemMBean
- DistributedRegionMBean
- MemberGroupMBean
- DistributedLockServiceMBean
- Management MBean with Scope=ALL which allows various DS-wide operations.
- Proxy to MBeans on managed nodes
- Built in platform MBeans
- Custom MBeans
Inbuilt Federation Mechanism
GemFire federation mechanism is the process by virtue of which we can achieve single agent view.The protocol used is internal to GemFire and does not interfere with any application and manager implementation.
GemFire federation takes care of the following functionality
- MBean proxy creation
- MBean state propagation
- notifications propagation
- operation invocation.
GemFire federation is not limited to predefined MBeans only. In fact users can write their own MBeans and GemFire will provide a federated view of their MBean.
MBean/Proxy Naming Convention
Each GemFire MBean will follow a particular naming convention for easier grouping
e.g. GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName>
At the managing node this MBean will be registered with GemFire/<memberId> as domain
e.g. GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName>
Application Program Interfaces
GemFire Management API represent the Gemfire System view to a JMX user. However it does not intend to provide functionality which is otherwise present in JMX. It only provides a gateway into various services exclusively offered by GemFire Management.
The entry point to GemFire Management is through interface ManagementService
ManagementService service = ManagementService.getManagementService(cache);
The implementation of getManagementService is a singleton for now but it allows us to eventually support multiple cache instances (probably multiple client caches and on peer cache). The resulting ManagementService instance will be specific to the provided cache and its distributed system. Also the instance will behave differently when the cache is a client cache vs a peer cache. When its a client cache then execution of a command string should be forwarded to the cache server instead of processed in the local JVM. More detailing needs to be done for client cache.
APIs useful at Managed Node
public class ManagementService { /** * It will a singleton instance across a Cache. This instance will * act as th einterface between user and GemFire Management. * * @param Cache * @return ManagementService */ public static ManagementService getManagementService(Cache cache) { } /** * This will register the MBean in default GemFire domain. * Any MBean which wishes to register itself in Gemfire * domain should use this method. * Note: The ObjectName will be modified here. Hence returning the * modified ObjectName. */ public ObjectName registerMBean(Object object, ObjectName objectName) { } /** * For consistency and clean up of GemFire resources * use this method to unregister the MBean from GemFire Domain */ public void unregisterMBean(ObjectName objectName) { } /** * This method is used to mark an MBean for federation. * * Users may want to only register their MBean in GemFire domain. In that * case they don't have to call federate(); * * @param <T> * @param name MBean name * @param interfaceClass MBean interface class * @param notificationEmitter is the MBean a notification emitter * @return */ public <T> boolean federate(ObjectName name, Class<T> interfaceClass, boolean notificationEmitter) { } /** * This method is used to process a command * This takes an String is a parameter. * * @param commandString * @return */ public Object processComand(String commandString) { } /** * Interfaces to get all GemFire specific MBeans * */ public RegionMXBean getLocalRegionMXBean(Region region) { } public LockServiceMXBean getLocalLockServiceMXBean(String name){ } ........ }
APIs useful for MBean Federation
Usage:
MyCustomMXBean myMbean = new MyCustomMBean(); ManagementService service = ManagementService.getManagementService(cache); //To register in Gemfire domain // GemFire will modify the given ObjectName to append memberName ObjectName newObjectName service.register(myMbean, name); /** * To allow MyCustomMBean to be federated */ service.federate(newObjectName, MyCustomMXBean.class, false);
APIs useful at Managing Node
public class ManagementService { /** * This method will be invoked whenever a member wants to be a managing node. The * exception Management exception has to be handled by the caller. */ public void startManager() { } /** * This method will be invoked whenever a member stops being a managing node. * The exception Management exception has to be handled by the caller. * This will stop all the activities and clean up resources pertaining to * Managing Node */ public void stopManager() { } ........ /** * This will return a set of MBean proxy names for a given member name * * @param memberName */ public Set<ObjectName> queryProxyMBeans(DistributedMember memberName) { } /** * The method will return the proxy instance represented by the ObjectName. * * It will be an actual reference to the proxy instance rather than an * ObjectInstance. * * @param <T> * @param objectName * @return * @throws ManagementException * @throws InstanceNotFoundException */ public <T> T getMBeanProxy(DistributedMember memberName, ObjectName objectName, Class<T> interfaceClass) { } /** * This will return the last updated time of the proxyMBean * * @param memberName */ public long getLastUpdateTime(ObjectName objectName) { } public Object processComand(String commandString) { } /** * Methods to get remote MBean proxy reference */ public RegionMXBean getRemoteRegionMXBean(DistributedMember memberName, Region region) { } public LockServiceMXBean getRemoteLockServiceMXBean(DistributedMember memberName, String name) {} ....................... }
2. Architectural Description
Internal Details The following section is internal details of GemFire and not visible to end user directly |
For JMX data distribution, Gemfire's inbuilt Replicated Region support will be used. This will help in data distribution in one direction i.e. from Managed Node to Managing Node.
Replicated region fits into push based monitoring support easily.
For information flow from managed node to managing node, Gemfire's "Function Service" will be used. Functions are primarily for all operations and propagating attribute setters.
Transition from Managed Node to Managing Node
The mechanism for transition from Managed Node to Managing node will be part of Beaverton specs.
a) StartManagingNode : Will create Management resources namely Regions, proxies for all the connected members
b) StopManagingNode : Will remove all the Management resources.It will behave as a normal Managed Node and start pushing its local data to the hidden regions
Management Listener
A new member wide listener component to be introduced to listen on different GFE resource creation/removal/destroy. On particular events it will create/ destroy Mbeans. This will save us from unnecessary existence of Mbeans. However this will require some internal GFE changes.
Other internal components can also be initialized by this listener. e.g. HiddenRegion can be created by this listener on cache create event.
public final class InternalDistributedSystem{ private void initialize() { ...................... ManagementListener listener = new ManagementListener(); this.managementListener = listener; } } public enum ManagementEvent { CACHE_CREATE, REGION_CREATE, DISKSTORE_CREATE, CACHESERVER_CREATE, GATEWAYRECEIVER_CREATE, GATEWAYSENDER_CREATE, LOCKSERVICE_CREATE, CACHE_REMOVE, REGION_REMOVE, DISKSTORE_REMOVE, CACHESERVER_REMOVE, GATEWAYRECEIVER_REMOVE, GATEWAYSENDER_REMOVE, LOCKSERVICE_REMOVE } public class ManagementListener { public void handleEvent(ManagementEvent event){ } } class GemFireCacheImpl{ init(){ .............. system.getManagementListener().handleEvent(ManagementEvent.CACHE_CREATE, this); .......... } }
Federation Strategy
Three main functionalities of any MBean : Attributes, Operation and Notifications are federated in following way
- Attribute value are federated using GemFire Hidden Region
Central component for Federation is FederatedComponent which will store object state as well as some meta data like MBean interface , notification enabled etc. - Operations are federated using GemFire Function Service
- Notification are federated using Hidden Region and Function Service together.
Each member of the system will be represented by one hidden region. The regions can be named "MONITORING_REGION_<memberName>". Whichever node becomes a Managing Node it will replicate all the hidden regions created by its managed member. Hidden region will store MBean state and other meta-data about mbean and its notifications.
Federation at managing member
This section describes the events that happen when a node makes transition from Normal Node to Managing node
Query cluster to know all manageable members
Managing member has to discover set of members in the current distributed system. Very first step would be to know all manageable members. This includes members with fixed-persistent memberName as well as member with non-persistent ID. This section assumes mechanism for querying all member of system. The results of this operation should return memberName and its descriptor. The descriptor would tell following attributes of the member
- memberName
- Type : Persistent, Non-Persistent
- Services running : Peer, CacheServer, Locator, Managing or Managed
Creation of Hidden Regions
Managing node has to communicate to all managed node and exchange all mbean informration i.e. what all mbeans are hosted by each managed node. Once this set is established managing node can finalize the list of proxies that it needs to create.
As we will be using GemFire hidden regions while the Managing member comes up GII functionality will take care of this.
If Managing member is up and a new member comes up then MembershipListener will take care of this.Create Proxy for remote MBeans
For this purpose Java's Proxy Design pattern is used. Our proxies are similar to JMX proxies. Only underlying InvocationHandler is different.
JMX delegates all call to remote RMI connnection. Calls on Gemfire Management proxies will be delegated to the state containing FederatedComponent and FunctionService.
Reference : javax.management.MBeanServerInvocationHandler
Method Invocation in Proxy MBean
- All getter method on the proxy will be delegated to the contained FederationComponent
- All setter metohd on the proxy will be delegated to an instance of GenericMBeanFunction
- All other opertaions on the proxy will be delegated to an instance of GenericMBeanFunction. Flow same as setter functions
Method Invocationin Proxy MBean(If proxy is provided explicitly)
GemFire MBeans will implement an interface which will serve two purposes
a) System can identify GemFire specific MBeans
b) Any proxy implementation can be given by this MBean which will by pass above generic proxy implementation
All the getters, setters, and operations will be delegated to this proxy. As proxy will be written by the developer who develops the MBean he will have the information how to access the data locally.
Following listeners will trigger creation and deletion of proxy mbeans
- MembershipListener ( Whenever a member joins or departs we need to create /destroy proxies)
- ManagementCacheListener ( Whenever a region operation happens on the MONITORING_REGION)
- ManagingNodeTransitionListener ( Whenever a member becomes a Managing node)
Data Model
Gemfire MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics.
Non-Gemfire MBeans (Platform MBeans, User Defined MBeans) will be represented as it is.
Data Model - Managed Node
1 Data model
Each MBean state will be represented by a FederatedComponent as per the proposed model. FederatedComponent will also be behave as an entry for the hidden region for data distribution.
The relationship between MBean, FederatedComponent is depicted below.
2 Bridge Data model
For Gemfire MBeans an intermediary MBeanBridge will be implemented to act as a conduit between GFE resource and MBeans.
Note MBeanBridge is only for Gemfire MBeans for ease of programming. Other MBeans are free to have their implementation as they require. |
3 Federated MBeans
If a MBean is intended to be federated then Gemfire should have some means to get the state data to be propagated by hidden region.
For that an FederatedComponent is used.
It uses MBeanInfo to dynamically get attribute & attribute values and keep it in an ObjectState map as a key value pair.
In addition it also contains some meta data like MBean interface, notification capabilities etc.
4 Delta Approach and other optimizations
This is deferred till implementation phase after checking the number of attributes which gets change frequently. If the percentage. is quite high then Delta might not be a good idea.
Data Model - Managing Node
1 Proxy Data model
This will include proxies for Global scoped User Defined MBeans and of course Gemfire MBeans
The below diagram depicts the relationship between various components for Proxies.
- Delta Approach and other optimizations
This is deferred till implementation phase after checking the number of attributes which gets change frequently.
Push Model for MBean State
Generic MBeans
All user MBeans and Platform MBeans can be federated provided they follow the above mentioned contract with Gemfire.
Gemfire will internally scan through all such MBeans and push their state to the hidden region in a given frequency.
Gemfire Specific MBeans
MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics. Following diagram shows how M&M layer gets data from gemfire statistics. ManagementThread is used for pushing updates from managed node to manging node via hidden region replication. ManagementThread will group and batch all updated in-order to reduce network traffic.
Note MBeanBridge is only for Gemfire MBeans for ease of programming.They contain the MBean state. They can be updated by the Listeners directly as we are free to code the listener object |
- Non-Statistics Part
This part of MBean state is collected from Bridge component of underlying gemfire resources. - Statistics Part
Statistics Listener provided by Statistics framework is used for updating mbean state. There is not direct invocation or signaling mechanism between StatisticsSampler Thread and Management Thread. ManagementThread is responsible for getting updating statistics part of Mbean State.
StatisticsListener listener = new StatisticsListener() { public void handleNotification(StatisticsNotification notification) { // handle the notification } }; ValueMonitor monitor = new ValueMonitor() .addStatistic(GemFireStatistics.getCreatesDescriptor()) .addStatistic(GemFireStatistics.getDestroysDescriptor()) .addStatistic(GemFireStatistics.getEntriesDescriptor()) .addStatistic(GemFireStatistics.getServerStatistics().getCurrentClientsDescriptor() .addStatistic(GemFireStatistics.getServerStatistics().getSubscriptionQueueSizeDescriptor()); monitor.registerListener(listener);
MBean Operation on Remote MBeans
NotificationsTo understand the proposed notification federation model let's look briefly at the distributed management scope. It can be depicted roughly in the below mentioned layers.
Each layer runs on different VMs. GemFire implementation of proxy will be a little different than other JMX federation approaches. GemFire proxies inherently are local MBeans.It will emit notification when an event occurs. This event is mapped to notification emitted by the actual MBean on Managed Node side. So some component(in our model its NotificationHub) at Managed Node
Some conceptual details for remote notifications can be found at
Following diagram shows two flows
Diagram also shows EmailNotifier as another consumer of notification. Such notifiers can be installed on Managing Server where HubClient can use this to send emails to gemfire users or any other notification consumption mechanism other than JMX
|
3.Future Direction
Management FiltersManagement filter are defined to restrict or narrow-down the management and monitoring to limited set of resources or limited number of members. Filter will help to scale M&M in a large distributed system consisting of hundreds or member. Typical filter consists of inclusion and exclusion lists. It mimics firewall rules, hence inclusion and exclusion. Inclusion is processed first and defines what to include. Exclusion is processed second and defines what to exclude out of what was already defined is being included.
|