Introduction
Federation of MBeanServers means that Member-1 has a master MBeanServer which provides a proxied view of all the MBeans that Member-1 hosts. Federation will also involve cascading of operations and notifications across master and managed members.
This specification defines Federation as service for gemfire mbeans and some platform mbeans as well as any MBean as desired by the User.
Requirements
The Mbean federation strategy focus on the below mentioned requirements.
- It should be scalable.
- Performance impact of federation should be minimal. Push-based model is advisable as opposed to standard way of proxy like RMI Pull-method based proxies.
- User should be able to monitor the Distributed System from any third party tool compliant with JMX. User should be able to reach all mbeans from managing node as well as managed nodes
- Management API should provide mechanism for filtering out mbeans in-order to limit impact on scalability in large-systems.
Assumptions
- Strict consistency across managing member and managed members is not desired.
- Strict real time data is not mandated, its alright as the system would give a close enough picture.
Scope
This spec only points out federation strategy for Mbeans. The Mbean modeling and details about the Mbeans are mentioned in related document section.
Related Document
- https://cwiki.apache.org/confluence/display/GEODE/Proposed+GemFire+MBeans
- https://cwiki.apache.org/confluence/display/GEODE/MBeanServer+Federation+Strategy
- https://cwiki.apache.org/confluence/download/attachments/61309759/JMXFederation-POC.pdf
Terminology
Managed Node : Each member of Distributed system is a Managed node. It can be monitored independently as well as from a single agent view.
Managing Node : The member which will provide the single agent view to monitor and manage all the members. There can multiple managing nodes in the system
Hidden Region : This term is used for internal meta-data region that will be used for replication of MBean states.
1.Overview Of GemFire Federation
GemFire Services For Management & Monitoring
GemFire will provide following services for Management and Monitoring.
- Single Agent View of Distributed System MBeans
- Management Resources : Connectors and predefined MBeans.
- Inbuilt Federation Mechanism
Single-Agent View
With a single-agent view, the application or manager interacts with one MBeanServer. Application and manager development is much easier because they don't have to find the right MBeanServer to make a request on an MBean. That MBeanServer is responsible for aggregating all of the MBeans in the MBeanServers with which it is federated. This is very convenient because the application or manager can interact with a local MBeanServer that has services and adapters to interact with other local or remote MBeanServers. The location of the federated MBeanServers does not change how the application or manager is programmed to interact with its MBeanServer. For the sake of this discussion, we will refer to the single-agent MBeanServer as Managing Node. The MBeanServers that the master interacts with (and the application does not) will be referred to as Managed Node.
GemFire achieves Single-Agent View by creating MBean proxies at Managing Node side and keeping their state in sync with their Managed node counterpart.
Management Resources
Managed Node
Any manageable node which is not managing other members is managed node.
Managed node has following resources allocated so that it can answer jmx queries both locally and remotely.
- An RMI Connector which will allow JMX Client to connect directly to the node to and get access to all local MBeans.
- Local MBeans : representing the local monitored components on this node.Built in platform MBeans
- MemberMBean
- RegionMBean
- ClientServiceMBean
- LockServiceMBean
- DiskStoreMBean
- GatewaySenderMBean
- GatewayReceiverMBean
- LocatorMBean
- Management MBean with Scope=LOCAL
- Custom MBeans
Managing Node
Managing node has following extra resources allocated so that it can answer jmx queries
- An RMI Connector which will allow JMX Client to connect to and get access to all MBeans of the distributed system
- Local MBeans : representing the local monitored components on this node. Same as any other managed node
- Aggregate MBeans :Management MBean with Scope=ALL which allows various DS-wide operations.
- DistributedSystemMBean
- DistributedRegionMBean
- MemberGroupMBean
- DistributedLockServiceMBean
- Proxy to MBeans on managed nodes
- Built in platform MBeans
- Custom MBeans
Inbuilt Federation Mechanism
GemFire federation mechanism is the process by virtue of which we can achieve single agent view.The protocol used is internal to GemFire and does not interfere with any application and manager implementation.
GemFire federation takes care of the following functionality
- MBean proxy creation
- MBean state propagation
- notifications propagation
- operation invocation.
GemFire federation is not limited to predefined MBeans only. In fact users can write their own MBeans and GemFire will provide a federated view of their MBean.
MBean/Proxy Naming Convention
Each GemFire MBean will follow a particular naming convention for easier grouping
e.g. GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName>
At the managing node this MBean will be registered with GemFire/<memberId> as domain
e.g. GemFire:type=Member,service=LockService,name=<dlsName>,memberName=<memberName>
Application Program Interfaces
GemFire Management API represent the Gemfire System view to a JMX user. However it does not intend to provide functionality which is otherwise present in JMX. It only provides a gateway into various services exclusively offered by GemFire Management.
The entry point to GemFire Management is through interface ManagementService
ManagementService service = ManagementService.getManagementService(cache);
The implementation of getManagementService is a singleton for now but it allows us to eventually support multiple cache instances (probably multiple client caches and on peer cache). The resulting ManagementService instance will be specific to the provided cache and its distributed system. Also the instance will behave differently when the cache is a client cache vs a peer cache. When its a client cache then execution of a command string should be forwarded to the cache server instead of processed in the local JVM. More detailing needs to be done for client cache.
APIs useful at Managed Node
public
class
ManagementService {
/** * It will a singleton instance across a Cache. This instance will * act as th einterface between user and GemFire Management. * @param Cache * @return ManagementService */
public
static
ManagementService getManagementService(Cache cache){}
/** * This will register the MBean in default GemFire domain. * Any MBean which wishes to register itself in Gemfire * domain should use this method. * Note: The ObjectName will be modified here. Hence returning the * modified ObjectName. */
public
ObjectName registerMBean(Object object, ObjectName objectName){}
/** * For consistency and clean up of GemFire resources * use this method to unregister the MBean from GemFire Domain */
public
void
unregisterMBean(ObjectName objectName){}
/** * This method is used to mark an MBean for federation. * * Users may want to only register their MBean in GemFire domain. In that * case they don't have to call federate(); * * @param <T> * @param name MBean name * @param interfaceClass MBean interface class * @param notificationEmitter is the MBean a notification emitter * @return */
public
<T>
boolean
federate(ObjectName name, Class<T> interfaceClass,
boolean
notificationEmitter) {}
/** * This method is used to process a command * This takes an String is a parameter. * @param commandString * @return */
public
Object processComand(String commandString){}
/** * Interfaces to get all GemFire specific MBeans * */
public
RegionMXBean getLocalRegionMXBean(Region region) {}
public
LockServiceMXBean getLocalLockServiceMXBean(String name){} .... .... }
APIs useful for MBean Federation
Usage :
MyCustomMXBean myMbean =
new
MyCustomMBean(); ManagementService service = ManagementService.getManagementService(cache);
//To register in Gemfire domain // GemFire will modify the given ObjectName to append memberName ObjectName newObjectName service.register(myMbean, name); /** * To allow MyCustomMBean to be federated */ service.federate(newObjectName, MyCustomMXBean.class, false);
APIs useful at Managing Node
public class ManagementService { /** * This method will be invoked whenever a member wants to be a managing node. The * exception Management exception has to be handled by the caller. */ public void startManager() {} /** * This method will be invoked whenever a member stops being a managing node. * The exception Management exception has to be handled by the caller. * This will stop all the activities and clean up resources pertaining to * Managing Node * */ public void stopManager() {} .... .... /** * This will return a set of MBean proxy names for a given member name * @param memberName * @return */ public Set<ObjectName> queryProxyMBeans(DistributedMember memberName) {} /** * The method will return the proxy instance represented by the ObjectName. * * It will be an actual reference to the proxy instance rather than an * ObjectInstance. * * @param <T> * @param objectName * @return * @throws ManagementException * @throws InstanceNotFoundException */ public <T> T getMBeanProxy(DistributedMember memberName, ObjectName objectName, Class<T> interfaceClass){} /** * This will return the last updated time of the proxyMBean * * @param memberName * @return */ public long getLastUpdateTime(ObjectName objectName) {} public Object processComand(String commandString) {} /** * Methods to get remote MBean proxy reference */ public RegionMXBean getRemoteRegionMXBean(DistributedMember memberName ,Region region) {} public LockServiceMXBean getRemoteLockServiceMXBean(DistributedMember memberName, String name) {} .......... ............. }
GemFire Services For Management & MonitoringGemFire will provide following services for Management and Monitoring.
Single-Agent ViewWith a single-agent view, the application or manager interacts with one MBeanServer. Application and manager development is much easier because they don't have to find the right MBeanServer to make a request on an MBean. That MBeanServer is responsible for aggregating all of the MBeans in the MBeanServers with which it is federated. This is very convenient because the application or manager can interact with a local MBeanServer that has services and adapters to interact with other local or remote MBeanServers. The location of the federated MBeanServers does not change how the application or manager is programmed to interact with its MBeanServer. For the sake of this discussion, we will refer to the single-agent MBeanServer as Managing Node. The MBeanServers that the master interacts with (and the application does not) will be referred to as Managed Node. GemFire achieves Single-Agent View by creating MBean proxies at Managing Node side and keeping their state in sync with their Managed node counterpart. Management Resources
Inbuilt Federation MechanismGemFire federation mechanism is the process by virtue of which we can achieve single agent view.The protocol used is internal to GemFire and does not interfere with any application and manager implementation. GemFire federation takes care of the following functionality
Application Program InterfacesGemFire Management API represent the Gemfire System view to a JMX user. However it does not intend to provide functionality which is otherwise present in JMX. It only provides a gateway into various services exclusively offered by GemFire Management. The entry point to GemFire Management is through interface ManagementService The implementation of getManagementService is a singleton for now but it allows us to eventually support multiple cache instances (probably multiple client caches and on peer cache). The resulting ManagementService instance will be specific to the provided cache and its distributed system. Also the instance will behave differently when the cache is a client cache vs a peer cache. When its a client cache then execution of a command string should be forwarded to the cache server instead of processed in the local JVM. More detailing needs to be done for client cache.
|
Internal Details The following section is internal details of GemFire and not visible to end user directly |
2.Architectural Description
For JMX data distribution, Gemfire's inbuilt Replicated Region support will be used. This will help in data distribution in one direction i.e. from Managed Node to Managing Node. For information flow from managed node to managing node, Gemfire's "Function Service" will be used. Functions are primarily for all operations and propagating attribute setters. Transition from Managed Node to Managing NodeThe mechanism for transition from Managed Node to Managing node will be part of Beaverton specs. Management ListenerA new member wide listener component to be introduced to listen on different GFE resource creation/removal/destroy. On particular events it will create/ destroy Mbeans. This will save us from unnecessary existence of Mbeans. However this will require some internal GFE changes. Other internal components can also be initialized by this listener. e.g. HiddenRegion can be created by this listener on cache create event. Management Listener : public final class InternalDistributedSystem{ private void initialize() { ...................... ManagementListener listener = new ManagementListener(); this.managementListener = listener; } } public enum ManagementEvent { CACHE_CREATE, REGION_CREATE, DISKSTORE_CREATE, CACHESERVER_CREATE, GATEWAYRECEIVER_CREATE, GATEWAYSENDER_CREATE, LOCKSERVICE_CREATE, CACHE_REMOVE, REGION_REMOVE, DISKSTORE_REMOVE, CACHESERVER_REMOVE, GATEWAYRECEIVER_REMOVE, GATEWAYSENDER_REMOVE, LOCKSERVICE_REMOVE } public class ManagementListener { public void handleEvent(ManagementEvent event){ } } class GemFireCacheImpl{ init(){ .............. system.getManagementListener().handleEvent(ManagementEvent.CACHE_CREATE, this); .......... } } Federation StrategyThree main functionalities of any MBean : Attributes, Operation and Notifications are federated in following way
Each member of the system will be represented by one hidden region. The regions can be named "MONITORING_REGION_<memberName>". Whichever node becomes a Managing Node it will replicate all the hidden regions created by its managed member. Hidden region will store MBean state and other meta-data about mbean and its notifications. Federation at managing memberThis section describes the events that happen when a node makes transition from Normal Node to Managing node
Data ModelGemfire MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics. Non-Gemfire MBeans (Platform MBeans, User Defined MBeans) will be represented as it is.
Push Model for MBean State
MBean Operation on Remote MBeans
NotificationsTo understand the proposed notification federation model let's look briefly at the distributed management scope. It can be depicted roughly in the below mentioned layers.
Each layer runs on different VMs. GemFire implementation of proxy will be a little different than other JMX federation approaches. GemFire proxies inherently are local MBeans.It will emit notification when an event occurs. This event is mapped to notification emitted by the actual MBean on Managed Node side. So some component(in our model its NotificationHub) at Managed Node
Some conceptual details for remote notifications can be found at
Following diagram shows two flows
Diagram also shows EmailNotifier as another consumer of notification. Such notifiers can be installed on Managing Server where HubClient can use this to send emails to gemfire users or any other notification consumption mechanism other than JMX
|
3.Future Direction
Management FiltersManagement filter are defined to restrict or narrow-down the management and monitoring to limited set of resources or limited number of members. Filter will help to scale M&M in a large distributed system consisting of hundreds or member. Typical filter consists of inclusion and exclusion lists. It mimics firewall rules, hence inclusion and exclusion. Inclusion is processed first and defines what to include. Exclusion is processed second and defines what to exclude out of what was already defined is being included.
|