Introduction
Federation of MBeanServers means that Member-1 has a master MBeanServer which provides a proxied view of all the MBeans that Member-1 hosts. Federation will also involve cascading of operations and notifications across master and managed members.
This specification defines Federation as service for gemfire mbeans and some platform mbeans as well as any MBean as desired by the User.
Requirements
The Mbean federation strategy focus on the below mentioned requirements.
- It should be scalable.
- Performance impact of federation should be minimal. Push-based model is advisable as opposed to standard way of proxy like RMI Pull-method based proxies.
- User should be able to monitor the Distributed System from any third party tool compliant with JMX. User should be able to reach all mbeans from managing node as well as managed nodes
- Management API should provide mechanism for filtering out mbeans in-order to limit impact on scalability in large-systems.
Assumptions
- Strict consistency across managing member and managed members is not desired.
- Strict real time data is not mandated, its alright as the system would give a close enough picture.
Scope
This spec only points out federation strategy for Mbeans. The Mbean modeling and details about the Mbeans are mentioned in related document section.
Related Document
- https://cwiki.apache.org/confluence/display/GEODE/Proposed+GemFire+MBeans
- https://cwiki.apache.org/confluence/display/GEODE/MBeanServer+Federation+Strategy
- https://cwiki.apache.org/confluence/download/attachments/61309759/JMXFederation-POC.pdf
Terminology
Managed Node : Each member of Distributed system is a Managed node. It can be monitored independently as well as from a single agent view.
Managing Node : The member which will provide the single agent view to monitor and manage all the members. There can multiple managing nodes in the system
Hidden Region : This term is used for internal meta-data region that will be used for replication of MBean states.
1.Overview Of GemFire Federation
GemFire Services For Management & MonitoringGemFire will provide following services for Management and Monitoring.
Single-Agent ViewWith a single-agent view, the application or manager interacts with one MBeanServer. Application and manager development is much easier because they don't have to find the right MBeanServer to make a request on an MBean. That MBeanServer is responsible for aggregating all of the MBeans in the MBeanServers with which it is federated. This is very convenient because the application or manager can interact with a local MBeanServer that has services and adapters to interact with other local or remote MBeanServers. The location of the federated MBeanServers does not change how the application or manager is programmed to interact with its MBeanServer. For the sake of this discussion, we will refer to the single-agent MBeanServer as Managing Node. The MBeanServers that the master interacts with (and the application does not) will be referred to as Managed Node. GemFire achieves Single-Agent View by creating MBean proxies at Managing Node side and keeping their state in sync with their Managed node counterpart. Management Resources
Inbuilt Federation MechanismGemFire federation mechanism is the process by virtue of which we can achieve single agent view.The protocol used is internal to GemFire and does not interfere with any application and manager implementation. GemFire federation takes care of the following functionality
Application Program InterfacesGemFire Management API represent the Gemfire System view to a JMX user. However it does not intend to provide functionality which is otherwise present in JMX. It only provides a gateway into various services exclusively offered by GemFire Management. The entry point to GemFire Management is through interface ManagementService The implementation of getManagementService is a singleton for now but it allows us to eventually support multiple cache instances (probably multiple client caches and on peer cache). The resulting ManagementService instance will be specific to the provided cache and its distributed system. Also the instance will behave differently when the cache is a client cache vs a peer cache. When its a client cache then execution of a command string should be forwarded to the cache server instead of processed in the local JVM. More detailing needs to be done for client cache.
|
Internal Details The following section is internal details of GemFire and not visible to end user directly |
2.Architectural Description
For JMX data distribution, Gemfire's inbuilt Replicated Region support will be used. This will help in data distribution in one direction i.e. from Managed Node to Managing Node. For information flow from managed node to managing node, Gemfire's "Function Service" will be used. Functions are primarily for all operations and propagating attribute setters. Transition from Managed Node to Managing NodeThe mechanism for transition from Managed Node to Managing node will be part of Beaverton specs. Management ListenerA new member wide listener component to be introduced to listen on different GFE resource creation/removal/destroy. On particular events it will create/ destroy Mbeans. This will save us from unnecessary existence of Mbeans. However this will require some internal GFE changes. Other internal components can also be initialized by this listener. e.g. HiddenRegion can be created by this listener on cache create event. Management Listener : public final class InternalDistributedSystem{ private void initialize() { ...................... ManagementListener listener = new ManagementListener(); this.managementListener = listener; } } public enum ManagementEvent { CACHE_CREATE, REGION_CREATE, DISKSTORE_CREATE, CACHESERVER_CREATE, GATEWAYRECEIVER_CREATE, GATEWAYSENDER_CREATE, LOCKSERVICE_CREATE, CACHE_REMOVE, REGION_REMOVE, DISKSTORE_REMOVE, CACHESERVER_REMOVE, GATEWAYRECEIVER_REMOVE, GATEWAYSENDER_REMOVE, LOCKSERVICE_REMOVE } public class ManagementListener { public void handleEvent(ManagementEvent event){ } } class GemFireCacheImpl{ init(){ .............. system.getManagementListener().handleEvent(ManagementEvent.CACHE_CREATE, this); .......... } } Federation StrategyThree main functionalities of any MBean : Attributes, Operation and Notifications are federated in following way
Each member of the system will be represented by one hidden region. The regions can be named "MONITORING_REGION_<memberName>". Whichever node becomes a Managing Node it will replicate all the hidden regions created by its managed member. Hidden region will store MBean state and other meta-data about mbean and its notifications. Federation at managing memberThis section describes the events that happen when a node makes transition from Normal Node to Managing node
Data ModelGemfire MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics. Non-Gemfire MBeans (Platform MBeans, User Defined MBeans) will be represented as it is.
Push Model for MBean State
MBean Operation on Remote MBeans
NotificationsTo understand the proposed notification federation model let's look briefly at the distributed management scope. It can be depicted roughly in the below mentioned layers.
Each layer runs on different VMs. GemFire implementation of proxy will be a little different than other JMX federation approaches. GemFire proxies inherently are local MBeans.It will emit notification when an event occurs. This event is mapped to notification emitted by the actual MBean on Managed Node side. So some component(in our model its NotificationHub) at Managed Node
Some conceptual details for remote notifications can be found at
Following diagram shows two flows
Diagram also shows EmailNotifier as another consumer of notification. Such notifiers can be installed on Managing Server where HubClient can use this to send emails to gemfire users or any other notification consumption mechanism other than JMX
|
3.Future Direction
Management FiltersManagement filter are defined to restrict or narrow-down the management and monitoring to limited set of resources or limited number of members. Filter will help to scale M&M in a large distributed system consisting of hundreds or member. Typical filter consists of inclusion and exclusion lists. It mimics firewall rules, hence inclusion and exclusion. Inclusion is processed first and defines what to include. Exclusion is processed second and defines what to exclude out of what was already defined is being included.
|