3 Architectural Description

Internal Details The following section is internal details of GemFire and not visible to end user directly

For JMX data distribution, Gemfire's inbuilt Replicated Region support will be used. This will help in data distribution in one direction i.e. from Managed Node to Managing Node.
Replicated region fits into push based monitoring support easily.

For information flow from managed node to managing node, Gemfire's "Function Service" will be used. Functions are primarily for all operations and propagating attribute setters.

3.1 Transition from Managed Node to Managing Node

The mechanism for transition from Managed Node to Managing node will be part of Beaverton specs.
a) StartManagingNode : Will create Management resources namely Regions, proxies for all the connected members
b) StopManagingNode : Will remove all the Management resources.It will behave as a normal Managed Node and start pushing its local data to the hidden regions

3.2 Management Listener

public final class InternalDistributedSystem{

 private void initialize() {
   ......................
    ManagementListener listener = new ManagementListener();
    this.managementListener = listener;
  }
}


public enum ManagementEvent {
    CACHE_CREATE, 
    REGION_CREATE, 
    DISKSTORE_CREATE, 
    CACHESERVER_CREATE, 
    GATEWAYRECEIVER_CREATE, 
    GATEWAYSENDER_CREATE, 
    LOCKSERVICE_CREATE, 
    CACHE_REMOVE, 
    REGION_REMOVE, 
    DISKSTORE_REMOVE, 
    CACHESERVER_REMOVE, 
    GATEWAYRECEIVER_REMOVE, 
    GATEWAYSENDER_REMOVE, 
    LOCKSERVICE_REMOVE
}

public class ManagementListener { 
	
	public void handleEvent(ManagementEvent event){
        }
    }

class GemFireCacheImpl{

        init(){
        ..............
        system.getManagementListener().handleEvent(ManagementEvent.CACHE_CREATE, this);
        ..........
       }
 }

3.3 Federation Strategy

Three main functionalities of any MBean : Attributes, Operation and Notifications are federated in following way

• Attribute value are federated using GemFire Hidden Region
  Central component for Federation is FederatedComponent which will store object state as well as some meta data like MBean interface , notification enabled etc.
• Operations are federated using GemFire Function Service
• Notification are federated using Hidden Region and Function Service together.

Each member of the system will be represented by one hidden region. The regions can be named "MONITORING_REGION_<memberName>". Whichever node becomes a Managing Node it will replicate all the hidden regions created by its managed member. Hidden region will store MBean state and other meta-data about mbean and its notifications.

3.4 Federation at managing member

This section describes the events that happen when a node makes transition from Normal Node to Managing node

3.4.1 Query cluster to know all manageable members

Managing member has to discover set of members in the current distributed system. Very first step would be to know all manageable members. This includes members with fixed-persistent memberName as well as member with non-persistent ID. This section assumes mechanism for querying all member of system. The results of this operation should return memberName and its descriptor. The descriptor would tell following attributes of the member

• 1. memberName
  2. Type : Persistent, Non-Persistent
  3. Services running : Peer, CacheServer, Locator, Managing or Managed

3.4.2 Creation of Hidden Regions

Managing node has to communicate to all managed node and exchange all mbean informration i.e. what all mbeans are hosted by each managed node. Once this set is established managing node can finalize the list of proxies that it needs to create. 

As we will be using GemFire hidden regions while the Managing member comes up GII functionality will take care of this.
If Managing member is up and a new member comes up then MembershipListener will take care of this.

3.4.3 Create Proxy for remote MBeans

For this purpose Java's Proxy Design pattern is used. Our proxies are similar to JMX proxies. Only underlying InvocationHandler is different.
JMX delegates all call to remote RMI connnection. Calls on Gemfire Management proxies will be delegated to the state containing FederatedComponent and FunctionService.
Reference : javax.management.MBeanServerInvocationHandler

3.4.4 Method Invocation in Proxy MBean

1. All getter method on the proxy will be delegated to the contained FederationComponent
   1. 
2. All setter metohd on the proxy will be delegated to an instance of GenericMBeanFunction
   1. 
      
3. All other opertaions on the proxy will be delegated to an instance of GenericMBeanFunction. Flow same as setter functions
   

• Method Invocationin Proxy MBean(If proxy is provided explicitly)

  GemFire MBeans will implement an interface which will serve two purposes
  a) System can identify GemFire specific MBeans
  b) Any proxy implementation can be given by this MBean which will by pass above generic proxy implementation
  All the getters, setters, and operations will be delegated to this proxy. As proxy will be written by the developer who develops the MBean he will have the information how to access the data locally.

• Following listeners will trigger creation and deletion of proxy mbeans

  1. MembershipListener ( Whenever a member joins or departs we need to create /destroy proxies)
  2. ManagementCacheListener ( Whenever a region operation happens on the MONITORING_REGION)
  3. ManagingNodeTransitionListener ( Whenever a member becomes a Managing node)

Data Model

Gemfire MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics.

Non-Gemfire MBeans (Platform MBeans, User Defined MBeans) will be represented as it is.

• Data Model - Managed Node

1 Data model
Each MBean state will be represented by a FederatedComponent as per the proposed model. FederatedComponent will also be behave as an entry for the hidden region for data distribution.
The relationship between MBean, FederatedComponent is depicted below.

2 Bridge Data model
For Gemfire MBeans an intermediary MBeanBridge will be implemented to act as a conduit between GFE resource and MBeans.

 

Note MBeanBridge is only for Gemfire MBeans for ease of programming. Other MBeans are free to have their implementation as they require.

 

3 Federated MBeans
If a MBean is intended to be federated then Gemfire should have some means to get the state data to be propagated by hidden region.
For that an FederatedComponent is used. 
It uses MBeanInfo to dynamically get attribute & attribute values and keep it in an ObjectState map as a key value pair.
In addition it also contains some meta data like MBean interface, notification capabilities etc.

4 Delta Approach and other optimizations
This is deferred till implementation phase after checking the number of attributes which gets change frequently. If the percentage. is quite high then Delta might not be a good idea.

• Data Model - Managing Node

1 Proxy Data model
This will include proxies for Global scoped User Defined MBeans and of course Gemfire MBeans 
The below diagram depicts the relationship between various components for Proxies.

1. Delta Approach and other optimizations
   This is deferred till implementation phase after checking the number of attributes which gets change frequently.

Push Model for MBean State

• Generic MBeans

  All user MBeans and Platform MBeans can be federated provided they follow the above mentioned contract with Gemfire.
  Gemfire will internally scan through all such MBeans and push their state to the hidden region in a given frequency.

Gemfire Specific MBeans

MBean state consists of Statistical information maintained by Gemfire Statistics and other information like configuration and other dynamic information not covered by gemfire statistics. Following diagram shows how M&M layer gets data from gemfire statistics. ManagementThread is used for pushing updates from managed node to manging node via hidden region replication. ManagementThread will group and batch all updated in-order to reduce network traffic.

 

Note MBeanBridge is only for Gemfire MBeans for ease of programming.They contain the MBean state. They can be updated by the Listeners directly as we are free to code the listener object

1. Non-Statistics Part
   This part of MBean state is collected from Bridge component of underlying gemfire resources.
2. Statistics Part
   Statistics Listener provided by Statistics framework is used for updating mbean state. There is not direct invocation or signaling mechanism between StatisticsSampler Thread and Management Thread. ManagementThread is responsible for getting updating statistics part of Mbean State. 

StatisticsListener listener = new StatisticsListener() {
  public void handleNotification(StatisticsNotification notification) {
    // handle the notification
  }
};

ValueMonitor monitor = new ValueMonitor()
    .addStatistic(GemFireStatistics.getCreatesDescriptor())
    .addStatistic(GemFireStatistics.getDestroysDescriptor())
    .addStatistic(GemFireStatistics.getEntriesDescriptor())
    .addStatistic(GemFireStatistics.getServerStatistics().getCurrentClientsDescriptor()
    .addStatistic(GemFireStatistics.getServerStatistics().getSubscriptionQueueSizeDescriptor());

monitor.registerListener(listener);

MBean Operation on Remote MBeans

• MBean Operation-Function Mapping

  For the time being one function class GenericMBeanFunction is proposed.
  This will be used :

1. For all setter functions of a MBean
2. For all operations of a MBean
3. Notification client add/ remove

• Function Attributes

1. hasResult : It will always be true. Even if MBean operation has void as return type. This is to handle error conditions and suitable messaging
2. isHa : Always false

• Function Registration

  ManagementService instance will register the function

• Failed Function Results

  If by any means the function execution fails (e.g. by unreachable members) it will return UNDEFINED string as a result.

Notifications

To understand the proposed notification federation model let's look briefly at the distributed management scope.

It can be depicted roughly in the below mentioned layers.

Each layer runs on different VMs.

GemFire implementation of proxy will be a little different than other JMX federation approaches.

GemFire proxies inherently are local MBeans.It will emit notification when an event occurs.

This event is mapped to notification emitted by the actual MBean on Managed Node side. So some component(in our model its NotificationHub) at Managed Node
side should capture the notifications and send a message to Managing Node for actual JMX notification emit.

• Description

  Federation of notification is implemented using NotificationHub. Hub subscribes for notifications in the managed node and forwards them to managing node. On managing-node HubClient will fan-out the notification to correct MBean proxy. Proxy then uses standard JMX NotificationBroadcasterSupport class for forwarding up to JMX stack and JMX RMI Connector. 
  This approach allows us to achieve following requirements

  1. Does not disturb standard JMX notification flow
  2. Supports standard JMX notifications for RMI connector on managed node as well as manging node 
     
     Notification federation will be supported in a sub region of MONITORING_REGION. It will be named NOTIFICATION_REGION which will be an empty region.
     If the region is not an empty region we would have to delete the notification from the region which will cause an extra message.

• Assumptions

  1. MBean is using NotificationBroadcasterSupport for emitting notifications. Thus on client side Proxy generated also extends NotificationBroadcasterSupport.
  2. All notifications generated by MBean marked as Federated are emitted to Managing node. There is no filtering on type of notification within a single MBean

• Flow at Managed Member

• Flow at Managing Member

Notification Subscription

NotificationHub should not send all the notifications originating in its VM . Rather it should send messages for only those notifications for which a client has expressed interest.
Whenever client express interest for notifications of a particular MBean an instance of NotificationListener will be registered at Managed Node side.
This listener will do the job of listening to notifications locally and put data in the hidden notification region.

1. addNotificationListener
2. removeNotificationListener 
   
   Proxy Class : MBeanProxyInvocationHandler 

MBeanServer mbeanServer = ManagementFactory.getPlatformMBeanServer();


public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
  
    final Class methodClass = method.getDeclaringClass();
     if (methodClass.equals(NotificationBroadcaster.class) || methodClass.equals(NotificationEmitter.class))
        return invokeBroadcasterMethod(proxy, method, args);
      .............
      ...............
}

private Object invokeBroadcasterMethod(Object proxy, Method method, Object[] args) throws Exception {
  if (methodName.equals("addNotificationListener")) {
   mbeanServer.addNotificationListener(mbeanName, listener, filter, handback);
   delegateToFucntionService();// For registering a NotificationListener at Managed Node side
  }
  if (methodName.equals("removeNotificationListener")) {
   mbeanServer.removeNotificationListener(mbeanName, listener, filter, handback);
   delegateToFucntionService();// For removing  NotificationListener from  Managed Node side
  }
}

Some conceptual details for remote notifications can be found at 
Remote Notifications Section: http://mx4j.sourceforge.net/docs/ch03s04.html

• Gemfire alerts

  Gemfire alert mechanism will also use Federation services backed by replicated regions. Current approach uses peer-to-peer messaging. Earlier alerts were consumed by remote listeners installed on JMX Agent.Now those messages or alerts will be consumed by NotificationHub.
  A default AlertNotificationBroadcaster will be registered for each node. ManagerLogWriter.notifyAlertListeners will invoke the notification broadcaster.
  This broadcaster will collect all the alerts depending on the alert level depending on the "alertLevel" attribute of ManagementMBean.The alerts will be propagated to the notification region.

Following diagram shows two flows

1. Gemfire Warning messages getting federated as JMX Notifications across DS
2. Custom Statistics based Monitor generating federated JMX notifications. Monitor concept is derived from ResourceMonitor and also similar to JDK provided Monitor and GaugeMonitor MBeans. Difference is all logic is executed on ManagementThread instead of JDK Monitor where each object uses its own thread.

Diagram also shows EmailNotifier as another consumer of notification. Such notifiers can be installed on Managing Server where HubClient can use this to send emails to gemfire users or any other notification consumption mechanism other than JMX

• Network Partitioning Handling

  • Managing Node on Winning Side : It will only emit notifications about crashed members
  • Managing Node on Loosing Side : Managing Node will shutdown itself. Before shutdown the alert emitted will contain info about Network Partition with failure set information

3.Future Direction

Management Filters

Management filter are defined to restrict or narrow-down the management and monitoring to limited set of resources or limited number of members. Filter will help to scale M&M in a large distributed system consisting of hundreds or member.

Typical filter consists of inclusion and exclusion lists. It mimics firewall rules, hence inclusion and exclusion. Inclusion is processed first and defines what to include. Exclusion is processed second and defines what to exclude out of what was already defined is being included.

• Types of Filters

  Three types of filters are defined :

  FilterName	Functionality	Inclusion List	Exclusion List	Description
  Local MBean Filter	Determines what local MBeans will be registered	if ObjectName matches any then it will be created/registered; default: * (all)	if ObjectName matches any then it will be ignored; default: null	localMBeanInclusion: * localMBeanExclusion: null  So by default "*" causes all local MBeans to be included. This means means that every single MBean relevant to the local member will be instantiated and registered. For example, every Region will result in a RegionMBean. Since the exclusion is "null", none of the MBean types would be excluded. But you could change the localMBeanExclusion to be regex pattern ".*Region.*" which results in excluding of the RegionMBeans for each Region in the local member (because the ObjectName for RegionMBeans contains "Region"). This is applicable only when SCOPE attribute of ManagementMBean is LOCAL
  Remote MBean Filter	determines what remote MBeans will be federated by proxy MBeans	if ObjectName matches any then it will be created/registered; default: * (all)	if ObjectName matches any then it will be skipped; default: null	remoteMBeanInclusion: * remoteMBeanExclusion: null  The above are ignored if Scope is "Local". But if Scope is "All" then the local JVM is a management node and it will create a hidden region for every other member in the distributed system. It will then create federated versions of every MBean in those members as determined by remoteMBeanInclusion/remoteMBeanExclusion. remoteMBeanInclusion "*" means that all remote MBeans will be federated. remoteMBeanExclusion "null" means that no remote MBeans will be excluded from this.
  Managed Member Filter	determines what members will be managed (federated)	if memberName matches any then it will be managed (federated); default: * (all)	if memberName matches any then it will be ignored; default: null	memberInclusion: * memberExclusion: null  memberGroupInclusion: * memberGroupExclusion: null  Thus by default all members are included and federated if the local JVM is a management node. Meaning this node will create federated views of the MBeans (that match the remote mbean filter) in every member. If an exclusion other than null is set then any member matching that exclusion would be excluded from the federated view. The values of all these inclusion/exclusion properties should support regex pattern such as "Publisher.*" as well as constant strings such as "Server-1" as well as LISTS such as "Publisher.*;Subscriber.*" (note that someone needs to pick a reasonable delimiter for lists of regex patterns) or even a mix such as "Publisher-A;Subscriber-B;Server.*". memberGroupInclusion/memberGroupExclusion are more important for large memberships and allows someone to limit their monitoring or managing to one server group or one logical group of members (or even more than one server group).

  
  Exact syntax of these filters is still TBD. 

Example: if you defined the following... memberInclusion: ".\*Publisher.\*;.\*Subscriber.\*" memberExclusion: "Trade.\*" ...you would end up including members with member-ids such as: OptionPublisher-1 OptionPublisher-2 OptionPublisher-3 OptionSubscriber-1 StockPublisher-1 StockPublisher-2 But the following would be excluded: TradePublisher-1 TradePublisher-2 TradeSubscriber-1

• How filters are specified

1. Gemfire configuration mechanism

• Behavior on Filter

  Filters will be picked up when a node starts up and those filters will be in force during the lifetime of the member node. Change in filters will require member node restart.