...
JMX can be used to receive Geode severe, error or warning log messages as alerts as well as to access GatewaySenderMBean and GatewayReceiverMBean attributes
Alerts
...
Alert Level
JMX clients can be configured to receive Geode severe, error or warning log messages as alerts using a JMX NotificationListener.
...
Once the alert level has been set to 'warning', then warning log messages will be received by any JMX NotificationListener in the form of Notifications.
Here are some example using JConsole and Java.
...
Note: This example logs every Geode warning, not just ones from GatewaySenders.
GatewaySender
...
Log Messages and Notifications of Interest
GatewaySender Cannot Connect to GatewayReceiver
When a GatewaySender cannot connect to a remote GatewayReceiver, a message like below is logged.
[warning 2017/11/09 09:28:44.977 PST ln-1 <Event Processor for GatewaySender_ny_3> tid=0x4c] ny: Could not connect.
The attached AlertNotificationListener will log an alert like:
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=9; time stamp=1510362355433; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=Event Processor for GatewaySender_ny_3 tid=0x4c}]
Once the GatewaySender does connect to a remote GatewayReceiver, an info message like below is logged. This message contains the remote receiver's host and port as well as the number of failed attempts. Depending on what thread has been attempting to connect, it'll retry. For example, the AckReaderThread retries every second by default.
[info 2017/11/09 09:30:59.511 PST ln-1 <Event Processor for GatewaySender_ny_3> tid=0x4c] ny: Using Pooled Connection to 10.118.20.102:5382: Connection[10.118.20.102:5382]@439092143 after 1 failed connect attempts
or:
[info 2017/11/09 11:15:45.931 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] ny: Using Pooled Connection to 10.118.20.102:5470: Connection[10.118.20.102:5470]@1533322301 after 100 failed connect attempts
GatewaySender Loses Connection to GatewayReceiver
When GatewaySender loses connection to a remote GatewayReceiver, messages like below are logged.
[warning 2017/11/09 11:13:54.045 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] Pool unexpected Socket closed connection=Pooled Connection to 10.118.20.102:5382: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
[warning 2017/11/09 11:13:55.155 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] ny: Could not connect.
The attached AlertNotificationListener will log alerts like:
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=26; time stamp=1510364435961; message=Pool unexpected closed socket on server connection=Pooled Connection to 10.118.20.102:5181: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_3 tid=0x82}]
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=29; time stamp=1510364437068; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_3 tid=0x82}]
Remote Locator Is Not Running
When the remote locator is not running, the local locator will log a message like below. This message will be logged for each remote locator configured the remote-locators property that is unreachable.
[warning 2017/11/09 11:32:48.667 PST locator-ln <WAN Locator Discovery Thread> tid=0x54] Locator discovery task could not exchange locator information 10.118.20.102[10332] with localhost[10331] after 6 retry attempts. Retrying in 10,000 ms.
And the GatewaySender will log messages like below.
[warning 2017/11/09 11:39:44.314 PST ln-1 <Event Processor for GatewaySender_ny_0> tid=0x47] Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".
[severe 2017/11/09 11:39:44.317 PST ln-1 <Event Processor for GatewaySender_ny_0> tid=0x47] GatewaySender "ny" could not get remote locator information for remote site "1".
The attached AlertNotificationListener will log alerts like:
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=17; time stamp=1510364889713; message=Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".; userData={AlertLevel=warning, Member=ln-1, Thread=Event Processor for GatewaySender_ny_0 tid=0x47}]
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=19; time stamp=1510364889715; message=GatewaySender "ny" could not get remote locator information for remote site "1".; userData={AlertLevel=severe, Member=ln-1, Thread=Event Processor for GatewaySender_ny_0 tid=0x47}]
MBean Attributes
Accessing GatewaySender and GatewayReceiver Attributes
JMX clients can also access GatewaySenderMBean and GatewayReceiverMBean attributes.
Here are some example using JConsole and Java.
JConsole
GatewaySender Attributes
To view GatewaySender attributes:
GatewayReceiver Attributes
To view GatewayReceiver attributes:
Java
GatewaySender Attributes
The attached DumpGatewaySenders JMX client dumps GatewaySender attributes
MBean Attributes
JMX clients can also access GatewaySenderMBean and GatewayReceiverMBean attributes.
Here are some example using JConsole and Java.
JConsole
To view GatewaySender attributes:
To view GatewayReceiver attributes:
Java
The attached DumpGatewaySenders JMX client dumps GatewaySender attributes using code like:
Code Block | ||
---|---|---|
| ||
private void dumpGatewaySenders() throws Exception {
ObjectName patternName = ObjectName.getInstance("GemFire:service=GatewaySender,gatewaySender=*,type=Member,member=*");
Set<ObjectName> objectNames = this.mbs.queryNames(patternName, null);
System.out.println("The distributed system contains the following " + objectNames.size() + " senders:");
for (ObjectName objectName : objectNames) {
System.out.println("=======================================================");
System.out.println(objectName);
System.out.println();
System.out.println("Configuration Attributes:");
showAttributes(objectName, getGatewaySenderConfigurationAttributes());
System.out.println();
System.out.println("Queue Attributes:");
showAttributes(objectName, getGatewaySenderQueueAttributes());
}
}
private void showAttributes(ObjectName objectName, List<String> attributesToDump) throws Exception {
MBeanInfo info = this.mbs.getMBeanInfo(objectName);
MBeanAttributeInfo[] attributes = info.getAttributes();
for (MBeanAttributeInfo attribute : attributes) {
if (attributesToDump.isEmpty() || attributesToDump.contains(attribute.getName())) {
Object value = null;
try {
value = this.mbs.getAttribute(objectName, attribute.getName());
} catch (Exception e) {} /** throws NPE if the value is null **/
System.out.println("\t" + attribute.getName() + "->" + valueAsString(value));
}
}
} |
This code will dump output like below. The Configuration Attributes are the static configuration attributes. The Queue Attributes are the dynamic queue attributes.
Code Block |
---|
The distributed system contains the following 2 senders:
=======================================================
GemFire:service=GatewaySender,gatewaySender=ny,type=Member,member=ln-1
Configuration Attributes:
GatewayEventFilters->null
SenderId->ny
SocketBufferSize->524288
BatchConflationEnabled->false
PersistenceEnabled->false
DiskSynchronous->true
RemoteDSId->1
ManualStart->false
OverflowDiskStoreName->null
MaximumQueueMemory->100
SocketReadTimeout->0
BatchTimeInterval->1000
AlertThreshold->100
DispatcherThreads->5
OrderPolicy->KEY
Parallel->true
BatchSize->100
Queue Attributes:
EventQueueSize->219
Primary->true
Connected->true
Running->true
Paused->false
EventsReceivedRate->10701.0
EventsQueuedRate->10701.0
TotalEventsConflated->0
BatchesDispatchedRate->54.0
AverageDistributionTimePerBatch->358332
TotalBatchesRedistributed->1655
EventsExceedingAlertThreshold->0
GatewayReceiver->server-2:5334
=======================================================
GemFire:service=GatewaySender,gatewaySender=ny,type=Member,member=ln-2
... |
The attached DumpGatewayReceivers JMX client dumps GatewayReceiver attributes using code like:
Code Block | ||
---|---|---|
| ||
private void dumpGatewayReceiversdumpGatewaySenders() throws Exception { ObjectName patternName = ObjectName.getInstance("GemFire:service=GatewaySender,gatewaySender=GatewayReceiver*,type=Member,member=*"); Set<ObjectName> objectNames = this.mbs.queryNames(patternName, null); System.out.println("The distributed system contains the following " + objectNames.size() + " receiverssenders:"); for (ObjectName objectName : objectNames) { System.out.println("======================================================="); System.out.println(objectName); System.out.println(); System.out.println("Configuration Attributes:"); showAttributes(objectName, getGatewayReceiverConfigurationAttributesgetGatewaySenderConfigurationAttributes()); System.out.println(); System.out.println("ConnectionQueue Attributes:"); showAttributes(objectName, getGatewayReceiverConnectionAttributesgetGatewaySenderQueueAttributes()); } } |
This code will dump output like below. The Configuration Attributes are the static configuration attributes. The Connection Attributes are the dynamic connection attributes.
Code Block |
---|
The distributed system contains the following 1 receivers:
=======================================================
GemFire:service=GatewayReceiver,type=Member,member=ny-1
Configuration Attributes:
Port->5334
BindAddress->
SocketBufferSize->524288
MaximumTimeBetweenPings->60000
StartPort->5500
EndPort->5500
GatewayTransportFilters->null
Connection Attributes:
Running->true
ConnectionLoad->0.0
LoadPerConnection->0.0
QueueLoad->0.0
LoadPerQueue->1.0
DuplicateBatchesReceived->0
OutoforderBatchesReceived->0
ClientConnectionCount->12
ConnectionThreads->0
GetRequestAvgLatency->0
GetRequestRate->0.0
PutRequestAvgLatency->0
PutRequestRate->0.0
ThreadQueueSize->0
TotalConnectionsTimedOut->0
TotalFailedConnectionAttempts->0
TotalReceivedBytes->49850096
TotalSentBytes->145416
UpdateRequestsRate->0.0
DestroyRequestsRate->0.0
EventsReceivedRate->0.0
CreateRequestsRate->0.0
NumGateways->2
AverageBatchProcessingTime->1
ConnectedGatewaySenders->[server1(ln-2:54496)<v2>:1027, server2(ln-1:54472)<v1>:1026] |
gfsh
private void showAttributes(ObjectName objectName, List<String> attributesToDump) throws Exception {
MBeanInfo info = this.mbs.getMBeanInfo(objectName);
MBeanAttributeInfo[] attributes = info.getAttributes();
for (MBeanAttributeInfo attribute : attributes) {
if (attributesToDump.isEmpty() || attributesToDump.contains(attribute.getName())) {
Object value = null;
try {
value = this.mbs.getAttribute(objectName, attribute.getName());
} catch (Exception e) {} /** throws NPE if the value is null **/
System.out.println("\t" + attribute.getName() + "->" + valueAsString(value));
}
}
} |
This code will dump output like below. The Configuration Attributes are the static configuration attributes. The Queue Attributes are the dynamic queue attributes.
Code Block |
---|
The distributed system contains the following 2 senders:
=======================================================
GemFire:service=GatewaySender,gatewaySender=ny,type=Member,member=ln-1
Configuration Attributes:
GatewayEventFilters->null
SenderId->ny
SocketBufferSize->524288
BatchConflationEnabled->false
PersistenceEnabled->false
DiskSynchronous->true
RemoteDSId->1
ManualStart->false
OverflowDiskStoreName->null
MaximumQueueMemory->100
SocketReadTimeout->0
BatchTimeInterval->1000
AlertThreshold->100
DispatcherThreads->5
OrderPolicy->KEY
Parallel->true
BatchSize->100
Queue Attributes:
EventQueueSize->219
Primary->true
Connected->true
Running->true
Paused->false
EventsReceivedRate->10701.0
EventsQueuedRate->10701.0
TotalEventsConflated->0
BatchesDispatchedRate->54.0
AverageDistributionTimePerBatch->358332
TotalBatchesRedistributed->1655
EventsExceedingAlertThreshold->0
GatewayReceiver->server-2:5334
=======================================================
GemFire:service=GatewaySender,gatewaySender=ny,type=Member,member=ln-2
... |
GatewayReceiver Attributes
The attached DumpGatewayReceivers JMX client dumps GatewayReceiver attributes using code like:
Code Block | ||
---|---|---|
| ||
private void dumpGatewayReceivers() throws Exception {
ObjectName patternName = ObjectName.getInstance("GemFire:service=GatewayReceiver,type=Member,member=*");
Set<ObjectName> objectNames = this.mbs.queryNames(patternName, null);
System.out.println("The distributed system contains the following " + objectNames.size() + " receivers:");
for (ObjectName objectName : objectNames) {
System.out.println("=======================================================");
System.out.println(objectName);
System.out.println();
System.out.println("Configuration Attributes:");
showAttributes(objectName, getGatewayReceiverConfigurationAttributes());
System.out.println();
System.out.println("Connection Attributes:");
showAttributes(objectName, getGatewayReceiverConnectionAttributes());
}
} |
This code will dump output like below. The Configuration Attributes are the static configuration attributes. The Connection Attributes are the dynamic connection attributes.
Code Block |
---|
The distributed system contains the following 1 receivers:
=======================================================
GemFire:service=GatewayReceiver,type=Member,member=ny-1
Configuration Attributes:
Port->5334
BindAddress->
SocketBufferSize->524288
MaximumTimeBetweenPings->60000
StartPort->5500
EndPort->5500
GatewayTransportFilters->null
Connection Attributes:
Running->true
ConnectionLoad->0.0
LoadPerConnection->0.0
QueueLoad->0.0
LoadPerQueue->1.0
DuplicateBatchesReceived->0
OutoforderBatchesReceived->0
ClientConnectionCount->12
ConnectionThreads->0
GetRequestAvgLatency->0
GetRequestRate->0.0
PutRequestAvgLatency->0
PutRequestRate->0.0
ThreadQueueSize->0
TotalConnectionsTimedOut->0
TotalFailedConnectionAttempts->0
TotalReceivedBytes->49850096
TotalSentBytes->145416
UpdateRequestsRate->0.0
DestroyRequestsRate->0.0
EventsReceivedRate->0.0
CreateRequestsRate->0.0
NumGateways->2
AverageBatchProcessingTime->1
ConnectedGatewaySenders->[server1(ln-2:54496)<v2>:1027, server2(ln-1:54472)<v1>:1026] |
Important GatewaySender and GatewayReceiver Attributes
GatewaySender Attributes
Attribute | Description |
---|---|
Running | Whether or not the GatewaySender is running. If it is running, then events will be queued. This does not mean the GatewaySender is connected to a remote site. |
Connected | Whether or not the sender is connected to the remote receiver. |
EventQueueSize | The size of the sender event queue. This attribute should remain fairly steady. If it is continuously growing, then the put rate into the queue is faster than the take rate from the queue. |
TotalBatchesRedistributed | The number of batches redistributed. This attribute should be 0 most of the time. If it is a flat-line above 0, then it generally means the connection has been lost. |
GatewayReceiver Attributes
Attribute | Description |
---|---|
Running | Whether or not the GatewayReceiver is running. If it is running, then remote GatewaySenders can connect to it. |
NumGateways | The number of remote GatewaySenders connected to this GatewayReceiver. If no remote GatewaySenders are joining or leaving, then this attribute should remain steady. |
ConnectedGatewaySenders | The actual remote GatewaySenders connected to this GatewayReceiver. If no remote GatewaySenders are joining or leaving, then this attribute should remain steady. |
ClientConnectionCount | The actual number of remote Gateway Sender threads connected to this GatewayReceiver. If no remote GatewaySenders are joining or leaving, then this attribute should remain steady. |
gfsh
gfsh can be used to dump most of the above GatewaySender and GatewayReceiver statistics. The GatewaySender statistics include status, queued events and receiver location. The GatewayReceiver statistics include sender count and senders connected (gfsh can be used to monitor several important GatewaySender and GatewayReceiver statistics including Status and Queued Events. Also included are the Receiver Locator for the sender and the Sender's Connected for the receiver although this data is a String[] and not printed correctly).
To log these statistics using gfsh, execute the list gateways command like:
...