Versions Compared

Old Version 7

changes.mady.by.user Barry Oglesby

Saved on

compared with

New Version Current

changes.mady.by.user Barry Oglesby

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

GatewaySender Log Messages and Notifications of Interest

GatewaySender Cannot Connect to GatewayReceiver

Startup

Remote Locator Is Not Running

When the remote locator is not running, the local locator will log a message like below. This message will be logged for each remote locator configured the remote-locators property that is unreachableWhen a GatewaySender cannot connect to a remote GatewayReceiver, a message like below is logged.

[warning 2017/11/09 0911:2832:4448.977667 PST locator-ln-1 <Event<WAN ProcessorLocator for GatewaySender_ny_3>Discovery Thread> tid=0x4c0x54] ny: Could Locator discovery task could not connect.

The attached AlertNotificationListener will log an alert like:

exchange locator information 10.118.20.102[10332] with localhost[10331] after 6 retry attempts. Retrying in 10,000 ms.

And the GatewaySender will log messages like below.

[warning 2017/11/09 11:39:44.314 PST ln-1 <EventAllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=9; time stamp=1510362355433; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=Event Processor for GatewaySender_ny_30> tid=0x4c}]

Once the GatewaySender does connect to a remote GatewayReceiver, an info message like below is logged. This message contains the remote receiver's host and port as well as the number of failed attempts. Depending on what thread has been attempting to connect, it'll retry. For example, the AckReaderThread retries every second by default.

0x47] Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".
[severe [info 2017/11/09 0911:3039:5944.511317 PST ln-1 <Event Processor for GatewaySender_ny_3>0> tid=0x4c0x47] GatewaySender "ny:" Usingcould Poolednot Connectionget to 10.118.20.102:5382: Connection[10.118.20.102:5382]@439092143 after 1 failed connect attempts

or:

[info 2017/11/09 11:15:45.931 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] ny: Using Pooled Connection to 10.118.20.102:5470: Connection[10.118.20.102:5470]@1533322301 after 100 failed connect attempts

GatewaySender Loses Connection to GatewayReceiver

When GatewaySender loses connection to a remote GatewayReceiver, messages like below are logged.

remote locator information for remote site "1".

The attached AlertNotificationListener will log alerts like:

AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=17; time stamp=1510364889713; message=Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".; userData={AlertLevel=warning, Member=ln-1, Thread=[warning 2017/11/09 11:13:54.045 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3>0 tid=0x820x47}]
AllNotificationListener Pool unexpected Socket closed connection=Pooled Connection to 10.118.20.102:5382: Connection[DESTROYED]). Server unreachable:received notification[source=DistributedSystem(2); type=system.alert; sequence number=19; time stamp=1510364889715; message=GatewaySender "ny" could not connectget afterremote locator information for remote site "1".; userData={AlertLevel=severe, Member=ln-1, Thread=Event Processor for GatewaySender_ny_0 tid=0x47}]
GatewaySender Cannot Connect to GatewayReceiver

When a GatewaySender cannot connect to a remote GatewayReceiver, a message like below is logged.

[warning 2017/11/09 09:28:44.977 PST ln-1 <Event1 attempts[warning 2017/11/09 11:13:55.155 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x820x4c] ny: Could not connect.

The attached attached AlertNotificationListener will will log alerts an alert like:

AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=269; time stamp=15103644359611510362355433; message=Pool unexpected closed socket on server connection=Pooled Connection to 10.118.20.102:5181: Connection[DESTROYED]). Server unreachable: couldny: Could not connect after 1 attempts.; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_3 tid=0x820x4c}]AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=29; time stamp=1510364437068; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event

Once the GatewaySender does connect to a remote GatewayReceiver, an info message like below is logged. This message contains the remote receiver's host and port as well as the number of failed attempts. Depending on what thread has been attempting to connect, it'll retry. For example, the AckReaderThread retries every second by default.

[info 2017/11/09 09:30:59.511 PST ln-1 <Event Processor for GatewaySender_ny_33> tid=0x82}]

Remote Locator Is Not Running

When the remote locator is not running, the local locator will log a message like below. This message will be logged for each remote locator configured the remote-locators property that is unreachable.

[warning 2017/11/09 11:32:48.667 PST locator-ln <WAN Locator Discovery Thread> tid=0x54] Locator discovery task could not exchange locator information 10.118.20.102[10332] with localhost[10331] after 6 retry attempts. Retrying in 10,000 ms.

And the GatewaySender will log messages like below.

0x4c] ny: Using Pooled Connection to 10.118.20.102:5382: Connection[10.118.20.102:5382]@439092143 after 1 failed connect attempts

or:

[info 2017/11/09 11:15:45.931 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] ny: Using Pooled Connection to 10.118.20.102:5470: Connection[10.118.20.102:5470]@1533322301 after 100 failed connect attempts
Ongoing
GatewaySender Loses Connection to GatewayReceiver

When GatewaySender loses connection to a remote GatewayReceiver, messages like below are logged.

[warning 2017/11/09 11:13:54.045 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] Pool unexpected Socket closed connection=Pooled Connection to 10.118.20.102:5382: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
[warning 2017/11/09 11:13:55.155 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] ny: Could not connect.

The attached AlertNotificationListener will log alerts like:

AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=26; time stamp=1510364435961; message=Pool unexpected closed socket on server connection=Pooled Connection to 10.118.20.102:5181: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_3 tid=0x82}]
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=29; time stamp=1510364437068; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_3 tid=0x82}]
Batch Exceptions

Exceptions that occur on the GatewayReceiver while processing an event in a batch cause a warning like below to be logged on the GatewayReceiver. The stack trace shows a BatchException70 wrapping the real exception. In this case, a CacheWriterException occurred.

Code Block
languagejava
[warning 2017/11/09 16:28:46.265 PST ny-1 <ServerConnection on port 5342 Thread 4> tid=0x6a] Server connection from [identity(10.118.20.102(ln-1:12887)<v1>:1027,connection=1; port=56515]: Wrote batch exception: 
org.apache.geode.internal.cache.wan.BatchException70: Exception occurred while processing a batch on the receiver running on DistributedSystem with Id: 1, DistributedMember on which the receiver is running: 10.118.20.102(ny-1:12879)<v3>:1026
	at org.apache.geode.internal.cache.tier.sockets.command.GatewayReceiverCommand.cmdExecute(GatewayReceiverCommand.java:715)
	...
Caused by: org.apache.geode.cache.CacheWriterException: test cache writer exception
	at TestCacheWriter.process(TestCacheWriter.java:24)
	at TestCacheWriter.beforeCreate(TestCacheWriter.java:14)
	...

The same exception warning is logged on the GatewaySender along with a message identifying the event being processed like below.

Code Block
languagejava
[warning 2017/11/09 16:56:36.394 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x6a] Gateway Sender ParallelGatewaySender{id=ny,remoteDsId=1,isRunning =true} : Received ack for batch id 0 with one or more exceptions

[warning 2017/11/09 16:56:36.395 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x6a] A BatchException occurred processing events. Index of Array of Exception : 0
org.apache.geode.internal.cache.wan.BatchException70: Exception occurred while processing a batch on the receiver running on DistributedSystem with Id: 1, DistributedMember on which the receiver is running: 10.118.20.102(ny-1:13339)<v5>:1026
	at org.apache.geode.internal.cache.tier.sockets.command.GatewayReceiverCommand.cmdExecute(GatewayReceiverCommand.java:715)
	...
Caused by: org.apache.geode.cache.CacheWriterException: test cache writer exception
	at TestCacheWriter.process(TestCacheWriter.java:24)
	at TestCacheWriter.beforeCreate(TestCacheWriter.java:14)
	...

[warning 2017/11/09 16:56:36.407 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x6a] The event being processed when the BatchException occurred was:  SenderEventImpl[id=EventIDid=25bytes;threadID=0x10030|1;sequenceID=0;bucketId=48];action=0;operation=CREATE;region=/data;key=0;value=Trade[id=0; cusip=PVTL; shares=32; price=575.0191650390625];valueIsObject=1;numberOfParts=9;callbackArgument=GatewaySenderEventCallbackArgument [originalCallbackArg=null;originatingSenderId=2;recipientGatewayReceivers={1}];possibleDuplicate=false;creationTime=1510275395589;shadowKey= 161;timeStamp=1510275395587;acked=false;dispatched=false;bucketId=48]

The attached AlertNotificationListener receives these GatewaySender warnings as notifications like:

AlertNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=18; time stamp=1510275396393; message=Gateway Sender ParallelGatewaySender{id=ny,remoteDsId=1,isRunning =true} : Received ack for batch id 0 with one or more exceptions; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event[warning 2017/11/09 11:39:44.314 PST ln-1 <Event Processor for GatewaySender_ny_0>3 tid=0x470x6a}]
AlertNotificationListener Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".[severe 2017/11/09 11:39:44.317 PST ln-1 <Eventreceived notification[source=DistributedSystem(2); type=system.alert; sequence number=20; time stamp=1510275396394; message=A BatchException occurred processing events. Index of Array of Exception : 0; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_0>3 tid=0x47] GatewaySender "ny" could not get remote locator information for remote site "1".

The attached AlertNotificationListener will log alerts like:

0x6a}]
AlertNotificationListenerAllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=17; time stamp=1510364889713; message=Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".; userData={AlertLevel=warning, Member=ln-1, Thread=Event Processor for GatewaySender_ny_0 tid=0x47}]AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=19; time stamp=1510364889715; message=GatewaySender "ny" could not get remote locator information for remote site "1".alert; sequence number=22; time stamp=1510275396397; message=The event being processed when the BatchException occurred was: SenderEventImpl[id=EventIDid=25bytes;threadID=0x10030|1;sequenceID=0;bucketId=48];action=0;operation=CREATE;region=/data;key=0;value=Trade[id=0; cusip=PVTL; shares=32; price=575.0191650390625];valueIsObject=1;numberOfParts=9;callbackArgument=GatewaySenderEventCallbackArgument [originalCallbackArg=null;originatingSenderId=2;recipientGatewayReceivers={1}];possibleDuplicate=false;creationTime=1510275395589;shadowKey= 161;timeStamp=1510275395587;acked=false;dispatched=false;bucketId=48]; userData={AlertLevel=severewarning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_03 tid=0x470x6a}]

MBean Attributes

Accessing GatewaySender and GatewayReceiver Attributes

...