Versions Compared

Old Version 8

changes.mady.by.user Barry Oglesby

Saved on

compared with

New Version Current

changes.mady.by.user Barry Oglesby

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

GatewaySender Log Messages and Notifications of Interest

GatewaySender Cannot Connect to GatewayReceiver

Startup

Remote Locator Is Not Running

When the remote locator is not running, the local locator will log a message like below. This message will be logged for each remote locator configured the remote-locators property that is unreachableWhen a GatewaySender cannot connect to a remote GatewayReceiver, a message like below is logged.

[warning 2017/11/09 0911:2832:4448.977667 PST locator-ln-1 <Event<WAN ProcessorLocator for GatewaySender_ny_3>Discovery Thread> tid=0x4c0x54] ny: Could not connect.

The attached AlertNotificationListener will log an alert like:

Locator discovery task could not exchange locator information 10.118.20.102[10332] with localhost[10331] after 6 retry attempts. Retrying in 10,000 ms.

And the GatewaySender will log messages like below.

[warning 2017/11/09 11:39:44.314 PST ln-1 <EventAllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=9; time stamp=1510362355433; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=Event Processor for GatewaySender_ny_30> tid=0x4c}]

Once the GatewaySender does connect to a remote GatewayReceiver, an info message like below is logged. This message contains the remote receiver's host and port as well as the number of failed attempts. Depending on what thread has been attempting to connect, it'll retry. For example, the AckReaderThread retries every second by default.

[info 2017/11/09 09:30:59.511 PST ln-1 <Event Processor for GatewaySender_ny_3> tid=0x4c] ny: Using Pooled Connection to 10.118.20.102:5382: Connection[10.118.20.102:5382]@439092143 after 1 failed connect attempts

or:

0x47] Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".
[severe[info 2017/11/09 11:1539:4544.931317 PST ln-1 <AckReaderThread for : Event<Event Processor for GatewaySender_ny_3>0> tid=0x820x47] GatewaySender "ny:" Usingcould Poolednot Connectionget to 10.118.20.102:5470: Connection[10.118.20.102:5470]@1533322301 after 100 failed connect attempts

GatewaySender Loses Connection to GatewayReceiver

When GatewaySender loses connection to a remote GatewayReceiver, messages like below are logged.

[warning 2017/11/09 11:13:54.045 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x82] Pool unexpected Socket closed connection=Pooled Connection to 10.118.20.102:5382: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
remote locator information for remote site "1".

The attached AlertNotificationListener will log alerts like:

AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=17; time stamp=1510364889713; message=Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".; userData={AlertLevel=warning, Member=ln-1, Thread=[warning 2017/11/09 11:13:55.155 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3>0 tid=0x82] ny: Could not connect.

The attached AlertNotificationListener will log alerts like:

0x47}]
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=2619; time stamp=15103644359611510364889715; message=Pool unexpected closed socket on server connection=Pooled Connection to 10.118.20.102:5181: Connection[DESTROYED]). Server unreachable: could not connect after 1 attemptsGatewaySender "ny" could not get remote locator information for remote site "1".; userData={AlertLevel=warningsevere, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_30 tid=0x820x47}]
GatewaySender Cannot Connect to GatewayReceiver

When a GatewaySender cannot connect to a remote GatewayReceiver, a message like below is logged.

[warning 2017/11/09 09:28:44.977 PST ln-1 <Event Processor for GatewaySender_ny_3> tid=0x4c] ny: Could not connect.

The attached AlertNotificationListener will log an alert like:

AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=9AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=29; time stamp=1510364437068; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event1510362355433; message=ny: Could not connect.; userData={AlertLevel=warning, Member=ln-1, Thread=Event Processor for GatewaySender_ny_3 tid=0x4c}]

Once the GatewaySender does connect to a remote GatewayReceiver, an info message like below is logged. This message contains the remote receiver's host and port as well as the number of failed attempts. Depending on what thread has been attempting to connect, it'll retry. For example, the AckReaderThread retries every second by default.

[info 2017/11/09 09:30:59.511 PST ln-1 <Event Processor for GatewaySender_ny_33> tid=0x82}]

Remote Locator Is Not Running

When the remote locator is not running, the local locator will log a message like below. This message will be logged for each remote locator configured the remote-locators property that is unreachable.

0x4c] ny: Using Pooled Connection to 10.118.20.102:5382: Connection[10.118.20.102:5382]@439092143 after 1 failed connect attempts

or:

[info[warning 2017/11/09 11:3215:4845.667931 PST locator-ln <WAN Locator Discovery Thread>ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x540x82] Locatorny: discoveryUsing taskPooled could not exchange locator informationConnection to 10.118.20.102[10332] with localhost[10331] after 6 retry attempts. Retrying in 10,000 ms.:5470: Connection[10.118.20.102:5470]@1533322301 after 100 failed connect attempts
Ongoing
GatewaySender Loses Connection to GatewayReceiver

When GatewaySender loses connection to a remote GatewayReceiver, messages like below are loggedAnd the GatewaySender will log messages like below.

[warning 2017/11/09 11:3913:4454.314045 PST ln-1 <Event<AckReaderThread for : Event Processor for GatewaySender_ny_0>3> tid=0x470x82] Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]". Pool unexpected Socket closed connection=Pooled Connection to 10.118.20.102:5382: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts
[warning[severe 2017/11/09 11:3913:4455.317155 PST ln-1 <Event <AckReaderThread for : Event Processor for GatewaySender_ny_0>3> tid=0x470x82] GatewaySender "ny": couldCould not get remote locator information for remote site "1"connect.

The attached AlertNotificationListener will log alerts like:

AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=1726; time stamp=15103648897131510364435961; message=Remote locator host port information for remote site "1" is not available in local locator "localhost[10332]".Pool unexpected closed socket on server connection=Pooled Connection to 10.118.20.102:5181: Connection[DESTROYED]). Server unreachable: could not connect after 1 attempts; userData={AlertLevel=warning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_03 tid=0x470x82}]
AllNotificationListener received notification[source=DistributedSystem(2); type=system.alert; sequence number=1929; time stamp=15103648897151510364437068; message=GatewaySender "ny": couldCould not get remote locator information for remote site "1"connect.; userData={AlertLevel=severewarning, Member=ln-1, Thread=AckReaderThread for : Event Processor for GatewaySender_ny_03 tid=0x470x82}]
Batch Exceptions

Exceptions that occur on the GatewayReceiver while processing an event in a batch cause a warning like below to be logged on the GatewayReceiver. The stack trace shows a BatchException70 wrapping the real exception. In this case, a CacheWriterException occurred.

...

Code Block
languagejava
[warning 2017/11/09 16:56:36.394 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x6a] Gateway Sender ParallelGatewaySender{id=ny,remoteDsId=1,isRunning =true} : Received ack for batch id 0 with one or more exceptions


[warning 2017/11/09 16:56:36.395 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x6a] A BatchException occurred processing events. Index of Array of Exception : 0
org.apache.geode.internal.cache.wan.BatchException70: Exception occurred while processing a batch on the receiver running on DistributedSystem with Id: 1, DistributedMember on which the receiver is running: 10.118.20.102(ny-1:13339)<v5>:1026
	at org.apache.geode.internal.cache.tier.sockets.command.GatewayReceiverCommand.cmdExecute(GatewayReceiverCommand.java:715)
	...
Caused by: org.apache.geode.cache.CacheWriterException: test cache writer exception
	at TestCacheWriter.process(TestCacheWriter.java:24)
	at TestCacheWriter.beforeCreate(TestCacheWriter.java:14)
	...


[warning 2017/11/09 16:56:36.407 PST ln-1 <AckReaderThread for : Event Processor for GatewaySender_ny_3> tid=0x6a] The event being processed when the BatchException occurred was:  SenderEventImpl[id=EventIDid=25bytes;threadID=0x10030|1;sequenceID=0;bucketId=48];action=0;operation=CREATE;region=/data;key=0;value=Trade[id=0; cusip=PVTL; shares=32; price=575.0191650390625];valueIsObject=1;numberOfParts=9;callbackArgument=GatewaySenderEventCallbackArgument [originalCallbackArg=null;originatingSenderId=2;recipientGatewayReceivers={1}];possibleDuplicate=false;creationTime=1510275395589;shadowKey= 161;timeStamp=1510275395587;acked=false;dispatched=false;bucketId=48]

...