Avoid the queuing of dropped events by the primary gateway sender when the gateway sender is stopped

To be Reviewed By: July 9th16th, 2020

Authors: Alberto Gomez (alberto.gomez@est.tech)

Status: Draft  Draft | Discussion | Active | Dropped | Superseded

...

As described above, dropped events in the primary gateway sender are stored in a member variable. It is out of the scope of this RFC to change how those events are stored.

Solution 1 (original proposal, deprecated)

The solution proposes to change the primary gateway sender so that it does not store dropped events when it is stopped explicitly (not while starting). The reason is that these events could never end in the queue of any secondary gateway sender and will use memory unnecessarily.

In order to do so, it is proposed to add a new boolean member variable (mustQueueDroppedEvents) to the AbstractGatewaySender that will tell if the primary gateway sender must store dropped events or not.

• This member variable must mustQueueDroppedEvents must be set to false (do not store dropped events) in the primary and secondary gateway sender instances:
  • At gateway sender creation if the --manual-start option was used.
  • Right after stopping the gateway sender using the gfsh stop gateway sender command.
• This member variable must mustQueueDroppedEvents must be set to true (store dropped events) in the primary and secondary gateway sender instances:
  • At gateway sender creation if the --manual-start option was not used or set to false.
  • Right before a start gateway sender gfsh command is executed.

The start gateway sender and stop gateway sender gfsh commands would be modified as follows in order to set the member value of the gateway sender accordinglyvalue of mustQueueDroppedEvents as follows:

• Code will be added at the end of the current stop gateway sender gfsh command which will set mustQueueDroppedEvents to false the new member variable (after the gateway senders have been stopped).
• Code will be added at the beginning of the current start gateway sender gfsh command which will set mustQueueDroppedEvents to true the new member variable (before the gateway senders are started).

In case the GatewaySender Java API is used to start/stop gateway senders, in order to get the new behavior, given that the scope of the start/stop methods is the VM on which it is invoked, it will be necessary to set the mustQueueDroppedEvents accordingly (to true before starting all sender instances and to false after stopping all sender instances) on every VM. To set the value of the variable, the GatewaySender interface will offer the following new method: setMustQueueDroppedEvents(boolean mustQueue). If the new method is not used, the legacy behavior will prevail except if the gateway sender is started manually in which case dropped events will not be queued.

A draft PR of the solution can be found here: https://github.com/apache/geode/pull/5348

Changes and Additions to Public Interfaces

No changes to public interfaces are proposedTwo new methods: setMustQueueDroppedEvents(boolean) and mustQueueDroppedEvents() will be added to the GatewaySender public interface.

Performance Impact

As the proposal implies changing the implementation of the gfsh start gateway sender and  the stop gateway sender gfsh commands to be done in two steps, these commands may be slightly slower but not significantlyalthough not significantly.

Backwards Compatibility and Upgrade Path

The proposal does not affect the rolling upgrade and has not impacts in the regular rolling upgrade process.

Solution 2 (new proposal, agreed after discussions on this RFC )

The solution consists of, instead of storing dropped events in `tmpDroppedEvents` to later send batch removal messages when the primary gateway sender is not started, try to send the batch removal message when the event to be dropped is received. That way, when the sender is stopped for a long time and there are events coming, the memory of the `AbstractGatewaySender` will not grow with entries in the `tmpDroppedEvents` member.

In order to send the batch removal message directly, the `eventProcessor` for the `AbstractGatewaySender` must have been created. If it is not yet created because the sender was created with manual start set to true, while receiving events to be dropped, they will be stored in `tmpDroppedEvents` as there is no other choice. Nevertheless, in order to consume less memory, the event stored could be a simplified event containing only the necessary information to handle it.

A draft PR of the solution can be found here: https://github.com/apache/geode/pull/5486

Changes and Additions to Public Interfaces

No changes.

Performance Impact

No impacts foreseen.

Backwards Compatibility and Upgrade Path

The proposal does not affect the rolling upgrade and has not impacts in the regular rolling upgrade process.

...