To be Reviewed By: April 1st, 2020

Authors: Alberto Gomez (alberto.gomez@est.tech)

Status: Draft | Discussion | Active | Dropped | Superseded

Superseded by: N/A

Related: N/A

Problem

Gateway senders deliver events to gateway receivers over batches. Batches are filled with events retrieved from queues and then sent whenever the maximum configured size for the batch is reached or when it is due time according to configuration. With this mechanism, events belonging to the same transaction may be sent over different batches.

If there is a network split in a WAN deployment when the last batch received from a sender does not contain all the events for one or more transactions, the data in the receiving side will be inconsistent while the network split remains. Inconsistent data due to applying an incomplete number of the operations for a transaction could be very problematic for clients.

Anti-Goals

This proposal does not intend to replicate transactions using gateway senders and receivers. It just aims at making sure that transaction events are delivered atomically by gateway senders to gateway receivers.

It is also assumed that the events to be delivered atomically must be so by means of the same gateway sender. Therefore, the deployments that require this feature should configure replication such that the regions to which the data in the transactions belong must be collocated and must have the same gateway senders. Additionally, if serial gateway senders are used, they must be configured with just one dispatcher thread given that it would be impossible to fulfill the requirement if each event of a transaction is handled by a different dispatcher thread.

Solution

In order for the gateway senders to deliver batches that contain all the events for the transactions the following is proposed:

• After transaction commit and before events are sent to the gateway sender, mark the last event of the transaction (a new boolean attribute “lastEventInTransaction” may be added to GatewaySenderEventImpl)
  
• In the process of creating a batch, the gateway sender must keep track of all the transactions involved in the events it is putting in the batch.
• Once a batch is due to be sent (due to maximum configured size is reached or time), the gateway sender must check:
  • If  for every transaction to which events on the batch belong, the last event has been received, then the batch may be sent.
  • Otherwise, two options are proposed:
    • a) Keep reading events from the queue and put them in the batch until the above condition is fulfilled. Then send the batch. This option has the problem that the size of batches could grow a lot beyond the maximum size in situations of big transactions and high number of them.
    • b) Go to the queue and get selectively the events for the transactions in the batch that do not have all the events. Once these events are added to the batch, send it. This option is preferred over option a) because it should avoid the problem of batches much bigger than the maximum size configured.

Given that this behavior requires a given configuration of replication as mentioned above, it is proposed to make it configurable by means of a new parameter to the "create gateway sender" command (--group-transaction-events) and that is not activated by default.

Other solutions explored that have been discarded are:

• Send a new type of event inside batches -event transactions events, that will contain all the events of a given transaction. This solution, does not seem to fit well with the current code base as events are triggered on a region basis while transactions could contain events from several regions. The changes in the code required seemed to be much greater than the ones with the preferred solution.
• Make sure in the gateway receiver that events for a transaction are only applied if all the batches where they have been delivered, have been received. This solution seemed to be more complex than the one preferred and did not provide extra benefits.

Changes and Additions to Public Interfaces

A new method to GatewaySenderFactory, setGroupTransactionEvents() to create gateway senders with support for this new feature must be added.

Performance Impact

The performance of gateway senders may be impacted as there will be an extra cost of keeping track of transactions in batches and also of retrieving selectively events from the queue (if this option is selected).

Backwards Compatibility and Upgrade Path

The gateway senders will behave as before if the new parameter --group-transaction-events is not used.

Versioned support for GatewaySenderEventImpl must be added due to the new field added.

Prior Art

As said above, with the current behavior of gateway senders, on the event of network splits, clients will have to deal with data inconsistencies provoked by events not delivered atomically for the time the network split is in place.

FAQ

Answers to questions you’ve commonly been asked after requesting comments for this proposal.

Errata

What are minor adjustments that had to be made to the proposal since it was approved?