Page History

...

Technically rebalancing is the exchange of supply and demand messages between nodes. A demand message is sent from the node where partition has stale data to the node with that has the actual data. A supply message containing data is sent in response to the demand message. After processing the  supply message, the next demand message is sent, and so on, until there is nothing more to rebalance. We will look at this later in more detail.

...

Currently, different implementations of update counter logic exist for persistent mode and for in-memory mode.In this article we will only look at former. The main difference with latter is ability to track updates that are  applied out of order.

Each update to a partition receives a sequential number for ensuring data consistency between copies. A special structure called partition update counter is used to assign and increment update counters.

Partition A partition update counter has the following structure:

/** Low watermark. */
private final AtomicLong cntr = new AtomicLong();

...

Out of order updates range is held in the sequence only if an update with lower range is missing. For example, LWM=5, HWM=9, seq=(6,2) means updates 7, 8 were applied out of order and updates 6, 9 are still not applied.

There Note that there is an important invariant which can be derived from the statements above:

...

Let's imaging that out of order updates are not tracked and LWM doesn't existsexist. In this case, if the primary node fails before applying the update from Tx1, we will end up with update counter=5 and will lose missed updates after the node rejoins to the topology , because rebalancing will not start due to equal counters.

...

There are no guarantees that updates with already assigned counters will ever be applied.For example, if during the prepare phase some mapped nodes exit, then the transaction will be rolled back, ; or it is also possible for a node to crash during commit.

...

In the scenario on figure 3, Tx2 is committed before Tx1 had the chance to prepare, and the primary node fails.

As a part of the PME protocol on node left , when a node leaves, all existing gaps are removed after waiting for the completion of all active transactions completion. This  This happens in org.apache.ignite.internal.processors.cache.PartitionUpdateCounter#finalizeUpdateCounters.

...

It is important to ensure the atomicity when switching the primary node for partition on all grid nodes.  Otherwise,   otherwise the HWM >= LWM invariant will break, which ensures that if at least one stable copy is available, everyone else can restore its integrity from it.

...