Page History

...

[2] Updates counter based rebalancing.
In the second case which is relevant when baseline is not changed rebalancing is used to restore the consistency of the data if the node was absent for some time in the topology under the load. This might happen if node is crashed for example.

When the node re-joins the topology, it will have stale data, and the partition update counter will be analyzed to determine how much the partitions have “lagged” from other copies.
Depending on how strong the partitions are lagging behind, the second type of rebalance can use the write-ahead log as a data source for updates history. In this case, the rebalancing can be performed without complete reloading of the partition and is called “historical”.
Another scenario for the occurrence of [2] is the activation of the grid having nodes in topology with stale data.

Technically rebalancing is the exchange of supply and demand messages between nodes. A demand message is sent from node where partition has stale data or missing to node with actual data. Supply message containing data is send in response to demand message. After processing a supply message next demand message is send and so on until nothing more to rebalance. We will look at this later in details.

Update counter structure

Currently three different implementations of update counter logic exist: for persistent mode, for in-memory mode and for MVCC mode.

In this article we only look at former. The main difference with latter is ability to track updates applied out of order.

Each update to a partition receives a sequential number for a purpose of ensuring data consistency between copies. A special structure called partition update counter is used to assign and increment update counters.

...

/** High watermark. */
protected final AtomicLong reserveCntr = new AtomicLong();
High watermark or reservation counter is incrementincremented for each pending update, which even not guaranteed to succeed. Reservation counter is assigned during tx prepare phase.

...

Where is an important invariant which can be derived concluded from statements above:

HWM >= LWM

...