Page History

...

[2] Counter based re-balance
In the second case, which is relevant only for the enabled persistence mode, re-balance is used to restore the consistency of the data if the node was absent for some time in the topology under the load.
When the node re-joins the topology, the partition update counter will be analyzed to determine how much the partition has “lagged” from other copies.
Depending on how much the partition is lagging behind, the second type of rebalance can use the write-ahead log as a data source. In this case, the re-balance can be performed without a complete reloading of the partition and is called “historical”.
Another scenario for the occurrence of [2] is the activation of the grid, where some nodes may have stale data.
Re-balance is the exchange of supply and demand messages between nodes. A demand message is sent from node where partition has stale data to node with actual data. Supply messages is send in response to demand message.

Update counter structure

Update counter = [lwm, out-of-order updates tracking set, hwm]

Consider an example of counter status:

lwm = 5, ooo = (6.1) (8.2), hwm = 11

Lwm - low water mark - number of the last applied sequential update.
That is, in this partition all updates from 1 to 5 are present.
Partial - applied out-of-order updates. In the example in the partition there are updates 7,9,10
But skipped updates 6, 8, 11
Hwm = 11 indicates the number of updates made to the partition, some of which are still “in flight”.
highestApplied = 9 the deduced property - an update with the maximum serial number

Hwm is wound only on the primary node to ensure the monotonicity of the counter lwm <= hwm.

Due to the fact that it is possible to reorder transactions into one partition, updates can be applied in any order, which is why omissions are possible in the sequence of updates. To do this, a special structure has been introduced into the counter for tracking updates applied in random order.

Update counters are used to determine the state when the backup is behind the primaries. A lag can occur, for example, if the node went out under load and while there were no updates to one of the partitions for which it is the owner. If for each partition the number of updates is the same, then the partitions are identical.
If on one of the partitions the counter lags by k, then it means you need to use k lagging updates to synchronize the partitions. Only relevant for persistent mode.
For in-memory, the partition is always restored from scratch.
Thus, it is extremely important to correctly track update counters, since it depends on them whether a rebalance will be launched to restore the integrity of partitions [2].