Page History

...

Because transaction commits can reorder it's possible to have some updates applied out of order, as show by the example above. The out of order updates sequence is used for tracking such updatesthem.

Let's imaging out of order updates are not tracked and LWM doesn't exists. If primary node fails before applying update from Tx1 we end up with update counter=5 and will lose missed updates after node rejoins to the topology, because rebalancing will not start due to equal counters.

Restoring consistency

The LWM value is used to restore consistency by rebalancing calculate missing data when outdated node joins topology. In this scenario counter based rebalancing is used to achieve consistency.

On joining Joining node sends LWM for each partition as a part of partition map exchange protocol. Coordinator collects all LWMs and determines nodes with highest LWM , say maxLWMfor each partition. These nodes are elected as data suppliers for a given partition.

If coordinator detects lagging node it sends a partition map with partition state changed to MOVING , triggering a rebalance on correspondingjoining node. The missing number of update on each node is between it's LWM and maxLWMif updates for each partition is calculated as a difference between highest known LWM and LWM from joining node.

The simplified rebalancing flow is illustratred on fugure:

...