ConsistentCut splits WAL on 2 global areas - Before
and After
. It guarantees that every transaction committed Before
also will be committed Before
on every other node participated in the transaction. It means that an Ignite nodes can safely recover themself to the consistent Before
state without any coordination with each other.
The border between Before
and After
areas consists of two WAL records - ConsistentCutStartRecord
and ConsistentCutFinishRecord
. It guarantees that the Before
consists of:
ConsistentCutStartRecord
AND weren't included into ConsistentCutFinishRecord#after().
ConsistentCutStartRecord
and ConsistentCutFinishRecord
AND were included into ConsistentCutFinishRecord#before()
.On the picture below the Before
area consist of transactions colored to yellow, while After
is green.
/** */ public class ConsistentCutStartRecord extends WALRecord { /** Consistent Cut ID. */ private final UUID cutId; } /** */ public class ConsistentCutFinishRecord extends WALRecord { /** Consistent Cut ID. */ private final UUID cutId; /** * Collections of transactions committed BEFORE. */ private final Set<GridCacheVersion> before; /** * Collections of transactions committed AFTER. */ private final Set<GridCacheVersion> after; }
Picture bellow illustrates steps of the algorithm on single node:
lastFinishedCutId
holds previous ConsistentCutId
, or null.DistributedProcess
with message SnapshotOperationRequest
that holds new ConsistentCutId
(goal is to notify every node in a cluster about running incremental snapshot). topVer
on which ConsistentCut started.SnapshotOperationRequest#ConsistentCutId
by DiscoverySPI (by the DistributedProcess).ConsistentCutAwareMessage
#ConsistentCutId
by CommunicationSPI (by transaction messages - Prepare, Finish).ConsistentCutId
, every node: lastFinishedCutId
== id) for this id, skip if it has.ConsistentCutAwareMessage#
topVer
with local node order:topVer
it means the node joined after ConsistentCut started. Skip start ConsistentCut on this node.ConsistentCut
future.committingTx
(Goal is to not miss transactions. This collection doesn't remove transactions unlike IgniteTxManager#activeTx
does).ConsistentCutAwareMessage
(contains ConsistentCutId
).ConsistentCutStartRecord
to WAL with the received ConsistentCutId
.IgniteTxManager#activeTx.
Set listeners on those tx#finishFuture.
After
side.committingTxs
(contains transactions that are might be cleaned from IgniteTxManager#activeTx
).
Set listeners on those tx#finishFuture.
committingTxs
to null.
DistributedProcess
is running every node wraps outgoing transaction messages (Prepare, Finish) to ConsistentCutAwareMessage (
if transaction has not committed yet on sender node)
or ConsistentCutAwareTxFinishMessage (
if transaction has committed on a sender node)
. Messages contain info:ConsistentCutId
(to trigger ConsistentCut
on remote node, if not yet).ConsistentCutAwareTxFinishMessage
messages contains additionally txCutId
.
It set on the node that commits first (if null then transaction starts committing Before
Consistent Cut started, otherwise After
):committingTxs
if ConsistentCut is running and committingTxs
is not null:committingTxs
right before it is removed from IgniteTxManager#activeTx.
ConsistentCutAwareTxFinishMessage
Ignite marks the related transaction with message#txCutId.
check If transaction state is UNKNOWN or status is RECOVERY_FINISH, then complete ConsistentCut with exception.
topVer
, then put it into after
(topology changed after ConsistentCut started).txCutId
equals to local, then put transaction into after
, otherwise put into before
.ConsistentCutFinishRecord
into WAL with the collections ( before
, after
). ConsistentCut
future.ConsistentCut,
on every node:lastFinishedCutId
with the current id.
ConsistentCut
future becomes null.Consistent Cut is such cut that correctly finished on all baseline nodes - ConsistentCutStartRecord
and ConsistentCutFinishRecord
are written.
"Inconsistent" Cut is such a cut when one or more baseline nodes hasn't wrote ConsistentCutFinishRecord
. It's possible in cases:
tx.finalizationStatus
== RECOVERY_FINISH).On the picture below on left side is a diagram of sending transaction messages. Before sending message it checks whether cut is running. If it is then wraps the message, otherwise send ordinary message (PrepareRequest
in example).
Ignite transaction protocol includes multiple messages. But only some of them affects meaningful (relating to the algorithm) that change state of transactions (PREPARED, COMMITTED):
GridNearTxPrepareRequest / GridDhtTxPrepareRequest
GridNearTxPrepareResponse / GridDhtTxPrepareResponse
GridNearTxFinishRequest / GridDhtTxFinishRequest
Those messages are wrapped in ConsistentCutAwareMessage
that is prepared right before sending message on other node. They used the current ConsistentCutId
.
class ConsistentCutAwareMessage { /** Original transaction message. */ Message msg; /** Consistent Cut ID. */ UUID cutId; /** Cluster topology version on which Consistent Cut started. */ long topVer; }
Also some messages require to be combine with additional ConsistentCutId
to check it them on primary/backup node.
GridNearTxFinishRequest / GridDhtTxFinishRequest
GridNearTxPrepareResponse / GridDhtTxPrepareResponse
(for 1PC algorithm).Those messages are wrapped in ConsistentCutAwareTxFinishMessage
that is prepared right before transaction starts committing on first committing node. They used the current ConsistentCutId
for this setting. If current ConsistentCutId
is not null, then transaction starts committing after ConsistentCut started and it means that this transaction belongs the After
side.
class ConsistentCutAwareTransactionFinishMessage extends ConsistentCutAwareMessage { /** Consistent Cut ID after which transaction committed. */ @Nullable UUID txCutId; }