Table of Contents
ConsistentCut is a distributed algorithm that splits WAL on 2 global areas - Before
and After
. It guarantees that every transaction committed Before
also will be committed Before
on every other node participated in the transaction. It means that an Ignite node nodes can safely recover itself themself to the consistent Before
state without any coordination with each other nodes.
The border between Before
and After
areas consists of two WAL records - ConsistentCutStartRecord
and ConsistentCutFinishRecord
. It guarantees that the Before
consists consists of:1. transactions committed before ConsistentCutStartRecord
and
ConsistentCutStartRecord
AND weren't included into ConsistentCutFinishRecord#after()
...
.
...
ConsistentCutStartRecord
...
ConsistentCutFinishRecord
AND were included into ConsistentCutFinishRecord#before()
.On the picture below the Before
area consist of transactions colored to yellow, while After
is green.
Code Block | ||||
---|---|---|---|---|
| ||||
/** */ public class ConsistentCutStartRecord extends WALRecord { /** Marker that inits Consistent Cut ID. */ private final ConsistentCutMarkerUUID markercutId; } /** */ public class ConsistentCutFinishRecord extends WALRecord { /** Consistent Cut ID. */ private final UUID cutId; /** * Collections of transactions committed BEFORE. */ private final Set<GridCacheVersion> before; /** * Collections of transactions committed AFTER. */ private final Set<GridCacheVersion> after; } |
Picture bellow illustrates steps of the algorithm on single node:
lastFinishedCutId
holds previous ConsistentCutId
, or null.DistributedProcess
with special message a DistributedProcess with discovery message SnapshotOperationRequest
that holds new ConsistentCutMarker
ConsistentCutId
(goal is to notify every node in a cluster about running incremental snapshot). ConsistentCutMarker
by discoverythe SnapshotOperationRequest#ConsistentCutId
by DiscoverySPI (by the DistributedProcess).ConsistentCutMarker
by transaction message ( ConsistentCutAwareMessage#ConsistentCutId
by CommunicationSPI (by transaction messages - Prepare, Finish).ConsistentCutId
it starts local ConsistentCut: ConsistentCut
!= null) or finished (lastFinishedCutId
== id) for this idtopVersion
with received in marker. Skip if it is different.ConsistentCut
futurecommittingTx,
goal is to track COMMITTING+ transactions, that aren't part of IgniteTxManager#activeTx
ConsistentCutMarker
.ConsistentCutAwareMessage#topVer
with local node order:ConsistentCut
future. ConsistentCut != null
wraps outgoing messages to ConsistentCutAwareMessage
. It contains info:ConsistentCutId
(to start ConsistentCut on remote node, if not yet).txCutId
equals to null then transaction starts committing Before
Consistent Cut started, otherwise After
.ConsistentCutAwareMessage
that makes transaction committed (FinishRequest for 2PC, PrepareResponse for 1PC) sets tx#txCutId = message#txCutId
.ConsistentCut
future.removedActiveTxs
(This collection doesn't remove transactions unlike IgniteTxManager#activeTx
does).ConsistentCutStartRecord
to WAL ConsistentCutMarker
ConsistentCutId
.IgniteTxManager#activeTx
and committingTxs
.DistributedProcess
is alive every node signs output transaction messages:ConsistentCutMarker
(to trigger ConsistentCut
on remote node, if not yet).ConsistentCutMarker
(to trigger...) and transaction ConsistentCutMarker
(to notify nodes which side of cut this transaction belongs to).ConsistentCutMarker
on node that commits first.ConsistentCutMarker
and prepares before
, after
collections:before
sideafter
sideIgniteTxManager#activeTx
. Set listeners on those tx#finishFuture
.tx#status == ACTIVE
. It's guaranteed that such transactions belongs After side.removedActiveTxs
(contains transactions that are might be cleaned from IgniteTxManager#activeTx
). Set listeners on those tx#finishFuture
.removedActiveTxs
to null. We don't care of txs concurrently added to removedActiveTxs
, they just don't land into "before" or "after" set and will be excluded from recovery.removedActiveTxs
if ConsistentCut != null
and removedActiveTxs != null
:removedActiveTxs
right before it is removed from IgniteTxManager#activeTx
.tx#txCutId
equals to local, then put transaction into after, otherwise put into before.ConsistentCutFinishRecord
into WAL with the collections committingTxs
.ConsistentCut
futureConsistentCut
future becomes future becomes null....
tx.finalizationStatus
== RECOVERY_FINISH).Every ignite nodes tracks current ConsistentCutMarker
:
Code Block | ||||
---|---|---|---|---|
| ||||
class ConsistentCutMarker {
UUID id;
AffinityTopologyVersion topVer;
} |
id
is just a unique ConsistentCut
ID (is assigned on the node initiator).
topVer
is topology version on node initiator before Incremental Snapshot starts.
...
Ignite transaction protocol includes multiple messages. But only some of them affects meaningful (relating to the algorithm) that change state of transactions (PREPARED, COMMITTED):
...
Those messages are wrapped in ConsistentCutMarkerMessage
ConsistentCutAwareMessage
that is prepared right before sending message on other node. They used the current ConsistentCutMarker
for setting the marker.
Code Block | ||||
---|---|---|---|---|
| ||||
class ConsistentCutMarkerMessage {
Message msg;
ConsistentCutMarker marker;
} |
ConsistentCutId
.
Also some messages require to be signed with combine with additional ConsistentCutMarker
ConsistentCutId
to check it them on primary/backup node.:
GridNearTxFinishRequest / GridDhtTxFinishRequest
GridNearTxPrepareResponse / GridDhtTxPrepareResponse
(for 1PC algorithm).Those messages are wrapped in ConsistentCutMarkerFinishMessage
filled with txCutId
that is prepared right before transaction starts committing on first committing node. They used the current ConsistentCutMarker
for setting the txMarker
. txMarker
can be null, if ConsistentCutId
for this setting. If current ConsistentCutId
is not null, then transaction starts committing before ConsistentCut starts.after ConsistentCut started and it means that this transaction belongs the After
side.
Code Block | ||||
---|---|---|---|---|
| ||||
class ConsistentCutMarkerFinishMessage extends ConsistentCutMarkerMessage {ConsistentCutAwareMessage { /** Original transaction message. */ Message msg; /** Consistent Cut ID. */ UUID cutId; /** Consistent Cut ID after which transaction committed. */ @Nullable ConsistentCutMarkerUUID txMarker; } |
There are 2 records: ConsistentCutStartRecord
for Start event and ConsistentCutFinishRecord
for Finish event.
ConsistentCutStartRecord
: record is written to WAL in moment when ConsistentCut starts on a local node. It helps to limit amout of active transactions to check. But there is no strict guarantee for all transactions belonged to the BEFORE side to be committed before ConsistentCutStartRecord, and vice versa. This is the reason for having ConsistentCutFinishRecord.ConsistentCutFinishRecord
: This record is written to WAL after Consistent Cut stopped analyzing active transactions and storing them in a particular bucket (BEFORE or AFTER)....
txCutId;
/** Cluster topology version on which Consistent Cut started. */
long topVer;
} |
A new field added to IgniteInternalTx
Code Block | |||||
---|---|---|---|---|---|
| |||||
class IgniteInternalTx { /** */ public class ConsistentCutStartRecord extends WALRecord { / * @param ID of {@link ConsistentCut} AFTER which this transaction was committed, {@code null} if transaction * committed BEFORE. */ public void cutId(@Nullable UUID id); } |
Code Block | ||||
---|---|---|---|---|
| ||||
// Class is responsible for managing all stuff related to Consistent Cut. It's an entrypoint for transaction threads to check running consistent cut.
class ConsistentCutManager extends GridCacheSharedManagerAdapter {
// Current Consistent Cut. All transactions threads wraps outgoing messages if this field is not null. */
volatile @Nullable ConsistentCut cut;
// Entrypoint for handling received new Consistent Cut ID.
void handleConsistentCutId(UUID id);
} |
Code Block | ||||
---|---|---|---|---|
| ||||
class ConsistentCut extends GridFutureAdapter<WALPointer> { Set<GridCacheVersion> beforeCut; Set<GridCacheVersion> afterCut; Set<IgniteInternalFuture<IgniteInternalTx>> removedActive; }** Marker that inits Consistent Cut. */ private final ConsistentCutMarker marker; } /** */ public class ConsistentCutFinishRecord extends WALRecord { /** * Collections of TXs committed BEFORE the ConsistentCut (sent - received). */ private final Set<GridCacheVersion> before; /** * Collections of TXs committed AFTER the ConsistentCut (exclude). */ private final Set<GridCacheVersion> after; } |