...
Picture bellow illustrates steps of the algorithm on single node:
Image Modified
- Initial state:
- No concurrent ConsistentCut process is running.
lastFinishedCutId
lastFinishedCutId
holds previous ConsistentCutId
, or null.
- User starts a command for creating new incremental snapshot:
- Ignite node inits
a - a DistributedProcess with discovery message
SnapshotOperationRequest
that holds - that holds new
ConsistentCutId
- (goal is to notify every node in a cluster about running incremental snapshot).
- DistributedProcess store the topology version topVer on which ConsistentCut started.
- Process of creation of incremental snapshot consistent cut can be started by two events (what will happen earlier):
- Receive the
SnapshotOperationRequest#ConsistentCutId
by DiscoverySPI (by the DistributedProcess). - Receive the
ConsistentCutAwareMessage
#ConsistentCutId
by -
ConsistentCutAwareMessage#ConsistentCutId
by CommunicationSPI (by transaction messages - Prepare, Finish).
- On receiving the
ConsistentCutId
, every node: - Checks whether ConsistentCut has already started (ConsistentCut is running) or finished (lastFinishedCutId
- == id) for this id, skip if it has.
- In case ConsistentCut is inited by CommunicationSPI then compare the
ConsistentCutAwareMessage#
topVer
with local node order:- Local node order equals to new topVer on the moment when node joined to a cluster.
- If the order is higher than ConsistentCut
topVer
it means the node joined after ConsistentCut started. Skip start ConsistentCut on this node.
- On baseline nodes:
- In the message thread atomically inits ConsistentCut:
- creates new ConsistentCut
future- future.
- creates
committingTx
(Goal is to not miss transactions. - empty collection removedActiveTxs (This collection doesn't remove transactions
unlike - unlike IgniteTxManager#activeTx does
).starts wraps outgoing messages to ConsistentCutAwareMessage
(contains ConsistentCutId
- ).
- In the background thread:
- Writes a
ConsistentCutStartRecord
- ConsistentCutStartRecord to WAL
with the - with the received ConsistentCutId.
- Creates a copy (weakly-consistent)
of - of IgniteTxManager#activeTx.
Set - Set listeners on
those - those tx#finishFuture.
- For optimization it's safely exclude transactions that tx#status == ACTIVE. It's guaranteed that such transactions belongs After side.
- Creates a copy
of committingTxs
- of removedActiveTxs (contains transactions that are might be cleaned
from - from IgniteTxManager#activeTx).
Set - Set listeners on
those - those tx#finishFuture.
Set committingTxs
- Set removedActiveTxs to null.
- While the
DistributedProcess
is running every node wraps outgoing transaction messages (Prepare, Finish) to ConsistentCutAwareMessage (
if transaction has not committed yet on sender node)
or ConsistentCutAwareTxFinishMessage (
if transaction has committed on a sender node)
. Messages contain info:ConsistentCutId
(to trigger ConsistentCut
on remote node, if not yet).ConsistentCutAwareTxFinishMessage
messages contains additionally txCutId
.
It set on the node that commits first (if null then transaction starts committing Before
Consistent Cut started, otherwise After
):- For 2PC it is an originated node.
- For 1PC it is a backup node.
Fills committingTxs
if ConsistentCut is running and committingTxs
- We don't care of txs concurrently added to removedActiveTxs, they just don't land into "before" or "after" set and will be excluded from recovery.
- Fills removedActiveTxs if ConsistentCut is running and removedActiveTxs is not null:
- Every transaction is added into
committingTxs
- removedActiveTxsright before it is removed from IgniteTxManager#activeTx
.For every receiving ConsistentCutAwareTxFinishMessage
Ignite marks the related transaction with message#txCutId
before it starts handling the received message- .
- For every listening transaction,
the callback - the callback is called when transaction finished:
- check If transaction state is UNKNOWN or status is RECOVERY_FINISH, then complete ConsistentCut with exception.
- If transaction mapped to a higher topology version than
ConsistentCut - ConsistentCut topVer, then put it into
after
(topology changed - after
ConsistentCut started)- .
- if
tx#txCutId
equals - tx#txCutId equals to local, then put transaction
into - into after, otherwise put into before.
- After every listening transaction finished:
- Writes a
ConsistentCutFinishRecord
- ConsistentCutFinishRecord into WAL with
the collections - the collections ( before, after ).
- Completes ConsistentCut future.
- On non-baseline nodes (clients, non-baseline servers):
- In case ConsistentCut
future.- Note, that it continues to wrap messages even after local ConsistentCut finish.
- is inited by CommunicationSPI then compare the ConsistentCutAwareMessage#topVer with local node order:
- Local node order equals to new topVer on the moment when node joined to a cluster.
- If the order is higher than ConsistentCut topVer it means the node joined after ConsistentCut started. Skip start ConsistentCut on this node.
- In the message thread creates completed ConsistentCut future.
- On all nodes (clients, non-baseline, baseline):
- While ConsistentCut != null wraps outgoing messages to ConsistentCutAwareMessage. It contains info:
- ConsistentCutId (to start ConsistentCut on remote node, if not yet).
- Some messages sets additional field txCutId. It set on the node that commits first:
- For 2PC it is an originated node.
- For 1PC it is a backup node.
- If txCutId set to null then transaction starts committing Before Consistent Cut started, otherwise After.
- For receiving ConsistentCutAwareMessage that make transaction committed (FinishRequest for 2PC, PrepareResponse for 1PC) Ignite set tx#txCutId = message#txCutId before it starts handling the received message.
- After ConsistentCut future
After ConsistentCut - finish, DistributeProcess automatically notifies a node-initiator about local procedure has finished.
- After all nodes finished ConsistentCut, on on every node:
Updates - Updates lastFinishedCutId with the current id.
ConsistentCut
future becomes - ConsistentCut future becomes null.
- Stops signing outgoing transaction messages.
- Node initiator checks that every node completes correctly.
- If any node complete exceptionally - complete Incremental Snapshot with exception.
Consistent and inconsistent Cuts
...
- any errors appeared during processing local Cut.
- if a transaction is recovered with transaction recovery protocol (
tx.finalizationStatus
== RECOVERY_FINISH). - if transaction finished in UNKNOWN state.
- topology change.
Wrapping messages
On the picture below on left side is a diagram of sending transaction messages. Before sending message it checks whether cut is running. If it is then wraps the message, otherwise send ordinary message (PrepareRequest
in example).
Image Removed
Ignite transaction protocol includes multiple messages. But only some of them affects meaningful (relating to the algorithm) that change state of transactions (PREPARED, COMMITTED):
...
Those messages are wrapped in ConsistentCutAwareMessage
that is prepared right before sending message on other node. They used the current ConsistentCutId
.
...
language | java |
---|
title | ConsistentCutAwareMessage |
---|
...
.
...
Also some messages require to be combine with additional ConsistentCutId
to check it them on primary/backup node.:
GridNearTxFinishRequest / GridDhtTxFinishRequest
GridNearTxPrepareResponse / GridDhtTxPrepareResponse
(for 1PC algorithm).
Those messages are wrapped in ConsistentCutAwareTxFinishMessage
filled with txCutId
that is prepared right before transaction starts committing on first committing node. They used the current ConsistentCutId
for this setting. If current ConsistentCutId
is not null, then transaction starts committing after ConsistentCut started and it means that this transaction belongs the After
side.
...
Code Block |
---|
language | java |
---|
title | ConsistentCutAwareTxFinishMessageConsistentCutAwareMessage |
---|
|
class ConsistentCutAwareTransactionFinishMessage extends ConsistentCutAwareMessage {
ConsistentCutAwareMessage {
/** Original transaction message. */
Message msg;
/** Consistent Cut ID. */
UUID cutId;
/** Consistent Cut ID after which transaction committed. */
@Nullable UUID txCutId;
/** Cluster topology version on which Consistent Cut started. */
long topVer;
} |
Links
- ON DISTRIBUTED SNAPSHOTS, Ten H. LAI and Tao H. YANG, 29 May 1987