Page History

...

Paper [1] defines a distributed snapshot algorithm. It uses some definitions, let's describe them in terms of Ignite:

1. LocalState - set of committed transactions prior to a moment (WAL).
2. Message - transaction message (...FinishRequest for 2PC, ...PrepareResponse for 1PC)
   1. it is considered sent after
   changing LocalState
   1. all transaction DataRecords are written into WAL on sending node
   , and it changes LocalState on receiving node after delivering
   1. In Ignite such messages are FinishRequest for 2PC, PrepareResponse for 1PC.
   Channel - unidirectional connection between two Ignite nodes, where messages is sent (Communication SPI)
   1. .
   2. it is considered received after all transaction DataRecords are written into WAL on receiving node. .
3. ChannelState - for single channel it's a set of messages that was sent (changed LocalState of sending node), but not received yet (hasn't changed LocalState of receiving node).
   1. In Ignite we can think that incoming ChannelState is represented by active transactions in PREPARING+ state (the Message might be sent by other nodes, but isn't received yet).
4. GlobalSnapshot - set consist of (LocalState and ChannelState for every incoming channel) for every node.In Ignite the snapshot is represented IncrementalSnapshot - on Ignite node it is represented with 2 WAL records (ConsistentCutStartRecord commits the LocalState, ConsistentCutFinishRecord describes the ChannelState). It guarantees that every node in cluster includes in the snapshot:
   1. transactions physically committed before ConsistentCutStartRecord and weren't included into ConsistentCutFinishRecord#after();
   2. transactions physically committed between ConsistentCutStartRecord and ConsistentCutFinishRecord and were included into ConsistentCutFinishRecord#before().
5. Marker - mark that piggy backs on the Message, and notifies a node about running snapshot.
   1. In Ignite ConsistentCutMarkerMessage is used

      After IS start and before finish all PrepareRequest, FinishRequest are wrapper by ConsistentCutMarkerMessage instead of regular Message. This is done to notify target node ⁣⁣via communication channel about running IS.

In terms of Ignite, there are additional definitions:

1. Consistent Cut - Successful attempt of creating Global SnapshotIncrementalSnapshot.
2. Inconsistent Cut - Failed attempt of creating a Global SnapshotIncrementalSnapshot, due to inability to correctly describe a ChannelState.

...

1. Initial state:
   1. Ignite WAL are in consistent state relatively to previous full or incremental snapshot.
   2. Every Ignite node has local ConsistentCut future equals to null (node is WHITE).
   3. Empty collection committingTxs (Set<GridCacheVersion>) that goal is to track COMMITTING+ transactions, that aren't part of IgniteTxManager#activeTx.
2. Ignite node inites inits a GlobalSnapshotIncrementalSnapshot, by starting DistributedProcess (by discovery IO):
   creates a new with special message holds new ConsistentCutMarker.
4. prepares a marker message that contains the marker and transmits this message to other nodes.
Every nodes starts a local snapshot process after receiving the marker message (whether by discovery, or by communication with transaction message) 
1. Atomically: creates new ConsistentCut future (node becomes RED), creates committingTxs, starts signing outgoing messages with the ConsistentCutMarker.
2. Write a snapshot record to WAL with the received ConsistentCutMarker (commits LocalState).
3. Collect of active transactions - concat of IgniteTxManager#activeTx and committingTxs 
4. Prepares 2 empty collections - before andafter cut (describes ChannelState).
5. While global Consistent Cut is running every node signs output transaction messages:
   1. Prepare messages signed with the ConsistentCutMarker (to trigger ConsistentCut on remote node, if not yet).
   2. Finish messages signed with the ConsistentCutMarker (to trigger...) and transaction ConsistentCutMarker (to notify nodes which side of cut this transaction belongs to).
   3. Finish messages is signed on node that commits first (near node for 2PC, backup or primary for 1PC).
6. For every collected active transaction, node waits for Finish message, to extract the ConsistentCutMarker and fills before, after collections:
   1. if received marker is null or differs from local, then transaction on before side
   2. if received color equals to local, then transaction on after side
7. After all transactions finished:
   1. Writes a finish WAL record with ChannelState (before, after). 
   2. Stops filling committingTxs.
   3. Completes ConsistentCut future, and notifies a node-initiator about finishing local procedure (with DistributedProcess protocol).
8. After all nodes finished ConsistentCut, every node stops signing outgoing transaction messages - ConsistentCut future becomes null (node is WHITE again).

Consistent and inconsistent Cuts

...