...

• CDC utility will be started and automatically restarted in the case of failure by the OS or some external tools to provide stable change event processing.
• CDC feature may be used for the deployment that has WAL only.
• At the start of the CDC first consumed event will be the first event available in the WAL archive.
• The lag between the record change and CDC consumer notification will depend on segment archiving timeout and requires additional configuration from the user.
• CDC failover depends on the WAL archive segment count. If the CDC application will be down a relatively long time it possible that Ignite deletes certain archive segments,
  therefore consumer can't continue to receive changed records and must restart from the existing segments.

Online (real-time) CDC

In case of using separate process for capturing data changes from WAL archives makes the lag between CDC event happens and consumer notified about it is relatively big. it's proposed to provide opportunity to capture data and notify consumers directly from Ignite process. It helps minimize the lag by cost of additional memory usage.

User interface

1. IgniteConfiguration#cdcConsumer - implementation of the CdcConsumer interface.
2. IgniteConfiguration#cdcBufSize - size of the buffer used by CDC to store captured changes. Default is (walSegCount * walSegSize), for the default values it is 640MB.
3. Logs: 
1. initialization info
2. switch between working modes.
4. metrics: 
1. ordinary CDC metrics (count of captured WAL segments and entries)
2. current working mode
3. used buffer space
4. lag between buffer and WAL archive (segments)
5. lag between writing to WAL and capturing by CDC (milliseconds)
6. last captured WALPointer

Segments

Note, there is a confusion of using “segment” word:

1. WAL segments are represented as numerated files. Size of WAL segments is configured with DataStorageConfiguration#walSegmentSize.
2. ReadSegment is a slice of the mmap WAL segment. It contains WAL records to sync with the actual file. Size of the segment differs from time to time and its maximum can be configured with DataStorageConfiguration#walBuffSize.

CdcWorker

CdcWorker is a thread responsible for collecting WAL records, transforming them into cdc events, submitting them to the CdcConsumer. The worker has 2 modes:

1. BUFFER_MODE - consumes WAL records from the CdcBufferQueue, that is filled directly from the WAL manager.
2. ARCHIVE_MODE - consumes WAL records from archived WAL segments.
1. Note, that the CdcBufferQueue is being filled in background in this mode.

Initialization

1. CdcWorker initialized with CdcConsumerState#loadWalState.
2. Initial mode is ARCHIVE_MODE. It switches to the CdcBufferQueue after:
1. the loaded pointer is not reached in the archive.
2. OR the head of the buffer queue is less than the loaded pointer. 

Capturing from the buffer (wal-sync-thread)

1. In wal-sync-thread (the only reader of mmap WAL), under the lock that synchronizes preparing ReadSegment and rolling the WAL segment, to guarantee there are no changes in the underlying buffer.
2. Offers a deep copy of flushing ReadSegments to the CdcWorker.
3. CdcWorker checks remaining capacity and the buffer size
4. If the size fits the capacity then store the offered buffer data into the Queue. 
5. Otherwise: 
   1. remove from the queue tail segments to free space for the offered buffer
   2. store the head of the offered buffer as nextHead (WALPointer)
   3. It captures data from the Queue while nextHead is not reached.
   4. Switch to the ARCHIVE_MODE.

Body loop (cdc-worker-thread)

1. BUFFER_MODE:
   1. polls the Queue, transforms ReadSegment data to Iterator<CdcEvent>, pushes them to CdcConsumer.
   2. Optimization: transform segment buffers to CDC events in background (to reduce the buffer usage). CdcConsumer should be async then?
2. ARCHIVE_MODE:
   1. similar to CdcMain - await archived segments
   2. submits the read WAL records to the CdcConsumer.
   3. for every segment/record checks a condition to switch to the bufferMode:
      1. check the loaded WALPointer after initialization
      2. OR while nextHead is not reached
3. In both modes it persists CdcConsumerState. Policy for committing the progress: by WAL segment.

Discussion Links

http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-IEP-59-CDC-Capture-Data-Change-tc49677.html

...