Page History

...

1. IgniteConfiguration#cdcConsumer - implementation of the CdcConsumer interface.
2. IgniteConfiguration#cdcBufSize - size of the buffer used by CDC to store captured changes. Default is (walSegCount * walSegSize), for the default values it is 640MB.
3. Logs: 
1. initialization info
2. switch between working modes.
4. metrics: 
1. ordinary CDC metrics (count of captured WAL segments and entries)
2. current working mode
3. used buffer space
4. lag between buffer and WAL archive (segments)
5. lag between writing to WAL and capturing by CDC (milliseconds)
6. last captured WALPointer

...

CdcWorker is a thread responsible for collecting WAL records, transforming them into cdc events, submitting them to the CdcConsumer. The worker has 2 modes:

1. bufferMode BUFFER_MODE - consumes WAL records from the CdcBufferQueue, that is filled directly from the WAL manager.
2. archiveMode ARCHIVE_MODE - consumes WAL records from archived WAL segments.
1. Note, that the CdcBufferQueue is being filled in background in this mode.

...

1. CdcWorker initialized with CdcConsumerState#loadWalState.
2. Initial mode is archiveMode ARCHIVE_MODE. It switches to the CdcBufferQueue after:
1. the loaded pointer is not reached in the archive.
2. OR the head of the buffer queue is less than the loaded pointer. 

...

1. In wal-sync-thread (the only reader of mmap WAL), under the lock that synchronizes preparing ReadSegment and rolling the WAL segment, to guarantee there are no changes in the underlying buffer.
2. Offers a deep copy of flushing ReadSegments to the CdcWorker.
3. CdcWorker checks remaining capacity and the buffer size
4. If the size fits the capacity then store the offered buffer data into the Queue. 
5. Otherwise: 
   1. remove from the queue tail segments to free space for the offered buffer
   2. store the head of the offered buffer as nextHead (WALPointer)
   3. It captures data from the Queue while nextHead is not reached.
   4. Switch to the archiveMode ARCHIVE_MODE.

Body loop (cdc-worker-thread)

1. bufferModeBUFFER_MODE:
   1. polls the Queue, transforms ReadSegment data to Iterator<CdcEvent>, pushes them to CdcConsumer.
   2. Optimization: transform segment buffers to CDC events in background (to reduce the buffer usage). CdcConsumer should be async then?
2. archiveModeARCHIVE_MODE:
   1. similar to CdcMain - await archived segments
   2. submits the read WAL records to the CdcConsumer.
   3. for every segment/record checks a condition to switch to the bufferMode:
      1. check the loaded WALPointer after initialization
      2. OR while nextHead is not reached
3. In both modes it persists CdcConsumerState. Policy for committing the progress: by WAL segment.

...