Page History

...

1. IgniteConfiguration#OnlineCdcConfiguration - CdcConsumer, keepBinary.
2. DataStorageConfiguration#onlineCdcBufSize - by default (walSegments * walSegmentSize). it’s now 640 MB by default.
   1. All non-archived segments are fitted in memory. If OnlineCDC requires more space than it, it looks like ordinary CDC process should be used instead.
3. DataRegionConfiguration#cdcMode - BACKGROUND, ONLINE (default is BACKGROUND)
   1. BACKGROUND - make hard links of archived segments into cdc directory, that is watched by the background ignite-cdc process.
   2. ONLINE - OnlineCDC enabled + still do BACKGROUND job (ignite-cdc runs in BACKUP mode)mode job.
4. Logs: 
   1. initialization (amount of records read during the restore)
   2. failure 
   3. buffer is full
5. Metrics: 
   1. ordinary cdc metrics (count of wal segments, wal entries)
   2. current buffer size
   3. status of CDC (on/off)
   4. last committed WALPointer
   5. lag between buffer and WAL archive (segments)
   6. lag between WAL and CDC consumer (milliseconds).

...

Online capturing of WALRecords

Entrypoint for WALRecords to be captured by CDC. Options are:

1. SegmentedRingByteBuffer (represents WAL segment) is During read of SegmentedRingByteBuffer after fsync is invoked. It is a multi-producer/single-consumer data structure, then the only place to built-in is read operations (invoked at the moment of fsync).
   
   1. + Relying on the consumer workflow we can guarantee order of events.
   2. + Consumer is a background thread, capturing records doesn't affect performance of transactional threads
   3. - Can't No opportunity to filter physical records at the entrypoint (might waste the buffer space). Must deserialize and Will filter them later before actual sending to a CdcConsumer.
   4. - The consumer is triggered by a schedule - every 500ms by default.
   5. - Logic has some differences depending on the WAL settings (mmap true/false, FULL_SYNC) 
2. Capturing in FileWriteAheadLogManager#log(WALRecord).
   1. + Capture logical records only
   2. + Common logic for all WAL settings  
   3. - Captures record in buffer in transactional threads - might affect performance
   4. - CDC process must sort events by WALPointer by self - maintain concurrent ordering data structure, and implementing waiting for closing WAL gaps before sending.
   5. - Send events before they actually flushed in local Ignite node - lead to inconsistency between main and stand-by clusters.

First option is proposed to use.

CdcWorker

CdcWorker is a thread responsible for collecting WAL records, transforming them to CdcEvents and submitting them to a CdcConsumer. The worker collects records in the queue.

...