Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • CDC utility will be started and automatically restarted in the case of failure by the OS or some external tools to provide stable change event processing.
  • CDC feature may be used for the deployment that has WAL only.
  • At the start of the CDC first consumed event will be the first event available in the WAL archive.
  • The lag between the record change and CDC consumer notification will depend on segment archiving timeout and requires additional configuration from the user.
  • CDC failover depends on the WAL archive segment count. If the CDC application will be down a relatively long time it possible that Ignite deletes certain archive segments,
    therefore consumer can't continue to receive changed records and must restart from the existing segments.

Online (real-time) CDC

In case of using separate process for capturing data changes from WAL archives makes the lag between CDC event happens and consumer notified about it is relatively big. it's proposed to provide opportunity to capture data and notify consumers directly from Ignite process. It helps minimize the lag by cost of additional memory usage.

User interface

  1. IgniteConfiguration#cdcConsumer - implementation of the CdcConsumer interface.
  2. IgniteConfiguration#cdcBufSize - size of the buffer used by CDC to store captured changes. Default is (walSegCount * walSegSize), for the default values it is 640MB.
  3. Logs: 
    1. initialization info
    2. switch between working modes.
  4. metrics: 
    1. ordinary CDC metrics (count of captured WAL segments and entries)
    2. current working mode
    3. used buffer space
    4. lag between buffer and WAL archive (segments)
    5. lag between writing to WAL and capturing by CDC (milliseconds)
    6. last captured WALPointer

...

  1. In wal-sync-thread (the only reader of mmap WAL), under the lock that synchronizes preparing ReadSegment and rolling the WAL segment, to guarantee there are no changes in the underlying buffer.
  2. Offers a deep copy of flushing ReadSegments to the CdcWorker.
  3. CdcWorker checks remaining capacity and the buffer size
  4. If the size fits the capacity then store the offered buffer data into the Queue. 
  5. Otherwise: 
    1. remove from the queue tail segments to free space for the offered buffer
    2. store the head of the offered buffer as nextHead (WALPointer)
  6. If the size fits the capacity then store the offered buffer data into the Queue. 
  7. If there is not enough space for the segment buffer:
    1. It captures data from the segment buffers while nextHead is not reached.
    2. Switch to the archiveMode.

Body loop

  1. bufferMode:
    1. polls queue, transforms ReadSegment data to Iterator<CdcEvent>, pushes them to CdcConsumer.
    2. Optimization: transform segment buffers to CdcEvents in background (to reduce the buffer usage). CdcConsumer should be async then?
  2. archiveMode:
    1. similar to CdcMain - await archived segments
    2. submits the read WALRecords to the CdcConsumer.
    3. for every segment/record checks a condition to switch to the bufferMode:
      1. check the loaded WALPointer after initialization
      2. OR while nextHead is not reached
  3. In both modes it persists CdcConsumerState. Policy for committing the progress is the same - by WAL segment.

...