Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


IDIEP-104
Author
Sponsor
Created 26/05/2023
Status
Status
colourGrey
titleDRAFT


Table of Contents

Motivation

...

  1. Checks metadata (mappings, binary_meta, caches - can check inside Ignite, not reading files), prepare updates if any.
  2. Polls the Queue, transforms ReadSegment data to Iterator<CdcEvent>, pushes them to CdcConsumer.
  3. If CdcConsumer#onEvents returns true:
    1. Persists CdcConsumerState.
    2. Write RealtimeCdcRecord record to WAL with the WALPointer.
  4. Optimization: transform segment buffers to CdcEvents in background (to reduce the buffer usage). CdcConsumer should be async then?

Try restart Ignite's internal CDC processswitch to the realtime mode:

  1. User sends the command to switch modes
  2. Ignite does initialization - CdcWorker, buffer
  3. Writes TryStartRealtimeCdcRecord into WAL and rollover current segment(since this record realtime cdc becomes active again).
  4. Ignite monitors the CDC directory, awaits while segment with the record cleaned - it means ignite-cdc.sh reach the record and stops capturing the data.
  5. If buffer is not overflowed in this moment - Ignite enables CDCConsumer and starts sending the records
  6. Otherwise, ordinal stop is invoked (with writing StopRealtimeCdc record)

...