Page History

...

1. Explicit stop ignite-cdc.sh
2. Explicit stop OnlineCDC
   
   

Restart OnlineCDC after buffer overflow:

1. Try to switch Ignite from BACKGROUND to ONLINE mode:
   1. ./control.sh --cdc online --try-switch-to-online
2. Command will return immediately, but doesn't guarantee success of the switch. User should check logs and metrics here.

User interface

Ignite

1. IgniteConfiguration#OnlineCdcConfiguration - CdcConsumer, keepBinary.
2. DataStorageConfiguration#onlineCdcBufSize - by default (walSegments * walSegmentSize). it’s now 640 MB by default.
   1. All non-archived segments are fitted in memory. If OnlineCDC requires more space than it, it looks like ordinary CDC process should be used instead.
3. DataRegionConfiguration#cdcMode - BACKGROUND, ONLINE (default is BACKGROUND)
   1. BACKGROUND - make hard links of archived segments into cdc directory, that is watched by the background ignite-cdc process.
   2. ONLINE - OnlineCDC enabled + still do BACKGROUND mode job.
4. Logs: 
   1. initialization (amount of records read during the restore)
   2. failure 
   3. buffer is full
   4. switch from BACKGROUND to ONLINE mode.
5. Metrics:Metrics: 
   1. ordinary cdc metrics (count of wal segments, wal entries)
   2. current buffer size
   3. status of CDC (on/off)
   4. last committed WALPointer
   5. lag between buffer and WAL archive (segments)
   6. lag between WAL and CDC consumer (milliseconds).

...

control.sh

1. CdcOnline subcommand
   1. ./control.sh --
   enable/disable.
   1. cdc online [ --enable ] [ --disable ] [ --try-switch-to-online ] 

Segments

Note, there is a confusion of using “segment” word:

...

1. Checks metadata (mappings, binary_meta, caches - can check inside Ignite, not reading files), prepare updates if any.
2. Polls the Queue, transforms ReadSegment data to Iterator<CdcEvent>, pushes them to CdcConsumer.
3. If CdcConsumer#onEvents returns true:
   1. Persists CdcConsumerState.
   2. Write OnlineCdcRecord record to WAL with the WALPointer.
4. Optimization: transform segment buffers to CdcEvents in background (to reduce the buffer usage). CdcConsumer should be async then?

Try switch BACKGROUND to ONLINE mode:

1. User sends the command to switch modes
2. Ignite does initialization - CdcWorker, buffer
3. Writes TryStartOnlineCdcRecord into WAL and rollover current segment(since this record OnlineCdc becomes active again).
4. Ignite monitors the CDC directory, awaits while segment with the record cleaned - it means ignite-cdc.sh reach the record and stops capturing the data.
5. If buffer is not overflowed in this moment - Ignite enables CDCConsumer and starts sending the records
6. Otherwise, ordinal stop is invoked (with writing StopOnlineCdc record)

OnlineCdcRecord extends WALRecord {
	private WALPointer last;
}

StopOnlineCdcRecord extends WALRecord {
	private WALPointer last;
}

TryStartOnlineCdcRecord extends WALRecord {
	
}

ignite-cdc in BACKUP mode

1. Parses WAL records, looking for OnlineCdcRecord and StopOnlineCdcRecord
2. For OnlineCdcRecord - clears obsolete links from CDC directory
3. For StopOnlineCdcRecord - switch to ACTIVE mode, start capturing from the last WALPointer (from previous OnlineCdcRecord).

ignite-cdc in ACTIVE mode

1. Capturing WAL records
2. Looking for TryStartOnlineCdcRecord - after reaching it, persist CdcConsumerState locally, switch to BACKUP mode.

Meta Storage

1. Online CDC status - ON / OFF
2. Committed pointer (confirmed by CdcConsumer).

...