ID | IEP-104 |
Author | |
Sponsor | |
Created | 26/05/2023 |
Status | DRAFT |
IEP-59 Change Data Capture defines CDC that runs in near realtime. The background process ignite-cdc
awaits WAL segments to be archived for data capturing. The awaiting leads to the lag between the moment event happens and consumer is notified about it. This lag can be relatively big (1s-10s seconds). It's proposed to provide opportunity to capture data and notify consumers directly from the Ignite node process. It will minimize the lag by cost of additional memory usage.
Enable realtime CDC on cluster:
ignite-cdc
(it automatically switches to the PASSIVE
mode)Ignite node restart after failure:
Run CDC with only ignite-cdc.sh process:
./control.sh –cdc realtime off
Command stops Ignite internal cdc process, CDC relies on ignite-cdc only (it automatically switches to the ACTIVE state).
Try restart realtime CDC after working with online ignite-cdc.sh:
./control.sh --cdc realtime on
Command will return immediately, but it doesn't guarantee success of the switch. It might fallback to using the ignite-cdc
only again. User should check logs and metrics here.
Ignite
IgniteConfiguration#CdcConfiguration
- CdcConsumer, keepBinary.DataStorageConfiguration#cdcBufSize
- by default (walSegments * walSegmentSize). it’s now 640 MB by default. ignite-cdc:
control.sh
Note, there is a confusion of using “segment” word:
DataStorageConfiguration#walSegmentSize
.ReadSegment
is a slice of the mmap WAL segment. It contains WAL records to sync with the actual file. Size of the segment differs from time to time and its maximum can be configured with DataStorageConfiguration#walBuffSize
.On Ignite start during memory restore (in the main thread):
CdcConfiguration#cdcConsumer
is not null, then create CdcProcessor
.CdcProcessor
read from the Metastorage the last persisted CdcConsumerState
.CdcState#enabled
is false then skip initialization.CdcState == null
then initialize.GridCacheDatabaseSharedManager#performBinaryMemoryRestore
.Entrypoint for WALRecords to be captured by CDC. Options are:
FileWriteAheadLogManager#log(WALRecord).
First option is proposed to use.
CdcWorker is a thread responsible for collecting WAL records and submitting them to a CdcConsumer
. The worker collects records in the queue.
Capturing from the buffer (wal-sync-thread):
Otherwise, stop realtime CDC:
CdcConsumerState
with (enabled=false, last send WALPointer)StopRealtimeCdcRecord
into WAL (use the prepared CdcConsumerState).Body loop (cdc-worker-thread):
RealtimeCdcRecord
record to WAL with the WALPointer.Try switch to the realtime mode:
TryStartRealtimeCdcRecord
into WAL and rollover current segment (since this record realtime cdc becomes active again).RealtimeCdcRecord extends WALRecord { private WALPointer last; } StopRealtimeCdcRecord extends WALRecord { private WALPointer last; } TryStartRealtimeCdcRecord extends WALRecord { }
RealtimeCdcRecord
and StopRealtimeCdcRecord
CdcRecord
- clears obsolete links from CDC directoryStopRealtimeCdcRecord
- switch to ACTIVE mode, start capturing from the last WALPointer (from previous RealtimeCdcRecord).TryStartRealtimeCdcRecord
- after reaching it, persist CdcConsumerState locally, switch to PASSIVE mode.class CdcWorker { private final CdcConsumer consumer; private final long checkFreq; // Invoked in wal-sync-thread. public void offer(ReadSegment seg) { // Check capacity, adding segment to the queue. } // online-cdc-thread public void body() { // Polling queue, push to CdcConsumer, writing CdcState to MetaStorage. } }
// Describe project risks, such as API or binary compatibility issues, major protocol changes, etc.
// Links to discussions on the devlist, if applicable.
// Links to various reference documents, if applicable.
// Links or report with relevant JIRA tickets.