...
In case of using separate process for capturing data changes from WAL archives makes the lag between CDC event happens and consumer notified about it is relatively big. It's proposed to provide opportunity to capture data and notify consumers directly from Ignite process. It helps minimize the lag by cost of additional memory usage.
Enable OnlineCDC on cluster:
ACTIVE_READ_ONLY
mode.BACKUP
mode.ACTIVE
state.Note, that ignite-cdc.sh can be run in 2 modes - BACKUP
, ACTIVE
(default):
BACKUP
is used as backup process for OnlineCDC, and then such process may fetch CDC configuration from IgniteConfiguration. Case is async replication between master and stand-by clusters.ACTIVE
is used as independent process that doesn’t rely on OnlineCDC, has its own configuration. Case is filling a cold data lake.Ignite node restart after failure:
...
Stop OnlineCDC and use ignite-cdc instead:
...
Stop both CDC - Online and ignite-cdc:
...
Ignite
IgniteConfiguration#OnlineCdcConfiguration
- CdcConsumer, keepBinary.DataStorageConfiguration#onlineCdcBufSize
- by default (walSegments * walSegmentSize). it’s now 640 MB by default. DataRegionConfiguration#cdcMode
- BACKGROUND, ONLINE (default is BACKGROUND)BACKGROUND
- make hard links of archived segments into cdc directory, that is watched by the background ignite-cdc process.ONLINE
- OnlineCDC enabled + do BACKGROUND job (ignite-cdc runs in BACKUP mode).ignite-cdc
CdcConfiguration#mode
- ACTIVE, BACKUP (default ACTIVE)control.sh
Note, there is a confusion of using “segment” word:
DataStorageConfiguration#walSegmentSize
.ReadSegment
is a slice of the mmap WAL segment. It contains WAL records to sync with the actual file. Size of the segment differs from time to time and its maximum can be configured with DataStorageConfiguration#walBuffSize
.On Ignite start during memory restore (in the main thread):
DataRegionConfiguration#cdcMode
== ONLINE, then create CdcProcessor
.CdcProcessor
read from the Metastorage the last persisted CdcConsumerState
.CdcState == null OR CdcState#enabled
is false then skip initialization.GridCacheDatabaseSharedManager#performBinaryMemoryRestore
.SegmentedRingByteBuffer
(represents WAL segment) is multi-producer/single-consumer data structure.FileWriteAheadLogManager#log(WALRecord).
CdcWorker is a thread responsible for collecting WAL records, transforming them to CdcEvents
and submitting them to a CdcConsumer
. The worker collects records in the queue.
Capturing from the buffer (wal-sync-thread):
...
Otherwise, stop online CDC:
...
Also, it's possible to stop Online CDC using command in control.sh. In this case it also writes StopOnlineCdcRecord.
Body loop (cdc-worker-thread):
OnlineCdcRecord
record to WAL with the WALPointer.Code Block | ||||
---|---|---|---|---|
| ||||
OnlineCdcRecord extends WALRecord {
private WALPointer last;
}
StopOnlineCdcRecord extends WALRecord {
private WALPointer last;
}
|
OnlineCdcRecord
and StopOnlineCdcRecord
OnlineCdcRecord
- clears obsolete links from CDC directoryStopOnlineCdcRecord
- switch to ACTIVE mode, start capturing from the last WALPointer (from previous OnlineCdcRecord).Code Block | ||||
---|---|---|---|---|
| ||||
class CdcWorker {
private final CdcConsumer consumer;
private final long checkFreq;
// Invoked in wal-sync-thread.
public void offer(ReadSegment seg) {
// Check capacity, adding segment to the queue.
}
// online-cdc-thread
public void body() {
// Polling queue, push to CdcConsumer, writing CdcState to MetaStorage.
}
}
|
...