...
In case of using separate process for capturing data changes from WAL archives makes the lag between CDC event happens and consumer notified about it is relatively big. it's proposed to provide opportunity to capture data and notify consumers directly from Ignite process. It helps minimize the lag by cost of additional memory usage.
IgniteConfiguration#cdcConsumer
- implementation of the CdcConsumer interface.IgniteConfiguration#cdcBufSize
- size of the buffer used by CDC to store captured changes. Default is (walSegCount * walSegSize), for the default values it is 640MB....
DataStorageConfiguration#walSegmentSize
.DataStorageConfiguration#walBuffSize
.CdcWorker is a thread responsible for collecting WAL records, transforming them into CdcEventscdc events, submitting them to the CdcConsumer
. The worker has 2 modes:
bufferMode
- consumes WAL records from the CdcBufferQueue
, that is filled directly from the WAL manager.archiveMode
- consumes WAL records from archived WAL segments.CdcBufferQueue
is being filled in background in this mode....
CdcConsumerState#loadWalState
.archiveMode
. It switches to the CdcBufferQueue
after:...
ReadSegment
and rolling the WAL segment, to guarantee there are no changes in the underlying buffer.ReadSegments
to the CdcWorker
.CdcWorker
checks remaining capacity and the buffer sizeWALPointer
)Queue
while nextHead is not reached.archiveMode
.Body loop (cdc-worker-thread)
bufferMode
:Queue
, transforms ReadSegment data to Iterator<CdcEvent>
, pushes them to CdcConsumer
.CdcConsumer
should be async then?archiveMode
:CdcMain
- await archived segmentsCdcConsumer
.bufferMode
:WALPointer
after initializationCdcConsumerState
. Policy for committing the progress: by WAL segment....