Page History

...

In case of using separate process for capturing data changes from WAL archives makes the lag between CDC event happens and consumer notified about it is relatively big. It's proposed to provide opportunity to capture data and notify consumers directly from Ignite process. It helps minimize the lag by cost of additional memory usage.

Issues to solve:

1. Behavior after the CDC buffer is full. Options are:
   1. Stop online CDC and delegate capturing to the ignite-cdc.sh process (CdcMain, based on WAL archives)
   2. Temporary switch to the CdcMain, and switch back to online CDC after closing the gap.
2. From which point online CDC starts capturing:
   1. Check local persisted OnlineCdcConsumerState - find last captured WALPointer.
   2. What if the pointer is less than any pointer recovered during Ignite node startup?
   3. What if ignite-cdc.sh streamed before node stop?

User interface:

1. IgniteConfiguration#cdcConsumer - implementation of the CdcConsumer interface.
2. IgniteConfiguration#cdcBufSize - size of the buffer used by CDC to store captured changes. Default is (walSegCount * walSegSize), for the default values it is 640MB.
3. Logs: 
1. Initialization info.
2. Switch between working modes.
4. metrics: 
1. Ordinary CDC metrics (count of captured WAL segments and entries).
2. Current working mode.
3. Used buffer space.
4. Lag between buffer and WAL archive (segments).
5. Lag between writing to WAL and capturing by CDC (milliseconds).
6. Last captured WALPointer.

...