ID | IEP-59 |
Author | |
Sponsor | Nikolay Izhikov|
Created | 14.10.2020 |
Status | DRAFT |
Many use-cases build on observation and processing changed records.
These use-cases include but not limited by
For now, such scenarios are hard to implement in Ignite.
The only solution that can help with it, for now, is a Continuous Query.
Disadvantages of the CQ in described scenarios:
The convenient solution should be:
IgniteCDC is a new utility that should be run on the server node host. CDC utility watches by the appearance of the WAL archive segments.
On the segment archiving, utility iterates it using existing WALIterator and notificates CDCConsumer of each record from the segment.
CDCConsumer public API interface:
/** Consumer of data change events. */ @IgniteExperimental public interface DataChangeListener<K, V> { /** * @return Consumer ID. */ String id(); /** * Starts the consumer. * * @param configuration Ignite configuration. */ void start(IgniteConfiguration configuration, IgniteLogger log); /** * @return {@code True} if entry key and value should be keeped in binary format. */ boolean keepBinary(); /** * Handles entry changes events. * If this method return {@code true} then current offset will be stored and ongoing notifications after CDC application fail/restart * will be started from it. * * @param events Entry change events. * @return {@code True} if current offset should be commited. */ boolean onChange(Iterable<EntryEvent<K, V>> events); /** * Stops the consumer. * This methods can be invoked only after {@link #start(IgniteConfiguration, IgniteLogger)}. */ void stop(); } /** * Event for single entry change. * * @param <K> Key type. * @param <V> Value type. */ @IgniteExperimental public interface EntryEvent<K, V> { /** * @return Key for the changed entry. */ public K key(); /** * @return Value for the changed entry. */ public V value(); /** * @return {@code True} if event fired on primary node for partition containing this entry. * @see <a href="https://ignite.apache.org/docs/latest/configuring-caches/configuring-backups#configuring-partition-backups">Configuring partition backups.</a> */ public boolean primary(); /** * @return Operation type. */ EntryEventType operation(); /** * @return Cache ID. * @see org.apache.ignite.internal.util.typedef.internal.CU#cacheId(String) */ long cacheId(); /** * @return Expire time. */ long expireTime(); } /** * Event for single entry change. * * @param <K> Key type. * @param <V> Value type. */ @IgniteExperimental public interface EntryEvent<K, V> { /** * @return Key for the changed entry. */ public K key(); /** * @return Value for the changed entry. */ public V value(); /** * @return {@code True} if event fired on primary node for partition containing this entry. * @see <a href="https://ignite.apache.org/docs/latest/configuring-caches/configuring-backups#configuring-partition-backups">Configuring partition backups.</a> */ public boolean primary(); /** * @return Operation type. */ EntryEventType operation(); /** * @return Cache ID. * @see org.apache.ignite.internal.util.typedef.internal.CU#cacheId(String) */ long cacheId(); /** * @return Expire time. */ long expireTime(); }
Risks and Assumptions
https://dev.mysql.com/doc/refman/8.0/en/mysqlbinlog.html
https://debezium.io/documentation/reference/1.2/architecture.html
https://jdbc.postgresql.org/documentation/head/replication.html
https://www.oracle.com/middleware/technologies/goldengate.html