Page History

...

CDC application works as a separate process.
CDC relies on the existing Ignite mechanism - WAL.
IEP Scope - deliver local data change events to a local consumer.
CDC keeps consumer offset in a special file.
WAL process will start from this offset on restart.
To prevent interference between the WAL archive process and CDC Ignite will create a hard link to the newly created segment in a special folder.
After success processing, CDC will delete this link.
Note, data will be removed from the disk only after CDC and Ignite will remove the link to a segment from both corresponding folders.
To manage minimal event gap new configuration timeout introduced - WalForceArchiveTimeout.
Flag to distinguish DataEntry on primary and backup nodes introduced.
All public APIs market with @IgniteExperimental to be able to improve it based on real-world usage feedback.
CDC consumer will be notified about binary metadata changes (Phase 2).
Configuration parameter "Maximum CDC folder size" will be implemented to prevent disk volume exceed.
CDC folder resolved using the logic as Ignite node does.
CDC application should be restarted by the OS mechanism in case of any error (destination unavailability, for example)
Initially, single CDC consumer supported. Support of several concurrently running consumers will be implemented in Phase2.

CDCConsumer public API interface:

...

Code Block

language	java
title	CDCConsumer.java

/** Consumer of data change events. */
@IgniteExperimental
public interface DataChangeListener<K, V> {
    /**
     * @return Consumer ID.
     */
    String id();

    /**
     * Starts the consumer.
     *
     * @param configuration Ignite configuration.
     */
    void start(IgniteConfiguration configuration, IgniteLogger log);

    /**
     * @return {@code True} if entry key and value should be keeped in binary format.
     */
    boolean keepBinary();

    /**
     * Handles entry changes events.
     * If this method return {@code true} then current offset will be stored and ongoing notifications after CDC application fail/restart
     * will be started from it.
     *
     * @param events Entry change events.
     * @return {@code True} if current offset should be commited.
     */
    boolean onChange(Iterable<EntryEvent<K, V>> events);

    /**
     * Stops the consumer.
     * This methods can be invoked only after {@link #start(IgniteConfiguration, IgniteLogger)}.
     */
    void stop();
}

/**
 * Event for single entry change.
 *
 * @param <K> Key type.
 * @param <V> Value type.
 */
@IgniteExperimental
public interface EntryEvent<K, V> {
    /**
     * @return Key for the changed entry.
     */
    public K key();

    /**
     * @return Value for the changed entry.
     */
    public V value();

    /**
     * @return {@code True} if event fired on primary node for partition containing this entry.
     * @see <a href="https://ignite.apache.org/docs/latest/configuring-caches/configuring-backups#configuring-partition-backups">Configuring partition backups.</a>
     */
    public boolean primary();

    /**
     * @return Operation type.
     */
    EntryEventType operation();

    /**
     * @return Cache ID.
     * @see org.apache.ignite.internal.util.typedef.internal.CU#cacheId(String)
     */
    long cacheId();

    /**
     * @return Expire time.
     */
    long expireTime();
}

...

Page tree

Versions Compared

Old Version 12

New Version 13

Key