...

Current state: Under Discussion

Discussion thread: here [TBD]

JIRA: here 

Motivation

...

• At least once delivery to the destination cluster must be supported
• Ability to Can specify a list of topics or regular expressionstopic whitelist as a regular expression (Same as in MirrorMaker)

Basic Implementation Components

...

Starts an AdminClient in a separate thread when the connector is first started (prior to determining the number of tasks to run) to determine the topics and partitions available on the source cluster which match the supplied topic list. After the first check of the available partitions, periodically runs (at a configurable interval) to check if the matching topics or partitions on the source cluster have changed. Triggers a task reconfiguration if it detects that there has been a change in the matching topic partitions .(or, leaders - if option is selected to reconfigure on leader change)

Source Task

Each source task will be started with a list of topic partitions to subscribe to. On task start will start a KafkaConsumer and manually assign (rather than the high level subscribe()) these topic partitions to the consumer. On task start, will check the Kafka connect offset storage for an existing offset for each topic partition - if found then it will resume reading the topic partition at this offset, otherwise it will begin from the beginning or the end of the topic (configurable). Each time the source task's poll() method is called, it will poll the Kafka consumer for available messages, and return them to the connect framework to be delivered to the Destination cluster.

Note that offset checkpointing is managed by Kafka Connect. In order to ensure that 

Configuration Options

Common Options

...