...

For Mirror Maker, an internal rebalance listener which implements ConsumerRebalanceCallback is wired in by default to avoid duplicates on consumer rebalance. User could still specify a custom listener class in command line argument. The internal rebalance listener will call that custom listener after it finishes the default logic.

Proposed Changes

Image RemovedImage Added

• No data channel, each mirror maker thread consume messages, process them and then send them.
• Scale by creating more mirror maker thread.

Pros: Simple structure.

Cons: consumer and producer are coupled, which is less flexible. More TCP connection and memory footprintEach mirror maker thread has a consumer instance associate with it, the thread will be responsible for decompression, message handling and compression. All mirror maker thread share a producer.

The only source of truth for offsets in consume-then-send pattern is target cluster. The offsets should only be committed after having received the confirmation from the target cluster. For the new producer, the confirmation is the ACK from target cluster.

...

Whether message handler will be added is still under discussion. The benefit is that with this small add-on, user can chose to do some work such as filtering/reformatting in the pipeline once rather than do that in each client for multiple times. The downside of adding the message handler is that it more or less an addition to the basic mirror maker function.

Offset commit thread

The offsets commit thread should execute periodically. Because the consumer offset is committed per SourcePartition/ConsumerGroup. A Map[TopicPartition, UnackedOffsetList] will be needed to track the offsets that is ready for commit.

...