...

For Mirror Maker, an internal rebalance listener which implements ConsumerRebalanceCallback is wired in by default to avoid duplicates on consumer rebalance. User could still specify a custom listener class in command line argument. The internal rebalance listener will call that custom listener after it finishes the default logic.

Proposed Changes

Proposal 1

Image Removed

• A single consumer thread distribute messages into different data channel queues based on source target partition hash.
• multiple producer threads take responsibility of handling message and produce them to target cluster.
• Scale by creating more mirror maker instances or processing threads.

Pros: Decouple consume and process of messages. Allow adjustment on processing/consuming thread ratio.

Cons: More complicated structure than proposal 2.

Proposal 2

• No data channel, each mirror maker thread consume messages, process them and then send them.
• Scale by creating more mirror maker thread.

...

Cons: consumer and producer are coupled, which is less flexible. More TCP connection and memory footprint.

Common in both proposals

The only source of truth for offsets in consume-then-send pattern is target cluster. The offsets should only be committed after having received the confirmation from the target cluster. For the new producer, the confirmation is the ACK from target cluster.

...

The changes are backward compatible.

Rejected Alternatives

NoneImage Added