Page History

...

Each mirror maker thread has a consumer instance associate with it, the thread will be responsible for decompression, message handling, compression and compressionoffset commit. All mirror maker threads share a producer.

...

Whether message handler will be added is still under discussion. The benefit is that with this small add-on, user can chose to do some work such as filtering/reformatting in the pipeline once rather than do that in each client for multiple times. The downside of adding the message handler is that it is more or less an addition to the basic mirror maker function.

Offset commit

...

Each mirror maker thread maintains the consumer offsets for the partitions they are consuming from and commit the offsets The offsets commit thread should execute periodically. Because the consumer offset is committed per SourcePartition/ConsumerGroup. A Map[TopicPartition, UnackedOffsetList] will be needed to track the offsets that is ready for commit.

...