...

Kafka's mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. The following diagram shows how to use the MirrorMaker tool to mirror a source Kafka cluster into a target (mirror) Kafka cluster. The tool uses a Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the local (target) cluster using an embedded Kafka producer.

There is also documentation on the MirrorMaker at https://kafka.apache.org/documentation.html#basic_ops_mirror_maker.

How to set up a mirror

Setting up a mirror is easy - simply start up the mirror-maker processes after bringing up the target cluster. At minimum, the mirror maker takes one or more consumer configurations, a producer configuration and either a whitelist or a blacklist. You need to point the consumer to the source cluster's ZooKeeper, and the producer to the mirror cluster's ZooKeeper (or use the broker.list parameter).

bin/kafka-runmirror-classmaker.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClusterProducer.config --whitelist=".*"

...

Shallow iteration and producer compression (Kafka 0.7)

(N/A for Kafka 0.8, see JiraserverIssueskeyJIRA issue KAFKA-732 )

Our recommendation is to enable shallow iteration in the mirror maker's consumer. This means that the mirror maker's consumer will not attempt to decompress message-sets of compressed messages. It will simply re-publish these messages as is.

If you enable shallow iteration, you must disable compression in the mirror maker's producer, or message-sets can become double-compressed.

...

The consumer offset checker tool is useful to gauge how well your mirror is keeping up with the source cluster. Note that the --zkconnect argument should point to the source cluster. For 's ZooKeeper (DC1 in this scenario). Also, if the topic is not specified, then the tool prints information for all topics under the given consumer group. For example:

bin/kafka-runconsumer-offset-classchecker.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect localhostdc1-zookeeper:2181 --topic test-topic
KafkaMirror,topic1,0-0 (Group,Topic,BrokerId-PartitionId)
Group           Topic                          Pid Offset          logSize         Lag             Owner
KafkaMirror = KafkaMirror_jkoshy-ld-1320972386342-beb4bfc9-0
  Consumer offset = 561154288
    test-topic                     0   5               5             = 561,154,288 (0.52G)
0               none
KafkaMirror     test-topic    Log size = 2231392259
              1   3 = 2,231,392,259 (2.08G)
     Consumer lag = 1670237971
    4              = 1,670,237,971 (1.56G)
BROKER INFO
0 -> 127.0.0.1:9092

...

 1               none
KafkaMirror     test-topic                     2   6               9               3               none