Table of Contents |
---|
Kafka's mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. The following diagram shows how to use the MirrorMaker tool to mirror a source Kafka cluster into a target (mirror) Kafka cluster. The tool uses a Kafka consumer to consume messages from the source cluster, and re-publishes those messages to the local (target) cluster using an embedded Kafka producer.
There is also documentation on the MirrorMaker at https://kafka.apache.org/documentation.html#basic_ops_mirror_maker.
How to set up a mirror
Setting up a mirror is easy - simply start up the mirror-maker processes after bringing up the target cluster. At minimum, the mirror maker takes one or more consumer configurations, a producer configuration and either a whitelist or a blacklist. You need to point the consumer to the source cluster's ZooKeeper, and the producer to the mirror cluster's ZooKeeper (or use the broker.list parameter).
Code Block |
---|
bin/kafka-runmirror-classmaker.sh kafka.tools.MirrorMaker --consumer.config sourceCluster1Consumer.config --consumer.config sourceCluster2Consumer.config --num.streams 2 --producer.config targetClusterProducer.config --whitelist=".*" |
...
Use the --num.streams option to specify the number of mirror consumer threads to create. Note that if you start multiple mirror maker processes then you may want to look at the distribution of partitions on the source cluster. If the number of consumption streams is too high per mirror maker process, then some of the mirroring threads will be idle by virtue of the consumer rebalancing algorithm (if they do not end up owning any partitions for consumption).
Shallow iteration and producer compression (Kafka 0.7)
(N/A for Kafka 0.8, see JIRA issue KAFKA-732 )
Our recommendation is to enable shallow iteration in the mirror maker's consumer. This means that the mirror maker's consumer will not attempt to decompress message-sets of compressed messages. It will simply re-publish these messages as is.
If you enable shallow iteration, you must disable compression in the mirror maker's producer, or message-sets can become double-compressed.
...
The consumer offset checker tool is useful to gauge how well your mirror is keeping up with the source cluster. For Note that the --zkconnect argument should point to the source cluster's ZooKeeper (DC1 in this scenario). Also, if the topic is not specified, then the tool prints information for all topics under the given consumer group. For example:
Code Block |
---|
bin/kafka-runconsumer-offset-classchecker.sh kafka.tools.ConsumerOffsetChecker --group KafkaMirror --zkconnect localhostdc1-zookeeper:2181 --topic test-topic KafkaMirror,topic1,0-0 (Group,Topic,BrokerId-PartitionId) Group Topic Pid Offset logSize Lag Owner KafkaMirror = KafkaMirror_jkoshy-ld-1320972386342-beb4bfc9-0 Consumer offset = 561154288 test-topic 0 5 5 = 561,154,288 (0.52G) 0 none KafkaMirror test-topic Log size = 2231392259 1 3 = 2,231,392,259 (2.08G) Consumer lag = 1670237971 4 = 1,670,237,971 (1.56G) BROKER INFO 0 -> 127.0.0.1:9092 |
...
1 none
KafkaMirror test-topic 2 6 9 3 none
|