Note: The embedded consumer approach is deprecated by the MirrorMaker approach in 0.7.1. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring+(MirrorMaker) for details.

Kafka mirroring

Kafka's mirroring feature makes it possible to maintain a replica of an existing Kafka cluster. This is generally a good practice in production environments, as it increases the durability and availability of data in case the primary Kafka cluster fails. It also facilitates isolation in asymmetric producer-consumer configurations - i.e., if you have a very large number of consumers, it is preferrable to have these clients consume from a mirror as opposed to the primary Kafka cluster.

The Kafka mirror cluster uses an embedded Kafka consumer to consume messages from a source cluster, and re-publishes those messages to the local cluster using an embedded Kafka producer.

...

The following demo uses the sample configurations provided for the Kafka quick-start, and the following configuration files:
mirror-server.properties, mirror-consumer.properties, mirror-producer.properties, mirror-zookeeper.properties.

• Start the primary cluster:
  bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
• Start the mirror cluster:
  bin/zookeeper-server-start.sh config/mirror-zookeeper.properties
JMX_PORT=8888 bin/kafka-server-start.sh config/mirror-server.properties config/mirror-consumer.properties config/mirror-producer.properties
• If you now send messages to topics on the source cluster, the mirror cluster will eventually mirror those topics.

...