...

Kafka Brokers today rely on Apache Zookeeper. Many folks in the community have expressed a desire to either change the zkclient and start using Apache Curator or allowing other systems like etcd, consul, Apache Cassandra and others to handle the role Zookeeper is currently playing. By allowing the brokers to have both a way to plug-in another server for storing the meta data and also for leader election then we can have the ability to-do this.

Proposed Changes

This KIP proposes approach for isolating coordination related functionality to separate modules. These modules should come with a public interface that can have pluggable implementations.

Zookeeper has advanced low-level primitives for coordinating distributed systems – ephemeral nodes, key-value storage, watchers. Such primitives may not be available in other consensus frameworks. At the same time such low-level primitives (especially ephemeral nodes) are error prone and usually a cause of subtle bugs in Kafka coordination code.

That's why instead of focusing on question “how Kafka does coordination with Zookeeper” it is proposed to concentrate on question “what general problems of distributed systems are solved in Kafka by means of Zookeeper”. Having defined interface boundaries this way, we'll be able to hide implementation details under concrete realizations, developed with corresponding built-in facilities available in particular tools (e.g. ephemeral node or TTLs).

It is proposed to separate such high-level concerns:

1. Group membership protocol (kafka brokers form a cluster; consumer connectors form a consumer group)
2. Leader election (electing controller among brokers)
3. Distributed key-value storage (topic config storage etc etc etc)
4. Data-change listeners (triggering events - partition reassignment, catching up isr-s etc)

Public Interfaces

Below each module is presented by its interfaces.

1. Gr

...

Compatibility, Deprecation, and Migration Plan

...