Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka Brokers today rely on Apache Zookeeper. Many folks in the community have expressed a desire to either change the zkclient and start using Apache Curator or allowing other systems like etcd, consul, Apache Cassandra and others to handle the role Zookeeper is currently playing. By allowing the brokers to have both a way to plug-in another server for storing the meta data and also for leader election then we can have the ability to-do this.

Proposed Changes

This KIP proposes approach for isolating coordination related functionality to separate modules. These modules should come with a public interface that can have pluggable implementations.

Zookeeper has advanced low-level primitives for coordinating distributed systems – ephemeral nodes, key-value storage, watchers. Such primitives may not be available in other consensus frameworks. At the same time such low-level primitives (especially ephemeral nodes) are error prone and usually a cause of subtle bugs in Kafka coordination code.

That's why instead of focusing on question “how Kafka does coordination with Zookeeper” it is proposed to concentrate on question “what general problems of distributed systems are solved in Kafka by means of Zookeeper”. Having defined interface boundaries this way, we'll be able to hide implementation details under concrete realizations, developed with corresponding built-in facilities available in particular tools (e.g. ephemeral node or TTLs).

It is proposed to separate such high-level concerns:

1. Group membership protocol (kafka brokers form a cluster; consumer connectors form a consumer group)
2. Leader election (electing controller among brokers)
3. Distributed key-value storage (topic config storage etc etc etc)
4. Data-change listeners (triggering events - partition reassignment, catching up isr-s etc)

Public Interfaces

Below each module is presented by its interfaces.

1. Gr

Compatibility, Deprecation, and Migration Plan

The use of zookeeper.connect in server.properties will still be honored. In the code we should have the default value of the new configuration of which class to invoke as  

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.