Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-4060

Released: <Kafka Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP removes the Zookeeper dependency from Kafka Streams. Currently, for Kafka Streams API accesses Zookeeper directly to create internal topics. We will use the client admin API introduced in KIP-4 to manage the internal topics via Kafka broker instead of directly accessing Zookeeper.

Public Interfaces

This KIP does not change any public interfaces.

Proposed Changes

We add a new KafkaStreams client which is used for internal topic management in Kafka Streams API. We will remove the dependency to Zookeeper from Kafka Streams API..

Compatibility, Deprecation, and Migration Plan

This feature is not backward compatible.

Zookeeper related config value, ZOOKEEPER_CONNECT_CONFIG in StreamsConfig, is deprecated.

In the current version of Kafka Streams user app needs to have proper access privileges to the zookeeper in order to manage internal topics. After the changes user app will not need such privileges, however, user app will need new privileges in order to use Client Admin to manage the internal topics. Streaming application should have the following privileges for Client Admin:

• Create topics

• Delete Topics

• Fetch topic metadata

Test Plan

The same tests that are currently used will be used for testing the new internal topic creation.

Rejected Alternatives

N/A.