Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-4060

Released: <Kafka Version>

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP removes the Zookeeper dependency from Kafka Streams. Currently, for Kafka Streams API accesses Zookeeper directly to create internal topics. We will use the client admin API introduced in KIP-4 to manage the internal topics via Kafka broker instead of directly accessing Zookeeper.

Public Interfaces

Zookeeper related config value, ZOOKEEPER_CONNECT_CONFIG in StreamsConfig, is deprecated.

Proposed Changes

We add a new KafkaStreams client which is used for internal topic management in Kafka Streams API. We will remove the dependency to Zookeeper from Kafka Streams API..

Compatibility, Deprecation, and Migration Plan

This feature is backward compatible from API perspective and no changes need to be done to streaming apps. However, if kafka cluster is secured the cluster admin should grant required privileges to the streaming app as described below.

Zookeeper related config value, ZOOKEEPER_CONNECT_CONFIG in StreamsConfig, is deprecated.

In the current version of Kafka Streams user app needs to have proper access privileges to the zookeeper in order to manage internal topics. After the changes proposed in this KIP user app will not need such privileges, however, user app will need new privileges in order to use Client Admin to manage the internal topics. Streaming application should have the following privileges for Client Admin:

• Create topics

• Delete Topics (Without delete topic privilege streaming apps would still work if they don't have to delete topics.)

User information need to be set the same way it is set in the streaming app if the cluster is secured.

Test Plan

The same tests that are currently used will be used for testing the new internal topic creation.

Rejected Alternatives

N/A.