Status

Current state: Accepted

Discussion thread: here

JIRA:  Unable to render Jira issues macro, execution error. ,  Unable to render Jira issues macro, execution error.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The KIP-500 effort to remove Kafka's dependency on Apache ZooKeeper requires some changes to the way Kafka system administration tools work.  All of the functionality that was previously available through direct ZooKeeper access must now be accessible through other means.

Several previous KIPs have prepared the way for this change.  KIP-455, KIP-497, KIP-554, and KIP-543 added broker-side APIs for operations that had previously only been supported via direct Zookeeper access.  KIP-555 deprecated the --zookeeper flag in administrative tools where it was still supported.

Now, we must take the final steps towards removing the ZooKeeper flags from the administrative tools.  This step will pave the way for the bridge release, a release which can be upgraded to a ZooKeeper-free state.

Public Interfaces

The --zookeeper flag will be removed from all the places where it was deprecated in KIP-555, with only one exception which will be described in a little bit.

We will also remove support for the --authorizer in the ACL command. This was previously allowed for writes directly to the Zookeeper-based authorizer. Now support for changing ACLs is supported through the admin client (see KIP-332: Update AclCommand to use AdminClient API).

Here is a table of all the changes:

kafka-preferred-replica-election.sh

kafka-preferred-replica-election.sh will be removed.  It has been deprecated in favor of kafka-leader-election.sh for some time.

kafka-configs.sh changes

Unlike all the other tools, kafka-configs.sh will keep the --zookeeper flag.  It will be allowed only in two scenarios:

1. In order to make dynamic configuration updates before and brokers are registered in ZooKeeper.
2. In order to make SCRAM configuration updates before any brokers are registered in ZooKeeper.

The command will not succeed if any brokers are registered.  It is intended to be used before any brokers are started, to set up SSL and SCRAM settings in ZooKeeper.

The rationale behind keeping the --zookeeper flag for this use-case is that we don't yet have an equivalent replacement.  Although we are developing a new broker-side SCRAM API, this will only be usable once the brokers have started.

kafka-topics.sh changes

kafka-topic.sh will no longer support setting configurations.  This functionality has been deprecated for a long time in favor of simply using kafka-configs.sh.  This transition is a good time to remove this deprecated functionality, since it has never been supported in conjunction with --bootstrap-server.

We will support --if-exists and --if-not-exists when --bootstrap-server is specified.  Previously, this was only supported if --zookeeper was specified.

kafka-acls.sh changes

In kafka-acls.sh, support for directly updating ACLs through AclAuthorizer extensions will be deprecated and eventually removed in a major release. Instead users will be expected to update ACLs through the admin client by providing the --bootstrap-server argument.

Compatibility, Deprecation, and Migration Plan

This is an incompatible change.  Therefore, it must be done in the next major release of Kafka, the 3.0 release.

The flags that are being removed immediately (excluding kafka-acls.sh) have already been deprecated for at least one release. 

The migration plan is to use --bootstrap-server for these commands instead.

Rejected Alternatives

Do not allow bootstrapping network connection settings

We could have disabled the --zookeeper flag for all uses, including bootstrapping network connection settings.  However, this would have caused problems for people who are currently using this functionality.