THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: "Under Discussion"
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here
Motivation
Kafka has a dependency on Zookeeper 3.6.3, which reached its end of life in December 2022. We would like to upgrade Zookeeper to version 3.8.1 which is the latest release of the 3.8.x versions.
Zookeeper 3.8.1 server supports clients no older than 3.5.x and Zookeeper 3.8.1. clients support server versions no older than 3.5.x.
ZooKeeper clients from 3.5.x onwards are fully compatible with 3.8.x servers. The upgrade from 3.6.x and 3.7.x can be executed as usual, no particular additional upgrade procedure is needed. ZooKeeper 3.8.x clients are compatible with 3.5.x, 3.6.x and 3.7.x servers as long as you are not using new APIs not present these versions.
In comparison, Zookeeper 3.6.3 server supports clients no older than 3.4.x and Zookeeper 3.6.3 clients support server versions no older than 3.5.x
ZooKeeper clients from 3.4 and 3.5 branch are fully compatible with 3.6 servers. The upgrade from 3.5.7 to 3.6.0 can be executed as usual, no particular additional upgrade procedure is needed. ZooKeeper 3.6.0 clients are compatible with 3.5 servers as long as you are not using new APIs not present in 3.5.
Public Interfaces
No public interfaces are being changed.
Proposed Changes
Similarly to https://github.com/apache/kafka/pull/12620/files we would like to upgrade to 3.8.1.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users? Users who use Kafka clusters with Zookeeper clients older than 3.5.x won't be able to communicate with a Zookeeper cluster using 3.8.1. As mentioned in the accompanying JIRA ticket Apache Kafka has been using Zookeeper since version 2.4, everything above and including this version should be stable. It is acceptable to break compatibility with Apache Kafka versions prior to 2.4 as they are considered beyond their end of life and are not maintained (source: Time Based Release Plan#WhatIsOurEOLPolicy).
Notable changes in Zookeeper 3.7.0 related to security
- - previously there could be only one SASL SuperUser. With this change multiple SuperUsers could be specified via a configuration.
- - quotas which were not previously enforced are now enforced.
- - Kerberos authentication did not work over SSL, but has now been fixed.
- - user enforced authentication was only available for SASL before this change.
Notable changes in Zookeeper 3.8.0 related to security
- - instead of storing password as plaintext use password protected files.
- If we are changing behavior how will we phase out the older behavior? It should gradually be phased out as users update their Kafka versions
- If we need special migration tools, describe them here. N/A
- When will we remove the existing behavior? N/A
Test Plan
We ran the following test on the latest trunk of Kafka with Zookeeper 3.6.3 and Zookeeper 3.8.1:
1) Start 1 Zookeeper node on an m5.4xlarge machine
2) Start 1 Kafka broker on a different m5.4xlarge machine
3) Using 4 admin clients sequentially create up to 2000 topics with 1 partition
4) Using 4 admin clients sequentially change the number of partitions on all 2000 topics to 2
5) Using 4 admin clients sequentially delete all topics
Zookeeper 3.8.1 request latency (PROPOSED)
Zookeeper 3.6.3 request latency (CURRENT)
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way. N/A