THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
| Table of Contents |
|---|
Status
Current state: "Under DiscussionAccepted"
Discussion thread: here
Voting thread:here
JIRA: here
PR: here
Motivation
Kafka has a dependency on Zookeeper 3.6.3, which reached its end of life in December 2022. We would like to upgrade Zookeeper to version 3.8.1 2 which is the latest release of the 3.8.x versions.
Zookeeper 3.8.1 2 server supports clients no older than 3.5.x (i.e. Apache Kafka version 2.4.0) and Zookeeper 3.8.12. clients support server versions no older than 3.5.x (i.e. Apache Kafka version 2.4.0).
| Code Block |
|---|
ZooKeeper clients from 3.5.x onwards are fully compatible with 3.8.x servers. The upgrade from 3.6.x and 3.7.x can be executed as usual, no particular additional upgrade procedure is needed. ZooKeeper 3.8.x clients are compatible with 3.5.x, 3.6.x and 3.7.x servers as long as you are not using new APIs not present these versions. |
...
| Code Block |
|---|
ZooKeeper clients from 3.4 and 3.5 branch are fully compatible with 3.6 servers. The upgrade from 3.5.7 to 3.6.0 can be executed as usual, no particular additional upgrade procedure is needed. ZooKeeper 3.6.0 clients are compatible with 3.5 servers as long as you are not using new APIs not present in 3.5. |
Public Interfaces
No public interfaces are being changed.
Proposed Changes
Similarly to https://github.com/apache/kafka/pull/12620/files we would like to upgrade to 3.8.12.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
Users of Kafka clusters with Zookeeper clients older than 3.5.x won't be able to communicate with a 3.8.1 2 Zookeeper cluster. As mentioned in the accompanying JIRA ticket Kafka has been using Zookeeper clients 3.5.x since version 2.4 so versions above and including it should be safe for this upgrade. It is acceptable to break compatibility with Kafka versions prior to 2.4 as they are considered beyond their end of life and are not maintained. (source: Time Based Release Plan#WhatIsOurEOLPolicy).
A Zookeeper cluster is separate from a Kafka cluster.
For example, a user wanting to migrate from a Kafka cluster with version 2.3 and a Zookeeper cluster with version 3.4 to a Kafka cluster with version 3.X (where X is the Kafka version containing this KIP) and Zookeeper cluster with version 3.8.1 would first have to upgrade their Kafka cluster to 3.4.0 (latest version of the time of writing), then update their Zookeeper cluster to 3.8.1 and lastly update their Kafka cluster again from 3.4.0 to 3.X.
As another example, a user wanting to migrate from a Kafka cluster with version 2.4 and a Zookeeper cluster with version 3.5 to a Kafka cluster with version 3.X and Zookeeper cluster with version 3.8.1 could upgrade their Kafka cluster to 3.X and then their Zookeeper cluster to 3.8.1 or vice versa.
It is worth mentioning that Kafka is still tested for cluster upgrades between versions 0.8.2 and 3.4.
The table below makes the assumption you want to carry out the upgrades using a rolling restart
| Zookeeper cluster (< 3.5) | Zookeeper cluster (>= 3.5) | |
|---|---|---|
| Kafka cluster (< 2.4) |
|
|
|
|
|
|
|
| |
| Kafka client (< 2.4) |
|
|
|
|
|
|
Kafka cluster (>= 2.4) |
|
|
|
|
|
|
|
Kafka client (>= 2.4) |
|
|
|
|
|
This is a compatibility matrix between Kafka and Zookeeper versions.
The two coloured rectangles indicate versions of Kafka and Zookeeper which are compatible with one another. When a new version of Kafka having a dependency on Zookeeper 3.8.2 is released the bridge release will be any of the version where the two squares intersect.
Below is an analysis of changes introduced in 3.7.0 and 3.8.0 and whether they will introduce breaking changes in Kafka or not.
...
- If we are changing behavior how will we phase out the older behavior? It should gradually be phased out as users update their Kafka versions
- If we need special migration tools, describe them here. N/A
- When will we remove the existing behavior? N/A
Test Plan
We ran the following test on the latest trunk of Kafka with Zookeeper 3.6.3 and Zookeeper 3.8.12:
1) Start 1 Zookeeper node on an m5.4xlarge machine
...
5) Using 4 admin clients sequentially delete all topics
Zookeeper 3.8.1 2 request latency (PROPOSED)
...
Rejected Alternatives
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way. N/A