Status
Current state: "Under Discussion"
Discussion thread: here
JIRA: None (don't think this needs a Jira ticket)
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Kafka 3.7.x was meant to be the last minor release within the 3.x series followed then by a 4.0 release where Apache Zookeeper will be removed from the code base.
During the "Road to Kafka 4.0" discussion in the mailing list (https://lists.apache.org/thread/mxltb2k5y17q89vg0k9492dfc8z66jw1), it became apparent that there are some features that are not yet present in KRaft that the community is relying on to successfully operate and run Kafka. Thus, making it either hard or not feasible for a subset of the community to move to KRaft (or at least fully test KRaft) before Zookeeper's complete removal in 4.0.0. Some community members might only be able to try KRaft when Zookeeper is gone, leaving only a downgrade as a way out path if something doesn't work as expected (instead of config switching)
This KIP proposes to create at least one more minor version within the 3.x series ,3.8.0 release, and the scope of this KIP is to define which are the respective KIPs (mostly around KRaft) that should be part of it so the community can safely migrate to a potential 4.0 version.
This way community members can safely test KRaft without the need to perform downgrades or manual workarounds, as ZooKeeper is meant to be removed in 4.0.
Public Interfaces
No direct interfaces will be changed by this KIP
Proposed Changes
We propose the following:
- Have a release 3.8.x after 3.7.0
- The aim of this release would be KRaft strategic feature parity with Zookeeper
- Once 3.8.0 is branched, immediately start with the 4.0.x release
- 3.8.x would be the last minor release within the 3.x series.
This assumes all must have features / KIPs are implemented in 3.8. Otherwise we would need to inject a 3.9 version in between 3.8 and 4.0
Scope
The following KIPs were identified by the community as blockers items that would need to be present in a 3.x release before moving to a potential 4.0 release.
We would aim to have all these in a 3.8 release. However, in the case that this is not possible and some of these blockers are still not resolved, we should then create a new minor version on the 3.x series (namely 3.9).
The following list is not an exhaustive list of new features 3.8 would contain (this release will most probably contain other features and KIPs).
KIP | Status |
---|---|
KIP-853: KRaft Controller Membership Changes | Under Discussion |
A way of enabling unclean leader election by default in KRaft (Could be KIP-966 or others) | N/A |
Timeline
Release 3.7.0: TBD, probably early January 2024
Release 3.8.0: 3 to 4 months after 3.7 (or earlier if all required KIPs are already merged)
Release 4.0.0: 3 to 4 months after 3.8 branch is created
This assumes all must have features / KIPs are implemented in 3.8. Otherwise we would need to inject a 3.9 version in between 3.8 and 4.0
Compatibility, Deprecation, and Migration Plan
Not relevant for this KIP
Test Plan
This release will follow the same testing plan as usual, but we encourage community members who haven't migrated yet to KRaft, to test the Release Candidates (or even specific trunk builds) in order to detect as many potential problems with KRaft when used under these, potentially, new scenarios.
Rejected Alternatives
Some rejected alternatives are:
Parallel Development
- Have a multiple release heads: keep working on 3.x branches while also working on 4.x branches.
- This will create a huge overhead in terms of management and conflict resolution
Mark 4.0 as Experimental Release
- Mark release 4.0 as experimental and work towards a 4.x which would be stable
- We never did anything like this
- This would create confusion
- Risk of losing credibility among the community