...

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA:

...

Apache Kafka 2.1 has been chosen as the baseline because it was released many years ago and it includes support for KIP-320 (required for proper fencing support when consuming) and KIP-110 (zstd is a popular compression mechanism).

Since some protocol APIs are used both for clients and inter-broker replication (e.g. Fetch, ListOffsets, etc.), this proposal also implies a minimum inter.broker.protocol.version  of 2.1. That said, the removal of zk mode (also planned for Apache Kafka 4.0) will have even stricter requirements when it comes to this.

We believe there is merit in having a general protocol API versioning policy that can be applied mechanically on each new major version - we intend to submit a separate KIP for that. It would only apply from Apache Kafka 5.0 to give the ecosystem plenty of time to adapt.

...

The Java client is part of the Apache Kafka repository, it is released alongside the broker and it generally supports the latest protocol versions exposed by the broker. Given that, we know that the Java client has supported the protocol API versions baseline set by Apache Kafka 2.1 as part of that release in November 2018 and no further analysis is required.

The inter.broker.protocol.version would have to be 2.1 or higher.

Proposed Changes

As stated in the Public Interfaces  sections, a number of protocol API and inter broker protocol versions would no longer be supported. Kafka brokers would return the UNSUPPORTED_VERSION error when they receive a request with any of the removed API versions. Similarly, the Java clients (producer, consumer and admin) would throw UnsupportedVersionException when interacting with brokers that do not support the newly established minimum protocol API versions. A broker configured with an unsupported inter-broker protocol version would fail during start-up with an error.

Compatibility, Deprecation, and Migration Plan

...

Similarly, users should ensure their brokers are running at least Apache Kafka 2.1 before upgrading their Java clients to Apache Kafka 4.0.

Finally, users should ensure the brokers are running at least Apache Kafka 2.1 and inter.broker.protocol.version is set to 2.1 or higher before upgrading brokers to Apache Kafka 4.0.

Test Plan

• Client compatibility system tests will updated so that 2.0.x clients fail with an UNSUPPORTED_VERSION error and older versions are removed.
• Client compatibility system tests will updated so that 4.0.x clients fail with an UNSUPPORTED_VERSION error when dealing with Apache Kafka brokers that are too old (2.0 and/or 1.x).
• Broker upgrade tests will be updated so that upgrades from 2.0.x fail with an appropriate error and upgrades from older broker versions are removed.
• Protocol API integration tests will be updated so that the highest unsupported version and lowest supported version are tested and older versions are not.
• Protocol API unit tests will continue to cover all versions, but the expectations will be updated for unsupported versions.
• Test simple flows end to end with the clients mentioned in this proposal against a cluster with this KIP implemented.

...