...

The inter.broker.protocol.version would have to be 2.1 or higher.

Finally, a couple of observability changes:

• Introduce metric kafka.network:type=RequestMetrics,name=DeprecatedRequestsPerSec,request=(api-name),version=(api-version),clientSoftwareName=(client-software-name),clientSoftwareVersion=(client-software-version)
• Add boolean field requestApiVersionDeprecated  to the request header section of the request log (alongside requestApiKey , requestApiVersion , requestApiKeyName , etc.).

Proposed Changes

As stated in the Public Interfaces  sections, a number of protocol API and inter broker protocol versions would no longer be supported. Kafka brokers would return the UNSUPPORTED_VERSION error when they receive a request with any of the removed API versions. Similarly, the Java clients (producer, consumer and admin) would throw UnsupportedVersionException when interacting with brokers that do not support the newly established minimum protocol API versions. A broker configured with an unsupported inter-broker protocol version would fail during start-up with an error.

...

Users relying on one of the clients included in this document will have to ensure they are using the minimum version described (or a newer version) with the right configuration* before upgrading their Kafka clusters to Apache Kafka 4.0. Users relying on clients not included in this document will have to ensure they are using a version that does not rely on protocol API versions that are being removed as part of this proposal.

The following metrics are used to determine both questions:

...

 If the newly introduced DeprecatedRequestsPerSec metric is 0, the upgrade is safe. If it's not zero, users can inspect the request log for additional details. The newly introduced field requestApiVersionDeprecated makes it convenient to find the relevant request log entries.

As stated previously, the UNSUPPORTED_VERSION error will be returned in case incompatible clients are used with Kafka clusters running Apache Kafka 4.0.

...

Finally, users should ensure the brokers are running at least Apache Kafka 2.1 and inter.broker.protocol.version is set to 2.1 or higher before upgrading brokers to Apache Kafka 4.0.

* This is relevant for clients that do not auto negotiate protocol api versions and default to older versions - Sarama is one such client.

Test Plan

• Client compatibility system tests will updated so that 2.0.x clients fail with an UNSUPPORTED_VERSION error and older versions are removed.
• Client compatibility system tests will updated so that 4.0.x clients fail with an UNSUPPORTED_VERSION error when dealing with Apache Kafka brokers that are too old (2.0 and/or 1.x).
• Broker upgrade tests will be updated so that upgrades from 2.0.x fail with an appropriate error and upgrades from older broker versions are removed.
• Protocol API integration tests will be updated so that the highest unsupported version and lowest supported version are tested and older versions are not.
• Protocol API unit tests will continue to cover all versions, but the expectations will be updated for unsupported versions.
• Test simple flows end to end with the clients mentioned in this proposal against a cluster with this KIP implemented.

...