...

Add the new transaction version and group version features. For every metadata version, we will map to the corresponding transaction and group version. For MVs that existed before these features, we map the new features to version 0.

Fixing KAFKA-17011 and KAFKA-17492

As part of KIP-584, brokers expose a range of supported versions for each feature. For example, metadata.version might be supported from 1 to 21. (Note that feature level ranges are always inclusive, so this would include both level 1 and 21.)

These supported ranges are supposed to be able to include 0. The 0 version has two meanings, one is that this feature isn’t completed and another is to disable this feature. For example, it should be possible for a broker to support a kraft.version of [0, 1]. However, in older software versions, there is an assertion in org.apache.kafka.common.feature.SupportedVersionRange that prevents this. This causes problems when the older software attempts to deserialize an ApiVersionsResponse containing such a range.

In order to To resolve this dilemma, we will create a new version 4 for ApiVersionsRequest. Clients which send v4 promise to be able to handle ranges including 0. Clients which send v3 will never see a range that starts with 0. Such ranges will be translated into ranges starting with 1 instead.

Similarly, for BrokerRegistrationRequest, we will create a new version 4. Servers which accept v4 promise to be able to handle ranges including 0. Servers which do not will get ranges that only start with 1, rather than 0.

there are two approaches:

1. Modify the minimum version from 0 to 1 to pass old client validation. The advantage is that the client will receive an error message, but this removes the ability to disable the feature.
2. Omit features version 0 in requests. The downside of this solution is that it won't trigger any error messages.

We chose the first solution in KAFKA-17011, so we created a new version 4 for ApiVersionsRequest. Clients sending v4 will be able to handle ranges including 0, while clients sending v3 will never see ranges starting with 0. These ranges will instead be translated to start at 1.

Similarly, for BrokerRegistrationRequest, we will create a new version 4. Servers which accept v4 promise to be able to handle ranges including 0. Servers which do not will get ranges that only start with 1, rather than 0. 

There is another issue with BrokerRegistrationRequest: when a new version broker sends a BrokerRegistrationRequest to an old version controller, the new request sets the minSupportedVersion to 1. However, the old controller doesn’t support the version range of minSupportVersion to maxSupportVersion, leading to a dilemma.

In KAFKA-17492, we implemented the second solution while keeping the new version v4 for ApiVersionsRequest. Clients sending versions v0-v3 with minSupportedVersion 0 will have that version omitted.

Similarly, for BrokerRegistrationRequest, we retain version v4. Servers that accept v4 can handle ranges including 0, while servers using versions v0-v3 will omit features with version 0 in the BrokerRegistrationRequestFinally, if the maximum supported version is 0, the feature is simply not exposed as supported at all via the ApiVersionsResponse or the BrokerRegistrationRequest. Likewise, if a finalized feature is 0, it is not returned in the ApiVersionsResponse.

Compatibility, Deprecation, and Migration Plan

...