...

In order for a client to successfully talk to a broker, it needs to know what versions of various protocols does the broker support. The KIP suggests the following to achieve that.

1. A new ApiVersionQueryRequest ApiVersionRequest and Response type (version 0) will be added to allow clients to query the broker for supported API request types and versions.

   Code Block
   ApiVersionQueryRequestApiVersionRequest => ApiKeys ApiKeys = [INT16]   ApiVersionQueryResponseApiVersionResponse => ApiVersions ApiVersions = [ApiVersion] ApiVersion = ApiKey MinVersion MaxVersion ApiKey = INT16 MinVersion = INT16 MaxVersion = INT16

   1.  

2. ApiVersionQueryRequestApiVersionRequest.ApiKeys semantics:

   1. Empty Null array (size 0-1): return ApiVersions for all APIs supported by broker.

   2. Array of some ApiKeys: return ApiVersions for the requested API keys only.

3. ApiVersionQueryResponseApiVersionResponse..MinVersion-MaxVersion semantics:
   1. MinVersion and MaxVersion dictates the lowest and highest supported API versions for the API key (inclusive).
   2. All versions in the MinVersion-MaxVersion must be supported by the broker. Specific version may be deprecated through protocol documentation but must still be supported (broker must not disconnect the client and return a valid protocol response, although it is fair valid to return an error code if the specific API supports it).

4. Clients are recommended to use latest common supported API version.

5. Deprecation of a protocol version will be done by marking a protocol version as deprecated in protocol documentation. Documentation shall also be used to indicate a protocol version that must not be used, or for any such information. For instance, say 0.9.0 had protocol versions [0] for api key 1. On trunk, version 1 of the api key was added. Users running off trunk started using version 1 of the api and found out a major bug. To rectify that version 2 of the api is added to trunk. For some reason, it is now deemed important to have version 2 of the api in 0.9.1 as well. To do so, version 1 and version 2 both of the api will be backported to the 0.9.1 branch. 0.9.1 broker will return 0 as min supported version for the api and 2 for the max supported version for the api. However, the version 1 should be clearly marked as deprecated on its documentation. It will be client's responsibility to make sure they are not using any such deprecated version to the best knowledge of the client at the time of development (or alternatively by configuration). 
   Note that backporting of protocols has not been done in Kafka so far and in practice it can not be done until the Java client is changed so that it doesn't blindly use the latest protocol version. Otherwise, if new request versions were added to 0.9.0.2, the client would not be able to talk to a 0.9.0.1 broker.

6. Supported protocol versions obtained from a broker, is good only for current connection on which that information is obtained. In the event of disconnection, the client should Supported protocol versions obtained from a broker, is good only for current connection on which that information is obtained. In the event of disconnection, the client should obtain the information from the broker again, as the broker might have upgraded/ downgraded in the mean time.

7. If a client faces connection closure on sending ApiVersionQueryRequest it could be due to following reasons.
   1. Broker does not support the ApiVersionQueryRequest API, the client should use some other method of determining the API versions to use (a safe default, configuration, error out, probing).
   2. Connection closure was caused by issues other than unknown API request.

...

1. Versions of ApiVersion api should never be deprecated, as clients will rely on the api to inquire supported versions.
2. Java clients will use this API to assert that current API versions are all supported by the brokers it is talking to, and raise an exception if they are not. Java clients check will test the new API.
3. The new API will be tested with librdkafka's implementation as well.

Public Interfaces

Addition of ApiVersionQueryRequest ApiVersionRequest and Response version 0.

Compatibility, Deprecation, and Migration Plan

• This is an optional interface and does not affect existing clients.
• Updated clients that add support for the new ApiVersionQueryRequestApiVersionRequest will be able to adapt their functionality based on supported broker functionality.

...

1. Client sends metadata request and gets to know what brokers it needs to talk to for producing, consuming, etc.
2. Client opens a connection to those brokers and sends ApiVersionQueryRequestApiVersionRequest. Cl
3. Client collects ApiVersionQueryResponse ApiVersionResponse from all brokers it is interested in and builds a state that will look something like below.

   ApiVersionQueryResponseApiVersionResponse received:

   B1 -> (0, 0, 3), (1, 2, 3)
   B2-> (0, 1, 2), (1, 0, 3), (2, 0, 0)
   Versions state built by client.
   (0, 1, 2) // ApiKey, MinVersion across brokers, Max version across brokers
   (1, 2, 3)
4. Client is expected to have a map of feature to supported ApiVersions. Something like this.
   Feature1 -> {(0, 3, 3), (1, 2, 3)} // ApiKey, MinVersion supported, Max version supported
   Feature2 -> {(0, 0, 1), (1, 2, 3)}
   Note, that this KIP is not proposing to provide this information to clients. It is clients responsibility to maintain this information.
5. Using information from 3 and 4, client can see if it can use a feature or not. For the client in example, the result will be as below.
   Feature1 can not be used as version 3 of api_key 0 is not supported by all interested brokers.
   Feature 2 can be used.

...

• Include meta key-value tags in response ("broker.id=...", "message.max.bytes=...", ...)
• Include user-defined tags in response ("aws.zone=...", "rack=...", ..)
• Return array of supported versions, instead of min and max of supported versions, in MetadataResponse.
• Make java clients backwards compatible. This is a much wider discussion and warrants a separate KIP by itself.

Other options that were widely discussed.

...