...

1. There is currently no way for a Kafka client to know which API version the broker supports. This means a client might not be able to perform its desired functionality, nor report any meaningful errors back to the application. Which leads in to problem number two:..the next problem, problem #2.

2. Unsupported requests are not handled gracefully by the broker. A broker receiving a protocol request it does not support will simply close the TCP connection to the client. From the client's point of view it has no way of knowing that the broker did not support the request, it could just as well be a network or software problem causing the connection reset.

...

ProtocolVersionRequest => (empty)
 
ProtocolVersionResponse => [ApiKey ApiName [ApiVersion]]
  ApiKey => int16                // API protocol key
  ApiName => string              // Human readable form of API, e.g. "ProduceRequest"
  ApiVersion => int16            // Supported versions in reverse order of preferredness, most preferred version firstlast (thus typically descendingascending order)
  ApiDeprecatedVersion                    => int16  // A value of -1 in the array should not be treated as a version but a barrier to indicate that any
                        // sub-sequent versions are deprecated, e.g.: [5,4,-1,3,2,1,0] means 5 & 4 are okay while versions 3-0 are deprecated.Deprecated versions
 
Example in pseudo format:
 
ProtocolVersionResponse =>
[
 { ApiKey: 0, ApiName: "ProduceRequest", ApiVersion: [2,1,0], ApiDeprecatedVersion: [] },
 { ApiKey: 1, ApiName: "FetchRequest", ApiVersion: [1,0], ApiDeprecatedVersion: [] },
 { ApiKey: 2, ApiName: "OffsetRequest", ApiVersion: [2], ApiDeprecatedVersion: [0,-1, 1] },
  ...
]

Proposed Changes

...

This KIP proposes a new protocol request and response to be added to the Kafka protocol, ProtocolVersionRequest and ProtocolVersionResponse. A client may use this new API to query the broker for its supported protocol request types and versions and use the information to select a suitable set of conforming API requests and versions for further communication with the broker.

If authentication is enabled, client must have permissions enabling "describe" action on "cluster" resource.

The ProtocolVersionResponse is only valid for the current connection to the broker; the client will need to reissue the ProtocolVersionRequest upon reconnect or when connecting to other brokers.

...