...

In order for a client to successfully talk to a broker, it needs to know what version versions of protocol various protocols does the broker support. This in turn leads to following requirements.

Informing client of unknown protocol request or unsupported version

As mentioned before, broker on receiving a protocol request it does not support will simply close the TCP connection to the client. From the client's point of view it has no way of knowing that the broker did not support the request, it could just as well be a network or software problem causing the connection reset.

...

The KIP suggests the following to achieve that.

1. Use existing ApiVersion as a unique identifier of protocols' versions supported by a broker. ApiVersion has a monotonically increasing id, which is increased by 1 every time a new version of a protocol is added.

   sealed trait ApiVersion extends Ordered[ApiVersion] {
     val version: String // Version string, also has information on internal version. E.g., "0.9.0.X", "0.10.0-IV0"
     val id: Int // Monotonically increasing id. E.g., "0.10.0-IV0" has id 4
   }

2. Maintain protocol documentation per ApiVersion. Each time a new version of any protocol is added, a new ApiVersion, with id as last ApiVersion's id +1, is created and a protocol documentation for the new ApiVersion is created with documentation on newly added version of protocol. The new protocol documentation for the latest ApiVersion will be then added to main protocol documentation page. This procedure on adding protocol documentation for each new ApiVersion will be documented.

3. Document requirements on deprecating a version of a protocol. Deprecation of a protocol will be done by marking a protocol version as deprecated in protocol documents corresponding to the ApiVersions in which the protocol version is deprecated. This procedure on deprecating a protocol version will be documented.
4. On receiving a request for an unknown protocol request type, or for an unsupported version of a known type,

...

1. broker

...

1. will respond with an empty response only including the common Length+CorrId header with Length=0, instead of closing the connection. The effect of this empty message received by the client is that the client will fail to parse the response (since it will expect a proper non-empty reply) and throw an error to the application with the correct context - that of a protocol parsing failure for request type XYZ. All existing clients should be able to handle this gracefully without any alterations to the code, while updated clients supporting the proposals in this KIP can handle the empty response accordingly by returning a "Request not supported by server" error to the application. The level of detail in the returned error message to the application naturally varies between client implementations but is still by far better than a closed connection which says nothing of the underlying error.

This is a nice to have part of this KIP and not a blocker.

Informing client of protocol versions supported by a broker

The information can be provided in various ways. However after considering many alternatives, mentioned in rejected alternatives, the KIP suggests the following.

Maintain protocol versions per broker version

Protocol versions supported by a broker version will be documented, which will be versioned with broker version. Every time there is a version change in any of the protocols, documentation version needs to be updated and linked to main documentation page. Deprecation of a protocol version will be done via marking the version as deprecated on the protocol documentation.

Communicate broker version to clients

A client can deduce protocol versions supported by a broker by knowing its broker version. Note that broker version is only valid for the current connection to the broker, as the broker might upgrade/ downgrade to a different version during a re-connect. This leads to an interesting question, how can a broker convey broker version to its clients. There are a few options to achieve this.

1. New Metadata Request and Response version, v1, will be added to provide ApiVersion information to clients.

   Code Block
   MetadataRequest => Topics Topics => NullableArrayOf(STRING)   MetadataResponse => ApiVersion, BrokerEndPoints, TopicsMetaData // ApiVersion is added ApiVersion => ProtocolVersion, BuildDescription // E.g., 4,

2. Use (versionInt, versionString), e.g., (0x00100000, "0.10.0-IV0"). Client will have to parse versionString to get internal version, if they want to. Comparing versions will be tricky, due to internal versions. This requires adding and managing another entity in Kafka codebase that keeps track of broker version, apart from ApiVersion.

Use ApiVersion info which has one to one mapping with broker version, (versionId, versionString), e.g., (4,

3. "0.10.0-IV0"

). versionId is a monotonically increasing number that is bumped every time version for any protocol is bumped. This can re-use existing ApiVersion.Use (versionInt, versionId, versionString), e.g., (0x00100000, 4,

4. ProtocolVersion => INT16 // E.g., 4 BuildDescription => STRING // "0.10.0-IV0"

). Combines option 1 and 2.

Based on above options, we are looking at 3 things to identify a broker version, i.e., versionInt, versionId and versionString. Whether we use versionInt or versionId, client needs to keep track of what each versionInt/versionId supports. The major difference between versionInt and versionId is that versionInt do not care about internal versions, while versionId does. Since internal version is maintained as of now, going with Option 2 or 3 makes sense. Among option 2 and 3, option 2 has the minimal information required, the KIP suggests to go with option 2. 

Summary of the changes proposed as part of this KIP

1. , this should only be used for error messages, etc. BrokerEndPoints => Same as existing TopicsMetaData => Same as existing

2. A client willing to and capable of handling multiple protocol versions should send a Metadata Request v1 or above with topics as null for each new connection to a broker. A broker supporting Metadata Request v1 and above will respond with MetadataResponse that will contain ApiVersion, i.e., ProtocolVersion and BuildDescription, and will NOT contain any TopicsMetadata. Note that broker only provides information on its ProtocolVersion and BuildDescription, while responding to a MetadataRequest.
3. Client should get broker's ApiVersion from the broker's MetadataResponse and use the info to determine protocol version it should use to communicate with the broker. Note that ApiVersion obtained for a broker from the broker's MetadataResponse is good only for the current connection. In the event of disconnection, the client should obtain the ApiVersion information from the broker again, as the broker might have upgraded/ downgraded in the mean time
4. Every time a protocol version changes, for any request/response, ApiVersion will be bumped up.
5. Protocol documentation will be versioned with ApiVersion. Every time there is a ApiVersion change, protocol documentation version needs to be updated and linked to main documentation page.
6. Deprecation of protocol version will be done via marking the version as deprecated on the protocol documentation.
7. On getting unknown protocol version, broker will send an empty response, instead of simply closing client connection.
8. Metadata response will be enhanced to also contain broker version, i.e., (versionId, versionString), of the responding broker.
9. Metadata request with a null array can be used to fetch metadata response with only broker version information and no topic/broker info. Note that this will require changing MetadataRequest from ArrayOf[STRING] to NullableArrayOf[STRING].
On receiving a metadata request with null array, broker will respond without topics metadata, as this can lead to unnecessary penalty on large clusters
10. .

Public Interfaces

Changes to MetadataRequest and MetadataResponse

MetadataRequest and response version will be bumped. On receiving a MetadataRequest with new version, broker will respond with new MetadataResponse containing ApiVersion associated with the broker version.

...

. This information can be used by client to determine which protocol versions are supported by the broker.

Compatibility, Deprecation, and Migration Plan

...