...

In order for a client to successfully talk to a broker, it needs to know what versions of various protocols does the broker support. The KIP suggests the following to achieve that.

1. New Metadata Request and Response version, v1, will be added to provide information on supported protocol versions to clients.

   Code Block

   MetadataRequest => api_keys, topics api_keys => [INT16] topics => [STRING]   MetadataResponse => api_versions, brokers, topic_metadata api_versions => [single_api_versions] single_api_versions => api_key, min_version, max_version api_key => INT16 min_version => INT16 max_version => INT16 brokers => node_id host port node_id => INT32 host => STRING port => INT32 topic_metadata => topic_error_code topic [partition_metadata] topic_error_code => INT16 topic => STRING partition_metadata => partition_error_code partition_id leader [replicas] [isr] partition_error_code => INT16 partition_id => INT32 leader => INT32

   Metadata Request v1 can be used by clients to ask for supported protocol versions by a broker. Client can send Metadata Request, v1, with api_keys set to one of the following values.

   1. Empty array, this will get the client supported versions for all ApiKeys.

   2. Array of some ApiKeys, this will get client supported versions for the specified ApiKeys.

   3. Null array, this will not get any information on ApiKeys. 

2. Broker on receiving a Metadata Request v1, will respond with MetadataResponse v1. The response will contain information on supported versions for api keys specified in the request, if an empty array of api keys is sent, then broker will respond with info on all api keys, and if null array is sent for api keys, then no information on protocol versions is added to the response. Note that broker only provides information on its supported protocol versions.

3. Clients can choose to not receive topics metadata by specifying a null array for topics in Metadata Request v1.

4. Deprecation of a protocol version

5. Use existing ApiVersion as a unique identifier of protocols' versions supported by a broker. ApiVersion has a monotonically increasing id, which is increased by 1 every time a new version of a protocol is added.

   sealed trait ApiVersion extends Ordered[ApiVersion] {
     val version: String // Version string, also has information on internal version. E.g., "0.9.0.X", "0.10.0-IV0"
     val id: Int // Monotonically increasing id. E.g., "0.10.0-IV0" has id 4
   }

6. Maintain protocol documentation per ApiVersion. Each time a new version of any protocol is added, a new ApiVersion, with id as last ApiVersion's id +1, is created and a protocol documentation for the new ApiVersion is created with documentation on newly added version of protocol. The new protocol documentation for the latest ApiVersion will be then added to main protocol documentation page. This procedure on adding protocol documentation for each new ApiVersion will be documented.

   For instance say we have two brokers, BrokerA has ApiVersion 4 and BrokerB has ApiVersion 5. This means we should have protocol documentations for ApiVersions 4 and 5. Say we have the following as protocol documentation for these two versions.

   Sample Protocol Documentation V4
   Version: 4 // Comes from ApiVersion
   REQ_A_0: ...
   REQ_A_1: ...
   RESP_A_0: ...
   RESP_A_1: ...

   Sample Protocol Documentation V5
   Version: 5 // Comes from ApiVersion
   REQ_A_1: ...
   REQ_A_2: ...
   RESP_A_1: ...
   RESP_A_2: ...

   All a client needs to know to be able to successfully communicate with a broker is what is the supported ApiVersion of the broker. Say via some mechanism, discussed below, client gets to know that BrokerA has ApiVersion 4 and BrokerB has ApiVersion 5. With that information, and the available protocol documentations for those ApiVersions client can deduce what protocol versions does the broker supports. In this case client will deduce that it can use v0 and v1 of REQ_A and RESP_A while talking to BrokerA, while it can use v1 and v2 of REQ_A and RESP_A while talking to BrokerB.

7. Document requirements on deprecating a version of a protocol. Deprecation of a protocol

   will be done by marking a protocol version as deprecated in protocol

   documents corresponding to the ApiVersions in which the protocol version is deprecated. This procedure on deprecating

   documentation. Documentation shall also be used to indicate a protocol version

   will be documented.
8. On receiving a request for an unknown protocol request type, or for an unsupported version of a known type, broker will respond with an empty response only including the common Length+CorrId header with Length=0, instead of closing the connection. The effect of this empty message received by the client is that the client will fail to parse the response (since it will expect a proper non-empty reply) and throw an error to the application with the correct context - that of a protocol parsing failure for request type XYZ. All existing clients should be able to handle this gracefully without any alterations to the code, while updated clients supporting the proposals in this KIP can handle the empty response accordingly by returning a "Request not supported by server" error to the application. The level of detail in the returned error message to the application naturally varies between client implementations but is still by far better than a closed connection which says nothing of the underlying error.

9. New Metadata Request and Response version, v2, will be added to provide ApiVersion information to clients. ApiVersion will be composed of ProtocolVersion, an int, and BuildDescription, a String. BuildDescription should only be used where readability is a concern, like, logs, error-messages, tool outputs, etc.

   Code Block

   MetadataRequest => Topics Topics => NullableArrayOf(STRING)   MetadataResponse => ApiVersion, BrokerEndPoints, TopicsMetaData // ApiVersion is added ApiVersion => ProtocolVersion, BuildDescription // E.g., 4, "0.10.0-IV0" ProtocolVersion => INT16 // E.g., 4 BuildDescription => STRING // "0.10.0-IV0", this should only be used for error messages, etc. BrokerEndPoints => Same as existing TopicsMetaData => Same as existing

10. A client willing to and capable of handling multiple protocol versions should send a Metadata Request v2 or above with topics as null for each new connection to a broker. A broker supporting Metadata Request v2 and above will respond with MetadataResponse that will contain ApiVersion, i.e., ProtocolVersion and BuildDescription, and will NOT contain any TopicsMetadata. Note that broker only provides information on its ProtocolVersion and BuildDescription, while responding to a MetadataRequest.

11. that must not be used, or for any such information.For instance, say 0.9.0 had protocol versions [0] for api key 1. On trunk, version 1 of the api key was added. Users running of trunk started using version 1 of the api and found out a major bug. To rectify that version 2 of the api is added to trunk. For some reason, it is now deemed important to have version 2 of the api should be added to 0.9.1 as well. To do so, version 1 and version 2 both of the api will be backported to the 0.9.1 branch. 0.9.1 broker will return 0 as min supported version for the api and 2 for the max supported version for the api. However, the version 1 should be clearly marked as deprecated on its documentation. It will be client's responsibility to make sure they are not using any such deprecated version.

12. Supported protocol versions obtained from a broker, is good only for current connection on which that information is obtained

    Client should get broker's ApiVersion from the broker's MetadataResponse and use the info to determine protocol version it should use to communicate with the broker. Note that ApiVersion obtained for a broker from the broker's MetadataResponse is good only for the current connection

    . In the event of disconnection, the client should obtain the

    ApiVersion

    information from the broker again, as the broker might have upgraded/ downgraded in the mean time.

13. If a client faces connection closure on sending MetadataRequest Vn, where n is >=1, seeking supported protocol versions, it could be due to following reasons.
    1. Broker does not support version Vn of MetadataRequest. It is advised that client starts from V1 of metadata request and incrementally try above versions for connection closures.
    2. Connection closure was caused by issues other than unknown MetadataRequest version.

Public Interfaces

Changes to MetadataRequest and MetadataResponse

MetadataRequest and response version MetadataResponse v1 will be bumped. On receiving a MetadataRequest with new version, broker will respond with new MetadataResponse containing ApiVersion associated with the broker. This information can be used by client to determine which protocol versions are supported by the broker.added.

Compatibility, Deprecation, and Migration Plan

• Existing clients will get the error propagation benefits of not closing the connection for unknown protocol versions. (graceful errors).
• Updated clients that add support for new MetadataRequest will be able to adapt their functionality based on broker functionality and provide meaningful error propagation to applications when proper broker support is lacking.

Rejected Alternatives

• Include meta key-value tags in response ("broker.id=...", "message.max.bytes=...", ...)
• Include user-defined tags in response ("aws.zone=...", "rack=...", ..)
• Using a versionInt instead of ApiVersion's id as broker protocol version.

• Use a new protocol request and response, ProtocolVersionRequest and ProtocolVersionResponse. A client may use this new API to query the broker for its supported protocol request types and versions and use the information to select a suitable set of conforming API requests and versions for further communication with the broker.

• Return array of supported versions, instead of min and max of supported versions, in MetadataResponse.