Page History

...

1. Use existing ApiVersion as a unique identifier of protocols' versions supported by a broker. ApiVersion has a monotonically increasing id, which is increased by 1 every time a new version of a protocol is added.

   sealed trait ApiVersion extends Ordered[ApiVersion] {
     val version: String // Version string, also has information on internal version. E.g., "0.9.0.X", "0.10.0-IV0"
     val id: Int // Monotonically increasing id. E.g., "0.10.0-IV0" has id 4
   }

2. Maintain protocol documentation per ApiVersion. Each time a new version of any protocol is added, a new ApiVersion, with id as last ApiVersion's id +1, is created and a protocol documentation for the new ApiVersion is created with documentation on newly added version of protocol. The new protocol documentation for the latest ApiVersion will be then added to main protocol documentation page. This procedure on adding protocol documentation for each new ApiVersion will be documented.

   For instance say we have two brokers, BrokerA has ApiVersion 4 and BrokerB has ApiVersion 5. This means we should protocol documentations for ApiVersions 4 and 5. Say we have the following as protocol documentation for these two versions.

   Sample Protocol Documentation V4
   Version: 4 // Comes from ApiVersion
   REQ_A_0: ...
   REQ_A_1: ...
   RESP_A_0: ...
   RESP_A_1: ...

   Sample Protocol Documentation V5
   Version: 5 // Comes from ApiVersion
   REQ_A_1: ...
   REQ_A_2: ...
   RESP_A_1: ...
   RESP_A_2: ...

   All a client needs to know to be able to successfully communicate with a broker is what is the supported ApiVersion of the broker. Say via some mechanism, discussed below, client gets to know that BrokerA has ApiVersion 4 and BrokerB has ApiVersion 5. With that information, and the available protocol documentations for those ApiVersions client can deduce what protocol versions does the broker supports. In this case client will deduce that it can use v0 and v1 of REQ_A and RESP_A while talking to BrokerA, while it can use v1 and v2 of REQ_A and RESP_A while talking to BrokerB.

3. Document requirements on deprecating a version of a protocol. Deprecation of a protocol will be done by marking a protocol version as deprecated in protocol documents corresponding to the ApiVersions in which the protocol version is deprecated. This procedure on deprecating a protocol version will be documented.
4. On receiving a request for an unknown protocol request type, or for an unsupported version of a known type, broker will respond with an empty response only including the common Length+CorrId header with Length=0, instead of closing the connection. The effect of this empty message received by the client is that the client will fail to parse the response (since it will expect a proper non-empty reply) and throw an error to the application with the correct context - that of a protocol parsing failure for request type XYZ. All existing clients should be able to handle this gracefully without any alterations to the code, while updated clients supporting the proposals in this KIP can handle the empty response accordingly by returning a "Request not supported by server" error to the application. The level of detail in the returned error message to the application naturally varies between client implementations but is still by far better than a closed connection which says nothing of the underlying error.

5. New Metadata Request and Response version, v1, will be added to provide ApiVersion information to clients. ApiVersion will be composed of ProtocolVersion, an int, and BuildDescription, a String. BuildDescription should only be used where readability is a concern, like, logs, error-messages, tool outputs, etc.

   Code Block

   MetadataRequest => Topics Topics => NullableArrayOf(STRING)   MetadataResponse => ApiVersion, BrokerEndPoints, TopicsMetaData // ApiVersion is added ApiVersion => ProtocolVersion, BuildDescription // E.g., 4, "0.10.0-IV0" ProtocolVersion => INT16 // E.g., 4 BuildDescription => STRING // "0.10.0-IV0", this should only be used for error messages, etc. BrokerEndPoints => Same as existing TopicsMetaData => Same as existing

6. A client willing to and capable of handling multiple protocol versions should send a Metadata Request v1 or above with topics as null for each new connection to a broker. A broker supporting Metadata Request v1 and above will respond with MetadataResponse that will contain ApiVersion, i.e., ProtocolVersion and BuildDescription, and will NOT contain any TopicsMetadata. Note that broker only provides information on its ProtocolVersion and BuildDescription, while responding to a MetadataRequest.
7. Client should get broker's ApiVersion from the broker's MetadataResponse and use the info to determine protocol version it should use to communicate with the broker. Note that ApiVersion obtained for a broker from the broker's MetadataResponse is good only for the current connection. In the event of disconnection, the client should obtain the ApiVersion information from the broker again, as the broker might have upgraded/ downgraded in the mean time.

...