Page History

...

The client does not know what ApiVersions versions the brokers support as the ApiVersions is used for this purpose. Today, the client sends an ApiVersionsRequest (AVR) with the latest schema it is aware of. When the broker receives it, it deserializes the AVR The broker handles it with the correct version if it knows it or fail back to version 0 otherwise, and sends back the response with the corresponding version to the clientsends back an ApiVersionsResponse v0 with an `UNSUPPORTED_VERSION` error to the client. When the client receives such error, it retries the whole process with the ApiVersionsRequest v0. It means that the broker will ignore won't get any additional information if a newer version is used that it does not know about yet. As an example, if version 4 of AVR would be released and used by the client, the broker wouldn't get the client name and the client version even though they would be passed on the wire.To circumvent this, we propose to update the logic of the broker to fail back to the latest version it knows instead of using version 0. This is possible if every new version (version N) of the request is always prefixed by its predecessor (version N-1). Oppositely, the client continues to fail back to the oldest . To circumvent this, we propose to add the supported version of the ApiVersionsRequest in the response sent back to the client alongside the error. The client will be able to leverage this version to send back the correct ApiVersionsRequest instead of defaulting to version 0.

At the moment, the ApiVersionsRequest is handled in two different places in the broker: 1) in the SaslServerAuthenticator (when used); and 2) in the KafkaApis. Both places will be updated to ensure that all clients work. We have decided to not refactor the handling of the ApiVersionsRequest for now and it leave it for further improvements.   version it knows (version 0).

Metadata

We propose to attach the various metadata captured to the connection alongside existing metadata such as the principal or the listener. A registry will be created to store metadata about all the active connections. Connections will be removed when they are closed.

...

We propose to validate the client name and the client version with ([\.\-_a-zA-Z0-9])+, and to close the connection and log the error if they are not valid. The validation may sound brutal but as the metadata are fixed in the client, the error should only happen during the development of the client.

Metrics & Log

The various metrics described above will be created based on the metadata available in the connection registry. Metrics will be removed when they are inactive (gauge equals to zero). The request log will be extended to include the metadata collected.

Client (Java)

ApiVersions Request/Response Handling

The client continues to As mentioned earlier, the client will use the latest version of the ApiVersionsRequest it knows and continues provided on the ApiVersionsResponse to fail back to the latest one instead of defaulting to version 0 when it can not parse the ApiVersionsRequest it receives from the broker.

When SASL is used, the (Java) client sends two ApiVersionsRequest to the broker. The first one is sent by the SaslClientAuthenticator and the second one is send by the NetworkClient when the KafkaChannel is established. The SaslClientAuthenticator always sends version 0 of the AVR. We have decided to not change this for now and to only update the second call which always happens. The reasoning behind this choice is to avoid multiplying the round trip when an unknown version is used by the client, version 0 always works.

Name and Version

The client uses the version provided in the `kafka/kafka-version.properties` file and the name `apache-kafka-java`.

...

Add a new request to communicate the client metadata to the broker

Instead of piggy backing on the ApiVersionsRequest, we could implement a new Request/Response only for this purpose. This request would need to be sent as early as possible when the connection is established in order to have the information in the broker. Concretely, it means that it would be sent right after the ApiVersionsRequest/Response round trip and before any other request is sent. It A new separate request/response could be used for the purpose. This option has been discarded because it would add another round trip to the broker before the client can proceed with its regular stuff. It also would require to be done before the authentication (TLS AuthN aside) and thus requiring specific treatment, similarly to the ApiVersionsRequest.in the establishment of the KafkaChannel.