...

To circumvent this, we propose to update the logic of the broker to fail back to the latest version it knows instead of using version 0. This is possible if every new version (version N) of the request is always prefixed by its predecessor (version N-1). Oppositely, the client continues to fail back to the oldest version it knows (version 0).

of the request instead of failing back to version 0 when the brokers do not know about the version used to serialise the request. This is possible if every new version (version N) of the request is always prefixed by its predecessor (version N-1). This way, the broker will always be able to leverage the information it is aware of. The broker can send back the response corresponding the the version it has used to parse the request. The client won't be able to parse it with the version it expects but will be able to parse it with version 0 (its failback version).

Validation

Metrics & Log

Client

The client is responsible to provide its name and its version. For the Java client, we will rely on the metadata defined in the `kafka/kafka-version.properties` property file which is shipped in the jar.

In the broker, the name and the version of the client will be attached to the connection alongside already existing metadata such as the principal. This makes them reusable to later purposes (e.g. extending existing metrics).

Metadata

We propose to attach the various metadata captured to the connection alongside existing metadata such as the principal or the listener. A connection registry will be created to store metadata about all the active connections. Connections will be removed when they are closed.

Validation

We propose to validate the client name and the client version with ([\.\-_a-zA-Z0-9])+, and close the connection and log the error if they are not valid.

Metrics & Log

The various metrics described above will be created based on the metadata available in the connection registry. Metrics will be removed when they are inactive (gauge equals to zero). The request log will be extended to include the metadata collected.

Client

ApiVersions Request/Response Handling

The client continues to use the latest version of the ApiVersionsRequest it knows and continues to fail back to version 0 when it can not parse the ApiVersionsRequest it receives from the broker.

Name and Version

The client uses the version provided in the `kafka/kafka-version.properties` file and the name `apache-kafka-java`Metrics will be added and the request log will be extended to make the metadata available to the operators.

Compatibility, Deprecation, and Migration Plan

...