Status

Current state: Under Discussion

Discussion thread: discussion threadhere [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-79751 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

We still have some users using very old Kafka client libraries, such as Kafka 8 and Kafka 9. We need a mechanism to force users to upgrade their client libraries, or and forbid outdated client libraries.

Some The old libraries are still supported by the new latest brokers, but can cause problems when we starting using some new features. Some examples are:

1. Kafka 8 / 9 / 10 consumer hangs when the message contains message header (

   Jira
   server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-6739

   )
2. LZ4 is not correctly handled in Kafka 8 and Kafka 9 (

   Jira
   server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key KAFKA-3160

   )
3. Performance penalty of converting message format from V3 to V1 or V2 for the old consumers (KIP-31 - Move to relative offsets in compressed message sets) 

In general, we think it's reasonable to give cluster administrator the capability to forbid some old client library versionversions.

In this KIP, we propose to provide client API version to authorizer. So that, the user defined authorizer can decide allowing or not allowing the operation based on client library versions.add a new broker configuration to specify the minimum client version. 

Public Interfaces

A public interface is any change to the following:

• Configuration

  Add a new broker configuration:

• Binary log format

  No

• The network protocol and api behaviorNo

  Client requests older than the specified versions will be rejected by the broker. 

• Any class in the public packages under clientsConfiguration, especially client configuration

  No

• Monitoring

  No

...

• Anything else that will likely break existing users in some way when they upgrade
  No

Proposed Changes

Public Interfaces and classes

Authorizer

...

...

All the changes of this KIP are in Kafka broker:

1. Add min.api.version configuration in Kafka broker
2. Allow dynamic update of min.api.version configuration
3. Reject the API requests of older versions. The broker will return an UNSUPPORTED_VERSION error to the client, and log an INFO level log message.
4. In ApiVersionResponse, the MinVersion of each API is set to the current setting of min.api.version.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
  No impact to the users that are not using user-defined authorizer. For the users that are using user-defined authorizer, they will have to modify their authorizer code to use the new interface.By default, all the existing client libraries are allowed. If the administrator set 

Rejected Alternatives

Alternative 1 -

...

Provide api versions to authorizer

Provide client API version to authorizer. So that, the user-defined authorizer can block clients based on api versions. This is more flexible. But there are 2 concerns: 1) Blocking old clients is not a security issue. It doesn't make much sense to add this feature to authorizer. 2) For most users, it's easier to set a broker configuration than write a user-defined authorizer.

Alternative 2 - Use Kafka release version rather than API version

Users are more familiar with Kafka release versions. So far, the API version is mostly a Kafka internal thing. It's more convenient for the users to use Kafka release versions than API versions.

However, we can't use Kafka release versions here, because Kafka clients do not report the release versions to brokers. Brokers can only see the clients' API versions. An API version doesn't map a single release version. For example, a user sets the min Kafka version of produce request to Kafka 1.1. She would expect the broker will reject Kafka 1.0 producers. However, both Kafka 1.1 and Kafka 1.0 are using api version 5. The broker can't distinguish the 2 versions. So, Kafka 1.0 producers are still allowed. This can be version confusing to the users.

Moreover, there are 3rd party implementations of Kafka clients (e.g. go client, c++ client). Those client libraries have their own release versions, and are not related with any specific Java / Scala client version.

The method proposed in this KIP provides more flexibility to the cluster administrators. For example, they can forbid old consumers, but still allow old producers; or forbid old clients for a set of topics (e.g. the topics that are using new message format), while still allow for the other topics.

Alternative 2 - Provide the authorizer the entire Request object

From the request object, users can many other information about the request (e.g. the message content for produce request), in addition to the api version. The concern is may expose too much broker internal details to the authorizer, and may be abused.