Status

Current state: Draft

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Apache Kafka has many kinds of resources with names or identifiers, such as topics and consumer groups. However, unlike many systems, it is quite lax about specifying or enforcing maximum lengths for these identifiers. There are many ways in which accidentally or maliciously huge identifiers could cause problems, such as inflating the size of the data structures used by the group coordinator for managing group membership, or making command output unreadable. This KIP proposes setting maximum lengths for all resource names and identifiers present in the requests of the Kafka protocol.

The practical limit today for most identifiers is the serialization limit of a string in the Kafka protocol. For example, if you try to create a topic with a massively long name, a runtime exception is thrown in the admin client like this:

java.lang.RuntimeException: 'name' field is too long to be serialized
        at org.apache.kafka.common.message.CreateTopicsRequestData$CreatableTopic.addSize(CreateTopicsRequestData.java:548)
        at org.apache.kafka.common.message.CreateTopicsRequestData.addSize(CreateTopicsRequestData.java:207)
        at org.apache.kafka.common.protocol.SendBuilder.buildSend(SendBuilder.java:218)
        at org.apache.kafka.common.protocol.SendBuilder.buildRequestSend(SendBuilder.java:187)
        at org.apache.kafka.common.requests.AbstractRequest.toSend(AbstractRequest.java:110)
        at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:608)
        at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:582)
        at org.apache.kafka.clients.NetworkClient.send(NetworkClient.java:542)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1302)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1516)
        at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1459)
        at java.base/java.lang.Thread.run(Thread.java:840)

Although a topic name limit is applied by the broker, it is still possible to get an ugly client-side exception. As a result, it seems appropriate to apply the proper limit on the broker, and apply a looser sanity check on the client. For other identifiers such as group IDs, it is only the serialization limit which prevents excessively long identifiers because there is no broker-side limit defined.

Proposed Changes

This KIP proposes enforcing maximum lengths for all resource names and identifiers, starting in Apache Kafka 5.0. The suggested maximum values are chosen to be sufficiently high that they are very unlikely to impact any existing users, while providing stronger enforcement of limits in the Kafka protocol.

It is proposed that all resource names for which configuration properties and ACLs can be defined take the same maximum length as currently enforced for topic names, which is 249 characters. This is still a generous limit for other identifiers such as client IDs, so the KIP proposes the same limit for almost all resource names and identifiers.

The following table summarises the proposal:

A single new error code RESOURCE_IDENTIFIER_TOO_LARGE  is defined to use whenever any of the limits is breached, as opposed to a separate error code for each case. The exception message will provide more detail about which identifier was too long.

A broker configuration resource.identifier.limit.enable  is added which allows administrators of Kafka 4.x clusters to apply the limit in advance of Kafka 5.0, either to enforce the limit earlier or to evaluate whether they encounter problems with their existing workloads when the limits are applied. By setting the limits generously, it is anticipated that there will be no such problems.

Public Interfaces

Configuration

Broker configuration

Kafka protocol changes

Error codes

The following new error code is defined:

• RESOURCE_IDENTIFIER_TOO_LARGE  (to be assigned) - The resource name or identifier is too large.

This new error code can be returned in the response from any Kafka protocol request which breaches the limit for a resource name or identifier. The versions of the requests and responses are not bumped by this KIP.

The associated exception ResourceIdentifierTooLargeException  is a subclass of ApiException . It is not a retriable exception.

The exception will be thrown in the client if the supplied value exceeds the serialization sanity check (32767 characters).

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.