Current state: Draft
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Apache Kafka has many kinds of resources with names or identifiers, such as topics and consumer groups. However, unlike many systems, it is quite lax about specifying or enforcing maximum lengths for these identifiers. There are many ways in which accidentally or maliciously huge identifiers could cause problems, such as inflating the size of the data structures used by the group coordinator for managing group membership, or making command output unreadable. This KIP proposes setting maximum lengths for all resource names and identifiers present in the requests of the Kafka protocol.
The practical limit today for most identifiers is the serialization limit of a string in the Kafka protocol. For example, if you try to create a topic with a massively long name, a runtime exception is thrown in the admin client like this:
java.lang.RuntimeException: 'name' field is too long to be serialized at org.apache.kafka.common.message.CreateTopicsRequestData$CreatableTopic.addSize(CreateTopicsRequestData.java:548) at org.apache.kafka.common.message.CreateTopicsRequestData.addSize(CreateTopicsRequestData.java:207) at org.apache.kafka.common.protocol.SendBuilder.buildSend(SendBuilder.java:218) at org.apache.kafka.common.protocol.SendBuilder.buildRequestSend(SendBuilder.java:187) at org.apache.kafka.common.requests.AbstractRequest.toSend(AbstractRequest.java:110) at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:608) at org.apache.kafka.clients.NetworkClient.doSend(NetworkClient.java:582) at org.apache.kafka.clients.NetworkClient.send(NetworkClient.java:542) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1302) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1516) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1459) at java.base/java.lang.Thread.run(Thread.java:840) |
Although a topic name limit is applied by the broker, it is still possible to get an ugly client-side exception. As a result, it seems appropriate to apply the proper limit on the broker, and apply a looser sanity check on the client. For other identifiers such as group IDs, it is only the serialization limit which prevents excessively long identifiers because there is no broker-side limit defined.
This KIP proposes enforcing maximum lengths for all resource names and identifiers, starting in Apache Kafka 5.0. The suggested maximum values are chosen to be sufficiently high that they are very unlikely to impact any existing users, while providing stronger enforcement of limits in the Kafka protocol.
It is proposed that all resource names for which configuration properties and ACLs can be defined take the same maximum length as currently enforced for topic names, which is 249 characters. This is still a generous limit for other identifiers such as client IDs, so the KIP proposes the same limit for almost all resource names and identifiers.
The following table summarises the proposal:
Resource name or identifier | Current limit | Proposed client sanity check | Proposed broker limit | Proposed error code when broker limit breached | Notes |
|---|---|---|---|---|---|
Topic name | 249 characters | 32767 characters | 249 characters -no change |
| |
Group ID | Serialization limit of a string | 32767 characters | 249 characters |
| |
Group member ID | Serialization limit of a string | Not applicable - generated by the client library | 36 characters |
| In KIPs 848, 932 and 1071, the client library is intended to generate a UUID to be used as the member ID. As a result, the maximum should be the length of a serialized UUID, such as |
Group instance ID | Serialization limit of a string | 32767 characters | 249 characters |
| |
Client ID | Serialization limit of a string | 32767 characters | 249 characters |
| |
Transactional ID | Serialization limit of a string | 32767 characters | 249 characters |
| |
Offset commit metadata | 4096 bytes | 32767 characters | 4096 bytes - no change |
| Controlled by the |
Rack ID | Serialization limit of a string | 32767 characters | 249 characters |
| |
Resource name for configs and ACLs | Serialization limit of a string | 32767 characters | 249 characters |
| |
ACL principal | Serialization limit of a string | 32767 characters | 249 characters |
| |
ACL host | Serialization limit of a string | 32767 characters | 249 characters |
|
A single new error code RESOURCE_IDENTIFIER_TOO_LARGE is defined to use whenever any of the limits is breached, as opposed to a separate error code for each case. The exception message will provide more detail about which identifier was too long.
A broker configuration resource.identifier.limit.enable is added which allows administrators of Kafka 4.x clusters to apply the limit in advance of Kafka 5.0, either to enforce the limit earlier or to evaluate whether they encounter problems with their existing workloads when the limits are applied. By setting the limits generously, it is anticipated that there will be no such problems.
| Configuration | Description | Values |
|---|---|---|
resource.identifier.limit.enable | Whether the cluster enforces maximum sizes for resource names and identifiers. | In Kafka 4.x: default In Kafka 5.0 and later: default |
The following new error code is defined:
RESOURCE_IDENTIFIER_TOO_LARGE (to be assigned) - The resource name or identifier is too large.This new error code can be returned in the response from any Kafka protocol request which breaches the limit for a resource name or identifier. The versions of the requests and responses are not bumped by this KIP.
The associated exception ResourceIdentifierTooLargeException is a subclass of ApiException . It is not a retriable exception.
The exception will be thrown in the client if the supplied value exceeds the serialization sanity check (32767 characters).
Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.