Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-3492

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-13 introduced client quotas in Kafka 0.9.0.0. Rate limits on producers and consumers are enforced to prevent clients saturating the network or monopolizing broker resources. The current implementation allocates quotas to client-ids. This works well in single user clusters or clusters that use PLAINTEXT where all users have the same identity. But since client-id is unauthenticated and can be set to any value by the client, multi-tenant secure installations require quotas to be enforced for authenticated principals to guarantee fair allocation of resources and prevent denial-of-service.

This KIP adds an option to apply quotas based on authenticated principal to prevent users generating heavy traffic from monopolizing resources and impacting the performance of other users in a multi-tenant cluster.

Public Interfaces

Configuration Options

A configuration option quota.type will be added with supported values "client-id" and "user". This option can be used to choose between the existing client-id based implementation and the new authenticated user principal based implementation.  The default value will be client-id to be consistent with Kafka 0.9.0.x and 0.10.0.x.

Default quota configs will apply to user principals if quota.type=user.

Metrics

If quota.type=user, quota related metrics will be generated for user principals rather than client-ids.

Metric names currently use the tag client-id. The tag user will be set instead when quota.type=user. This makes the metrics tag name identical to quota.type and leaves existing metrics for client-ids unchanged.

Sensor names are currently a sensor type concatenated with the client id value, eg. FetchThrottleTime-clientA. This will be modified to include quota type using the format sensor.type-quota.type-quota.id. For example sensor name  FetchThrottleTime-client-id-clientA will be the throttle time sensor for clientA when quota.type=client-id and FetchThrottleTime-user-user1 will be the throttle time sensor for user1 when quota.type=user. Since sensor names are not reflected in JMX names, the change in sensor names for client ids will not impact externally visible metrics.

Tools

kafka-configs.sh will be extended to support quota overrides for user principals.  A new entity type “users” will be added which provides quota configuration similar to the existing entity type "clients".

Proposed Changes

User Principal 

Authenticated user principal will be obtained from the Session object when quota.type=user. Base64-encoded hex string version of the Principal will be used so that it can be used as a node name in Zookeeper and as the metric name without placing any restrictions on the characters allowed in the principal.  For PLAINTEXT, the principal is "anonymous" by default and quotas will be applied for that principal. But principal can be overridden using a custom principal builder even for PLAINTEXT, enabling different user quotas, for example, for connections from different IP addresses.

Quota Configuration

Quotas are currently configured as the rate limits for producers and consumers based on their client-id. The same set of limits will be configurable for user principals. If quota.type=user, the configuration specified for user principals will be used. Default options will applied to either client-id or user principal based on the value of quota.type.

Quota Identifier

Quota configuration and metrics currently use client-id as the unique key, enforcing one quota for all clients with the same client-id. This will be replaced with a new quota-id. Each quota-id is associated with a pair of producer and consumer rate limits which may be config overrides or the default quota.  Quota id is set to either client-id or the encoded user principal based on the value of quota.type.

Quota Persistence in Zookeeper

Client-id based quota configuration overrides will continue be stored under /config/clients. Quota configuration overrides for user principals will be stored under /config/users. Only one of these will be processed and watched by the brokers depending on the value of quota.type. Note that Base64-encoded hex version of the user principal will be used as node name under /config/users to cope with Zookeeper naming restrictions. The non-encoded user principal with be stored as a property to make it easy to identify the actual user associated with the path.

Tools

kafka-configs.sh will be extended to support a new entity type "users". Configuration format for users will be the same as for the existing clients entity, but the entity will be stored in Zookeeper under /users.

Compatibility, Deprecation, and Migration Plan

quota.type is set to  client-id as default to be consistent with Kafka 0.9.0.x and 0.10.0.x. Hence the existing quota configurations will apply. If quota.type is set to user in broker configuration and default or new quotas are configured for users, clients may be throttled based on the quota limits for user principals. But no client API changes are necessary to work with the new implementation.

Rejected Alternatives

Hierarchical quotas with configurable sub-quotas for clients of a user

It may be useful to allocate sub-quotas for clients of a user to enable a user to run multiple rate-limited clients. Since this adds complexity to the implementation and configuration and there are no requirements to support this feature at the moment, a simpler single-level user quota was preferred.