Table of Contents |
---|
Status
Current state: [One of "Under Discussion", "Accepted", "Rejected"]
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: KAFKA-3492
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
KIP-13 introduced client quotas in Kafka 0.9.0.0. Rate limits on producers and consumers are enforced to prevent clients saturating the network or monopolizing broker resources. The current implementation allocates quotas to client-ids. This works well in single user clusters or clusters that use PLAINTEXT where all users have the same identity. But since client-id is unauthenticated and can be set to any value by the client, multi-tenant secure installations require secure quotas to be enforced for authenticated principals to guarantee fair allocation of resources and prevent denial-of-service.
With the introduction of security in Kafka 0.9, the identity of Kafka clients is the user principal. User principal is an authenticated user or a grouping of unauthenticated users chosen by the broker using a configurable PrincipalBuilder
and is currently used for ACLs. Client-id is a logical grouping of clients with a meaningful name that is used in client metrics and logs. Multi-user systems have a hierarchy - user owns zero or more clients. <user-principal, client-id
> defines a safe group of clients. The shorter unsafe client-id is sufficient in client metrics and logs, but quotas should be allocated to safe groups to avoid clients of one user throttling clients of another user with the same client-id.
This KIP addresses proposes the following extensions changes to the existing implementation:
- The option to apply quotas based on authenticated principal instead of client-idQuota configuration for user principal. This prevents users generating heavy traffic from monopolizing resources and impacting the performance of other users in a multi-tenant cluster.
- Sub-quotas Quota overrides for clients of an authenticated user. Like the current client-id implementation, this enables a user to rate-limit some producers or consumers to ensure that they don’t impact other more critical clients. For instance, users may be able to rate-limit an auditing client running in the background, leaving resources always available for a critical event processing client.
Public Interfaces
Configuration Options
A configuration option quota.secure
will be added to choose between the existing client-id based implementation and the new authenticated-principal based implementation. The default value will be false
to be consistent with Kafka 0.9.0.x.
Default quota configs will apply to authenticated user principals if quota.secure=true.
quota.producer.default
: Ifquota.secure=false
, this is the default producer quota for each unique client- id. Otherwise, this is the default quota for each authenticated principal.quota.consumer.default
: Ifquota.secure=false
, this is the default consumer quota for each client-id. Otherwise, this is the default quota for each authenticated principal.
Metrics
- Client-id quotas for users without a user quota override. Existing quota configuration for client-ids will continue to be applied to users with unlimited quota, if no user-specific defaults are configured. Defaults may also be specified for client-ids that do not span multiple users.
The goal of this KIP is to provide a single unified quota implementation with a simple configuration for client-id based quotas, a similar simple configuration for user quotas and the flexibility to define more complex quota configurations.
Public Interfaces
Overview
Quotas can be set at <user, client-id
>, user
or client-id
levels. For a given client connection, the most specific quota matching the connection will be applied. For example, if both a <user, client-id
> and a user
quota match a connection, the <user, client-id
> quota will be used. Otherwise, user
quota takes precedence over client-id
quota. The order of precedence is:
/config/users/<user>/clients/<client-id>
/config/users/<user>
/config/users/<default>/clients/
<client-id>
/config/users/<default>/clients/<default>
/config/users/
<default>
/config/clients/<client-id>
/config/clients/
<default>
Quota Entity
Quotas are currently configured for client-ids. All clients with the same client-id are currently grouped together as a quota entity, enforcing one quota for all clients with the same client-id. This KIP proposes to define quotas for safe client groups which share the same user-principal and client-id and user-level quotas that limit the total quota of users in addition to the existing client-id quotas.
Configuration Options
Default quotas for users and client-ids will be added as dynamic properties. The existing default configuration options for client-id quotas will be deprecated and these properties will be applied only if default user quota is unlimited and the dynamic client-id defaults are not specified.
Deprecate static properties for client-id default quotas
quota.producer.default, quota.consumer.default
: Default client-id producer/consumer quota is currently applied to each unique client-id across all users. This client-id
default will be applied only if default user quota is unlimited and no user-specific defaults are configured. These properties will be deprecated and will be applied only if dynamic default quotas are not configured for clients.
Dynamic configuration for default quotas
- Default quota for
<user, client-id>
will be stored in Zookeeper at/config/users/<default>/clients/<default>.
- Default user quota will be stored in Zookeeper at
/config/users
/<default>
. - Default client-id quota will be stored in Zookeeper at
/config/clients
/<default>
. If not specified, the broker propertiesquota.producer.default, quota.consumer.default
will be used as the default client-id quota for clients of users with unlimited quota.
Metrics
Quota related metrics are currently generated for client-ids and use the tag "client-id
". URL-encoded user principal will be added with the tag "user".
Both user and client-id tags will be specified when <user, client-id>
quotas are applied, only user
tag will be specified if user quota is applied and the existing client-id
tag will be retained if client-id quotas are applied.
Sensor names are currently a sensor type concatenated with the client id value, eg. FetchThrottleTime-clientA. This will be modified to use quota-id
instead of client-id
. This is not a public interface change since sensor names are not reflected in JMX metricsWhen quota.secure=true
, quota related metrics will be generated for authenticated principals rather than client-ids.
Tools
kafka-configs.sh
will be extended to support authenticated user quotas and sub-specific quotas for clients of a user <user, client-id>
. A new entity type “users
” will be added . The with the same key-value pairs supported for users
will beas the existing "clients
" entity type:
producer_byte_rate=p
: The total rate limit for the user’s producers is set top
MB/secwithout a client-id quota overrideconsumer_byte_rate=c
: The total rate limit for the user’s consumers is set toc
MB/secclient_without a client-id quota override
For example:
bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'producer_byte_
...
rate=
...
1024,consumer_byte_rate=2048'
--entity-name user1
--entity-type users
Default quotas for users or clients or <user, client-id> can be configured by omitting entity name. For example:
bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=10000,consumer_byte_rate=20000'
--entity-type users
Quotas for clients of a user can be configured by specifying entity types "users"
and “clients
” in the same command line . For example, the following command sets quotas for <user2, clientA>
:
bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=10,consumer_byte_rate=20' --entity-name clientA --entity-type clients --entity-name user2 --entity-type users
The existing entity type "clients
" will be retained to set client-id
quotas which are used when user quotas are not overridden.
Proposed Changes
User
Proposed Changes
...
Principal
Authenticated user principal will be obtained from the Session
when quota.secure=true
. Base64 object. URL-encoded hex string version of the Principal will be used so that it can be used as a node name in Zookeeper and as the metric name in metrics without placing any restrictions on the characters allowed in the principal.
Characters that cannot be used for Zookeeper node names or metrics (eg. *) will be percent-encoded. Encoded user principal will be cached in Session.
For PLAINTEXT, the principal is "ANONYMOUS" by default and quotas will be applied for that principal. But principal can be overridden using a custom principal builder even for PLAINTEXT, enabling different user quotas, for example, for connections from different IP addresses.
Quota Configuration
Quotas are currently configured as the total rate limits (p, c)
for all the producer or consumers with a specific for producers and consumers based on their client-id. Default values are specified in server.properties (quota.producer.default=defaultP, quota.consumer.default=defaultC)
for client-ids which don’t have a config override. Producer quotas and consumer quotas can be configured independently and default values are applied when an override is not specified. In the examples below, both are overridden together for simplicity.
- Current Implementation: Client-id based quota (
quota.secure=false
):{ clientA : (pA, cA) }
pA
(orcA
) is the total producer (or consumer) rate limit for all clients with client-idclientA
- Clients with a different client-id
clientX
with no overrides use the default rate limit of(defaultP, defaultC)
.
- Authenticated-principal based quota
(quota.secure=true)
:{ user1 : (p1,c1) }
p1
(orc1
) is the total producer (or consumer) rate limit for all clients with the authenticated principaluser1
- The total rate limit for all clients with another authenticated principal
user10
with no overrides is the default rate limit of(defaultP, defaultC)
.
- Hierarchical quotas for clients of a user
(quota.secure=true):
{ user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B)}}
p2A
(orc2A
) is the producer (or consumer) rate limit for the clients with client-idclientA
AND user principaluser2
(p2 –p2A –p2B)
is the total producer rate limit for clients with any client-id other thanclientA/clientB
with the user principaluser2
. Only the total quota for these clients is enforced, per-client quota is enforced only forclientA
andclientB
. Similarly for consumer quotas.- All clients with another authenticated principal
user10
will share the default rate limit of(defaultP, defaultC)
. - The basic authenticated-principal based quota (2) is simply a special case of hierarchical quotas when no client-specific quotas are specified for a user.
rate limits can be configured for clients without a config override. The same set of limits will be configurable each quota-id.
Quota configuration for a client with client-id clientX and user principal userN is determined by the following sequence (this example is for producer, similar sequence is applied to consumer):
- If quota override is defined for <userN, clientX> in
/config/users/userN/clients/clientX
, this quota is allocated for the sole use of <userN, clientX>. - If user quota override is defined for userN, clientX in
/config/users/userN
shares this quota with other clients of userN - If <user, client-id> quota is defined in
/config/users/<default>clients/clientX
, this quota is allocated for the sole use of <userN, clientX> - If default <user, client-id> quota is defined in
/config/users/<default>clients
/<default>
, this quota is allocated for the sole use of <userN, clientX> - If default user quota is defined in
/config/users/
, clientX shares this default quota with other clients of userN<default>
- If client-id quota override is defined for clientX in
/config/clients/clientX
, this quota is shared across client-id <clientX> of all users - If dynamic client-id default is configured in
/config/clients/
, this default quota is shared across client-id <clientX> of all users<default>
- If
quota.producer.default is
configured for the broker in server.properties, this is shared across client-id <clientX> of all users - Client is not throttled
Use cases:
- Simple client-id based quotas are configured using client-id quota override, dynamic client-id default and static
quota.producer.default :
(steps 6, 7, 8, 9) - Simple user-principal based quotas are configured using user quota override and user quota default
:
(steps 2, 5, 9) - More specific <user, client-id> quotas and defaults for users and client-ids can be configured if required: (steps 1 - 9)
Code Block | ||||
---|---|---|---|---|
| ||||
// Default user quota
// Zookeeper persistence path /config/users/<default>
{
"version":1,
"config": {
"producer_byte_rate":"10000",
"consumer_byte_rate":"20000"
}
}
|
...
Code Block | ||||
---|---|---|---|---|
| ||||
// Quotas for user1 (without subclient-quotasid overrides). // Zookeeper persistence path /config/users/<encoded-user1> { "version":1, "config": { "+" : {"producer_byte_rate":"1024", "consumer_byte_rate":"2048"} } } |
Code Block | ||||
---|---|---|---|---|
| ||||
// User-level Quotasquotas for user2 (with sub-quotas) // Zookeeper persistence path /config/users/<encoded-user2> { "version":1, "config": { "producer_byte_rate":"4096", "+" : {"consumer_byte_rate":"8192" } } // Quota override for <user2, clientA> // Zookeeper persistence path /config/users/<encoded-user2>/clients/clientA { "version":1, "config": { "producer_byte_rate":"102410", "consumer_byte_rate":"2048"}, "clientA" : {30" } } // Quota override for <user2, clientB> // Zookeeper persistence path /config/users/<encoded-user2>/clients/clientB { "version":1, "config": { "producer_byte_rate":"1020", "consumer_byte_rate":"20"}40" } } |
Code Block | ||||
---|---|---|---|---|
| ||||
// Quotas for client-id clientA of users without user quota override. // Zookeeper persistence path /config/clients/clientA { "version":1, "clientBconfig" : { "producer_byte_rate":"30100", "consumer_byte_rate":"40200"} } } |
In the sample configuration above:
- Total rate limits for all clients with user principal user1 is (1024, 2048).
- Total rate limits for all clients with user principal user2 without additional client-id quota is (4096, 8192).
- The rate limits for clients with user principal user2 AND client-id clientA is (10, 20).
- Clients of user2 with client-id other than clientA and clientB share the quota (4096, 8192).
- Total rate limits for all clients of user3 is the default (
10000, 20000
), since no config override is specified. - If default user quota is not specified or is unlimited, clients of user3 use client-id quota configuration. For example quota for client-id clientA of user3 is (100, 200) and this is shared with clients of other users with client-id clientA. And quota for client-id clientB of user3 without a client-id override is (
quota.producer.default, quota.consumer.default
)- In a multi-user cluster, defaults can also be specified for
<user, client-id>
quotas which treat clientA of user4 as a different group from clientA of user2.
- In a multi-user cluster, defaults can also be specified for
Quota Identifier
Quota configuration and metrics sensors currently use client-id
as the unique key, enforcing one quota for all clients with the same client-id
. This will be replaced with a new quota-id
that includes user principal. Each quota-id
is associated with a pair of producer and consumer rate limits which may be config overrides or the default quota.
quota
...
- Client
-id
is used as quota-id (Current implementation) - Each client–id with a quota override uses the overridden limits
- Each client-id without an override gets the default limits
(defaultP, defaultC)
quota.secure=true
- the concatenation of url-encoded user principal and client-id for
<user, client-id>
quota, just the client-id forclient-id
quotas and just the encoded user principal foruser
quotas. Similarly, quota-related metrics names will include the tagsclient-id
and/oruser
. - In the example:
- All clients of user1 share the quota-id user1:
- Authenticated principal is used as quota-id if a sub-quota is not specified for the client-id. Otherwise quota-id is the concatenation of principal and client-id. Note that user principals will be base64 encoded hex in the actual implementation, but are shown here as names for readability.
Example:
{user1 : { total: (p1,c1) }, user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B) }}
- All clients of
user1
use the quota-iduser1
. This shares(p1, c1)
amongst the clients of user1 with no individual limits for client-ids. - clientA of user2 uses the quota-id
user2clientA
. This gives clients ofuser2
with client-idclientA
a quota of(p2A, c2A).
Similarly for clients with client-idclientB
.clientX
user2:clientA - clientC of user2 uses the quota-id user2: since it does not have a client quota override, sharing a quota with other clients of user2.
- clientA of user3 uses the quota-id :clientA.
clientY
ofuser2
also uses the same quota-iduser2
. HenceclientX
andclientY
share(p2-p2A-p2B, c2-c2A-c2B)
All clients of a differentuser10
use the quota-iduser10
. This shares(defaultP, defaultC)
amongst the clients ofuser10
.
- All clients of
Quota Persistence in Zookeeper
Client-id based quotas quota configuration overrides will continue be stored under /config/clients,
but these will be applied only to clients . Authenticated user quotas of users without a quota override and only if default user quota is unlimited. Quota configuration overrides for user principals will be stored under /config/users
. Only one of these will be processed and watched by the brokers depending on the value of quota.secure
/<user>
. Note that Base64url-encoded hex version of the user principal will be used as node name under /config/users
to cope with Zookeeper naming restrictions. Default user quotas will be stored under /config/users
/<default>
. Quota overrides for clients of a user will be stored under /config/users/<user>/clients
.
Configuration change notifications will be generated for changes to quota configuration similar to the current notifications for client-id
quotas. JSON for change notification will be modified to provide entity path instead of specifying entity type and name separately.
Code Block | ||||
---|---|---|---|---|
| ||||
// Change notification for default user quota
{
"version":2,
"entity_path": "users/<default>"
}
// Change notification for user quota of user1
{
"version":2,
"entity_path": "users/user1"
}
// Change notification for quota of <user2, clientA>
{
"version":2,
"entity_path": "users/user2/clients/clientA"
}
// Change notification for default client-id quota
{
"version":2,
"entity_path": "clients/<default>"
}
// Change notification for client-id quota of clientA
{
"version":2,
"entity_path": "clients/clientA"
} |
Tools
kafka-configs.sh
will be extended to support a new entity type "users".
Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying total user quota and possibly some client quotasrate limits, validate these and convert them to the equivalent JSON for persistence in Zookeeper.
Compatibility, Deprecation, and Migration Plan
The existing entity “clients
” will continue to be supported to set client-id quotas for users with unlimited quota. The tool will be extended to accept multiple entity types to configure <user, client-id> quotas. The tool will also be updated to configure default quotas (/config/users/
<default>
, /config/clients
/<default>
, /config/users
/<default>/clients/
).<default>
Compatibility, Deprecation, and Migration Plan
No quota.secure
is set to false as default to be consistent with Kafka 0.9.0.x. Hence the existing quota configurations will apply if new secure quotas are not defined. If quota.secure
is set to true
and default or new quotas are configured for users, clients may be throttled based on the quota limits. But no client API changes are necessary to work with the new implementation. Existing client-id quota configuration will be processed from Zookeeper and since users have unlimited quota as default, existing client-id quotas will be applied to all clients with the client-id.
Rejected Alternatives
...
Broker configuration property to choose either client-id
...
This KIP proposes to use a broker configuration option to switch between client-id based quotas and authenticated-principal based quotas for simplicity. An alternative would be to define a unified configuration where client-id based quotas are a special case of a unified quota config with the same username applied to all clients. The internal quota implementation will use common code for both options with only the quota-id being different. But the externally visible configuration and defaults are much simpler to define with separate options that are consistent with 0.9.0.x since it is unlikely that a cluster would support both.
or user quota
Single user clusters require only client-id based quotas and some secure clusters may require only overall user quotas. A simple switch between these two provides a simpler configuration and implementation. But a server option that changes the semantics of quota scan be confusing and it is hard to configure a cluster without actually knowing which quota mode is being used. Since hierarchical quotas are useful in multi-user clusters, it is better to handle both client-id quotas and user quotas with semantics suitable to both secure and plaintext clusters.
Add quota-id to Kafka protocol to define a new grouping
Both client-id and user principal are groupings of clients currently used for other purposes. User-principals are used for ACLs and client-ids are used for client metrics and logs. It may be useful to define a quota group that doesn't have to align with existing uses of user principal or client-id. But this require addition of a new concept of quota-id into the Kafka protocol that is authenticated. Use of existing ids simplifies configuration and keeps the definitions consistent.