Table of Contents |
---|
Status
Current state: Under Discussion Accepted
Discussion thread: here
JIRA: KAFKA-3492
...
/config/users/<user>/clients/<client-id>
/config/users/<user>
/config/users/<default>/clients/
<client-id>
/config/users/<default>/clients/
<client-id>
<default>/config/users/
<default>
/clients/config/clients/<client-id>
/config/clients/
<default>
Quota Entity
Quotas are currently configured for client-ids. All clients with the same client-id are currently grouped together as a quota entity, enforcing one quota for all clients with the same client-id. This KIP proposes to define quotas for safe client groups which share the same user-principal and client-id and user-level quotas that limit the total quota of users in addition to the existing client-id quotas.
...
Dynamic configuration for default quotas
- Default user quota for
<user, client-id>
will be stored in Zookeeper at the top level config for/config/users
./<default>/clients/<default>.
- Default If not specified, user quota will be unlimited and
<user, client-id>
defaults will apply. Default user quota can be updated dynamically. - Default quota for
<user, client-id>
will be stored in/config/users/<default>/clients.
If not specified, client-id defaults will apply. - stored in Zookeeper at
/config/users
/<default>
. - Default client-id quota will be stored in Zookeeper at
/config/clients
/<default>
Default client-id quota will be stored in Zookeeper at the top level config for/config/clients
. If not specified, the broker propertiesquota.producer.default, quota.consumer.default
will be used as the default client-id quota for clients of users with unlimited quota.
...
- If quota override is defined for <userN, clientX> in
/config/users/userN/clients/clientX
, this quota is allocated for the sole use of <userN, clientX>. - If user quota override is defined for userN, clientX in
/config/users/userN
shares this quota with other clients of userN - If default user <user, client-id> quota is defined in in
/config/users/<default>clients/clientX
, clientX shares this default quota with other clients of userNquota is allocated for the sole use of <userN, clientX> - If default If <user, client-id> quota is defined in
/config/users/<default>clients
/clientX<default>
, this quota is allocated for the sole use of <userN, clientX> - If default <user, client-id> user quota is defined in in
/config/users/
<default>clients
, this quota is allocated for the sole use of <userN, clientX> clientX shares this default quota with other clients of userN<default>
- If client-id quota override is defined for clientX in
/config/clients/clientX
, this quota is shared across client-id <clientX> of all users - If dynamic client-id default is configured in
/config/clients/
, this default quota is shared across client-id <clientX> of all users<default>
- If
quota.producer.default is
configured for the broker in server.properties, this is shared across client-id <clientX> of all users - Client is not throttled
...
- Simple client-id based quotas are configured using client-id quota override, dynamic client-id default and static
quota.producer.default :
(steps 6, 7, 8, 9) - Simple user-principal based quotas are configured using user quota override and user quota default
:
(steps 2, 35, 9) - More specific <user, client-id> quotas and defaults for users and client-ids can be configured if required: (steps 1 - 9)
...
Code Block | ||||
---|---|---|---|---|
| ||||
// Default user quota // Zookeeper persistence path /config/users/<default> { "version":1, "config": { "producer_byte_rate":"10000", "consumer_byte_rate":"20000" } } |
Code Block | ||||
---|---|---|---|---|
| ||||
// Quotas for user1 (without client-id overrides).
// Zookeeper persistence path /config/users/<encoded-user1>
{
"version":1,
"config": {
"producer_byte_rate":"1024",
"consumer_byte_rate":"2048"
}
}
|
Code Block | ||||
---|---|---|---|---|
| ||||
// TopUser-level total quotas for user2 // Zookeeper persistence path /config/users/<encoded-user2> { "version":1, "config": { "producer_byte_rate":"4096", "consumer_byte_rate":"8192" } } // Quota override for <user2, clientA> // Zookeeper persistence path /config/users/<encoded-user2>/clients/clientA { "version":1, "config": { "producer_byte_rate":"10", "consumer_byte_rate":"30" } } // Quota override for <user2, clientB> // Zookeeper persistence path /config/users/<encoded-user2>/clients/clientB { "version":1, "config": { "producer_byte_rate":"20", "consumer_byte_rate":"40" } } |
Code Block | ||||
---|---|---|---|---|
| ||||
// Quotas for client-id clientA of users without user quota override.
// Zookeeper persistence path /config/clients/clientA
{
"version":1,
"config": {
"producer_byte_rate":"100",
"consumer_byte_rate":"200"
}
}
|
...
Client-id based quota configuration overrides will continue be stored under /config/clients,
but these will be applied only to clients of users without a quota override and only if default user quota is unlimited. Quota configuration overrides for user principals will be stored under /config/users/<user>
. Note that url-encoded version of the user principal will be used as node name under /config/users
to cope with Zookeeper naming restrictions. Default user quotas will be stored under /config/users
/<default>
. Quota overrides for clients of a user will be stored under /config/users/<user>/clients
.
...
Code Block | ||||
---|---|---|---|---|
| ||||
// Change notification for default user quota { "version":2, "entity_path": "users/<default>" } // Change notification for user quota of user1 { "version":2, "entity_path": "users/user1" } // Change notification for quota of <user2, clientA> { "version":2, "entity_path": "users/user2/clients/clientA" } // Change notification for default client-id quota { "version":2, "entity_path": "clients/<default>" } // Change notification for client-id quota of clientA { "version":2, "entity_path": "clients/clientA" } |
...
kafka-configs.sh
will be extended to support a new entity type "users".
Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying rate limits, validate these and convert them to the equivalent JSON for persistence in Zookeeper. The existing entity “clients
” will continue to be supported to set client-id quotas for users with unlimited quota. The tool will be extended to accept multiple entity types to configure <user, client-id> quotas. The tool will also be updated to configure default quotas at the top-level quotas (/config/users or /
<default>
, /config/clients
/<default>
, /config/users
/<default>/clients/
).<default>
Compatibility, Deprecation, and Migration Plan
...