...
Quotas can be set at <user, client-id
>, user
or client-id
levels. For a given client connection, the most specific quota matching the connection will be applied. For example, if both a <user, client-id
> and a user
quota match a connection, the <user, client-id
> quota will be used. Otherwise, user
quota takes precedence over client-id
quota. The order of precedence is:
/config/users/<user>/clients/<client-id>
/config/users/<user>
/config/users
/config/clients/<client-id>
/config/clients
Quota Entity
Quotas are currently configured for client-ids. All clients with the same client-id are currently grouped together as a quota entity, enforcing one quota for all clients with the same client-id. This KIP proposes to define quotas for safe client groups which share the same user-principal and client-id. In a single user cluster, this retains the current semantics of client-id quotas.
Configuration Options
Default quotas for users and client-ids will be added as dynamic properties. The existing default configuration options for client-id quotas will be applied deprecated and these properties will be applied only if default user quota is unlimited . Default quotas for users will be added as dynamic properties.
...
and the dynamic client-id defaults are not specified.
Deprecate static properties for client-id default quotas
quota.producer.default, quota.consumer.default
: Default client-id producer/consumer quota is currently applied to each unique client-id across all users. This will be modified to be a per-user quota for each unique client-id of each user. This client-id
default will is applied only if default user quota is unlimited.
...
. These properties will be deprecated and will be applied only if dynamic default quotas are not configured for clients.
Dynamic configuration for default quotas
- Default user quota will be stored in Zookeeper at the top level config for
/config/users
. If not specified, user quota will be unlimited and client-id defaults will apply. Default user quota can be updated dynamically. - Default
...
- All clients have unlimited quota by default
- If default quotas are configured for users, these default quotas are allocated to each user principal.
- If default user quota is unlimited, client-id quota will be stored in Zookeeper at the top level config for
/config/clients
. If not specified, the broker propertiesquota.producer.default, quota.consumer.default
are allocated to each unique will be used as the default client-id of each userquota for clients of users with unlimited quota.
Metrics
Quota related metrics are currently generated for client-ids and use the tag client-id
. The metrics tag will be changed to quota-id
and the value will include base-64 encoded user principal.
...
bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=1024,consumer_byte_rate=2048'
--entity-name user1
--entity-type users
Default quotas for users or clients can be configured by omitting entity name. For example:
bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=10000,consumer_byte_rate=20000'
--entity-type users
...
- If quota override is defined for <userN, clientX> in
/config/users/userN/clients/clientX
, this quota is allocated for the sole use of <userN, clientX>. - If user quota override is defined for userN, clientX for userN, clientX in
/config/users/userN
shares this quota with other clients of userN - If default user quota is unlimiteddefined in
/config/users
, clientX shares this default quota with other clients of userN - If client-id quota override is defined for clientX in
/config/clients/clientX
, this quota is allocated for the sole use of <userN, clientX> - If dynamic client-id default is configured in
/config/clients
, this default quota is allocated for the sole use of <userN, clientX> - If
quota.producer.default is
configured for the broker in server.properties, this default quota is allocated for the sole use of <userN, clientX> - Client is not throttled
...
- Simple client-id based quotas are configured using client-id quota override, dynamic client-id default and static
quota.producer.default :
(steps 4, 5, 66, 7) - Simple user-principal based quotas are configured using user quota override and user quota default
:
(steps 2, 3, 67) - More specific <user, client-id> quotas and defaults for users and client-ids can be configured if required: (steps 1 - 67)
Code Block | ||||
---|---|---|---|---|
| ||||
// Default user quota // Zookeeper persistence path /users { "version":1, "config": { "producer_byte_rate":"10000", "consumer_byte_rate":"20000" } } |
...
Code Block | ||||
---|---|---|---|---|
| ||||
// Change notification for default user quota { "version":2, "entity_path": "users" } // Change notification for user quota of user1 { "version":2, "entity_path": "users/user1" } // Change notification for quota of <user2, clientA> { "version":2, "entity_path": "users/user2/clients/clientA" } // Change notification for default client-id quota { "version":2, "entity_path": "clients" } // Change notification for client-id quota of clientA { "version":2, "entity_path": "clients/clientA" } |
...
kafka-configs.sh
will be extended to support a new entity type "users".
Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying rate limits, validate these and convert them to the equivalent JSON for persistence in Zookeeper. The existing entity “clients
” will continue to be supported to set client-id quotas for users with unlimited quota. The tool will be extended to accept multiple entity types to configure <user, client-id> quotas. The tool will also be updated to configure default user quotas at the top-level (/config/users or
/config/clients
).
Compatibility, Deprecation, and Migration Plan
...