...
producer_byte_rate=p
: The total rate limit for the user’s producers set top
MB/secconsumer_byte_rate=c
: The total rate limit for the user’s consumers set toc
MB/secclient_producer_byte_rates=clientA:pA,clientB:pB
: The rate limit for the user’s producers with client-idclientA
is set to pA MB/sec. Similarly for clientB. Other clients sharep-pA-pB
MB/sec.client_consumer_byte_rates=clientA:cA,clientB:cB
: The rate limit for the user’s consumers with client-idclientA
is set to cA MB/sec. Other clients sharec-cA-cB
MB/sec.
Proposed Changes
Authenticated Principal
Authenticated user principal will be obtained from the Session
when quota.secure=true
. Base64-encoded hex string version of the Principal will be used so that it can be used as a node name in Zookeeper and as the metric name without placing any restrictions on the characters allowed in the principal.
Quota Configuration
Quotas are currently configured as the total rate limits (p, c)
for all the producer or consumers with a specific client-id. Default values are specified in server.properties (quota.producer.default=defaultP, quota.consumer.default=defaultC)
for client-ids which don’t have a config override. Producer quotas and consumer quotas can be configured independently and default values are applied when an override is not specified. In the examples below, both are overridden together for simplicity.
- Client-id based quota (
quota.secure=false
):{ clientA : (pA, cA) }
pA
(orcA
) is the total producer (or consumer) rate limit for all clients with client-idclientA
- Clients with a different client-id
clientX
use the default rate limit of(defaultP, defaultC)
.
- Authenticated-principal based quota
(quota.secure=true)
:{ user1 : (p1,c1) }
p1
(orc1
) is the total producer (or consumer) rate limit for all clients with the authenticated principaluser1
- The total rate limit for all clients with another authenticated principal
user10
is the default rate limit of(defaultP, defaultC)
.
- Hierarchical quotas for clients of an user
(quota.secure=true):
{ user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B)}}
p2A
(orc2A
) is the producer (or consumer) rate limit for the clients with client-idclientA
AND user principaluser2
(p2 –p2A –p2B)
is the total producer rate limit for clients with any client-id other thanclientA/clientB
with the user principaluser2
. Only the total quota for these clients is enforced, per-client quota is enforced only forclientA
andclientB
. Similarly for consumer quotas.- All clients with another authenticated principal
user10
will share the default rate limit of(defaultP, defaultC)
. - The basic authenticated-principal based quota (2) is simply a special case of hierarchical quotas when no client-specific quotas are specified for a user.
The configuration above for authenticated principal quotas will be represented in JSON as:
Code Block | ||||
---|---|---|---|---|
| ||||
// Quotas for user1 (without sub-quotas)
{
"version":1,
"config": {
"total" : {"producer_byte_rate":"1024","consumer_byte_rate":"2048"}
}
}
// Quotas for user2 (with sub-quotas)
{
"version":1,
"config": {
"total" : {"producer_byte_rate":"1024","consumer_byte_rate":"2048"},
"clientA" : {"producer_byte_rate":"10","consumer_byte_rate":"20"},
"clientB" : {"producer_byte_rate":"30","consumer_byte_rate":"40"}
}
} |
Quota Identifier
Quota configuration and metrics currently use client-id as the unique key, enforcing one quota for all clients with the same client-id. This will be replaced with a new quota-id. Each quota-id is associated with a pair of producer and consumer rate limits which may be config overrides or the default quota.
quota.secure=false
- Client-id is used as quota-id (Current implementation)
- Each client–id with a quota override uses the overridden limits
- Each client-id without an override gets the default limits
(defaultP, defaultC)
quota.secure=true
- Authenticated principal is used as quota-id if a sub-quota is not specified for the client-id. Otherwise quota-id is the concatenation of principal and client-id. Note that user principals will be base64 encoded hex in the actual implementation, but are shown here as names for readability.
Example:
{user1 : { total: (p1,c1) }, user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B) }}
- All clients of
user1
use the quota-iduser1
. This shares(p1, c1)
amongst the clients of user1 with no individual limits for client-ids. clientA
ofuser2
uses the quota-iduser2clientA
. This gives clients ofuser2
with client-idclientA
a quota of(p2A, c2A).
Similarly for clients with client-idclientB
.clientX
ofuser2
uses the quota-iduser2
.clientY
ofuser2
also uses the same quota-iduser2
. HenceclientX
andclientY
share(p2-p2A-p2B, c2-c2A-c2B)
- All clients of a different
user10
use the quota-iduser10
. This shares(defaultP, defaultC)
amongst the clients ofuser10
.
- All clients of
Quota Persistence in Zookeeper
Client-id based quotas will continue be stored under /config/clients
. Authenticated user quotas will be stored under /config/users
. Only one of these will be processed and watched by the brokers depending on the value of quota.secure
.
Tools
kafka-configs.sh
will be extended to support a new entity type "users".
Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying total user quota and possibly some client quotas, validate these and convert them to the equivalent JSON for persistence in ZookeeperDescribe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
Compatibility, Deprecation, and Migration Plan
...