...

• producer_byte_rate=p : The total rate limit for the user’s producers set to p MB/sec
• consumer_byte_rate=c : The total rate limit for the user’s consumers set to c  MB/sec
• client_producer_byte_rates=clientA:pA,clientB:pB : The rate limit for the user’s producers with client-id clientA is set to pA MB/sec. Similarly for clientB. Other clients share p-pA-pB MB/sec.
• client_consumer_byte_rates=clientA:cA,clientB:cB : The rate limit for the user’s consumers with client-id clientA is set to cA MB/sec. Other clients share c-cA-cB MB/sec.

Proposed Changes

Authenticated Principal 

Authenticated user principal will be obtained from the Session when quota.secure=true. Base64-encoded hex string version of the Principal will be used so that it can be used as a node name in Zookeeper and as the metric name without placing any restrictions on the characters allowed in the principal. 
 

Quota Configuration

Quotas are currently configured as the total rate limits (p, c) for all the producer or consumers with a specific client-id. Default values are specified in server.properties (quota.producer.default=defaultP, quota.consumer.default=defaultC) for client-ids which don’t have a config override. Producer quotas and consumer quotas can be configured independently and default values are applied when an override is not specified. In the examples below, both are overridden together for simplicity.

1. Client-id based quota (quota.secure=false): { clientA : (pA, cA) }
   • pA (or cA) is the total producer (or consumer) rate limit for all clients with client-id clientA
   • Clients with a different client-id clientX use the default rate limit of (defaultP, defaultC).
2. Authenticated-principal based quota (quota.secure=true): { user1 : (p1,c1) }
   • p1 (or c1) is the total producer (or consumer) rate limit for all clients with the authenticated principal user1
   • The total rate limit for all clients with another authenticated principal user10 is the default rate limit of (defaultP, defaultC).
3. Hierarchical quotas for clients of an user (quota.secure=true):{ user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B)}}
   • p2A (or c2A) is the producer (or consumer) rate limit for the clients with client-id clientA AND user principal user2
   • (p2 –p2A –p2B) is the total producer rate limit for clients with any client-id other than clientA/clientB with the user principal user2. Only the total quota for these clients is enforced, per-client quota is enforced only for clientA and clientB. Similarly for consumer quotas.
   • All clients with another authenticated principal user10 will share the default rate limit of (defaultP, defaultC).
   • The basic authenticated-principal based quota (2) is simply a special case of hierarchical quotas when no client-specific quotas are specified for a user.

The configuration above for authenticated principal quotas will be represented in JSON as:

// Quotas for user1 (without sub-quotas)
{
  "version":1,
  "config": {
    "total" : {"producer_byte_rate":"1024","consumer_byte_rate":"2048"}
  }
}
// Quotas for user2 (with sub-quotas)
{
  "version":1,
  "config": {
    "total" : {"producer_byte_rate":"1024","consumer_byte_rate":"2048"},
    "clientA" : {"producer_byte_rate":"10","consumer_byte_rate":"20"},
    "clientB" : {"producer_byte_rate":"30","consumer_byte_rate":"40"}
  }
} 

Quota Identifier

Quota configuration and metrics currently use client-id as the unique key, enforcing one quota for all clients with the same client-id. This will be replaced with a new quota-id. Each quota-id is associated with a pair of producer and consumer rate limits which may be config overrides or the default quota.  

quota.secure=false

• Client-id is used as quota-id (Current implementation)
• Each client–id with a quota override uses the overridden limits
• Each client-id without an override gets the default limits (defaultP, defaultC)

quota.secure=true

• Authenticated principal is used as quota-id if a sub-quota is not specified for the client-id. Otherwise quota-id is the concatenation of principal and client-id. Note that user principals will be base64 encoded hex in the actual implementation, but are shown here as names for readability.

• Example: {user1 : { total: (p1,c1) },  user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B) }}

  • All clients of user1 use the quota-id user1. This shares (p1, c1) amongst the clients of user1 with no individual limits for client-ids.
  • clientA of user2 uses the quota-id user2clientA. This gives clients of user2 with client-id clientA a quota of (p2A, c2A). Similarly for clients with client-id clientB.

  • clientX of user2 uses the quota-id user2. clientY of user2 also uses the same quota-id user2. Hence clientX and clientY share (p2-p2A-p2B, c2-c2A-c2B)
  • All clients of a different user10 use the quota-id user10. This shares (defaultP, defaultC) amongst the clients of user10.

Quota Persistence in Zookeeper

Client-id based quotas will continue be stored under /config/clients. Authenticated user quotas will be stored under /config/users. Only one of these will be processed and watched by the brokers depending on the value of quota.secure.

Tools

kafka-configs.sh will be extended to support a new entity type "users". Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying total user quota and possibly some client quotas, validate these and convert them to the equivalent JSON for persistence in ZookeeperDescribe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

...