...

KIP-13 introduced client quotas in Kafka 0.9.0.0. Rate limits on producers and consumers are enforced to prevent clients saturating the network or monopolizing broker resources. The current implementation allocates quotas to client-ids. This works well in single user clusters or clusters that use PLAINTEXT where all users have the same identity. But since client-id is unauthenticated and can be set to any value by the client, multi-user tenant secure installations require quotas to be enforced for authenticated principals to guarantee fair allocation of resources and prevent denial-of-service.

...

1. The option to apply quotas based on authenticated principal instead of client-id. This prevents users generating heavy traffic from monopolizing resources and impacting the performance of other users in a multi-user tenant cluster.
2. Sub-quotas for clients of an authenticated user.  Like the current client-id implementation, this enables a user to rate-limit some producers or consumers to ensure that they don’t impact other more critical clients.  For instance, users may be able to rate-limit an auditing client running in the background, leaving resources always available for a critical event processing client.

...

A configuration option quota.secure will be added to choose between the existing client-id based implementation and the new authenticated-principal based implementation.  The default value will be false to be consistent with Kafka 0.9.0.x.

The default Default quota configs will apply to authenticated user principals if quota.secure=true.

...

• producer_byte_rate=p : The total rate limit for the user’s producers is set to p MB/sec
• consumer_byte_rate=c : The total rate limit for the user’s consumers is set to c  MB/sec
• client_producer_byte_rates=clientA:pA,clientB:pB : The rate limit for the user’s producers with client-id clientA is set to pA MB/sec. Similarly for clientB. Other clients of the user share p-pA-pB MB/sec.
• client_consumer_byte_rates=clientA:cA,clientB:cB : The rate limit for the user’s consumers with client-id clientA is set to cA MB/sec. Other clients of the user share c-cA-cB MB/sec.

Proposed Changes

...

1. Client-id based quota (quota.secure=false): { clientA : (pA, cA) }
   • pA (or cA) is the total producer (or consumer) rate limit for all clients with client-id clientA
   • Clients with a different client-id clientX use the default rate limit of (defaultP, defaultC).
2. Authenticated-principal based quota (quota.secure=true): { user1 : (p1,c1) }
   • p1 (or c1) is the total producer (or consumer) rate limit for all clients with the authenticated principal user1
   • The total rate limit for all clients with another authenticated principal user10 is the default rate limit of (defaultP, defaultC).
3. Hierarchical quotas for clients of an a user (quota.secure=true):{ user2 : { total : (p2, c2), clientA : (p2A, c2A), clientB : (p2B, c2B)}}
   • p2A (or c2A) is the producer (or consumer) rate limit for the clients with client-id clientA AND user principal user2
   • (p2 –p2A –p2B) is the total producer rate limit for clients with any client-id other than clientA/clientB with the user principal user2. Only the total quota for these clients is enforced, per-client quota is enforced only for clientA and clientB. Similarly for consumer quotas.
   • All clients with another authenticated principal user10 will share the default rate limit of (defaultP, defaultC).
   • The basic authenticated-principal based quota (2) is simply a special case of hierarchical quotas when no client-specific quotas are specified for a user.

The configuration above for authenticated principal quotas will be represented in JSON as: 

// Quotas for user1 (without sub-quotas)
{
  "version":1,
  "config": {
    "total" : {"producer_byte_rate":"1024","consumer_byte_rate":"2048"}
  }
}
// Quotas for user2 (with sub-quotas)
{
  "version":1,
  "config": {
    "total" : {"producer_byte_rate":"1024","consumer_byte_rate":"2048"},
    "clientA" : {"producer_byte_rate":"10","consumer_byte_rate":"20"},
    "clientB" : {"producer_byte_rate":"30","consumer_byte_rate":"40"}
  }
} 

...

Client-id based quotas will continue be stored under /config/clients. Authenticated user quotas will be stored under /config/users. Only one of these will be processed and watched by the brokers depending on the value of quota.secure. Note that Base64-encoded hex version of the user principal will be used as node name under /config/users to cope with Zookeeper naming restrictions.

Tools

kafka-configs.sh will be extended to support a new entity type "users". Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying total user quota and possibly some client quotas, validate these and convert them to the equivalent JSON for persistence in Zookeeper.

...

quota.secureis set to  false as default to be consistent with Kafka 0.9.0.x. Hence the existing quota configurations will apply if new secure quotas are not defined. If quota.secure  is set to true and default or new quotas are configured for users, clients may be throttled based on the new quota configurationslimits. But no client API changes are necessary to work with the new implementation.

...