...

kafka-configs.sh will be extended to support authenticated user quotas and sub-quotas for clients of a user.  A new entity type “users” will be added .  The with the same key-value pairs supported for users will beas the existing "clients" entity type:

• producer_byte_rate : The total rate limit for the user’s producers
• consumer_byte_rate : The total rate limit for the user’s consumers

Sub-quotas can be set for clients of a user 's clients by specifying the rate limits with both user and client entities in a single command.

The existing entity type "clients" will be retained for backward compatibility. But quotas set for clients are used only for users without a config override and only if default user quota is unlimited.

Proposed Changes

User Principal 

can be configured by specifying entity types "users" and “clients”  in the same command line . For example, the following command sets quotas for <user2, clientA>:

bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=10,consumer_byte_rate=20' --entity-name clientA --entity-type clients --entity-name user2 --entity-type users

The existing entity type "clients" will be retained to set client-id quotas which are used when user quotas are not overridden.

Proposed Changes

User Principal 

Authenticated user principal will be obtained from the Session object. URL-Authenticated user principal will be obtained from the Session object. URL-encoded string version of the Principal will be used so that it can be used as a node name in Zookeeper and in metrics without placing any restrictions on the characters allowed in the principal. Characters that cannot be used for Zookeeper node names or metrics (eg. *) will be percent-encoded. Encoded user principal will be cached in Session. For PLAINTEXT, the principal is "ANONYMOUS" by default and quotas will be applied for that principal. But principal can be overridden using a custom principal builder even for PLAINTEXT, enabling different user quotas, for example, for connections from different IP addresses.

...

Configuration change notifications will be generated for changes to the quota configuration similar to the current notifications for client-id quotas. Changes to client-id sub-quotas of a user specify users as the entity_type and the sub-path of the node containing both user and client-id as entity_name . The Note that changes to sub-quota affect both the sub-quotas of the particular <user, client-id> as well as the remainder quota allocated to the user's clients without a sub-quota override.

// Change notification for user quota of user1
{
    "version":1,
    "entity_type": "users",
    "entity_name": "user1"
}
// Change notification for client sub-quota of <user2, clientA> that impacts clientA as well as clients of user2 without a sub-quota override
{
    "version":1,
    "entity_type": "users",
    "entity_name": "user2/clients/clientA"
 } 

Tools

...

"users",
    "entity_name": "user2/clients/clientA"
 } 

Tools

kafka-configs.sh will be extended to support a new entity type "users". Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying rate limits, validate these and convert them to the equivalent JSON for persistence in Zookeeper. Changes to user quotas and client sub-quotas will be validated to ensure that the total quota of the user is not exceeded by the clients with sub-quotas. The existing entity “clients” will continue to be supported to set client-id quotas for users with unlimited quota.. Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying rate limits, validate these and convert them to the equivalent JSON for persistence in Zookeeper. The existing entity “clients” will continue to be supported to set client-id quotas for users with unlimited quota. Sub-quotas for clients of a user can be configured by specifying entity types "users" and “clients”  in the same command line . For example, the following command sets quotas for <user2, clientA>:bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config 'producer_byte_rate=10,consumer_byte_rate=20' --entity-name clientA --entity-type clients --entity-name user2 --entity-type users

Compatibility, Deprecation, and Migration Plan

...