Page History

...

1. Quota configuration for user principal. This prevents users generating heavy traffic from monopolizing resources and impacting the performance of other users in a multi-tenant cluster.
2. Sub-quotas Quota overrides for clients of an authenticated user.  Like the current client-id implementation, this enables a user to rate-limit some producers or consumers to ensure that they don’t impact other more critical clients.  For instance, users may be able to rate-limit an auditing client running in the background, leaving resources always available for a critical event processing client.
3. Client-id quotas for users with unlimited without a user quota override. Existing quota configuration for client-ids will continue to be applied to users with unlimited quota, but these will be applied as per-user quota for clients that share both user principal and client-id.
4. Shared quotas for groups of clients that span multiple users will no longer be supported.

...

kafka-configs.sh will be extended to support authenticated user quotas and sub-specific quotas for clients of a user <user, client-id>.  A new entity type “users” will be added with the same key-value pairs as the existing "clients" entity type:

• producer_byte_rate : The total rate limit for the user’s producers without a client-id quota override
• consumer_byte_rate : The total rate limit for the user’s consumers without a client-id quota override

Quotas Sub-quotas for clients of a user can be configured by specifying entity types "users" and “clients”  in the same command line . For example, the following command sets quotas for <user2, clientA>:

...

Quota configuration for a client with client-id clientX and user principal userN is determined by the following sequence (this example is for producer, similar sequence is applied to consumer):

1. If client-id sub- quota override is defined for clientX of for <userN, clientX> this sub- quota is allocated for the sole use of <userN, clientX>.
2. If user quota override is defined for userN, clientX shares this quota with other clients of userN
3. If quota.user.producer.default is not unlimited, clientX shares this default quota with other clients of userN
4. If client-id quota override is defined for clientX, this quota is allocated for the sole use of <userN, clientX>
5. If quota.producer.default is configured, this default quota is allocated for the sole use of <userN, clientX>
6. Client is not throttled

...

• Simple client-id based quotas are configured using client-id quota override and quota.producer.default : (steps 4, 5, 6)

• Simple user-principal based quotas are configured using user quota override and quota.user.producer.default : (steps 2, 3, 6)
• Hierarchical quotas can be set using sub-quotas and combinations of user/client-id quotas and defaults More specific <user, client-id> quotas and defaults for users and client-ids can be configured if required: (steps 1 - 6)

// Quotas for user1 (without subclient-quotasid overrides).
// Zookeeper persistence path /users/<encoded-user1>
{
    "version":1,
    "config": {
        "producer_byte_rate":"1024",
        "consumer_byte_rate":"2048",
        "user_principal" : "user1"
    }
}

// Top-level total quotas for user2
// Zookeeper persistence path /users/<encoded-user2>
{
    "version":1,
    "config": {
        "producer_byte_rate":"4096",
        "consumer_byte_rate":"8192",
        "user_principal" : "user2"
    }
}
// Quota Sub-Quotasoverride for <user2, clientA>
// Zookeeper persistence path /users/<encoded-user2>/clients/clientA
{
    "version":1,
    "config": {
        "producer_byte_rate":"10",
        "consumer_byte_rate":"30"
    }
}
// Quota Sub-Quotasoverride for <user2, clientB>
// Zookeeper persistence path /users/<encoded-user2>/clients/clientB
{
    "version":1,
    "config": {
        "producer_byte_rate":"20",
        "consumer_byte_rate":"40"
    }
} 

// Quotas for client-id clientA of users without config override if default user quota is unlimitedoverride. 
// Zookeeper persistence path /clients/clientA
{
    "version":1,
    "config": {
        "producer_byte_rate":"100",
        "consumer_byte_rate":"200"
    }
}

...

1. Total rate limits for all clients with user principal user1 is (1024, 2048).
2. Total rate limits for all clients with user principal user2 without additional client-id quota is (4096, 8192).
   
   • The rate limits for clients with user principal user2 AND client-id clientA is (10, 20).
   • Clients of user2 with client-id other than clientA and clientB share the remaining quota (40564096, 81328192).
3. Total rate limits for all clients of user3  is (quota.user.producer.default, quota.user.consumer.default) configured in server.properties, since no config override is specified.
4. If default user quota is unlimited, clients of user3 use client-id quota configuration. For example quota for client-id clientA of user3 is (100, 200). And quota for client-id clientB of user3 without a client-id override is (quota.producer.default, quota.consumer.default)
   • In a single-user cluster, this provides the same semantics as the current client-id implementation
   • In a multi-user cluster,  quotas are now per-user, treating clientA of user4 as a different group from clientA of user2.

...

• quota-id is the concatenation of url-encoded user principal and client-id. Clients-ids without a sub- quota override share the user's quota and hence use the encoded user principal as quota-id.
• In the example (non-encoded user principal is used here for readability):
  • All clients of user1 share the quota-id user1
  • clientA of user2 uses the quota-id user2clientA
  • clientC of user2 uses the quota-id user2 since it does not have a client quota override, sharing a quota with other clients of user2.
  • clientA of user3 uses the quota-id user3clientA

...

Client-id based quota configuration overrides will continue be stored under /config/clients, but these will be applied only to clients of users without a quota override and only if default user quota is unlimited. Quota configuration overrides for user principals will be stored under /config/users. Note that url-encoded version of the user principal will be used as node name under /config/users to cope with Zookeeper naming restrictions. The non-encoded user principal will be stored as a property to make it easy to identify the actual user associated with the path. Sub-quotas Quota overrides for clients of a user will be stored under /config/users/<user>/clients.

Configuration change notifications will be generated for changes to quota configuration similar to the current notifications for client-id quotas. Changes to client-id sub- quotas of a user specify users as the entity_type and the sub-path of the node containing both user and client-id as entity_name. Note that changes to sub-quota affect both the sub-quotas of the particular <user, client-id> as well as the remainder quota allocated to the user's clients without a sub-quota override.

// Change notification for user quota of 

// Change notification for user quota of user1
{
    "version":1,
    "entity_type": "users",
    "entity_name": "user1"
}
// Change notification for client sub-quota of <user2, clientA> that impacts clientA as well as clients of user2 without a sub-quota override}
// Change notification for quota of <user2, clientA>
{
    "version":1,
    "entity_type": "users",
    "entity_name": "user2/clients/clientA"
 } 

...

kafka-configs.sh will be extended to support a new entity type "users". Quota configuration for users will be provided as key-value pairs to be consistent with other configuration options. Hence no new command line arguments will be added to the tool. The tool will parse the key-value pairs specifying rate limits, validate these and convert them to the equivalent JSON for persistence in Zookeeper. Changes to user quotas and client sub-quotas will be validated to ensure that the total quota of the user is not exceeded by the clients with sub-quotas. The existing entity “clients” will continue to be supported to set client-id quotas for users with unlimited quota.

...