This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: KAFKA-4195

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka currently supports quotas by data volume.  Clients that produce or fetch messages at a byte rate that exceeds their quota are throttled by delaying the response by an amount that brings the byte rate within the configured quota. However, if a client sends requests too quickly (e.g., a consumer with fetch.max.wait.ms=0), it can still overwhelm the broker even though individual request/response size may be small. It will be useful to additionally support throttling by request rate to ensure that broker resources are not monopolized by some users/clients.

Public Interfaces

Request rate quotas

The current produce and fetch quota limits are based on byte rate within a quota window. It may be harder to estimate sensible values for request rates for configuring quotas. While 5 MB/second byte rates for producer/consumer are meaningful, 10 requests/second is perhaps less meaningful.  For simpler configuration, quotas for requests will be configured as a percentage of time within a quota window that a client is allowed to use. This approach keeps the code consistent with the existing quota implementation, while making it simpler for administrators to configure quotas for different clients/users.

Default quotas

By default, clients will not be throttled based on request rate, but defaults can be configured using the dynamic default properties at <client-id>, <user> and <user, client-id> levels. Defaults as well as overrides are stored as dynamic configuration properties in Zookeeper alongside the other rate limits.

Metrics and sensors

Two new metrics and corresponding sensors will be added to track request-rate and throttle-time of each quota entity for quota type Request. These will be handled similar to the metrics and sensors for Produce/Fetch.

Tools

kafka-configs.sh will be extended to support request quotas.  A new quota property will be added, which can be applied to <client-id>, <user> or <user, client-id>:

• request_time_percent : The percentage of time for requests from the user or client within a quota window

For example:

bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config 'request_time_percent=1.0' --entity-name user1 --entity-type users

Default quotas for <client-id>, <user> or <user, client-id> can be configured by omitting entity name. For example:

bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config 'request_time_percent=10.0' --entity-type users

Proposed Changes

Quota entity

Request quotas will be supported for <client-id>, <user> and <user, client-id> similar to the existing produce/fetch rate quotas.  In addition to produce and fetch rates, an additional quota property will be added for request rate throttling. As with produce/fetch quotas, request quotas will be per-broker. Defaults can be configured using the dynamic default properties at <client-id>, <user> and <user, client-id> levels.

Request rate quotas

Quotas for requests will be configured as a percentage of time within a quota window that a client is allowed to use. For example, with the default configuration of a 1 second quota window size and 8 I/O threads handling requests, the total time a broker can spend processing requests is 8 seconds across all the threads. If user alice has a request quota of 1 percent, the total time all clients of alice can spend in the request handler in any one second window is 80 milliseconds. When this time is exceeded, a delay is added to the response to bring alice’s usage within the configured quota. The maximum delay added to any response will be the window size.  The calculation of delay will be the same as the current rate calculation:

• If O is the observed usage and T is the target usage over a window of W, to bring O down to T, we need to add a delay of X to W such that: O * W / (W + X) = T.
• Solving for X, we get X = (O - T)/T * W.

Sample configuration in Zookeeper

The version number for quota configuration will be increased from 1 to 2.

// Quotas for user1
// Zookeeper persistence path /config/users/<encoded-user1>
{
    "version":2,
    "config": {
        "producer_byte_rate":"1024",
        "consumer_byte_rate":"2048",
		"request_time_percent" : "1.0"
    }
}

Co-existence of multiple quotas

Produce and fetch byte rate quotas will continue to be applied as they are today. Request rate throttling will be applied on top if necessary. For example, if a large number of small produce requests are sent followed by a very large one, both request quota and produce byte rate quota may be violated by the same request. The produce byte rate delay is applied first. Request rate delay is computed only after the produce delay,. During this time, the quota window time would have moved forward, while the request handling time for this request stays constant. The request rate quota is perhaps no longer violated (or the delay may be lower due to the first delay already applied). The remaining delay if any is applied to the response.

Metrics and sensors

Two new metrics and corresponding sensors will be added to track request-time and throttle-time of each quota entity for quota type Request.  The request-time sensor will be configured with the quota for the user/client so that quota violations can be used to add delays to the response.

Metrics and sensors will be expired as they are today for Produce/Fetch quotas.

Compatibility, Deprecation, and Migration Plan

What impact (if any) will there be on existing users?

None

If we are changing behavior how will we phase out the older behavior?

By default, clients are not throttled on request rate. Quotas can be configured dynamically if required.

Test Plan

One set of integration and system tests will be added for request throttling. Since most of the code can be reused from existing producer/consumer quota implementation and since quota tests take a significant amount of time to run, one test for testing the full path should be sufficient.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.