Status

Current state: "Under DiscussionAccepted"

Discussion thread: here

JIRA: KAFKA-4195

...

An additional metric exempt-request-time will also be added for each quota entity for the quota type Request to track the time spent processing requests which are exempt from throttling. This will capture the total time for requests from all users/clients that are exempt from throttling so that administrators can view the full CPU utilization using the combination of two metrics.of exempt requests as well.

Producers and consumers currently Clients will expose average and maximum producer/fetch request throttle time as JMX metrics similar to the current produce/fetch throttle time metrics. The existing . These metrics will be updated to reflect the total throttle time for produce and fetch requests including both the producer or consumer including byte-rate throttling and processing request time throttling . Two new metrics request-throttle-time-max and request-throttle-time-min will be added to reflect total request processing time based throttling for all request types including produce/fetchfor all requests of the producer/consumer. Similar metrics may be added for the admin client in future.

Tools

kafka-configs.sh will be extended to support request quotas.  A new quota property will be added, which can be applied to <client-id>, <user> or <user, client-id>:

...

ApiVersions Response (Version: 1) => throttle_time_ms error_code [api_versions] 
  throttle_time_ms => INT32 (new)
  error_code => INT16
  api_versions => api_key min_version max_version 
    api_key => INT16
    min_version => INT16
    max_version => INT16
  throttle_time_ms => INT32 (new) 

CreateTopics Response (Version: 2) => throttle_time_ms [topic_errors] 
  throttle_time_ms => INT32 (new)
  topic_errors => topic error_code error_message 
    topic => STRING
    error_code => INT16
    error_message => NULLABLE_STRING

...

Metrics and sensors will be expired as they are today for Produce/Fetch quotas.

On the client side, a new sensor named request-throttle-time will be added to track total request throttle time returned in all responses. This is in addition to the sensor used to track produce/fetch throttle times, which will continue to be supported. These existing produce/fetch throttle times will include total throttling time for both bandwidth and utilization for produce/fetch requests. New metrics request-throttle-time-avg and request-throttle-time-max will be added and these will include throttle times across all requests including produce/fetchthe existing produce and fetch sensors will track total throttle time of all requests from producers and consumers respectively. This will include both bandwidth as well as utilization based throttling. Throttle time recording will be moved to NetworkClient with appropriate sensor parameters so that produce or fetch sensors are updated based on whether the client corresponds to a producer or consumer. This will also enable addition of similar sensors/metrics for admin clients in future.

Compatibility, Deprecation, and Migration Plan

...