Versions Compared

Old Version 7

changes.mady.by.user Mahsa Seifikar

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (jsancio)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents


Status

Current state: Under DiscussionAccepted

Vote thread: NA https://lists.apache.org/thread/kyrvq8ps35d83xco9xr4j9m0qx794t7t

Discussion thread: https://lists.apache.org/thread/8ky7t8xybgy2omkqld1fbtk16op9p5qo

...

The controller thread is responsible for critical cluster coordination tasks, including leadership elections, topic metadata updates, partition assignments, and Raft-based state changes. While we have metrics for event queue and processing durations — kafka.controller:type=ControllerEventManager,name=EventQueueTimeMs tracks the time an event waits in the queue before processing, and kafka.controller:type=ControllerEventManager,name=EventQueueProcessingTimeMs measures the time an event takes to be processed after taking've been retrieved from the event queue. However, there is currently no metric to measure how much time the controller spends actively processing work versus is idle either waiting for new events or waiting Raft callbacks while processing time. This lack of visibility makes it challenging to assess the controller’s performance, detect potential bottlenecks, or optimize resource allocation in high-load scenarios.

This KIP proposes introducing a new metric to monitor the controller thread’s idleness ratio.

Public Interfaces

Monitoring

NameTypeDescription
kafka.controller:type=ControllerEventManager,name=
ControllerThreadIdleRatio
AvgIdleRatio
Gauge
TimeRatioThe idle ratio measures the proportion of time the controller thread is
idle relative to the total time (idle+active). Values range
not actively processing an event using the existing metric org.apache.kafka.raft.internals.TimeRatio which tracks idleness over the actual measurement interval.

The metric is calculated using the following formula and value ranges from 0 to 1, where 0 indicates the controller thread is

fully busy and 1 indicates it is completely idle.

constantly processing without breaks, 1 signifies the controller spends most of its time waiting for work:


                      controller idle ratio = idle_time/(active_time+idle_time)


The components are defined as follows:

  • active_time: This is the total time the controller thread actively spends processing events or performing computation work rather than waiting for callbacks or external operations
  • idle_time: This is the total time the controller thread is not executing any instructions while waiting for either new events to appear in the queue or Raft callback. 


Compatibility, Deprecation, and Migration Plan

...