...

(1) kafka.controller:type=KafkaController,name=ControllerState

type: gauge

value: reporting the state the controller is in is in, i.e. the event that is currently being processed. Some actions like partition reassignment may take a while and include many events (potentially interleaved with other events), but that doesn't change the fact that at most one event is processed at a time.

Valid states:
0 - idle
1 - starting
2 - resigning
3 - broker change
4 - topic creation
5 - topic deletion
6 - partition reassigning
7 - auto leader balancing
8 - manual leader balancing
9 - controlled shutdown

...

(8). kafka.controller:type=ControllerStats,name=ManualLeaderBalancingRateAndTimeMs
type: timer
value: rate and latency for the controller to manually balance the leaders

(9). kafka.controller:type=ControllerStats,name=QueueSize

 type: gauge

value: the size of the queue

(10). kafka.controller:type=ControllerStats,name=QueueTimeMs

type: histogram

value: how long an event is waiting in the controller queue before being processed

ControllerChannelManager Metrics

We also want to know the size of the queue in ControllerChannelManager:

(119) kafka.controller:type=ControllerChannelManager,name=TotalQueueSize

type: gauge

(1210) kafka.controller:type=ControllerChannelManager,name=QueueSize,brokerId=10

...

We are introducing new metrics so there is no compatibility impact.

Rejected Alternatives

1. Don't add these metrics: it's currently difficult to detect these issues, they impact cluster health and the overhead of the proposed metrics is low.
2. Use Kafka metrics instead of Yammer metrics: most of the broker metrics use Yammer Metrics so it makes sense to stick with that until we have a plan on how to migrate them all to Kafka Metrics.

Future work

1. KAFKA-5028 introduced a queue for Controller events. It would be useful to have a gauge for the queue size and a histogram for how long an event waits in the queue before being processed. However, we are in the process of making additional changes to improve the handling of soft failures and there's a possibility that the controller queue could be replaced by a broker queue for all ZK communication. We will see how that develops before deciding which metrics should be exposed. In the meantime, the ControllerState and other metrics should provide enough information to issue an alert if the Controller is not healthy.

...