Current state: Under Discussion
Discussion thread: thread
JIRA:
When a partition becomes offline it is important to quickly determine what customers/service are being impacted - possibly reducing overall disruption. Additionally, while many sources recommend instrumenting alerting on the current OfflinePartitionCount
metric as reported by the active controller, this metric is not tagged in any way. However, as an example, it may be beneficial to only have alerts on a subset of topics or disable alerting for certain test topics. Further as topics are often synonymous with differing use cases (e.g. metrics, logs, etc), having the ability to associate an offline partition with a topic would provide new granularity for debugging, alerting, and SLO/SLA reporting. This KIP proposes to tag the current OfflinePartitionsCount
metric by the topic name associated with the offline partition(s) for these reasons.
The current OfflinePartitionsCount
metric will be tagged by the topic name of the offline partition(s) if at least one partition is offline for that given topic. The current (untagged) metric will still remain, reporting the sum of all offline partitions across all topics. No monitoring changes will be needed to maintain the current functionality.
Current MBean | Proposed MBeans |
---|---|
kafka.controller:type=KafkaController |
|
The proposed changes are based on the requirements:
${TOPIC}
has offline partitions, a Gauge
exists reporting the number of offline partitions under the MBean name kafka.controller:type=KafkaController,topic=${TOPIC}
Gauge
should be purged from the Yammer metric registry. The only changes required for this is the replacement of the current atomic OfflinePartitionsCount
to a concurrent map within the KafkaController. Additionally, some logic will needed to ensure gauges are created/purged as needed.
In my ad-hoc process notation, this flow can largely be define in the diagram below.
No impact will be seen on existing users and no migration plan will be necessary.
OfflinePartitionsCount
metric, representing the sum of all offline partitions will remain as before as a constant indicator. ControllerContext
after each ControllerEvent