Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

NameDescription
kafka.server:type=broker-metadata-metrics,name=publisher-error-countReports the number of errors encountered by the BrokerMetadataPublisher while publishing a new MetadataImage based on the MetadataDelta 
kafka.server:type=broker-metadata-metrics,name=listener-batch-load-error-countReports the number of errors encountered by the BrokerMetadataListener while generating a new MetadataDelta based on the log it has received thus far.
kafka.controller:type=KafkaController,name=ForceRenounceCountReports the number of times this controller node has renounced leadership of the metadata quorum owing to an error encountered during event processing

Proposed Changes

ControllerForceRenounceCount and MetadataPublisherErrorCount 

The MetadataPublisherErrorCount metric reflects the count of errors encountered while publishing a new version of the MetadataImage using the metadata log.

Both these metrics can be used to set up alerts so that affected nodes are visible and needed remedial actions can be performed on them.

Controllers

Any errors during metadata processing on the Active Controller cause it to renounce the quorum leadership. These are different than the general Raft elections being triggered due to other reasons like a roll. Repeated elections being caused due to errors in the active controller could point to issues in the metadata log generation handling logic and having visibility into these would be helpful. The ControllerForceRenounceCount metric reflects the number of times a controller node has had to renounce quorum leadership due to an error in the event processing logic.

The ForceRenounceCount metric will be incremented anytime the controller is going to resign as a result of handling exceptions in Event Processing. This metric will reflect the count of force resignations that a controller (both leader and non-leader) underwent since the last restart.

https://github.com/apache/kafka/blob/14d2269471141067dc3c45300187f20a0a051777/metadata/src/main/java/org/apache/kafka/controller/QuorumController.java#L409 

...

The publisher-error-count metric will be incremented by one every time there is an error in publishing a new MetadataImage . This metric will reflect the count of cumulative errors since the broker started up.

https://github.com/apache/kafka/blob/14d2269471141067dc3c45300187f20a0a051777/core/src/main/scala/kafka/server/metadata/BrokerMetadataPublisher.scala#L125

...

The listener-batch-load-error-count metric will be incremented every time there is an error in loading batches and generating MetadataDelta from them. This metric will reflect the count of cumulative errors since the broker started up.

https://github.com/apache/kafka/blob/14d2269471141067dc3c45300187f20a0a051777/core/src/main/scala/kafka/server/metadata/BrokerMetadataListener.scala#L112

...