Status

Current state: Under DiscussionAccepted (vote thread)

Discussion thread: here

JIRA: https://issues.apache.org/jira/browse/KAFKA-9106

...

This KIP proposes to make the set of metrics visible in JMX dynamically configurable via a set of whitelist/blacklist regular expressions on the mbean names.

...

Two new configurations will be introduce to allow filtering the set of metrics exposed via JMX.

The configurations will also be marked as dynamic to allow updates at runtime.would apply to brokers, as well as clients, streams, and connect.

kafka.metrics.jmx.whitelist=<regular expression of mbeans to expose> (defaults to ".*")

kafka.metrics.jmx.blacklist=<regular expression of mbeans to hide> (defaults to "", takes precedence over the whitelist)These configurations will apply to both JMX metrics exposed via Yammer metrics as well as JMX metrics exposed via Kafka's own metrics library. 

By default, this would not change the set of metrics exposed via JMX: all metrics would be exposed.Updates to the configurations at runtime would both add and remove JMX metrics as needed to reflect the new filters.

All metrics continue to be collected in the background regardless of those settings, so this only affects the visibility in JMX and not the actual metric values.

Brokers specifics:

• Configurations will apply to JMX metrics exposed via Yammer metrics as well Kafka's internal metrics library.

• Configurations will be marked as dynamic to allow updates at runtime: adding and removing JMX metrics as needed to reflect the new filters.

Clients, streams, and connect:

• Configuration will apply to all the metrics exposed in JMX metrics via Kafka's internal metrics library.

Example

To hide log-level metrics LogEndOffset, LogStartOffset, and NumLogSegments, as well as any partition level metrics, except  UnderMinIsr, one could exclude those by specifying the following property in their server.properties file:

metrics.jmx.blacklist=(kafka.log:type=Log,name=(LogEndOffset|LogStartOffset|NumLogSegments),.*\

                            |kafka.cluster:type=Partition,name=(?!UnderMinIsr),.*)

Proposed Changes

A new KafkaYammerMetrics class will be added to hold a Kafka-specific Yammer metrics registry, replacing the default global yammer metrics registry, which allows us to override the JMX behavior of Yammer metrics.

...