Status

Current state: Draft

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Metric Reporter plugins have the ability to collect metrics from various Kafka components (broker, connect) and Kafka client libraries (producer, consumer, admin client, streams) and expose/send those metrics to other systems.

However, the current reporter interface does not give much context about the system exposing those metrics. For instance, it is currently difficult to infer which component or library is exposing those metrics: a broker, a connect worker, or a client library running within any of those components.

There are two issues we would like to address:

1. Letting metric reporters know which component is exposing the metrics

The stock metrics reporter implementation – JmxReporter already gets passed some additional context via the JMX prefix (e.g. kafka.server kafka.streams, kafa.admin.client, etc.) at instantiation. However, this information is not available to additional metrics reporters configured at startup.

2. Exposing the context in which those metrics are collected

For client metrics, it is not straightforward for metrics reporter plugins to know which service the client is run in, e.g. for a connect sink task, the metrics reporter configured in the Kafka consumer has no information about connect worker in which it runs.

This makes it impractical to use metrics reporter plugins to correlate all the metrics exposed by a service.

For Kafka brokers, we currently resort to various workarounds to get the broker id (the config is manipulated before passed to the metrics reporter) or cluster id (only accessible via ClusterResourceListener).

Some of that information is valuable to expose in metrics and it would be useful to have a consistent mechanism to provide this context to metrics reporters.

To address those issues, we propose to add a mechanism to pass additional metadata to metrics reporters, so they can properly infer the context in which metrics are registered and which component exposes them.

We also propose to add new client configurations, so any application can propagate metrics context metadata from the parent application down to the metric reporter instantiated by the client.

Public Interfaces

MetricsReporter interface changes

To expose additional metadata fields to metric reporter plugins, we propose to add a new callback method to the MetricsReporter interface

public interface MetricsReporter extends Reconfigurable, AutoCloseable {
    [...]
    
    /**
     * Callback method providing context metadata for the
     * service or library exposing metrics
     *
     * @param metricsContext the metric context
     */
    @InterfaceStability.Evolving
    default void contextChange(MetricsContext metricsContext) {}
}

The MetricsContext will encapsulate this additional metadata as a map of key-value pairs.

/**
 * MetricsContext encapsulates additional metadata about metrics exposed via a
 * {@link org.apache.kafka.common.metrics.MetricsReporter}
 * 
 * The metadata map provides following information:
 * - a <code>_namespace</node> field indicating the component exposing metrics
 *   e.g. kafka.server, kafka.consumer
 *   {@link JmxReporter} uses this as prefix for mbean names
 *
 * - for clients and streams libraries: any freeform fields passed in via
 *   client properties in the form of `metrics.context.<key>=<value>
 *
 * - for kafka brokers: kafka.broker.id, kafka.cluster.id
 * - for connect workers: connect.kafka.cluster.id, connect.group.id
 */
@InterfaceStability.Evolving
public interface MetricsContext {

    /* predefined fields */
    String NAMESPACE = "_namespace"; // metrics namespace, formerly jmx prefix
    
    /**
     * Returns metadata fields
     */
    public Map<String, String> metadata();
}

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.