Status

Current state: Under Discussion

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

With the delivery of KIP-714, cluster operators can now pull metrics from a broker (sent by clients) via a plugin. KIP-714 dramatically simplifies observing client behavior via metrics.  Still, it only accounts for metrics from clients (admin, consumer, and producer), leaving out applications that embed client instances, such as Kafka Streams. While the importance of capturing client metrics is evident, there should be a mechanism for capturing the enclosing application metrics, as reviewing both sets will be required to get a complete performance picture. This proposal aims to provide a mechanism for applications containing Kafka clients to supplement the client metrics with some application-specific metrics.

Public Interfaces

In this KIP, we propose to add a new method to the Admin, Consumer, and Producer  interfaces. The KafkaAdminClient, KafkaConsumer, and KafkaProducer  will implement this new method.

Proposed Changes

The method registerAdditionalMetrics  will be added to the Admin, Consumer,  and Producer  interfaces.

...

package org.apache.kafka.clients.producer;

public interface Producer<K, V> extends Closeable {

/**
 * Register additional application metrics available for subscription
 *
 * @param metrics the application metrics to register
 *
 */
void registerAdditionalMetrics(Collection<KafkaMetric> metrics)

Compatibility, Deprecation, and Migration Plan

Since this is a new interface and opt-in behavior, no backward compatibility concerns are anticipated.

Test Plan

There will be unit and integration tests added to ensure correct behavior.

Rejected Alternatives

We could have used a new NetworkClient object inside Kafka Streams to connect to a broker and push Kafka Streams metrics. However, this approach is hacky and not general purpose, as it would only solve the issue for Kafka Streams and not third-party applications. Also, additional connections to a broker incur a cost, so we abandoned this idea.