Current state: Under Discussion
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
With the delivery of KIP-714, cluster operators can now pull metrics from a broker (sent by clients) via a plugin. KIP-714 dramatically simplifies observing client behavior via metrics. Still, it only accounts for metrics from clients (admin, consumer, and producer), leaving out applications that embed client instances, such as Kafka Streams. While the importance of capturing client metrics is evident, there should be a mechanism for capturing the enclosing application metrics, as reviewing both sets will be required to get a complete performance picture. This proposal aims to provide a mechanism for applications containing Kafka clients to supplement the client metrics with some application-specific metrics.
In this KIP, we propose to add a new method to the Admin, Consumer, and Producer interfaces. The KafkaAdminClient, KafkaConsumer, and KafkaProducer will implement this new method.
The method registerAdditionalMetrics will be added to the Admin, Consumer, and Producer interfaces.
package org.apache.kafka.clients.admin;
public interface Admin extends AutoCloseable {
/**
* Register additional application metrics available for subscription
*
* @param metrics the application metrics to register
*
*/
void registerAdditionalMetrics(Collection<KafkaMetric> metrics) |
package org.apache.kafka.clients.consumer;
public interface Consumer<K, V> extends Closeable {
/**
* Register additional application metrics available for subscription
*
* @param metrics the application metrics to register
*
*/
void registerAdditionalMetrics(Collection<KafkaMetric> metrics) |
package org.apache.kafka.clients.producer;
public interface Producer<K, V> extends Closeable {
/**
* Register additional application metrics available for subscription
*
* @param metrics the application metrics to register
*
*/
void registerAdditionalMetrics(Collection<KafkaMetric> metrics) |
Since this is a new interface and opt-in behavior, no backward compatibility concerns are anticipated.
There will be unit and integration tests added to ensure correct behavior.
We could have used a new NetworkClient object inside Kafka Streams to connect to a broker and push Kafka Streams metrics. However, this approach is hacky and not general purpose, as it would only solve the issue for Kafka Streams and not third-party applications. Also, additional connections to a broker incur a cost, so we abandoned this idea.