Status
Current state: Under Discussion
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Motivation
Currently, the consumer reports a metric of the lag between the high watermark of a log and the consumer offset. It will be useful to report a similar lead metric between the consumer offset and the start offset of the log. If this number gets close to 0, it's an indication that the consumer may stall or lose data soon. It would be useful to know per partition lead as well.
Public Interfaces
There is no programmatic public interface change. We are only adding new metrics.
Proposed Changes
Add min lead metric across all partitions to KafkaConsumer.the metric name would be:
"records-lead-min"
Also, add per partition lead metrics to KafkaConsumer. The metric names would be:
"TOPIC-PARTITION_ID.records-lead"
"TOPIC-PARTITION_ID.records-lead-avg"
"TOPIC-PARTITION_ID.records-lead-min"
The way to get the metrics is the same as getting other metrics. For example:
TopicPartition tp = new TopicPartition("topic", 0); HashMap<String, String> tags = new HashMap<>(); tags.put("client-id", "metricTestConsumer0"); double currentLead = kafkaConsumer.metrics().get(new MetricName(tp + ".records-lead", "consumer-fetch-manager-metrics", "", tags)).value() double averageLead = kafkaConsumer.metrics().get(new MetricName(tp + ".records-lead-avg", "consumer-fetch-manager-metrics", "", tags)).value() double minLead = kafkaConsumer.metrics().get(new MetricName(tp + ".records-lead-min", "consumer-fetch-manager-metrics", "", tags)).value()
When the consumer does not own a partition anymore, the lead metric will be removed.
Compatibility, Deprecation, and Migration Plan
The change is fully backwards compatible.
Rejected Alternatives
None