Page History

...

kafka.consumer:type=consumer-fetch-manager-metrics,client-id="{client-id}",topic="{topic}"

records-latency-max

kafka.consumer:type=consumer-fetch-manager-metrics,partition="{partition}",topic="{topic}",client-id="{client-id}"

...

Report latency metrics in the Kafka Consumer at the client (max latency) and partition level,  similar to how consumer lag is currently reported.

Since Kafka 0.10.0 the Kafka Producer will by default add a value for the timestamp property of a ProducerRecord unless it’s otherwise provided by the user.  The default value is the KIP-32 introduced a timestamp field to the Kafka message format.  The timestamp could be provided by the user, the KafkaProducer, or the Broker, depending on how the Broker configuration message.timestamp.type is defined.  When defined by Kafka the timestamp value uses current wallclock time represented as a unix epoch long in milliseconds returned by the Java Standard Library call to System.getCurrentMillis().  In the general case where we assume records are produced and consumed in the same timezone, the same network, and on machines with We assume that all application and Broker machines enable active clock synchronization services , then and that the latency may be calculated by taking the difference of current wall clock time with the timestamp from a fetched recordConsumerRecord.  When latency is calculated as negative then the metric value will be reported as NaN.

UNIX epoch time is always represented as UTC time, and is therefore agnostic of any machine's particular default timezone.

...

1. It only works in cases where offsets are committed back to Kafka.  If an app or stream processor uses its own offset management then the current offset of a partition cannot be obtained from Kafka.
2. It can only predict an estimate.  Accuracy improves with higher fidelity lookup tables (latest offsets looked up more frequently).

Disclosure: I (Sean Glover) am the author of the Kafka Lag Exporter project and the "Monitor Kafka Consumer Group Latency with Kafka Lag Exporter" blog post by Lightbend.