...

Discussion thread: here 

JIRA: KAFKA-9983, KAFKA-10054

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

The following metrics would be added:

• record-e2e-latency-max min [ms]
• record-e2e-latency-p99 max [ms] (99th percentile)
• record-e2e-latency-p90 p99 [ms] (90th percentile)
• record-e2e-latency-min p90 [ms] 

These will be exposed on the processor-node-level with the following tags:

...

In all cases the metrics will be computed at the end of the operation or subtopology, once the processing has been complete.

Proposed Changes

Imagine a simple 3-node subtopology with source node O, filter node F, aggregation A, and sink node I. For any record flowing through this with record timestamp t, let t_O be the system (wallclock) time when it is sent from the source topic, t_A be the time when it is finished being processed by the aggregator node, and t_I be the time when it leaves the sink node for the output or repartition topic. The end-to-end latency at operator O for a given record is defined as 

...