...
Discussion thread: here
JIRA: KAFKA-9983, KAFKA-10054
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
The following metrics would be added:
- record-e2e-latency-max min [ms]
- record-e2e-latency-p99 max [ms] (99th percentile)
- record-e2e-latency-p90 p99 [ms] (90th percentile)
- record-e2e-latency-min p90 [ms]
These will be exposed on the processor-node-level with the following tags:
...
In all cases the metrics will be computed at the end of the operation or subtopology, once the processing has been complete.
Proposed Changes
Imagine a simple 3-node subtopology with source node O, filter node F, aggregation A, and sink node I. For any record flowing through this with record timestamp t, let tO be the system (wallclock) time when it is sent from the source topic, tA be the time when it is finished being processed by the aggregator node, and tI be the time when it leaves the sink node for the output or repartition topic. The end-to-end latency at operator O for a given record is defined as
...