...

We will also expose these metrics on the processor-node-level for stateful operators at the recording level DEBUG TRACE with the following tags:

• type = stream-processor-node-metrics
• thread-id=[threadId]
• task-id=[taskId]
• processor-node-id=[processorNodeId]

In all cases the metrics will be computed at the end of the operation or subtopology. In the case of task-level metrics for example, this means the metric reflects the end-to-end-latency at the time it leaves the sink node.

For those of you wondering what TRACE level metrics are, we will be adding this new metrics level as part of the KIP.

Proposed Changes

Imagine a simple 3-node subtopology with source node O, filter node F, aggregation A, and sink node I. For any record flowing through this with record timestamp t, let t_O be the system (wallclock) time when it is sent from the source topic, t_A be the time when it is finished being processed by the aggregator node, and t_I be the time when it leaves the sink node for the output or repartition topic. The end-to-end latency at operator O for a given record is defined as 

...