...

These will be exposed on the task-level at the recording level INFO with the following tags

• type = stream-task-metrics
• thread-id=[threadId]
• task-id=[taskId]

We will also expose these metrics on the processor-node-level for stateful operators at the recording level TRACE with  with the following tags:

• type = stream-processor-node-metrics
• thread-id=[threadId]
• task-id=[taskId]
• processor-node-id=[processorNodeId]

These will be reported for source and terminal operators at the recording level INFO

We will also expose these metrics for stateful operators at the recording level TRACE (which is also being added as part of this KIP)

In all cases the metrics will be computed at the end of the operation or subtopology. In the case of task-level metrics for example, this means the metric reflects the end-to-end-latency at the time it leaves the sink node.For those of you wondering what TRACE level metrics are, we will be adding this new metrics level as part of the KIP, once the processing has been complete.

Proposed Changes

Imagine a simple 3-node subtopology with source node O, filter node F, aggregation A, and sink node I. For any record flowing through this with record timestamp t, let t_O be the system (wallclock) time when it is sent from the source topic, t_A be the time when it is finished being processed by the aggregator node, and t_I be the time when it leaves the sink node for the output or repartition topic. The end-to-end latency at operator O for a given record is defined as 

...