General architecture

In addition to the user view of the metrics (see https://docs.google.com/document/d/1voyUIQ2DrWkoY-BsJwM8YvF4gGKB76CDG8BYL8XBc7A/edit#heading=h.vv2fbulkp7t 

 and https://beam.apache.org/documentation/sdks/javadoc/2.6.0/org/apache/beam/sdk/metrics/package-frame.html), here is a diagram that shows the architecture of the metrics system inside the runners:


Metrics Pusher

There was a recent addition to this architecture in some runners (Spark and Flink for now) called Metrics Pusher  (see https://s.apache.org/runner_independent_metrics_extraction) represented on the left side of the diagram:




  • No labels