Page History

...

Currently, Sink Connectors include only putBatchLatency metric to measure the time to sink a batch of records into a external system.

In other to correlate these metrics observe connector performance and measure a complete end-to-end latency of the Sink Connector there is a need for additional measures:

• record e2e latency: wall-clock time - record timestamp  to evaluate how late records are processed.

• convert time: latency to convert + transform batch of records to know how long is taking to convert and transform records

With these metrics, it will be clearer how much latency the sink connector introduces, and where the bottleneck may be.

In the case of source connectors, convert time can be added to improve the latency monitoring and have parity with the metrics on the sink connectors.

...

The following metrics would be added at the Task Level:

• sink-record-e2e-latency-min [ms]

• sink-record-e2e-latency-max [ms]

• sink-record-e2e-latency-avg [ms]

• sink-record-convert-time-max [ms]

• sink-record-convert-time-avg [ms]

• source-record-convert-time-max [ms]

• source-record-convert-time-avg [ms]

...

Process stage already has a latency metric: put-batch-time 

To measure record e2e latency sink-record-e2e-latency , it's proposed to measure the different between record timestamp and current system time (wall-clock) at the beginning of the convert stage as it is when records are iterated already.

...