THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
The following metrics would be added at the Task Level:
kafka.connect:type=sink-
...
task-metrics,connector="{connector}",task="{task}"
| Attribute name | Description |
|---|---|
|
...
| The maximum latency of a record, measuring by comparing the record timestamp with the system time when it has been received by the Sink task |
|
...
| The average latency of a record, measuring by comparing the record timestamp with the system time when it has been received by the Sink task |
|
...
| The maximum time taken by this task to convert and transform a record. |
|
...
| The average time taken by this task to convert and transform a record. |
kafka.connect:type=source-task-metrics,connector="{connector}",task="{task}"
| Attribute name | Description |
|---|---|
|
...
| The maximum time in milliseconds taken by this task to transform and convert for a record. |
|
...
| The average time in milliseconds taken by this task to transform and convert for a record. |
Proposed Changes
Sink Connectors have 3 stages:
polling: gather batch of consumer records
convert/transform: convert records individually to generic
SinkRecordand , apply transformers, and prepare batchesprocess: put record batches into a external system.
Process stage already has a latency metric: put-batch-time
To measure record latency sink-record-latency , it's proposed to measure the different between record timestamp and current system time (wall-clock) at the beginning of the convert stage as it is when records are iterated already.
Convert latency sink-record-convert-transform-time measures the convert and transformation per-record.
Polling can be monitored with Consumer fetch metrics, e.g. fetch-latency-avg/max
...
On the other hand, in the Source Connector has:
polling: gather batch of records from external system
- transform/convert: apply transformations and convert records to
ProducerRecordindividually - send records: send records to Kafka topics individually
Polling records stage already has a latency metric: poll-batch-time .
source-record-transform-convert-time metric will measure the transformations applied and conversion from SourceRecord into ProducerRecord .
...