...
This proposal aims to enhance Kafka's monitoring capabilities by introducing a new metric to track the replication offset lag for a given topic-partition. The metric will be calculated by taking the difference between the LEO of a partition, which will be periodically updated by calling Consumer.endOffsets(), and the LRO of a partition, which will be stored in an in-memory "cache" and updated during the task's producer callback.
Anchor | ||||
---|---|---|---|---|
|
The proposed changes involve the following steps: Anchor
LRO Tracking Mechanism:
- Introduce a new in-memory "cache" to store the LRO for each topic-partition.
- Update the producer callback logic to keep this LRO cache up-to-date.
LEO Acquisition during Poll Loop:
- Introduce a periodic poll mechanism to keep a fresh set of end offsets for all partitions
- An alternative is to call Consumer.endOffsets() at each poll, which would be inefficient and expensive.
- Introduce a periodic poll mechanism to keep a fresh set of end offsets for all partitions
- Calculate the replication offset lag as
LEO - LRO
Expose Replication Offset Lag Metric:
- Introduce a new MirrorMaker metric, named "replication_offset_lag", which represents the calculated replication offset lag.
...