THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.
New Metrics
| kafka.server:type=BrokerTopicMetrics, name=UploadTierLag,topic=([-.w]+),partition=([-.w]+) | The tier lag of a topic-partition is defined as the number of records in non-active segments not yet uploaded to the remote storage. |
| kafka.server:type=BrokerTopicMetrics, name=DeleteTierLag,topic=([-.w]+),partition=([-.w]+) | The tier lag of a topic-partition is defined as the number of records in non-active segments marked for deletion but not yet deleted from remote storage. |
Proposed Changes
Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
Reference implementation for UploadTierLag on topic level: https://github.com/satishd/kafka/commit/c96d3af4d02bf515a4355b14f33793211c5b3745
If we have 10 closed segments ready for upload the UploadTierLag will start as the sum of offsets in these 10 segments and as segments are uploaded we will decrease the lag by decrementing it for each segment we have issued an upload for.
On the archive path of Tiered Storage we go through segments which are eligible for upload in a sequential manner. This allows us to emit the UploadTierLag metric by taking the difference between the segment's base offset and the base offset of the first segment not eligible for tiering.
In a similar manner, when we expire segments from Tiered Storage we go through them in a sequential manner. We can emit the DeleteTierLag metric by taking the difference between the segment's base offset and the end offset + 1 of the last eligible for deletion segmentSimilarly, if we have 10 segments which have breached their retention the DeleteTierLag will start as the sum of offsets in these 10 segments and as segments are deleted we will decrease the lag by decrementing it for each segment we have issued a delete for.
Compatibility, Deprecation, and Migration Plan
...