THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Describe the problems you are trying to solve.
Problem
I am an operator of a Tiered Storage Kafka cluster and I We would like to know whether interactions with the remote tier, uploads and deletions, are progressing at a steady pace.
For example, if uploads are not progressing at a constant rate I will have data building up in local storage and I might need to take corrective actions (like adding more storage temporarily). Likewise, if deletes are not progressing at a constant rate this might indicate a problem with the retention settings of my topics which I would like to remedy.
Solution
To get this observability the progress Tiered Storage is making in uploading and deleting/expiring segments. To achieve this we would like to expose a tier-upload and tier-delete metrics from the point of view of Kafka. Since upload and deletion are carried out by plugins such metrics emitted by Kafka are only going to be best estimates on what it believes the state of the world is. We propose the new metrics to be emitted on per topic and per partition granularity to be able to narrow down which brokers are slow.
Other Tiered Storage metrics are emitted only at a topic level, but we would like to also expose these at a partition level because this will allow us to easily track whether a single broker is experiencing a slowdown or the issue is spread across the cluster.
We leave more detailed progress metrics to be emitted by the developers of tiered storage Tiered Storage plugins.
Public Interfaces
...
| kafka.server:type=BrokerTopicMetrics, name=UploadTierLag, topic=([-.w]+), partition=([-.w]+) | The tier lag of a topic-partition is defined as the number of records in non-active segments eligible for tiering not yet uploaded to the remote storage. |
| kafka.server:type=BrokerTopicMetrics, name=DeleteTierLag, topic=([-.w]+), partition=([-.w]+) | The tier lag of a topic-partition is defined as the number of records in non-active segments marked for deletion but not yet deleted from remote storage. |
...
Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
Constraints
The metric calculation will not acquire any locks. If it had to acquire locks then it would be contending with the archival/deletion paths of Tiered Storage.
Reference implementation
Reference implementation for UploadTierLag on topic level: https://github.com/satishd/kafka/commit/c96d3af4d02bf515a4355b14f33793211c5b3745
Details
The metrics for upload and delete will be updated on every respective archival and deletion cycle. If no archival or deletion happened in a particular cycle then the metric will not change.
On the archive path of Tiered Storage we go through segments which are eligible for upload in a sequential manner. This allows us to emit the UploadTierLag metric by taking the difference between the segment's base offset and the base offset of the first segment not eligible for tiering.
...