Discussion thread | https://lists.apache.org/thread/h2lgqgrvcptzj3c8q4cjzv5jopgtwx9o |
---|---|
Vote thread | TBDhttps://lists.apache.org/thread/09lfq4j7w23h6o1lmywl6k9691sqykhm |
ISSUE | https://github.com/apache/incubator-paimon/issues/742 |
Release | TBD |
...
Code Block | ||
---|---|---|
| ||
public class CommitMetrics { private final String GROUP_NAME = "commitMetricGroup"; private final String LAST_COMMIT_DURATION_METRIC = "lastCommitDuration"; ... private MetricGroup group private final Map<Integer, BucketMetricGroup> bucketMetricGroups = new MetricsGroupHashMap<>(GROUP_NAME); private final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>(); private final MetricGroup genericMetricGroup; private void registerCommitMetricsregisterGenericCommitMetrics() { group.gauge(LAST_COMMIT_DURATION_METRIC, new CommitDurationGauge()); ... } } private void registerTaggedCommitMetrics() { if(isNewBucket()) { registerTaggedMetrics(); } if(isNewBucket()) { registerTaggedMetrics(); } } } |
CommitMetrics
list includes commit duration, counter of CommitMetrics
list includes commit duration, counter of files / records etc.
Metric Name | Description | Type | Tagged | Unit | Update at |
lastCommitDuration | The time it took to complete the last commit. | Gauge | none | Ms | Timer starts before commit starting, update commit duration after the last commit finished. |
commitDuration | Distributions of the time taken by the last few commits. | Histogram | none | Ms | Timer starts before commit starting, update commit duration after each commit finished. |
lastCommitAttempts | The number of attempts the last commit made. | CounterGauge | none | Number | Increment by 1 when trying to commit once, clear the counter after the last commit finished. |
lastTableFilesAdded | Number of added table files in last commit, including newly created data files and compacted after. | Gauge | Partition, Bucket | Number | Collecting changes from committables |
lastTableFilesDeleted | Number of deleted table files in last commit, which comes from compacted before. | Gauge | Partition, Bucket | Number | Collecting changes from committables |
lastTableFilesAppended | Number of appended table files in last commit, which means the newly created data files. | Gauge | Partition, Bucket | Number | Collecting changes from committables |
lastTableFilesCommitCompated | Number of compacted table files in last commit, including compacted before and after. | Gauge | Partition, Bucket | Number | Collecting changes from committables |
lastChangelogFilesAppended | Number of appended changelog files in last commit | Gauge | Partition, Bucket | Number | Collecting changes from committables |
lastChangelogFileCommitCompacted | Number of compacted changelog files in last commit | Gauge | Partition, Bucket | Number | Collecting changes from committables |
totalTablesFiles | Number of total data files currently maintained on storage. | Counter | none | Number | Collecting changes from committables |
totalChangelogFiles | Number of total changelog files currently maintained on storage. | Counter | none | Number | Collecting changes from committables |
lastGeneratedSnapshots | Number of snapshot files generated in last commit, maybe 1 snapshot or 2 snapshots. | Gauge | none | Number | After collecting changes from committables |
totalSnapshots | Number of currently retained total snapshots. | CounterGauge | none | Number | When trying to commit, the counter will increment by the number of snapshots generated When expiring snapshots, the counter will decrement by the number of expiring snapshots. |
lastDeltaRecordsAppended | Delta records count in last commit with APPEND commit kind | Gauge | Partition, Bucket | Number | Preparing snapshot file with APPEND commit kind |
lastChangelogRecordsAppended | Changelog records count in last commit with APPEND commit kind | Gauge | Partition, Bucket | Number | Preparing snapshot file with APPEND commit kind |
lastDeltaRecordsCommitCompated | Delta records count in last commit with COMPACT commit kind | Gauge | Partition, Bucket | Number | Preparing snapshot file with COMPACT commit kind |
lastChangelogRecordsCommitCompated | Changelog records count in last commit with COMPACT commit kind | Gauge | Partition, Bucket | Number | Preparing snapshot file with COMPACT commit kind |
lastPartitionsWritten | Number of partitions written in last commit | Gauge | none | Number | After collecting changes from committables |
lastBucketsWritten | Number of buckets written in last commit | Gauge | none | Number | After collecting changes from committables |
...
Code Block |
---|
public class ScanMetrics { private final String GROUP_NAME = "scanMetricGroup"; private final String LAST_SCAN_SKIPPED_MANIFESTS_METRIC = "lastScanSkippedManifests"; ... private final Map<Integer, MetricGroupBucketMetricGroup> groupbucketMetricGroups = new MetricsGroupHashMap<>(GROUP_NAME); private void registerScanMetrics() { group.gauge(LAST_SCAN_SKIPPED_MANIFESTS_METRIC, new LastScanSkippedManifestsGauge()); ... } final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>(); private void registerTaggedScanMetrics() { if(isNewBucket()) { registerTaggedMetrics(); } if(isNewPartition()) { registerTaggedMetrics(); } } } |
ScanMetrics
list includes duration, data files and manifest files counter.
...
Code Block |
---|
public class CompactionMetrics { private final String GROUP_NAME = "compactionMetricGroup"; private final String LAST_TABLE_FILES_COMPACTED_BEFORE_METRIC = "lastTableFilesCompactedBefore"; ... private final Map<Integer, MetricGroupBucketMetricGroup> groupbucketMetricGroups = new MetricsGroupHashMap<>(GROUP_NAME); private final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>(); private void registerScanMetricsregisterTaggedCompactionMetrics() { group.gauge(LAST_TABLE_FILES_COMPACTED_BEFORE_METRIC, new LastTableFilesCompactedBeforeGauge() if(isNewBucket()) { registerTaggedMetrics(); } if(isNewPartition()) { registerTaggedMetrics(); ...} } } |
CompactionMetrics
list includes duration, and counter of files, sorted runs etc.
...