THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Discussion thread | https://lists.apache.org/thread/h2lgqgrvcptzj3c8q4cjzv5jopgtwx9o |
|---|---|
| Vote thread | https://lists.apache.org/thread/09lfq4j7w23h6o1lmywl6k9691sqykhm |
| ISSUE | https://github.com/apache/incubator-paimon/issues/742 |
| Release | TBD0.6 |
Motivation
Currently Paimon runs without monitoring and metrics out of box. In the production environment, users need to know how their Paimon Table behaves like what is the commit duration, how many files each commit added or deleted, the status of compaction operation, the duration of scan query, lag of streaming reading and so on. So we need to introduce a metrics system and more metrics for Paimon.
...
Metric Name | Description | Type | Tagged | Unit | Update at |
lastScanDuration | The time it took to complete the last scan planning. | Gauge | none | Ms | Timer starts before the scan planning starts, update after the last scan planning operation finished |
scanDuration | Distributions of the time taken by the last few scan planning operations. | Histogram | none | Ms | Timer starts before the scan planning starts, update after each planning finished |
lastScannedManifests | Number of scanned manifests files in the last scan planning. | Gauge | none | Number | Scan planning |
lastScanSkippedManifests | Number of skipped manifests files in the last scan planning. | Gauge | none | Number | Scan planning |
lastScanResultTableFiles | Number of result table files in the last scan planning. | Gauge | none | Number | Scan planning |
lastScanGenerateSplits | Number of splits generated by the last scan planning. | Gauge | none | Number | Scan planning |
CompactionMetrics
| Code Block |
|---|
public class CompactionMetrics {
private final String LAST_TABLE_FILES_COMPACTED_BEFORE_METRIC = "lastTableFilesCompactedBefore";
...
private final Map<Integer, BucketMetricGroup> bucketMetricGroups = new HashMap<>();
private final Map<BinaryRow, PartitionMetricGroup> partitionMetricGroups = new HashMap<>();
private void registerTaggedCompactionMetrics() {
if(isNewBucket()) {
registerTaggedMetrics();
}
if(isNewPartition()) {
registerTaggedMetrics();
}
}
} |
...
Metric Name | Description | Type | Tagged | Unit | Update at |
lastCompactionDuration | The time it took to complete the last compaction. | Gauge | none | Ms | Timer starts before compaction, update after the last compaction finished |
compactionDuration | Distributions of the time taken by the last few compactions. | Histogram | none | Ms | Timer starts before compaction, update after each compaction finished |
lastTableFilesCompactedBefore | Number of deleted files in last compaction | Gauge | none | Number | After getting compaction result. |
lastTableFilesCompactedAfter | Number of added files in last compaction | Gauge | none | Number | After getting compaction result. |
lastChangelogFilesCompacted | Number of changelog files compacted in last compaction | Gauge | none | Number | After getting compaction result. |
lastSortedRunslastCompactOutputLevel | Number of total sorted runs of all levels at last compaction | Gauge | none | Number | Updating levels after getting compaction result. |
lastLevel0FileslastCompactFileSize | Number of files at level 0 at Sum of file size compacted in last compaction | Gauge | none | Number | Updating levels after getting compaction result. |
...