THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Class Metrics is the core of metrics system, there are `MetricRegistry` and `MetricsReporter` MetricRegistry and MetricsReporter container in it. When the Metrics instance is initiating, the MetricRegistry is instantiated and metrics reporters are started.
...
| Code Block | ||
|---|---|---|
| ||
public class CommitMetrics {
private Metrics metrics;
private final String COMMIT_DURATION_METRIC = "commitDurationlastCommitDuration";
...
private void registerCommitMetrics(Metrics metrics) {
metrics.gauge(COMMIT_DURATION_METRIC, new CommitDurationTimer());
...
}
...
} |
...
Metric Name | Description | Type | Unit | Update at |
lastCommitDuration | The time it took to complete the last commit. | Gauge | Ms | Timer starts before commit starting, update commit duration after the last commit finished. |
lastCommitAttempts | The number of attempts the last commit made. | Counter | Number | Increment by 1 when trying to commit once, clear the counter after the last commit finished. |
numTableFilesAdded | Number of added table files in last commit, including newly created data files and compacted after. | Gauge | Number | Collecting changes from committables |
numTableFilesDeleted | Number of deleted table files in last commit, which comes from compacted before. | Gauge | Number | Collecting changes from committables |
numTableFilesAppended | Number of appended table files in last commit, which means the newly created data files. | Gauge | Number | Collecting changes from committables |
numTableFilesCompated | Number of compacted table files in last commit, including compacted before and after. | Gauge | Number | Collecting changes from committables |
numChangelogFilesAppended | Number of appended changelog files in last commit | Gauge | Number | Collecting changes from committables |
numChangelogFileCompacted | Number of compacted changelog files in last commit | Gauge | Number | Collecting changes from committables |
totalTablesFiles | Number of total data files currently maintained on storage. | Counter | Number | Collecting changes from committables |
totalChangelogFiles | Number of total changelog files currently maintained on storage. | Counter | Number | Collecting changes from committables |
numGeneratedSnapshots | Number of snapshot files generated in last commit, maybe 1 snapshot or 2 snapshots. | Gauge | NumberTrying to commit | After collecting changes from committables |
totalSnapshots | Number of currently retained total snapshots. | Counter | Number | When trying to commit, the counter will increment by the number of snapshots generated When expiring snapshots, the counter will decrement by the number of expiring snapshots. |
numTotalRecordsAppended | Total records count in last commit with APPEND commit kind | Gauge | Number | Preparing snapshot file with APPEND commit kind |
numDeltaRecordsAppended | Delta records count in last commit with APPEND commit kind | Gauge | Number | Preparing snapshot file with APPEND commit kind |
numChangelogRecordsAppended | Changelog records count in last commit with APPEND commit kind | Gauge | Number | Preparing snapshot file with APPEND commit kind |
numTotalRecordsCompated | Total records count in last commit with COMPACT commit kind | Gauge | Number | Preparing snapshot file with COMPACT commit kind |
numDeltaRecordsCompated | Delta records count in last commit with COMPACT commit kind | Gauge | Number | Preparing snapshot file with COMPACT commit kind |
numChangelogRecordsCompated | Changelog records count in last commit with COMPACT commit kind | Gauge | Number | Preparing snapshot file with COMPACT commit kind |
numPartitionsWritten | Number of partitions written in last commit | Gauge | NumberTrying to commit | After collecting changes from committables |
numBucketsWritten | Number of buckets written in last commit | Gauge | NumberTrying to commit | After collecting changes from committables |
ScanMetrics
| Code Block |
|---|
public class ScanMetrics {
private Metrics metrics;
private final String SCAN_FILES_METRIC = "scanFileslastScanDuration";
...
private void registerScanMetrics(Metrics metrics) {
metrics.counter(SCAN_FILES_METRIC, new ScanFilesCounter());
...
}
...
} |
...
Metric Name | Description | Type | Unit | Update at |
scanDurationlastScanDuration | The time it took to complete the scanninglast scan planning. | Gauge | Ms | Timer starts before planning starts, update after planning finished |
numTotalManifests | Number of scanned manifests files in the last scan planning. | Gauge | NumberPlanning | Scan planning |
numSkippedManifests | Number of skipped manifests files in the last scan planning. | Gauge | NumberPlanning | Scan planning |
numResultTableFiles | Number of result table files in the last scan planning. | Gauge | Number | Scan planning |
numGenerateSplits | Number of splits generated by the last scan planning. | Gauge | NumberPlanning | Scan planning |
CompactionMetrics
| Code Block |
|---|
public class CompactionMetrics {
private Metrics metrics;
private final String COMPACTED_FILES_METRIC = "compactedFileslastCompactionDuration";
...
private void registerCompactionMetrics(Metrics metrics) {
metrics.counter(COMPACTED_FILES_METRIC, new CompactedFilesCounter());
...
}
...
} |
...
Metric Name | Description | Type | Unit | Update at |
compactionDurationlastCompactionDuration | The time it took to complete the last compaction. | Gauge | Ms | Timer starts before the last compaction, update after compaction it finished |
numFilesCompactedBefore | Number of deleted files in last compaction | Gauge | Number | Triggering After getting compaction result. |
numFilesCompactedAfter | Number of added files in last compaction | Gauge | Number | Triggering After getting compaction result. |
numChangelogFilesCompacted | Number of changelog files compacted in last compaction | Gauge | Number | Triggering After getting compaction result. |
numSortedRuns | Number Current number of total sorted runs of all levels | CounterGauge | Number | Triggering compactionUpdating levels after getting compaction result. |
numLevel0Files | Number Current number of files at level 0 | CounterGauge | Number | Triggering compactionUpdating levels after getting compaction result. |
Flink connector metrics
Implement important source metrics in FLIP-33.
...