THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
We introduce CommitMetrics , CompactionMetrics , ScanMetrics as metrics set to measure the stats of Paimon table committing, compaction and scanning.
Common metrics
CommitMetrics
| Code Block | ||
|---|---|---|
| ||
public class CommitMetrics {
private Metrics metrics;
private final String COMMIT_DURATION_METRIC = "commitDuration";
...
private void registerCommitMetrics(Metrics metrics) {
metrics.gauge(COMMIT_DURATION_METRIC, new CommitDurationTimer());
...
}
...
} |
CommitMetrics list includes commit duration, counter of files / records etc.
Metric Name | Description | Type | Unit | Update at |
commitDuration | Commit | Gauge | Ms | Timer starts before commit starting, update commit duration after commit finished |
numTableFilesAdded | Number of added table files in this commit | Counter | Number | Collecting changes from committables |
numTableFilesDeleted | Number of deleted table files in this commit | Counter | Number | Collecting changes from committables |
numTableFilesAppended | Number of appended table files in this commit | Counter | Number | Collecting changes from committables |
numTableFilesCompated | Number of compacted table files in this commit | Counter | Number | Collecting changes from committables |
numChangelogFilesAppended | Number of appended changelog files in this commit | Counter | Number | Collecting changes from committables |
numChangelogFileCompacted | Number of compacted changelog files in this commit | Counter | Number | Collecting changes from committables |
numSnapshots | Number of snapshot files generated in this commit | Counter | Number | Trying to commit |
numTotalRecords | Total records count in this commit | Counter | Number | Preparing snapshot file |
numDeltaRecords | Delta records count in this commit | Counter | Number | Preparing snapshot file |
numChangelogRecords | Changelog records count in this commit | Counter | Number | Preparing snapshot file |
numPartitionsWritten | Number of partitions written in this commit | Counter | Number | Trying to commit |
numBucketsWritten | Number of buckets written in this commit | Counter | Number | Trying to commit |
ScanMetrics
| Code Block |
|---|
public class ScanMetrics {
private Metrics metrics;
private final String SCAN_FILES_METRIC = "scanFiles";
...
private void registerScanMetrics(Metrics metrics) {
metrics.counter(SCAN_FILES_METRIC, new ScanFilesCounter());
...
}
...
} |
ScanMetrics list includes duration, data files and manifest files counter.
Metric Name | Description | Type | Unit | Update at |
scanDuration | Scan Duration | Gauge | Ms | Timer starts before planning starts, update after planning finished |
numTotalManifests | Number of scanned manifests files | Counter | Number | Planning |
numSkippedManifests | Number of skipped manifests files | Counter | Number | Planning |
numResultTableFiles | Number of result table files | Counter | Number | Planning |
CompactionMetrics
| Code Block |
|---|
public class CompactionMetrics {
private Metrics metrics;
private final String COMPACTED_FILES_METRIC = "compactedFiles";
...
private void registerCompactionMetrics(Metrics metrics) {
metrics.counter(COMPACTED_FILES_METRIC, new CompactedFilesCounter());
...
}
...
} |
Commit Metrics
Scan Metrics
...
CompactionMetrics list includes duration, and counter of files, sorted runs etc.
Metric Name | Description | Type | Unit | Update at |
compactionDuration | Compaction duration | Gauge | Ms | Timer starts before compaction, update after compaction finished |
numFilesCompactedBefore | Number of deleted files in compaction | Counter | Number | Triggering compaction |
numFilesCompactedAfter | Number of added files in compaction | Counter | Number | Triggering compaction |
numChangelogFilesCompacted | Number of changelog files compacted | Counter | Number | Triggering compaction |
numSortedRuns | Number of sorted runs | Counter | Number | Triggering compaction |
numLevel0Files | Number of files at level 0 | Counter | Number | Triggering compaction |
Flink connector metrics
Implement important source metrics in FLIP-33.
Metric name | Description | Type | Unit | Update at |
numBytesIn | The total number of input bytes since the source started. | Counter | Number | FileStoreSourceSplitReader fetch finished. |
pendingRecords | The number of records that have not been fetched by the source. | Gauge | Number |
Compatibility, Deprecation, and Migration Plan
There are no changes to the public interface and no impact to existing users.
Test Plan
Will add unit tests and integration tests for metrics.
Rejected Alternatives
...