...
Histogram is a type of metric interface measure the statistical distribution of a set of values including the min, max, mean, standard deviation and quantilespercentile.
Code Block |
---|
/** The histogram allows to record values, get the current count of recorded values and create
* histogram statistics for the currently seen elements.*/
@Public
public interface Histogram extends Metric {
/**
* Update the histogram with the given value.
*
* @param value Value to update the histogram with
*/
void update(long value);
/**
* Get the count of seen elements.
*
* @return Count of seen elements
*/
long getCount();
/**
* Create statistics for the currently recorded elements.
*
* @return Statistics about the currently recorded elements
*/
HistogramStatistics getStatistics();
} |
Metrics
interface HistogramStatistics {
/**
* Return the number of values that the statistics computation is based on.
*/
int size();
/** Returns the mean value of the histogram observations. */
double mean();
/** Returns the standard deviation of the histogram distribution. */
double stdDev();
/** Returns the maximum value of the histogram observations. */
long max();
/** Returns the minimum value of the histogram observations. */
long min();
/**
* Returns the percentile value based on the histogram statistics.
*
* @param percentile percentile point in double. E.g., 0.75 means 75 percentile. It is up to the
* implementation to decide what valid percentile points are supported.
* @return Value for the given percentile
*/
long percentile(double percentile);
}
} |
Metrics
Class Metrics
is the core of metrics system, there are MetricRegistry
Class Metrics
is the core of metrics system, there are MetricRegistry
and MetricsReporter
container in it. When the Metrics
instance is initiating, the MetricRegistry
is instantiated and metrics reporters are started.
...
Metric Name | Description | Type | Unit | Update at | Update at | ||
lastCommitDuration | The time it took to complete the last commit. | Gauge | Ms | Timer starts before commit starting, update commit duration after the last commit finished. | |||
commitDuration | Distributions of the time taken by the last few commits. | Histogram | lastCommitDuration | The time it took to complete the last commit. | Gauge | Ms | Timer starts before commit starting, update commit duration after the last each commit finished. |
lastCommitAttempts | The number of attempts the last commit made. | Counter | Number | Increment by 1 when trying to commit once, clear the counter after the last commit finished. | |||
numTableFilesAdded | Number of added table files in last commit, including newly created data files and compacted after. | Gauge | Number | Collecting changes from committables | |||
numTableFilesDeleted | Number of deleted table files in last commit, which comes from compacted before. | Gauge | Number | Collecting changes from committables | |||
numTableFilesAppended | Number of appended table files in last commit, which means the newly created data files. | Gauge | Number | Collecting changes from committables | |||
numTableFilesCompated | Number of compacted table files in last commit, including compacted before and after. | Gauge | Number | Collecting changes from committables | |||
numChangelogFilesAppended | Number of appended changelog files in last commit | Gauge | Number | Collecting changes from committables | |||
numChangelogFileCompacted | Number of compacted changelog files in last commit | Gauge | Number | Collecting changes from committables | |||
totalTablesFiles | Number of total data files currently maintained on storage. | Counter | Number | Collecting changes from committables | |||
totalChangelogFiles | Number of total changelog files currently maintained on storage. | Counter | Number | Collecting changes from committables | |||
numGeneratedSnapshots | Number of snapshot files generated in last commit, maybe 1 snapshot or 2 snapshots. | Gauge | Number | After collecting changes from committables | |||
totalSnapshots | Number of currently retained total snapshots. | Counter | Number | When trying to commit, the counter will increment by the number of snapshots generated When expiring snapshots, the counter will decrement by the number of expiring snapshots. | |||
numTotalRecordsAppended | Total records count in last commit with APPEND commit kind | Gauge | Number | Preparing snapshot file with APPEND commit kind | |||
numDeltaRecordsAppended | Delta records count in last commit with APPEND commit kind | Gauge | Number | Preparing snapshot file with APPEND commit kind | |||
numChangelogRecordsAppended | Changelog records count in last commit with APPEND commit kind | Gauge | Number | Preparing snapshot file with APPEND commit kind | |||
numTotalRecordsCompated | Total records count in last commit with COMPACT commit kind | Gauge | Number | Preparing snapshot file with COMPACT commit kind | |||
numDeltaRecordsCompated | Delta records count in last commit with COMPACT commit kind | Gauge | Number | Preparing snapshot file with COMPACT commit kind | |||
numChangelogRecordsCompated | Changelog records count in last commit with COMPACT commit kind | Gauge | Number | Preparing snapshot file with COMPACT commit kind | |||
numPartitionsWritten | Number of partitions written in last commit | Gauge | Number | After collecting changes from committables | |||
numBucketsWritten | Number of buckets written in last commit | Gauge | Number | After collecting changes from committables |
...
Metric Name | Description | Type | Unit | Update at | Unit | Update at | |
lastScanDuration | The time it took to complete the last scan planning. | Gauge | Ms | Timer starts before the scan planning starts, update after the last scan planning operation finished | |||
scanDuration | Distributions of the time taken by the last few scan planning operations. | Histogram | lastScanDuration | The time it took to complete the last scan planning. | Gauge | Ms | Timer starts before the scan planning starts, update after each planning finished |
numTotalManifests | Number of scanned manifests files in the last scan planning. | Gauge | Number | Scan planning | |||
numSkippedManifests | Number of skipped manifests files in the last scan planning. | Gauge | Number | Scan planning | |||
numResultTableFiles | Number of result table files in the last scan planning. | Gauge | Number | Scan planning | |||
numGenerateSplits | Number of splits generated by the last scan planning. | Gauge | Number | Scan planning |
...
CompactionMetrics
list includes duration, and counter of files, sorted runs etc.
Metric Name | Description | Type | Unit | Update at | |
lastCompactionDuration | The time it took to complete the last compaction. | Gauge | Ms | Timer starts before compaction, update after the last compaction finished | |
compactionDuration | Distributions of the time taken by the last few compactions. | Histogram | Ms | Timer starts before compaction, update after | it each compaction finished |
numFilesCompactedBefore | Number of deleted files in last compaction | Gauge | Number | After getting compaction result. | |
numFilesCompactedAfter | Number of added files in last compaction | Gauge | Number | After getting compaction result. | |
numChangelogFilesCompacted | Number of changelog files compacted in last compaction | Gauge | Number | After getting compaction result. | |
numSortedRuns | Current number of total sorted runs of all levels | Gauge | Number | Updating levels after getting compaction result. | |
numLevel0Files | Current number of files at level 0 | Gauge | Number | Updating levels after getting compaction result. |
Flink connector metrics
Implement important source metrics in FLIP-33.
...