...

Currently Paimon runs without monitoring and metrics out of box. Users need to know how their Paimon Table behaves like what is the commit duration, how many files each commit generated, the performance of scan query and so on. So we need to introduce more metrics for Paimon.  

Public Interfaces

Measurable metric interfaces

Metric

Metric is the base measurable metric interface, which indicate a class is a metric.

public interface Metric {}

Gauge

Gauge is a type of metric interface provides a value of any type at a point in time.

/** Gauge calculates a specific value at a point in time. */
@Public
public interface Gauge<T> {

    /**
     * Calculates and returns the measured value.
     *
     * @return calculated value
     */
    T getValue();
}

Counter

Counter is a type of metric interface which is used to count values by incrementing and decrementing.

...

public interface MetricsReporter {
	/** Configure reporter after instantiating it.*/
     void open();

    /** Closes this reporter. */
    void close();

	/** Report the current measurements. This method is called periodically by the Metrics. */
	void report();
}

MetricsSet

CommitMetrics

...

Code Block
language java
public class CommitMetrics { private Metrics metrics; private final String COMMIT_DURATION_METRIC = "commitDuration"; ... private void registerCommitMetrics(Metrics metrics) { metrics.gauge(COMMIT_DURATION_METRIC, new CommitDurationTimer()); ... } ... }

ScanMetrics

public class ScanMetrics {
	private Metrics metrics;
	private final String SCAN_FILES_METRIC = "scanFiles";
	...
	private void registerScanMetrics(Metrics metrics) {
		metrics.counter(SCAN_FILES_METRIC, new ScanFilesCounter());
		...
	}
	...
}

CompactionMetrics

public class CompactionMetrics {
	private Metrics metrics;
	private final String COMPACTED_FILES_METRIC = "compactedFiles";
	...
	private void registerCompactionMetrics(Metrics metrics) {
		metrics.counter(COMPACTED_FILES_METRIC, new CommitDurationGaugeCompactedFilesCounter());
		...
	}
	...
}

Proposed Changes

Architecture

Metrics Register

Metrics value update

...

Metrics Registering

Take CommitMetrics as example, the CommitMetrics will be instantiated by TableCommitImpl, then commit related metrics will be registered by MetricRegistry in singleton Metrics. 

The Metrics has properties of MetricRegistry and MetricsReporters set. MetricRegistry maintains metrics map containers. Metrics registering is a process of putting metrics instances into the metric (gauge, counter) map container. 

Update metrics value

The CommitMetrics values will be updated around commit() operation, for example the commit starting time will be recorded before commit operation, CommitDuration value will be recorded after commit completing.

CompactionMetrics values will be updated around compaction operation, and ScanMetrics will be recorded through the scan planning process.

Report metrics to external backend

Each reporter instance has an timer task fetch metrics from the metrics containers periodically and report them out to the external backends. Paimon will has a default reporter backend with JMX, users can define their own MetricsReporter by implement `MetricsReporter` interface.

Metrics list

Commit Metrics

Scan Metrics

Compaction Metrics

Compatibility, Deprecation, and Migration Plan

...