You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Next »

Discussion thread
Vote thread
ISSUEhttps://github.com/apache/incubator-paimon/issues/742
Release


Motivation

Currently Paimon runs without monitoring and metrics out of box. Users need to know how their Paimon Table behaves like what is the commit duration, how many files each commit generated, the performance of scan query and so on. So we need to introduce more metrics for Paimon.  

Public Interfaces

Metric

Metric is the base measurable metric interface, which indicate a class is a metric.

public interface Metric {}

Gauge

Gauge is a type of metric interface provides a value of any type at a point in time.

/** Gauge calculates a specific value at a point in time. */
@Public
public interface Gauge<T> {

    /**
     * Calculates and returns the measured value.
     *
     * @return calculated value
     */
    T getValue();
}

Counter

Counter is a type of metric interface which is used to count values by incrementing and decrementing.

/** A Counter is a metric measured by incrementing and decrementing. */
@Public
public interface Counter extends Metric {

    /** Increment the current count by 1. */
    void inc();

    /**
     * Increment the current count by the given value.
     *
     * @param n value to increment the current count by
     */
    void inc(long n);

    /** Decrement the current count by 1. */
    void dec();

    /**
     * Decrement the current count by the given value.
     *
     * @param n value to decrement the current count by
     */
    void dec(long n);

    /**
     * Returns the current count.
     *
     * @return current count
     */
    long getCount();
}

Metrics

Class Metrics  is the core of metrics system, there are `MetricRegistry` and `MetricsReporter` container in it. When the Metrics  instance is initiating, the MetricRegistry  is instantiated and metrics reporters are started. 

Metrics reporters are configurable, users can use custom reporters, Paimon will provide a default metrics reporter of JMX metrics reporter. 

public class Metrics {

    /** The registry that holds the metrics. */
 	private final MetricRegistry registry;

    /** The metrics reporters container. */
  	private final List<MetricsReporter> reporters;

	/** Register metrics to MetricRegistry. 
 		@param name The name of metric.
		@param metric The metric to register.	
	*/
	public void registerMetrics(String name, Metric metric) {}
}

MetricRegistry

MetricRegistry is a class responsible for metrics registering, there is a metrics container in it. It provides register method for each type of measurable metric, registering metrics will put metrics to the metrics container.

public class MetricRegistry {

	/** Map of gauge metrics. */
    private final Map<String, Gauge<?>> gauges = new HashMap<>();

    /** Map of counter metrics. */
    private final Map<String, Counter> counters = new HashMap<>();

	/** Register gauge metric. */
 	public void gauge(String name, Gauge gauge) {}

	/** Register counter metric. */
	public void counter(String name, Counter counter) {}
}

MetricsReporter

MetricsReporter  is used to report metrics to external backend, Paimon will implement an out of box reporter as JMX `MetricsReporter`.

public interface MetricsReporter {
	/** Configure reporter after instantiating it.*/
     void open();

    /** Closes this reporter. */
    void close();

	/** Report the current measurements. This method is called periodically by the Metrics. */
	void report();
}

CommitMetrics / ScanMetrics / CompactionMetrics

public class CommitMetrics {
	private Metrics metrics;
	private final String COMMIT_DURATION_METRIC = "commitDuration";
	...
	private void registerCommitMetrics(Metrics metrics) {
		metrics.gauge(COMMIT_DURATION_METRIC, new CommitDurationGauge());
	}
	...
}

Proposed Changes

Architecture

Metrics Register


Metrics value update

Metrics report

Metrics list

Commit Metrics

Scan Metrics

Compaction Metrics

Compatibility, Deprecation, and Migration Plan

There are no changes to the public interface and no impact to existing users.

Test Plan

Will add unit tests and integration tests for metrics.

Rejected Alternatives

None

  • No labels