Motivation

It is useful for user to work with own metrics, not only with provided by Ignite. Current public metrics API doesn't expose any method to add or delete additional metrics.

The most important reason to provide custom is probably the convenience of collecting of desired metrics using one platform, the same client, through the same API. This feature can simplify user application.

Examples of custom metric usages

• Some metrics may not be provided yet and user registers own.
• Metrics for load balancing.
• Various metrics of business processes or related to user application.
• Administration metrics.
• Security-related metrics.

Examples of databases with custom metrics

Oracle DB , Postgres , Oracle Coherence , MS SQL Server , IBM DB2

Implementation proposal

General approach

1. Custom metrics are experimental.
2. Custom Metrics base on the New Metric System.
3. Allow to register essential ready-to-use metrics like LongMetric which can be exported by a providing metric exporter.
4. User can add own metrics into different registries.
5. Custom metrics are separated from the internals by registry name prefix "custom." (lower cased). 
6. Names of custom metrics and thier registries can't contain any spaces.
7. Management of custom metrics might require a permission.
8. If we expose only few simple metrics, we should document examples of effecient metrics implementations like LongAdder.

API

   1. APIs description

An API to expose internals read-noly metrics was already proposed. Might be joined with the custom metrics.

To give an user the ability to register additional metrics, we could either:

• Refactor a bit current metrics. Create interfaces for writable metric registry, metric manager. Create interfaces for some essential metrics (int, double, long, longAdder, boolean, object) and the gauge metrics (IntGauge, LongGauge, etc.).  are below.
• Provide tiny enough facade with consumers and suppliers for int, long, double value metrics.

These two approaches are shown below.

2. Obtaining Custom Metrics

package org.apache.ignite;

public interface Ignite {

   public IgniteMetrics metrics();

}

3. 

4. Minimal Custom Metric interface

package org.apache.ignite;

/**
 * Manages custom metrics. Provides read-only internal metrics.
 *
 * Metrics are grouped into registries. Every metric has full name which is the conjunction of registry name and metric name. 
 * 
 * Full names and registry names of custom metrics are always prefixed with "custom.". For example, if provided registry name is "a.b.c.mname", 
 * it is automatically converted to "custom.a.b.c.mname". If the name is "custom.a.b.c.mname", it is used as is. 
 * Where "custom.a.b.c" is a registry name and "mname" is a metric name. 
 */

@IgniteExperimental
public interface IgniteMetrics extends Iterable<ReadOnlyMetricRegistry> {
    /**
     * Adds a long value custom metric.
     *
     * @return New or previously registered long value metric. {@code Null} if previous metric is not a {@link LongConsumer}.
     */
    @Nullable LongConsumer longMetric(String registryName, String metricName, @Nullable String description);

    /**
     * Adds a double value custom metric.
     *
     * @return New or previously registered long value metric. {@code Null} if previous metric is not a {@link DoubleConsumer}.
     */
    @Nullable DoubleConsumer doubleMetric(String registryName, String metricName, @Nullable String description);

    /**
     * Adds a int value custom metric.
     *
     * @return New or previously registered long value metric. {@code Null} if previous metric is not an {@link IntConsumer}.
     */
    @Nullable IntConsumer booleanMetric(String registryName, String metricName, @Nullable String description);

    /**
     * Adds a long value custom metric.
     *
     * @return {@code True} if {@code supplier} has been registered as a new int metric. {@code False}, if a previous 
     * value supplier already exists.
     */
    boolean longMetric(String registryName, String metricName, LongSupplier supplier, @Nullable String description);

    /**
     * Adds a double value custom metric.
     *
     * @return {@code True} if {@code supplier} has been registered as a new int metric. {@code False}, if a previous 
     * value supplier already exists.
     */
    boolean doubleMetric(String registryName, String metricName, DoubleSupplier supplier, @Nullable String description);

    /**
     * Adds a int value custom metric.
     *
     * @return {@code True} if {@code supplier} has been registered as a new int metric. {@code False}, if a previous 
     * value supplier already exists.
     */
    boolean intMetric(String registryName, String metricName, BooleanSupplier supplier, @Nullable String description);
    
    /** Removes certain custom metric. */
    void removeCustomMetric(String registryName, String metricName);

    /** Removes entire custom metric registry. */
    void removeCustomRegistry(String registryName);

    /** Provides custom or internal read-only metric registry. */
    @Nullable ReadOnlyMetricRegistry findRegistry(String registryName);
}

Risks and Assumptions

• Custom metrics are just another way to hit performance and consume additional resources.
• There could be race conditions on metric registrations. Metrics with incompatible metric type can be concurrently registered with the same names.
• We do not provide read-only metric wrappers for the internal metrics. User can cast read-only metric to writeable implementation or interface and change value of an iternal metric. But we already have public Metric#reset(), IgniteSpiContext#removeMetricRegistry(). And current ReadOnlyMetricRegistry doesn't return write-protected metric implementation.
• We do not expose API of all internal metrics. At least with the first tickets.
• Custom metrics aren't stored and require re-registration after node restart. At least with the first tickets.

Further Steps

We already have implementations of more complex and useful metrics. We could also store custom metrics. Thus, the development stages might be:

1. Extending the initial API with more complex metrics like Histogram or HitRate.
2. Introduce a permission of custom metric management.
3. Storing registered custom metrics.
4. Allowing to change settings of configurable custom metrics like histograms.

References

1. IEP-35 Monitoring & Profiling
2. New Metric System
3. Ticket of a public metric API
4. IEP-116 : Ignite 3 metric

Discussion Links

Tickets