Motivation

It is useful for user to work with own metrics, not only with provided by Ignite. Current public metrics API doesn't expose any method to add or delete additional metrics.

The most important reason to provide custom is probably the convenience of collecting of desired metrics using one platform, the same client, through the same API. This feature can simplify user application.

Famous databases with custom metrics: Oracle DB , Postgres , Oracle Coherence , MS SQL Server , IBM DB2

Examples of custom metric usages:

• Some metrics may not be provided yet and user registers own.
• Metrics for load balancing.
• Various metrics of business processes or related to user application.
• Administration metrics.
• Security-related metrics.

Implementation proposal

1. Custom metrics are experimental.
2. Custom Metrics base on the New Metric System.
3. Allow to register essential ready-to-use metrics like LongMetric which can be exported by a providing metric exporter.
4. User can add own metrics into different registries.
5. Custom metrics are separated from the internals by registry name prefix "custom." (lower cased). 
6. Names of custom metrics and their registries can't contain any spaces.
7. Management of custom metrics might require a permission.
8. If we expose only few simple metrics, we should document examples of effecient metrics implementations like LongAdder.

Risks and Assumptions

• Custom metrics are just another way to hit performance and consume additional resources.
• There could be race conditions on metric registrations. Metrics with incompatible types can be concurrently registered with the same names.
• We do not provide read-only metric wrappers for the internal metrics. User can cast read-only metric to writeable implementation or interface and change value even of an iternal metric. But we already have public Metric#reset(), IgniteSpiContext#removeMetricRegistry(). And current ReadOnlyMetricRegistry doesn't return write-protected metric implementation.
• We do not expose API of all internal metrics. At least with the first tickets.
• Custom metrics aren't stored and require re-registration after node restart. At least with the first tickets.

API

1. APIs description

An API to expose internals read-only internal metrics was already proposed. Might be joined with the custom metrics.

To give an user the ability to register additional metrics, we could either:

• Refactor a bit our current metrics. Create interfaces for writable metric registry, metric manager. Create interfaces for metrics like int, double, long, longAdder, boolean, object and the gauge metrics (IntGauge, LongGauge, etc.).
• Provide tiny-enough facade with consumers and suppliers for int, long, double value metrics.

These two approaches are shown below.

2. Obtaining Custom Metrics

package org.apache.ignite;

public interface Ignite {

    IgniteMetrics metrics();

}

3.  Interfaces for existing metrics

Instead of #4, we could create interfaces for the existing metrics implementations.

3.1 IgniteMetric

package org.apache.ignite.metric;

/**
* Allows to manage custom metrics and to obtain read-only internal metrics.
*
* Note: Names of custom metric registries are required to start with 'custom.' (lower case) and may have additional
* dot-separated qualifiers. The prefix 'custon' is automatically added if missed. For example, if provided custom registry name
* is "a.b.c.mname", it is automatically extended to "custom.a.b.c.mname".
*
* Any custom name or dot-separated name part can't have any spaces and must not be empty. Spaces are removed.
*
* Examples of custom metric registry names: "custom", "custom.admin", "custom.admin.sessions", "custom.processes", etc.
*/
@IgniteExperimental
public interface IgniteMetrics extends Iterable<ReadOnlyMetricRegistry> {
    /**
    * Gets or creates custom metric registry named "custom." + {@code registryName}.
    *
    * @return {@link IgniteMetricRegistry} registry.
    */
    IgniteMetricRegistry customRegistry(String registryName);

    /**
    * Gets or creates custom metric registry named "custom.".
    *
    * @return {@link IgniteMetricRegistry} registry.
   */
    default IgniteMetricRegistry customRegistry() {
        return customRegistry(null);
    }

    /**
    * Gets metric registry including the Ignite's internal registries.
    *
    * @return Certain read-only metric registry.
    */
    @Nullable ReadOnlyMetricRegistry findRegistry(String registryName);

    /** */
    void removeCustomRegistry(String registryName);

    /** Removes custom metric registry with name '.custom'./
    default void removeCustomRegistry() {
        removeCustomRegistry(null);
    }
}

3.2 IgniteMetricRegistry

Probably should be named "MetricRegistry" with the renaming of internal "MetricRegistry" to "MetricRegistryImpl".  

package org.apache.ignite.metric;

/**
* Metric registry. Allows to get, add or remove metrics.
*
* @see IgniteMetrics
* @see ReadOnlyMetricRegistry
*/
@IgniteExperimental
public interface IgniteMetricRegistry extends ReadOnlyMetricRegistry {
    /** @return {@code True} if new metric was added. {@code False} is other metric already exists with the same name. */
    boolean gauge(String name, IntSupplier supplier, @Nullable String desc);

    /** @return {@code True} if new metric was added. {@code False} is other metric already exists with the same name. */
    boolean gauge(String name, LongSupplier supplier, @Nullable String desc);

    /** @return {@code True} if new metric was added. {@code False} is other metric already exists with the same name. */
    boolean gauge(String name, DoubleSupplier supplier, @Nullable String desc);

    /** @return {@code True} if new metric was added. {@code False} is other metric already exists with the same name. */
    <T> boolean gauge(String name, Supplier<T> supplier, Class<T> type, @Nullable String desc);

    /** @return {@code True} if new metric was added. {@code False} is other metric already exists with the same name. */
    boolean gauge(String name, BooleanSupplier supplier, @Nullable String desc);

    /**
    * @return New {@link IntValueMetric} or previous one with the same name. {@code Null} if previous metric exists and
    * is not a {@link IntValueMetric}.
    */
    @Nullable IntValueMetric intMetric(String name, @Nullable String desc);

    /**
    * @return New {@link LongValueMetric} or previous one with the same name. {@code Null} if previous metric exists and
    * is not a {@link LongValueMetric}.
    */
    @Nullable LongValueMetric longMetric(String name, @Nullable String desc);

    /**
    * @return New {@link LongValueMetric} or previous one with the same name. {@code Null} if previous metric exists and
    * is not a {@link LongValueMetric}.
    */
    @Nullable LongSumMetric longAdderMetric(String name, @Nullable String desc);

    /**
    * @return New {@link DoubleValueMetric} or previous one with the same name. {@code Null} if previous metric exists and
    * is not a {@link DoubleValueMetric}.
    */
    @Nullable DoubleValueMetric doubleMetric(String name, @Nullable String desc);

    /**
    * @return New {@link CustomMetric} or previous one with the same name. {@code Null} if previous metric exists and
    * is not a {@link CustomMetric}.
    */
    @Nullable <T> AnyValueMetric<T> objectMetric(String name, Class<T> type, @Nullable String desc);

    /** Removes metrics with the {@code name}.*/
    void remove(String name);

    /** Resets all metrics of this metric registry. */
    void reset();
}

3.3 Updatable metric interfaces list

To the package "org.apache.ignite.metric" we add:

• BooleanValueMetric
• IntValueMetric
• LongValueMetric
• LongSumMetric (a long adder)
• DoubleValueMetric
• AnyValueMetric<T> (an object metric)

Names like "LongMetric" or "ObjectMetric" we already have in the package "org.apache.ignite.spi.metric".

3.4 Examples of updatable metrics

package org.apache.ignite.metric;

/** Updatable object value metric. */
@IgniteExperimental
public interface AnyValueMetric<T> extends ObjectMetric<T> {
    /** Sets object metric value./
    void value(T value);
}

/** Updatable simple double value metric. */
@IgniteExperimental
public interface DoubleValueMetric extends DoubleMetric {
    /** Raises metric value. */
    void add(double value);

    /** Sets double metric value. */
    void value(double value);
}

/** Updatable long metric which is efficient with adding values. Calculates sum in {@link LongMetric#value()}. */
@IgniteExperimental
public interface LongSumMetric extends LongMetric {
    /** Raises metric value. */
    void add(long value);

    /** Increments metric value with 1L. */
    void increment();

    /** Decrements metric value with -1L. */
    void decrement();
}

/** Updatable simple long value metric. */
@IgniteExperimental
public interface LongValueMetric extends LongSumMetric {
    /** Sets long metric value. */
    void value(long value);
}

4. Minimal custom interface

Instead of #3, we could bring only a minimal metric management interface.

package org.apache.ignite;

/**
* Allows to manage custom metrics and to obtain read-only internal metrics.
* <p>
* Metrics are grouped into registries (groups). Every metric has full name which is the conjunction of registry name
* and the metric short name. Within a registry metric has only its own short name.
* <p>
* Note: Names of custom metric registries are required to start with 'custom.' (lower case) and may have additional
* dot-separated qualifiers. The prefix is automatically added if missed. For example, if provided custom registry name
* is "a.b.c.mname", it is automatically extended to "custom.a.b.c.mname".
* <p>
* Any custom name or dot-separated name part cannot have spaces and must not be empty. Spaces are removed.
* <p>
* Examples of custom metric registry names: "custom", "custom.admin", "custom.admin.sessions", "custom.processes", etc.
*/
@IgniteExperimental
public interface IgniteMetrics extends Iterable<ReadOnlyMetricRegistry> {

    /**
    * Adds a long value custom metric.
    *
    * @return New or previously registered long value metric. {@code Null} if previous metric is not a {@link LongConsumer}.
    */
    @Nullable LongConsumer longMetric(String registryName, String metricName, @Nullable String description);

    /**
    * Adds a double value custom metric.
    *
    * @return New or previously registered long value metric. {@code Null} if previous metric is not a {@link DoubleConsumer}.
    */
    @Nullable DoubleConsumer doubleMetric(String registryName, String metricName, @Nullable String description);

    /**
    * Adds a int value custom metric.
    *
    * @return New or previously registered long value metric. {@code Null} if previous metric is not an {@link IntConsumer}.
    */
    @Nullable IntConsumer booleanMetric(String registryName, String metricName, @Nullable String description);

    /**
    * Adds a long value custom metric.
    *
    * @return {@code True} if {@code supplier} has been registered as a new int metric. {@code False}, if a previous
    * value supplier already exists.
    */
    boolean longMetric(String registryName, String metricName, LongSupplier supplier, @Nullable String description);

    /**
    * Adds a double value custom metric.
    *
    * @return {@code True} if {@code supplier} has been registered as a new int metric. {@code False}, if a previous
    * value supplier already exists.
    */
    boolean doubleMetric(String registryName, String metricName, DoubleSupplier supplier, @Nullable String description);

    /**
    * Adds a int value custom metric.
    *
    * @return {@code True} if {@code supplier} has been registered as a new int metric. {@code False}, if a previous
    * value supplier already exists.
    */
    boolean intMetric(String registryName, String metricName, BooleanSupplier supplier, @Nullable String description);

    /** Removes certain custom metric. */
    void removeCustomMetric(String registryName, String metricName);

    /** Removes entire custom metric registry. */
    void removeCustomRegistry(String registryName);

    /** Provides custom or internal read-only metric registry. */
    @Nullable ReadOnlyMetricRegistry findRegistry(String registryName);
}

Code examples

Further Steps

We already have implementations of more complex and useful metrics. We could also store custom metrics. Thus, the development stages might be:

1. Extending the initial API with more complex metrics like Histogram or HitRate.
2. Introduce a permission of custom metric management.
3. Storing registered custom metrics.
4. Allowing to change settings of configurable custom metrics like histograms.

References

1. IEP-35 Monitoring & Profiling
2. New Metric System
3. Ticket of a public metric API
4. IEP-116 : Ignite 3 metric

Discussion Links

Tickets