...

Motivation

It is useful for user to work with own metrics, not only with provided by Ignite. Current The current public metrics API doesn't expose any method to add or delete additional metrics.

The most important reason to provide custom is probably the convenience of collecting of desired metrics using one platform, the same client, through the same API. This feature can simplify user application.

Famous databases with custom metrics: Oracle DB , Postgres , Oracle Coherence , MS SQL Server , IBM DB2

Examples of custom metric usages:

• Some metrics may not be provided yet and user registers own.
• Metrics for load balancing.
• Custom conflict resolver successes/fails.
• Various metrics of business processes or related to user application.
• Administration metrics.
• Security-related metrics.

Examples of databases with custom metrics

Oracle DB , Postgres , Oracle Coherence , MS SQL Server , IBM DB2

Implementation proposal

General approach

Implementation proposal

1. Custom Metrics based on the New Metric System.
2. Custom metrics are experimental.
3. Custom Metrics base on the New Metric System.
4. Allow to register essential ready-to-use metrics like LongMetric which can be exported by a providing metric exporter.
5. User can add own metrics into different registries.
6. Custom metrics are separated from the internals by registry name prefix "custom." (lower cased). 
7. Names of custom metrics and their registries can't contain any spaces.
8. Management of custom metrics might require a permission.
9. If we expose only few simple metrics, we should document examples of effecient metrics implementations like LongAdder.

Risks and Assumptions

Risks and Assumptions

• Custom metrics affect performance and take Custom metrics might be just another way to hit performance and consume additional resources.
• There could be race conditions on metric registrations. Metrics with incompatible types can be concurrently registered with the same names.
• We do not provide read-only metric wrappers for the internal metrics. User can cast read-only metric to writeable implementation or interface and change value even of an iternal metric. But we already have public Metric#reset(), IgniteSpiContext#removeMetricRegistry(). And current ReadOnlyMetricRegistry doesn't return write-protected metric implementation.expose API of all internal metrics. At least with the first tickets.
• Custom metrics aren't stored and require re-registration after node restart. We do not expose API of all internal metrics. At least with the first tickets.
• Custom metrics aren't stored and require re-registration after node restart. At least with the first tickets.

API

1. APIs description

An API to expose internals read-only metrics was already proposed. Might be joined with the custom metrics.

To give an user the ability to register additional metrics, we could either:

• Refactor a bit current metrics. Create interfaces for writable metric registry, metric manager. Create interfaces for essential metrics like int, double, long, longAdder, boolean, object and the gauge metrics (IntGauge, LongGauge, etc.).
• Provide tiny enough facade with consumers and suppliers for int, long, double value metrics.

These two approaches are shown below.

2. Obtaining Custom Metrics

...

public interface Ignite {

   public IgniteMetrics metrics();

}

3.  Interfaces for existing metrics.

Instead of #4, we could create interfaces for the existing metrics implementations.

3.1 IgniteMetric

package org.apache.ignite.metric;

...

3.2 IgniteMetricRegistry

Probably should be named "MetricRegistry" with renaming of internal "MetricRegistry" to "MetricRegistryImpl".  

...

3.3 New writeable metric intefraces

To the package "org.apache.ignite.metric" we add:

• BooleanValueMetric
• IntValueMetric
• LongValueMetric
• LongSumMetric (a long adder)
• DoubleValueMetric
• AnyValueMetric<T> (an object metric)

Names like "LongMetric" or "ObjectMetric" we already have in the package "org.apache.ignite.spi.metric".

3.4 Examples of new  writeable metric intefraces

...

API

The APIs are supposed to be experimental.

Interfaces for existing metrics

package org.apache.ignite.metric;

public interface IgniteMetrics extends Iterable<ReadOnlyMetricRegistry> {
    MetricRegistry customRegistry(String registryName);
    @Nullable ReadOnlyMetricRegistry findRegistry(String registryName);
    void removeCustomRegistry(String registryName);
}

Names like "LongMetric" or "ObjectMetric" we already have in the package "org.apache.ignite.spi.metric".

package org.apache.ignite.metric;

public interface MetricRegistry extends ReadOnlyMetricRegistry {
    IntMetric register(String name, IntSupplier supplier, @Nullable String desc);
    LongMetric register(String name, LongSupplier supplier, @Nullable String desc);
    DoubleMetric register(String name, DoubleSupplier supplier, @Nullable String desc);
    <T> ObjectMetric<T> register(String name, Supplier<T> supplier, Class<T> type, @Nullable String desc);
    BooleanMetric register(String name, BooleanSupplier supplier, @Nullable String desc);
    IntValueMetric intMetric(String name, @Nullable String desc);
    LongValueMetric longMetric(String name, @Nullable String desc);
    LongSumMetric longAdderMetric(String name, @Nullable String desc);
    DoubleValueMetric doubleMetric(String name, @Nullable String desc);
    <T> ObjectValueMetric<T> objectMetric(String name, Class<T> type, @Nullable String desc);
    void remove(String name);
    void reset();
}

Examples of updatable metrics

package org.apache.ignite.metric;

public interface ObjectValueMetric<T> extends ObjectMetric<T> {
    void value(T value);
}

public interface DoubleValueMetric extends DoubleMetric {
    void add(double value);
    void value(double value);
}

public interface LongSumMetric extends LongMetric {
    void add(long value);
    void increment();
    void decrement();
}

public interface LongValueMetric extends LongSumMetric {
    void value(long value);
}

Code examples

public static final class TestCustomMetricsService implements TestService {
    @IgniteInstanceResource
    private Ignite ignite;

    @ServiceContextResource
    private ServiceContext ctx;

    private AtomicReference<UUID> remoteId;

    private final AtomicInteger metricValue = new AtomicInteger();

    @Override public void init() throws Exception {
        remoteId = new AtomicReference<>();
          
        // Registers metric "custom.service.svc.filteredInvocation"
        ignite.metrics().customRegistry(regName(ctx.name())).gauge("filteredInvocation", metricValue::get, "Counter of speceific service invocation.");

        // Registers metric "custom.service.svc.loaded"
        ignite.metrics().customRegistry(regName(ctx.name())).gauge("loaded", () -> metricValue.get() >= 100, "Load flag.");

        // Registers metric "custom.service.svc.remote.classId"
        ignite.metrics().customRegistry(regName(ctx.name())).gauge("remote.classId", () -> remoteId.get(), UUID.class, "Remote system class id.");
    }

    @Override public void cancel() {
        refresh();

        ignite.metrics().customRegistry(regName(ctx.name())).remove(COUNTER_METRIC_NAME);
    }

    @Override public void refresh() {
        metricValue.set(0);

        remoteId.set(null);
    }

    @Override public void invoke(int param) {
        if (ctx.isCancelled())
            return;

        remoteId.compareAndSet(null, UUID.randomUUID());

        // Updates metric sometimes.
        if (!ctx.isCancelled() && param % 10 == 0)
            metricValue.set(param / 10);
    }

    private static String regName(String svcName) {
        return "service." + svcName;
    }
}

private static final class TestCustomMetricsComputeTask extends ComputeTaskAdapter<Void, Long> {
    private static final class TestComputeJob extends ComputeJobAdapter {
        @IgniteInstanceResource
        private Ignite ignite;

        @Override public Long execute() throws IgniteException {
            long val = 0;

            // Some job limit.
            long limit = 300 + ThreadLocalRandom.current().nextLong(700);

            // Registers metric "custom.task.test.current"
            LongValueMetric metricCur = ignite.metrics().customRegistry("task.test").longMetric("current", null);

            // Registers metric "custom.task.test.total.sum"
            LongSumMetric metricTotal = ignite.metrics().customRegistry("task.test").longAdderMetric("total.sum", null);

            // Registers metric "custom.task.test.ticks"           
            LongSumMetric metricTicks = ignite.metrics().customRegistry("task.test").longAdderMetric("ticks", null);

            while (!isCancelled() && val < limit) {
                // Does some job.
                try {
                    U.sleep(ThreadLocalRandom.current().nextInt(50));
                }
                catch (IgniteInterruptedCheckedException ignored) {
                    //No op.
                }

                long increment = ThreadLocalRandom.current().nextLong(100);

                val += increment;

                metricTicks.increment()
            }

            metricCur.value(val);

            metricTotal.add(val);

            return isCancelled() ? 0 : val;
        }
    }

/** Updatable simple long value metric. */
@IgniteExperimental
public interface LongValueMetric extends LongSumMetric {
    /** Sets long metric value. */
    void value(long value);
}

4. Minimal Custom Metric interface

Instead of #3, we could bring only a minimal interface.

...

/**
 * Allows to manage custom metrics and to obtain read-only internal metrics.
 * <p>
 * Metrics are grouped into registries (groups). Every metric has full name which is the conjunction of registry name 
 * and the metric short name. Within a registry metric has only its own short name.
 * <p>
 * Note: Names of custom metric registries are required to start with 'custom.' (lower case) and may have additional
 * dot-separated qualifiers. The prefix is automatically added if missed. For example, if provided custom registry name
 * is "a.b.c.mname", it is automatically extended to "custom.a.b.c.mname".
 * <p>
 * Any custom name or dot-separated name part cannot have spaces and must not be empty. Spaces are removed.
 * <p>
 * Examples of custom metric registry names: "custom", "custom.admin", "custom.admin.sessions", "custom.processes", etc.
 *
 * @see ReadOnlyMetricRegistry
 * @see IgniteMetricRegistry
 */
@IgniteExperimental
public interface IgniteMetrics extends Iterable<ReadOnlyMetricRegistry> {

...

}

Further Steps

We already have implementations of more complex and useful metrics. We could also store custom metrics. Thus, the development stages might be:

1. An API to expose internals read-only internal metrics was already suggested. Might be joined with the custom metrics. By the methods like "findRegistry" we can also return read-only internal metrics.
2. Extending the initial API with more complex metrics like Histogram or HitRate.
3. Introduce a permission of for custom metric management.
4. Storing registered custom metrics.
5. Allowing to change settings of configurable custom metrics like histograms.

...

1. IEP-35 Monitoring & Profiling
2. New Metric System
3. Ticket of a public metric API
4. IEP-116 : Ignite 3 metric

Discussion Links

https://lists.apache.org/thread/9zs988jbw3jc6fzj6zhz0t5g8bl3lx5t

Tickets

...