...

Solr currently uses the Java Dropwizard 4 library framework for metric collection and event measurement. There have been a number of problems with Solr still using Dropwizard and motivation to move off of it:

...

The prometheus exporter is a Solr specific process for getting metrics from Dropwizard and /admin/metrics and requires constant maintenance with any new metrics or big changes that Solr introduces. Also running this external process can be costly to maintain.

Proposed changes

OpenTelemetry module

We aim to leverage the Solr already uses Open Telemetry module or the OTel Java agent, which initializes the Open Telemetry SDK for traces, to also push OTel metrics via OTLP. Metrics is currently disabled in the OTel module but we will enable this for metrics. We will collect metrics with the Open Telemetry API and this transition will completely remove Dropwizard.as a module for distributed tracing which integrates the SDK with auto-configuration. We aim to leverage the Open Telemetry SDK and move its dependencies into core. 

Open Telemetry framework has 2 different interfaces. The API and the SDK.

• The API is used to measure/capture metrics with instruments in Solr. Open Telemetry supports different types of instruments which we will try to migrate from

...

• Dropwizard's equivalent. See Open Telemetry Meter section from its documentation.

SolrMetricsContext needs to be completely refactored instead of being wrapped around Dropwizard to being wrapped around Open Telemetry. Some custom metric types that Solr uses from Dropwizard might not be possible or too difficult to port over to OTel such as MetricsMaps or complex Gauges.

The concept of registries goes away and metrics and meters will be retrieved from a GlobalOpenTelemetry object to measure events.

Filtering

We will maintain filtering capabilities on the metrics API to enable users to limit the amount of data and metrics being scraped from the endpoint. Enhancements may include filtering by tags, preserving the concept of registries/groups with Open Telemetry, or specific metric names possibly. Need to see what the Open Telemetry SDK supports for filtering in Solr but also filtering is possible to happen at an exporter level with OTEL collector/

Deprecation of Prometheus Exporter

With the removal and deprecation of the Prometheus Exporter, users will be encouraged to adopt alternative solutions such as the OTel collector. For those requiring pre-aggregation filtering or custom solutions, tools like Telegraf can be recommended. Deprecating this also is a benefit of no longer needing to maintain this.

JVM Metrics Collection

JVM metrics can be collected using the Open Telemetry runtime-telemetry-java17, which gathers comprehensive metric sets from JFR and JMX. We can also programmatically filter what metrics we want based JfrFeatures available giving further filtering for users. See the runtime-telemetry-java17 JfrFeature table.

Snippet of these metrics output from OTel:

• The SDK is then used to configure and export the API measurements collected from Solr by providing a number of different exporters from OTLP, prometheus or even in-memory metric readers. If not the SDK, the Java agent can also be used as a “zero-code” option

With the SDK, Solr can be used as both to expose both push and pull implementations for metrics. Push with OTLP which is becoming an industry standard with multiple different plugins and tools for exporting these and pull with prometheus as an endpoint to scrape from (/admin/metrics)

SolrMetricsContext needs to be completely refactored instead of being wrapped around Dropwizard to being wrapped around Open Telemetry. Some custom metric types that Solr uses from Dropwizard might not be possible or too difficult to port over to OTel such as MetricsMaps or complex Gauges.

Metrics API /admin/metrics

The metrics endpoint will be changed to output prometheus metrics as the standard for pull metric pipelines. This will use Open Telemetry SDK’s prometheus exporters to transform the API to the prometheus data model.

Filtering

We will maintain filtering capabilities on the metrics API to enable users to limit the amount of data and metrics being scraped from the endpoint. Enhancements may include filtering by tags, preserving the concept of registries/groups with Open Telemetry, or specific metric names possibly. Need to see what the Open Telemetry SDK supports for filtering in Solr but also filtering is possible to happen at an exporter level with OTEL collector/

Deprecation of Prometheus Exporter

With the removal and deprecation of the Prometheus Exporter, users will be encouraged to adopt alternative solutions such as the OTel collector. For those requiring pre-aggregation filtering or custom solutions, tools like Telegraf can be recommended. Deprecating this also is a benefit of no longer needing to maintain this.

JVM Metrics Collection

JVM metrics can be collected using the Open Telemetry runtime-telemetry-java17, which gathers comprehensive metric sets from JFR and JMX. We can also programmatically filter what metrics we want based JfrFeatures available giving further filtering for users. See the runtime-telemetry-java17 JfrFeature table.

Snippet of these metrics output from OTel:

jvm_gc_duration_seconds_count{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java17",otel_scope_version="2.14.0-alpha"} 3
jvm_gc_duration_seconds_sum{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java17",otel_scope_version="2.14.0-alpha"} 0.013224165999999999

jvm_gc_duration_seconds_countbucket{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java17java8",otel_scope_version="2.14.0-alpha",le="0.01"} 3
jvm_gc_duration_seconds_sumbucket{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java17java8",otel_scope_version="2.14.0-alpha",le="0.1"} 0.0132241659999999993
jvm_gc_duration_seconds_bucket{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.14.0-alpha",le="1.0.01"} 3
jvm_gc_duration_seconds_bucket{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.14.0-alpha",le="10.0.1"} 3
jvm_gc_duration_seconds_bucket{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.14.0-alpha",le="1.0+Inf"} 3
jvm_gc_duration_seconds_bucketcount{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.14.0-alpha",le="10.0"} 3
jvm_gc_duration_seconds_bucket{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.14.0-alpha",le="+Inf"} 3
jvm_gc_duration_seconds_count{jvm_gc_action="end of minor GC",jvm_gc_name="G1 Young Generation",otel_scope_name="io.opentelemetry.runtime-telemetry-java8",otel_scope_version="2.14.0-alpha"} 3} 3

New metrics

Introducing new metrics that did not exist before. Metrics on replica state, overseer and zookeeper. For example something similar to below:

solr_core_is_leader{core=core1, host=localhost:8983, shard=shard1} 1
solr_core_is_leader{core=core2, host=localhost:8983,shard=shard1} 0
solr_core_state{core=core1, host=localhost:8983,shard=shard1, state=active} 1
solr_core_state{core=core1, host=localhost:8983,shard=shard1, state=recovery} 0

The prometheus exporter originally retrieved these metrics by scraping the /admin/collections handler and transforming it. We will expose these natively from /admin/metrics instead.

Use-case migration with Open Telemetry

...

Pull model with metrics API (GET /admin/metrics) and filtering

The GET /admin/metrics API will continue to exist, allowing users to scrape metrics with a pull based system. However, the removal of Dropwizard means the current format, its naming conventions and usage will change. The endpoint will now output Prometheus standard formatted metrics and filters will be around tags instead of regex. Below are some use-cases and how the user would migrate:

...