...

Users will now be open to the option of using Open Telemetry and its push model with OTLP metrics, similar to how traces and spans are being exported but must enable the Open Telemetry OTLP module.

Execution plan

Many large changes will take place including breaking backwards compatibility. We will create a feature branch for code reviews until this feature is ready for main. At a high level, the changes will be split into 3 major parts for code reviews:

Open Telemetry Dependencies and module refactoring

Solr needs to add a few main dependencies to instrument and collect metrics in Solr core:

io.opentelemetry:opentelemetry-api - Enables Solr to record telemetry across Solr through a Global Open Telemetry context.

opentelemetry-runtime-java17 - Enable this on startup and Java metrics are automatically recorded and collected with the Open Telemetry API

opentelemetry-exporter-prometheus - This is part of the Open Telemetry SDK needed to export metrics collection from the Open Telemetry API. We will only include the opentelemetry-exporter-prometheus and exclude other unneeded SDK dependencies to allow Solr to hold a Prometheus Metric Reader and expose these metrics to the /admin/metrics endpoint.

From CoreContainer we will need to refactor how Open Telemetry is initialized which is done through TracerConfigurator:loadTracer but creates a OpenTelemetrySdk that only tracing. This needs to be refactor to support tracing and metrics.

The Open Telemetry Module goes through an auto configuration initialization for OTLP. When a user enables this module, the initialization of Open Telemetry will need to also enable OTLP and the prometheus metric reader so that the /admin/metrics endpoint is still available. This requires making some custom changes to Open Telemetry auto configuration for metrics.

Metric wrappers and /admin/metrics endpoint

Before recording metrics and telemetry, we will need to create wrappers around Open Telemetry so that it is easy to record metrics going forward. Solr currently does this by wrapping Dropwizard around SolrMetricManager and SolrMetricContext. We will modify this with wrappers creating Open Telemetry instruments such as counter and gauges while also moving Dropwizard registries to Open Telemetry's equivalent called scope. The metricsHandler will also remove all of its PrometheusFormatter code and we will collect the Open Telemetry metrics in Prometheus format using the Prometheus Metric reader created from the original refactoring from coreContainer. This will lay the foundation of workable recordable and reachable prometheus metrics on the /admin/metrics endpoint.

Metric API instrumentation and Deprecation of Dropwizard

SolrMetricProducer has 219 implementations of this interface. We will need to go to each overridded initializeMetrics() call and migrate it to the Open Telemetry equivalent metric API for measuring events instead of Dropwizard. We will also remove Dropwizard as a dependency from Solr and all it's existing usages. This also means deprecation metric reporters and the prometheus exporter. Lastly we will need to rewrite and update all metric documentation.