Enable Metrics

In order to enable the metrics to be published as endpoint comment in the servlet in the web.xml

<servlet>
  <servlet-name>metrics</servlet-name>
  <servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
  <servlet-name>metrics</servlet-name>
  <url-pattern>/services/metrics/</url-pattern>
</servlet-mapping>

See: /openmeetings-web/src/main/webapp/WEB-INF/web.xml#L110

What kind of metrics are we collecting

Majority of the metrics collected (as specially the below detailed ones) at of the Prometheus type "histogram" https://prometheus.io/docs/practices/histograms/

A histogram automatically both:

• Duration (called $metricName_sum)
• Count

Initially you may find it's not quite trivial to calculate the duration for individual calls. But it makes sense once you review below. What we are interested is "rate pre min" and similar statistics. Not an individual call, but rate/average over a certain period of time.

How and what to measure in metrics in OpenMeetings

There are roughly 4 categories of Metrics enabled in OpenMeetings

Tomcat statistics

Basic Tomcat generics see: /openmeetings-web/src/main/java/org/apache/openmeetings/web/util/logging/TomcatGenericExports.java 

Example active_threads

HTTP request metrics -including Web Service calls-  via ServletFilter

All Servlet calls, including the WebService calls are available automatically in Prometheus. They are collected by a ServletFilter and published as metrics. 

Once collected in Prometheus you can filter all statistics and run queries on long running or size.

Example metrics count web service calls

Below query returns the total count of all calls using the query (below uses the wildcard on the path attribute to filter web service calls out, you could filter further if required)

webapp_metrics_filter_count{path=~"/openmeetings/services/.+"}

Example metrics web service calls duration over 1 min time window and graphed

In order to get the duration for those calls you would take the sum  (by a certain time period) divided by the count (by the same time period) and what you would get is:

Average duration within a 1min time window, for each of the calls => and graph is and you can adjust the time window also

rate(webapp_metrics_filter_sum{path=~"/openmeetings/services/.+"}[1m])
/
rate(webapp_metrics_filter_count{path=~"/openmeetings/services/.+"}[1m])

Application metrics based on annotations in Spring Beans

For Spring Beans I've added an annotation that uses spring-aop to inject bytecode/intercept the start and end of the method (see: /openmeetings-util/src/main/java/org/apache/openmeetings/util/logging/PrometheusAspect.java)

So for any Spring Bean, you can just annotate the method with the 2 annotations I created:

• @TimedDatabase
• @TimedApplication 

I created two, cause that way you can filter them and measure database queries. (There would be a way to use JDBC interceptors, but it's quite complicated with openJPA).

Example metrics type database simple count

You can obviously use the same way like above to plot those counts into duration and graph it.

Example metrics type application simple count

Application metrics based on manual metric

Unfortunate not all places in the code are spring beans. Also sometimes we have inner classes or similar issues.

In that case I've added a Util that you can add at the start and end of the Method invocation, to create another metric.

See: