Enable Metrics

Metrics are disable by default. The gathering of metrics as well as exposing them via HTTP for analysis needs to be enabled first.

Enable HTTP and WebService calls enable the Servlet Filter

Comment in the following block in the openmeetings/WEB-INF/web.xml

<!-- Start Prometheus Filter HTTP Servlet metrics
	<filter>
		<filter-name>prometheusFilter</filter-name>
		<filter-class>io.prometheus.client.filter.MetricsFilter</filter-class>
		<init-param>
			<param-name>metric-name</param-name>
			<param-value>webapp_metrics_filter</param-value>
		</init-param>
		<init-param>
			<param-name>help</param-name>
			<param-value>This is the help for your metrics filter</param-value>
		</init-param>
		<init-param>
			<param-name>buckets</param-name>
			<param-value>0.005,0.01,0.025,0.05,0.075,0.1,0.25,0.5,0.75,1,2.5,5,7.5,10</param-value>
		</init-param>
		<init-param>
			<param-name>path-components</param-name>
			<param-value>0</param-value>
		</init-param>
	</filter>

	<filter-mapping>
		<filter-name>prometheusFilter</filter-name>
		<url-pattern>/*</url-pattern>
	</filter-mapping>
	End Prometheus -->

Enable Exposing the metrics and enable Tomcat generic metrics

To enable the export of the metrics as well as some generic Tomcat export (for example around active threads) you need to comment in the following block in openmeetings/WEB-INF/In order to enable the metrics to be published as endpoint comment in the servlet in the web.xml

<!-- Start Prometheus export metrics HTTP
	<servlet>
  		<servlet-name>metrics</servlet-name>
  		<servlet-class>io.prometheus.client.exporter.MetricsServlet<class>org.apache.openmeetings.web.util.logging.OpenMeetingsMetricsServlet</servlet-class>
	</servlet>
	<servlet-mapping>
  		<servlet-name>metrics</servlet-name>
  		<url-pattern>/services/metrics/</url-pattern>
	</servlet-mapping>
	End Prometheus -->

This will expose the metrics atSee: http:/openmeetings-web/src/main/webapp/WEB-INF/web.xml#L110/$HOST:$PORT/openmeetings/services/metrics/ (be aware of the trailing "/" !)

Enable metrics for streaming and other application relevant metrics

In order to get insights on the application specific logic and streaming relevant components you need to additionally enable certain annotations by commenting in openmeetings/WEB-INF/classes/applicationContext.xml

<!-- Start annotation Prometheus metrics
	<aop:aspectj-autoproxy/>
	End annotation -->

View your metrics in Prometheus

Above will enable an endpoint to publish the metrics as HTTP endpoint. You can then use Prometheus (or other tools) to read it in and graph it.

Easiest is to point to it via a Prometheus that runs in a docker container. How to startup the docker container:

docker run --rm -it -p 9090:9090 -v /Users/Sebastian.wagner/Documents/mywork/openmeetings/_REPO/copy-files/
:/etc/prometheus/prometheus.yml prom/prometheus

Example local-prometheus.yml file to reference in above docker command (points to local running OpenMeetings instance)

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # Evaluate rules every 15 seconds.

  # Attach these extra labels to all timeseries collected by this Prometheus instance.
  external_labels:
    monitor: 'scalyr-blog'

rule_files:
  - 'prometheus.rules.yml'

scrape_configs:
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['localhost:9090']

  - job_name:       'openmeetings-local'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    metrics_path:  'openmeetings/services/metrics/'

    static_configs:
      - targets: ['192.168.1.66:5080']
        labels:
          group: 'production'

Change 192.168.1.66:5080 to your local running OpenMeetings instance. Below graphs and queries are done using this docker container.

What kind of metrics are we collecting

Majority of the metrics collected (as specially the below detailed ones) at are of the Prometheus type "histogram" https://prometheus.io/docs/practices/histograms/

...

Initially you may find it's not quite trivial to calculate the duration for individual calls. But it makes sense once you review below. What we are interested is "rate pre per min" and similar statistics. Not an individual call, but rate/average over a certain period of time.

...

Basic Tomcat generics see: /openmeetings-web/src/main/java/org/apache/openmeetings/web/util/logging/TomcatGenericExports.java 

Example active_threads

...

HTTP request metrics -including Web Service calls-  via ServletFilter

All Servlet calls, including the WebService calls are available automatically in Prometheus. They are collected by a ServletFilter and published as metrics. 

...

I created two, cause that way you can filter them and measure database queries. (There would be a way to use JDBC interceptors, but it's quite complicated with openJPA).

See for example some annotated methods in UserDao: /openmeetings-db/src/main/java/org/apache/openmeetings/db/dao/user/UserDao.java#L626

Example metrics type database

...

duration over 1 min rate

Image RemovedImage Added

You can obviously use the same way like above to plot those counts into duration and graph it.

Example metrics type application simple count

select different ones via the legend on the bottom.

Example metrics type application durations 1 min rate

rate(org_openmeetings_metrics_sum{type="application"}[1m])
/
rate(org_openmeetings_metrics_count{type="application"}[1m])

Image AddedImage Removed

Application metrics based on manual metric

Unfortunate not all places in the code are spring beans. Also sometimes we have inner classes or similar issues.

In that case I've added a Util that you can add at the start and end of the Method invocation, to create another metric.

At the start

		Histogram.Timer timer = PrometheusUtil.getHistogram() //
				.labels("RoomPanel", "onInitialize", "application").startTimer();
		try {

At the end

		} finally {
			timer.observeDuration();
		}

See :  RoomPanel::onInitialize()

Example - narrow down the filter of the metric in order to just plot this method

rate(org_openmeetings_metrics_sum{type="application",class="RoomPanel",method="onInitialize"}[1m])
/
  rate(org_openmeetings_metrics_count{type="application",class="RoomPanel",method="onInitialize"}[1m])

Image Added