...
Info |
---|
The current work is in a branch. |
Installations
Enable Metrics
In order to enable the metrics to be published as endpoint comment in the servlet in the web.xml
Code Block | ||||
---|---|---|---|---|
| ||||
<servlet>
<servlet-name>metrics</servlet-name>
<servlet-class>io.prometheus.client.exporter.MetricsServlet</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>metrics</servlet-name>
<url-pattern>/services/metrics/</url-pattern>
</servlet-mapping> |
See: /openmeetings-web/src/main/webapp/WEB-INF/web.xml#L110
What kind of metrics are we collecting
Majority of the metrics collected (as specially the below detailed ones) at of the Prometheus type "histogram" https://prometheus.io/docs/practices/histograms/
A histogram automatically both:
- Duration (called $metricName_sum)
- Count
Initially you may find it's not quite trivial to calculate the duration for individual calls. But it makes sense once you review below. What we are interested is "rate pre min" and similar statistics. Not an individual call, but rate/average over a certain period of time.
How and what to measure in metrics in OpenMeetings
There are roughly 4 categories of Metrics enabled in OpenMeetings
Tomcat statistics
All Servlet calls, including the WebService calls are available automatically in Prometheus. They are collected by a ServletFilter and published as metrics.
See: /openmeetings-web/src/main/java/org/apache/openmeetings/web/util/logging/TomcatGenericExports.java
Once collected in Prometheus you can filter all statistics and run queries on long running or size.
Example:
Below query returns the total count of all calls using the query (below uses the wildcard on the path attribute to filter web service calls out, you could filter further if required)
Code Block |
---|
webapp_metrics_filter_count{path=~"/openmeetings/services/.+"} |
In order to get the duration for those calls you would take the sum (by a certain time period) divided by the count (by the same time period) and what you would get is:
Average duration within a 1min time window, for each of the calls => and graph is and you can adjust the time window also
Code Block |
---|
rate(webapp_metrics_filter_sum{path=~"/openmeetings/services/.+"}[1m])
/
rate(webapp_metrics_filter_count{path=~"/openmeetings/services/.+"}[1m]) |
Application metrics based on annotations in Spring Beans
For Spring Beans I've added an annotation that uses spring-aop to inject bytecode/intercept the start and end of the method (see: /openmeetings-util/src/main/java/org/apache/openmeetings/util/logging/PrometheusAspect.java)
So for any Spring Bean, you can just annotate the method with the 2 annotations I created:
- @TimedDatabase
- @TimedApplication
I created two, cause that way you can filter them and measure database queries. (There would be a way to use JDBC interceptors, but it's quite complicated with openJPA).
Info |
---|
However this method only works if the relevant method is in a Spring Bean |
Example metrics I collect:To enable