Broker Statistics

The following statistics are provided as a basic useful minimum.

1. Peak bytes per second
2. Current bytes per second
3. Total bytes
4. Peak messages per second
5. Current messages per second
6. Total messages

These statistics will be generated for both delivered and received messages, per connection and aggregated per virtualhost and also for the entire broker. Delivered messages are those that have been sent to a client by the broker, and may be counted more than once die to rollbacks and re-delivery. Received messages are those that are published by a client and received by the broker.

Mechanism

The statistics are generated using a sample counter, which is thread-safe and is triggered on incoming message events. The totals for messages and data (bytes) are recorded as type long and the rates are recorded as double since they may be fractional.

The default sample period is one second, during which events are recorded cumulatively. This can be changed for all counters in the broker by setting the system property qpid.statistics.samplePeriod. Basically, rates are calculated as an average over this sample period, so if the traffic is very bursty, a small sample period will fail to capture events most of the time, and you will end up with inflated peak rates. Since we are calculating rates per-second, the sample period is 1000ms as a default, but if this is increased to, say, 10000ms then messages will be counted over ten second periods, and the resulting totals divided by ten to give the rate.

When all statistics generation is enabled, there will be 12 counters that are used, since statistics are calculated for the product of the following sets:

{ Broker, Virtualhost, Connection } x { Delivery, Receipt } x { Messages, Data }

Reporting

If statistics are being generated for the broker or for virtualhosts then periodic reporting can be enabled. This is generated at specified intervals, with the option to reset the statistics after outputting the current data.

[Statistics-Reporting] BRK-1008 : received : 4.395 kB/s peak : 4500 bytes total
[Statistics-Reporting] BRK-1008 : delivered : 4.395 kB/s peak : 4500 bytes total
[Statistics-Reporting] BRK-1009 : received : 45 msg/s peak : 45 msgs total
[Statistics-Reporting] BRK-1009 : delivered : 45 msg/s peak : 45 msgs total
[Statistics-Reporting] VHT-1003 : localhost : received : 1.465 kB/s peak : 1500 bytes total
[Statistics-Reporting] VHT-1003 : localhost : delivered : 1.465 kB/s peak : 1500 bytes total
[Statistics-Reporting] VHT-1004 : localhost : received : 15 msg/s peak : 15 msgs total
[Statistics-Reporting] VHT-1004 : localhost : delivered : 15 msg/s peak : 15 msgs total
[Statistics-Reporting] VHT-1003 : test : received : 0.977 kB/s peak : 1000 bytes total
[Statistics-Reporting] VHT-1003 : test : delivered : 0.977 kB/s peak : 1000 bytes total
[Statistics-Reporting] VHT-1004 : test : received : 10 msg/s peak : 10 msgs total
[Statistics-Reporting] VHT-1004 : test : delivered : 10 msg/s peak : 10 msgs total
[Statistics-Reporting] VHT-1003 : development : received : 1.953 kB/s peak : 2000 bytes total
[Statistics-Reporting] VHT-1003 : development : delivered : 1.953 kB/s peak : 2000 bytes total
[Statistics-Reporting] VHT-1003 : development : received : 1.953 kB/s peak : 2000 bytes total
[Statistics-Reporting] VHT-1003 : development : delivered : 1.953 kB/s peak : 2000 bytes total
[Statistics-Reporting] VHT-1004 : development : received : 20 msg/s peak : 20 msgs total
[Statistics-Reporting] VHT-1004 : development : delivered : 20 msg/s peak : 20 msgs total

Sample statistics report log messages

The reported statistics are output using the operational logging mechanism, and can therefore be extracted from the log files using the same mechanisms as are currently used for other such messages. The new message types are as follows:

• BRK-1008 - Broker data statistics (peak bytes per second and total bytes)
• BRK-1009 - Broker message statistics (peak messages per second and total messages)
• VHT-1003 - Virtualhost data statistics (peak bytes per second and total bytes)
• VHT-1004 - Virtualhost message statistics (peak messages per second and total messages)

Configuration

The statistics generation will be configured via the config.xml broker configuration mechanism.

<statistics>
    <generation>
        <!-- default false -->
        <broker>true</broker>
        <virtualhosts>true</virtualhosts>
        <connections>true</connections>
    </generation>
    <reporting>
        <period>3600</period><!-- seconds -->
        <reset>true</reset>
    </reporting>
</statistics>

It is not possible to enable statistics generation for a single virtualhost - all virtualhosts will generate statistics if the //statistics/generation/virtualhosts element is set to true.

It only makes sense to enable reporting if either broker or virtualhosts have statistics generation enabled also. However, the broker will simply ignore the reporting configuration if no statistics are being generated. If the //statistics/reporting/reset element is set to true then after reporting on the statistics in the log, the statistics will be reset to zero for the entire broker.

Even if statistics generation is completely disabled, it is still possible to activate statistics on an individual connection, while the broker is running. The JMX attribute statisticsEnabled on a connection MBean can be set to true which will start statistics generation, and totals and rates will be calculated from this point onward, or until it is set to false again.

Additionally, the following two system properties can be set to configure the statistics counter:

• qpid.statistics.samplePeriod - sample period in ms - default 1000
• qpid.statistics.disable - override configuration and disable statistics collection - default false (not set)

These two properties are exposed on the StatisticsCounter class as static fields DEFAULT_SAMPLE_PERIOD and DISABLE_STATISTICS.

Design

StatisticsCounter Class

This implements the counting of event data and generates the total and rate statistics. There should be one instance of this class per type of statistic, such as messages or bytes. The instance methods that are called to add an event are:

• void registerEvent()
  Registers a single event at the current system time, such as the arrival of a message.

• void registerEvent(long value)
  Registers a cumulative value, such as the size of a message, at the current system time.

• void registerEvent(long value, long timestamp)
  Registers a cumulative value at the specified time. As with the previous method, this could be 1, which will allow timestamped single events to be registered if the no-argument version is not sufficient.

There are three constructors:

• StatisticsCounter()
• StatisticsCounter(String name)
• StatisticsCounter(String name, long samplePeriod)

These are chained in that order, using a default name of counter and the default sample period of 2000 ms or set in the qpid.statistics.samplePeriod property.

To retrieve the data, there are methods to return the current rate, peak rate and total, as well as the start time, sample period and name of the counter, and also a method to reset the counter.

StatisticsGatherer Interface

This is implemented by the broker business object that is generating statistics. It provides the following methods:

• void initialiseStatistics()
• void registerMessageReceived(long messageSize, long timestamp)
• void registerMessageDelivered(long messageSize)
• StatisticsCounter getMessageDeliveryStatistics()
• StatisticsCounter getDataDeliveryStatistics()
• StatisticsCounter getMessageReceiptStatistics()
• StatisticsCounter getDataReceiptStatistics()
• void resetStatistics()
• void setStatisticsEnabled(boolean enabled)
• boolean isStatisticsEnabled()

These statistics are exposed using the separate JMX Mbean interfaces detailed below, which calls these methods to retrieve the underlying StatisticsCounter objects and return their attributes. This interface gives a standard way for parts of the broker to set up and configure statistics generation.

When creating these objects, there should be a parent/child relationship between them, normally from broker to virtualhost to connection. This means that the lowest level gatherer can record statistics (if enabled) and also pass on the notification to its parent object to allow higher level aggregation of statistics. When resetting statistics, this works in the opposite direction, with higher level gatherers also resetting all of their child objects. Note that this parent/choild relationship is never explicitly specified, and is dependant on the implementation of registerMessageDelivery and resetStatistics in order to allow more flexibility.

JMX Interface

The Qpid JMX interface level that supports statistics is 1.9. Each object (MBean) that can generate statistics will add the following attributes and operations:

The following MBeans have had statistics attributes added:

• ServerInformation
  The entire broker, statistics for all virtualhosts. The totals for the broker will be equal to the sum of the totals for all virtualhosts. Rates for the broker include messages from all virtualhosts, so the broker peak can be higher than individual virtualhosts, and possibly also higher than the sum, due to slight timing differences. This is to be expected.
  org.apache.qpid:type=ServerInformation,name=ServerInformation,*

• ManagedBroker
  This is just a misleadingly named virtualhost. Note that the sum of all connections statistics will not always equal the statistics for the virtualhost, as connection MBeans are discarded when the connection is closed.
  org.apache.qpid:type=VirtualHost.VirtualHostManager,VirtualHost=*,*

• ManagedConnection
  Individual connections, which can be selected on a per-virtualhost basis. The name of a connection is usually the Java toString() representation of the address of the remote internet socket making the connection, so it is usually not possible to determine these in advance. These allow enabling and disabling of statistics generation.
  org.apache.qpid:type=VirtualHost.Connection,VirtualHost=*,name=*

The JMX attributes that record statistics are always present, and will have a value of 0/0.0 if generation is not enabled, and the statisticsEnabled attribute will be set to false.

Usage

The JMX JConsole application can be used to view the attributes, both as discrete values or as graphs. Sample output is shown below, illustrating the virtualhost statistics for a broker with a producer and a consumer attached and sending messsages.

Virtualhost statistics attributes at broker startup

Virtualhost statistics graphs and attributes after sending and receiving messages through the broker

Testing

One caveat related to testing is that these statistics, apart from totals, are by their nature time dependent, and subject to race conditions, where sending messages as part of a test can overlap two sample periods, instead of one, causing seemingly incorrect results.

Unit

The StatisticsCounter class is suitable for unit testing. Tests checking class creation and operation check the total, rate and peak data, as well as other class properties. It is difficult to write a test to provably show thread safety when updating the counter with new events, but there are tests to check that out-of-order events will not cause problems with the data generated.

System

System testing covers the generation of statistics using a running broker, as well as the configuration mechanisms and the operation of the JMX interface to the data and the operational logging output. The following system tests have been written, all using a shared parent test case class, MessageStatisticsTestCase which sets up the JMX connection to the broker.

• MessageStatisticsTest
• MessageStatisticsDeliveryTest
• MessageStatisticsReportingTest
• MessageConnectionStatisticsTest
• MessageStatisticsConfigurationTest

These tests can be run with the following command, from the systests module directory:
$ ant test -Dtest=Message*Statistics*Test