Java Broker Unified Configuration

The Java broker currently has a mixed model for storage of configuration, where some things are held in (multiple) XML files, others are held in a VirtualHost's durable (message) storage, and some can even be held in both. This can create challenges in determining which configuration source should be used to retrieve particular information, and reconciling which configuration source should dominate in the event that overlaps occur.

The following page is intended to document analysis and design aspects of efforts to unify the configuration model of the Java broker and provide it a single consistent storage model for configuration, held in a (durable) configuration store. The design for the new configuration store is in New design for the Java Broker configuration.

This is core to an overall effort to improve usability of the product and increase its suitability for use in Cloud environments. Components of this work include removal of the XML based configuration, addition of a new configuration model for broker entities, consistent storage of that model information, and new management functionality enabling a self-service style operation of remote broker instances via means such as browser based management and a REST API in addition to improved JMX support.

Table of contents

Configuration Entities

Relationships

• A Broker will have collections of:
  • VirtualHosts
  • Ports
  • AuthenticationProviders

• Ports will have collections of:
  • Transports
  • Protocols
  • Connections
  • VirtualHostAliases

A particular Port may support one or more Protocols from those supported by the Broker. A Port may support multiple Transports, for example to allow SSL and non-SSL connections.

A VrtualHostAlias will serve as wiring between a Port and a VirtualHost. Clients will connect to a particular VirtualHostAlias, which will be linked with a particular VirtualHost instance. VirtualHosts may have multiple Aliases. When connecting via a particular VirtualHostAlias clients must use one of its supported AuthenticationMethods.

An AuthenticationMethod is essentially a supported SASL mechnism which forms a relationship between a VirtualHostAlias and a particular AuthenticationProvider, effectively indicating that a particular SASL mechanism can be used when connecting to a given VirtualHost. When a client connects to a particular Port on the Broker, the union of available Method->Providers for the VirtualHostAliases bound to the Port is returned to the client, which after authenticating will indicate the specific VirtualHostAlias it wishes to connect and a final validation will be peformed to ensure the mechanism used was supported by the target.

There is a restriction that on a given Port that supports a 0-X AMQP version that only one AuthenticationProvider can be associated with any given AuthenticationMethod name.

• VirtualHosts will have collections of:
  • Aliases
  • Exchanges.
  • Queues.
  • Connections.

• Exchanges will have collections of:
  • Bindings.
  • Publishers.

• Queues will have collections of:
  • Bindings.
  • Subscriptions.

• Connections will have collections of:
  • Sessions.

• Sessions will have collections of:
  • Subscriptions.
  • Publishers.

Class Diagram

All configured objects will implement the ConfiguredObject interface. This will provide:

• Immutable unique id (probably a UUID) making objects identifiable
• State
• Lifetime policy, creation date, time to live
• Durable indicator
• Zero or more attributes (name/value pairs where value is an Object).
• Zero or more variable (name/value pairs where value is a String).
• Listeners:
  • Attribute changed listener (fired after a update to an attribute)
  • State change listener (fired after a configured object undergoes state transition)
  • Child added/removed listener (fired after this object acquires a new child object or one is removed)

Other entities or other subsystems will be Listeners for state/attribute/child events. This will be useful for Management implementations and Store implementation allowing them to react to changes in the configuration and adjust themselves accordingly.

• A management implementation could react to the creation of a new child (e.g. new virtual host added to broker) by creating the corresponding management entity (e.g. MBean).
• A store implementation could react to the update of the attributes of an existing object (e.g. update of a queue's name) by updating the queue in the persistent store.

Attributes

ConfiguredObjects have attributes. These are mappings between names and objects.  These attributes are used by Real Objects to govern behaviour.

For instance a VirtualHost may have an attribute store-location to specify the location of its data-store. A queue might have an attribute max-queue-depth which limits the depth of  a queue.

When a real object looks up a attribute, if the attribute does not exist at the level of its ConfiguredObject, it will automatically search the ConfiguredObject's parent for a value and so forth until the search reaches the Broker's Configured Object. Continuing with the queue depth example, if the Queue's Configured Object does not have a max-queue-depth, the system will automatically try the Virtual Host for a max-queue-depth.  If the Virtual Host has no value, it will finally try the Broker.

Each ConfiguredObject will understand a list of well-known attribute names. These will be usable in all places (management interfaces, AMQP connections) as argument keys for specifying attribute values when creating ConfiguredObjects. There may also be aliases for these, for example to support legacy arguments used by existing tools/clients, (e.g. x-qpid-maximum-message-count) for certain attributes. When performing AMQP (0-10) queries on queues, all of the attributes (and any associated aliases, to allow for transition) of the Queue will be returned.

Variables

ConfiguredObjects also have variables. These are mappings between names and string values.

A typical use-case would be set a variable for a file system root (for a store location or logging directory) but could be used for other purposes. Variables are provided for operational flexibility to help a user move their configuration between environments by factoring out environment-specific values.

Attribute values which are strings will automatically interpolate variables into their values by substituting placeholders.  Real objects won't uses variables directly (that is they will never call ConfiguredObject#getVariable), but will always configure themselves with attributes. Like attributes if a variable has no value, the system will automatically search the ancestors of the Configured Object for a variable assignment.

Example, a user may define variable the following variable at Broker level:

STORE_HOME=/local/prodstore/

and configure their virtual host with attribute

store-location => ${STORE_HOME}/vh1

At run-time, the VirtualHost implementation will lookup the attribute store-location to discover the location of its store. The system will resolve the attrribute from the VirtualHost configured object, then look to interpolate the variable.  As the variable has no definition at the VirtualHost, it will search the ancestors of VirtualHost for a value (in this case from the Broker).

Finally, the Broker will provide an ability to 'bless' Java system properties or Shell environment variables whose names match a pattern as a variables.   This will give the user the ability to configure Qpid from the shell or command line.   This will allow them to use environment variables such as MYAPP_* or system properties -Dmyapp.* to externally control their configuration.

Life cycle

Entities should have a common life cycle state model, allowing them to be updated in a consistent fashion:

• ACTIVE
• QUIESCED
• STOPPED
• DELETED
• INITIALISING

Representing configured objects in the store

All configured objects will have the following logical representation in the store:

• Object Id
• Object type
• Parent(s) ID(s)
• Attributes
• Variables

Attributes will be a map of key value pairs which can be entity-dependent but will always include:

• Object state
• Durable
• Creation Time
• Time To Live
• Lifetime Policy

Time To Live for configured objects represents how long an object will live once the conditions of its Lifetime Policy are no longer satisfied (e.g. a queue with an auto-delete policy could be deleted a certain amount of time after consumer/session closes rather than immediately, or an otherwise permanent queue could be deleted after being idle for a configured period of time).

Configuration Attributes and Statistics

Each of the ConfiguredObjects will have a number of attributes associated with them, including the several noted above that apply to all objects, and a group of other type specific (queue, exchange etc) and possibly instance specific (priority queue, LVQ, sorted queue etc) attributes. Additionally each ConfiguredObject will separately maintain a number of statistics (which are not persisted) which track elements of its ongoing usage since broker startup.

The below tables (generated from the attached spreadsheet) map out the base attributes and statistics each object type will have:

Overall work plan

1. Update BDB+Derby message stores to use new configuration storage model, representing ConfiguredObjects in a more generic fashion through storage attributes as JSON rather than specific table columns as at present.
   • Maintain existing store interface for now, use a new layer shared between BDB+Derby to convert existing objects into the JSON format stored by the new store schema, allowing later change of the store interface using the same on-disk format.
   • Update message content and metadata references to queues to use ids rather than names (allowing queues to be renamed).
   • TODO: what about messages for queues that are definied in XML? (Possibly generate UUID based from name (nameUUIDFromBytes(byte[] )
2. Refine store upgrade process using a model whereby the broker automatically performs the upgrades between each store version upon startup (inc multiple upgrades spanning version gaps).
   • Isolate individual upgrade steps, remove version numbers form the tables and have a version table etc (See Robs patch for example)
   • Convert existing messages to use new id-reference to their associated queue.
3. Complete implementation of new plugin based JMX MBean adaption layer.
   • VirtualHost, Exchange, Queue etc
   • Updating of tests as necesssary and/or creation of new unit tests to replace old system tests where possible.
   • UserManagement, LoggingManagement MBeans still TBC
4. Extract the Operational Logging from the core broker and define as a plugin listening for configuration events.
5. (Complete) Definition of the REST management interface
6. (Complete) Implementation of the REST management interface
7. Design the desired layout for the new web management UI
8. Get the basic web management ui running
9. Make the ui 'pretty'

This is also covered diagramatically in the attached PDF

Notes

• Alternate Exchanges: Although the alternate-exchange is listed as an attribute of the Queue and Exchange above, it probably shouldnt be: the 'attribute' for any given Queue or Exchange will simply be the name of its alternate exchange, but this is not somethign we would want to store on the Queue or Exchange in question since we want to be able to rename Exchanges. Additionally, storing the alternate exchange on an object could make recovery difficult as a specific recovery order would be required which may ultimately be impossible to achieve (e.g. there may be exchange-exchange alternate loops). Instead of storing the alternate exchange as an attribute directly, what we really want to do is be able to define a relationship between a given Queue/Exchange and its alternate (not unlike the parent-child relationships ConfiguredObjects will already have). For this we probably want to seperately store an 'alternate' relationship in the store, relating a given Quue/Exchange with its alternate Exchange and which can be applied after creating the ConfiguredObjects during recovery.

• Queues in XML configuration: When doing upgrades (/before we get rid of the configuration file), it is possible there will be queues definied in the configuration file (rather than the store) which have messages. IDs will need made for these queues to allow matching id-referenced message data in the store. We will probably need to use something like a UUID generated from the queue name as the ID, eg using nameUUIDFromBytes(byte[] name).

• User vs Store configuration: The current intension is for the storage of configuration to be through JSON, however this is also the likely avenue to be taken for users being able to extract and replay their configuration in future through the management interfaces. There is a key difference in that user-facing configuration is still likely to use names for Queue/Exchange/VirtualHost etc configuration, whereas the store is going to mainly refer to objects by their ID to allow renaming etc.