QMFv2 Map Message Protocol
Introduction
This document describes the design of a proposed protocol for QMF based on map-messages (offered by the new C++ and Python APIs as well as the existing JMS API).
If adopted, this new protocol will change the formats of the messages used by QMF components to communicate. It will also change some of the message exchange patterns. It will not significantly impact the console and agent APIs and is intended to operate with applications that use the current QMF APIs.
Some highlights of the new design:
- Current QMF message bodies are in packed binary formats. While quite efficient, this style of formatting makes it difficult to make changes to the format and content for new features. The proposed format is based on encoded maps (a.k.a. dictionaries, field-tables) which are very easily extended and require less context to be useful.
- QMF currently requires the message broker to participate in the QMF protocol. The proposed protocol removes this requirement and will run properly on any AMQP message broker.
- QMF Agents currently publish periodic updates of their managed content to a globally accessible topic. This has security implications with regard to access to data. This is also inflexible in that updates to all data are sent at the same intervals. The proposed protocol removes the global publishing of data and introduces a subscription-query whereby a console may request that an agent publish certain data at a certain interval to an indicated target. Such requests can be subject to access control and may be focused on only the data that is needed for a particular application.
- The proposed protocol allows for more general use of data. For example:
- Free-form data, that has no object-identifier nor schema, can be transferred. This is useful for complex queries (joins, reports, etc.).
- Methods can be invoked against an agent in the absence of a managed object.
QMF Protocol
Use of Message Headers
Standard Message Properties
Message Property |
Use |
---|---|
correlation-id |
Used in request/response/indication sets to correlate responses and indications to their request. |
reply-to |
Used in requests to indicate the address for the response. |
content-type |
'amqp/map' or 'amqp/list' |
user-id |
Supplied in a request if authentication/authorization at the agent is appropriate. |
app-id |
'qmf2' |
Custom Application Headers
Application Header Key |
Use |
---|---|
method |
'request', 'response', or 'indication'. This field describes the message's role in a particular message-exchange pattern. |
partial |
Void. If this field is present, it indicates that the message does not contain the complete request or response (i.e. another message will follow using the same opcode and correlation-id) |
qmf.opcode |
QMF-specific operation code (see list below). The opcode defines what content, if any, is to be found in the message body. |
qmf.content |
If the opcode is a data indication, this field indicates what kind of data will be found in the message body. |
qmf.agent |
If this message is a data indication sent by an agent, this field contains the agent's name. |
QMF OpCodes
qmf.opcode field |
Message Body Data Type |
Sent By |
Sent To |
Description |
---|---|---|---|---|
_exception |
QMF_DATA |
Agent |
reply-to |
This general-purpose message can be sent by an agent in response to any request (query, subscription, method) if the request could not be completed for any reason. The QMF_DATA in the message body contains details of why the failure occurred. |
_agent_locate_request |
QMF_QUERY_PREDICATE |
Console |
QMF Topic |
A console may send an agent-locate-request in order to reach all available agents. The predicate may be used to limit the set of agents that will respond to the request. |
_agent_locate_response |
QMF_DATA |
Agent |
reply-to |
This is a response to an agent-locate-request. An agent will send an agent-locate-response if it received an agent-locate-request with a predicate that matches its characteristics. |
_agent_heartbeat_indication |
QMF_DATA |
Agent |
Topic |
Each agent periodically sends a heartbeat message to a topic to indicate that it is alive and connected. The content of the heartbeat message is the list of the agent's characteristics. |
_query_request |
QMF_QUERY |
Console |
Agent |
A console sends a query to an agent to request that the agent send data to the requester. |
_query_response |
List of <qmf.content> |
Agent |
reply-to |
The response to a query sent by a console. |
_subscribe_request |
QMF_SUBSCRIBE |
Console |
Agent |
A console sends a subscribe-request to an agent to receive data matched by a query. A subscription differs from a query request in that it continues to send updated information to the console when the data changes. |
_subscribe_response |
QMF_SUBSCRIPTION |
Agent |
reply-to |
When an agent receives a subscribe-request, it sends a subscribe-response granting (or refusing) the subscription. Should the subscription succeed, the response will contain an identifier for the subscription assigned by the Agent. Thereafter, it will send data-indication messages on the same correlation-id with updates when they happen or periodically. The first data-indication message sent by the agent will contain all matching data, subsequent data-indications will contain only those matching data that has changed since the last update. |
_subscribe_cancel_indication |
QMF_SUBSCRIPTION_ID |
Console |
Agent |
A console can request that a subscription it created be immediately cancelled. This message must have the same correlation-id as the original request, and contain the subscription identifier as assigned by the Agent. |
_subscribe_refresh_indication |
QMF_SUBSCRIPTION_ID |
Console |
Agent |
A console can keep a subscription alive by periodically refreshing it by sending a subscribe-refresh-indication. This message must have the same correlation-id as the original request, and contain the subscription identifier as assigned by the Agent. |
_data_indication |
List of <qmf.content> |
Agent |
reply-to or topic |
A data indication is sent by an Agent when 1) subscription data has changed and needs to be published, 2) an event has occurred and event data is being published, and 3) any other time an agent wants to send unsolicited data. |
_method_request |
QMF_METHOD_CALL |
Console |
Agent |
A console may invoke a method on an object managed by an agent. It may also invoke a method directly on the agent if appropriate. This message contains the method call including the input arguments. |
_method_response |
QMF_METHOD_RESULT |
Agent |
reply-to |
A method call always results in a single method result. This message carries either the output arguments from a successful method call or it holds an exception to describe a failure. |
QMF Content Types
qmf.content field |
Data Type |
Description |
---|---|---|
_schema_package |
STRING |
Schema package name |
_schema_id |
SCHEMA_ID |
Schema class identifier |
_schema_class |
SCHEMA_CLASS |
Schema class definition |
_object_id |
OBJECT_ID |
Managed object identifier |
_data |
QMF_DATA |
Data, managed and/or described or free-form |
_event |
QMF_EVENT |
Event |
_query |
QMF_QUERY |
QMFv2 Query |
Message Body Map Formats
SCHEMA_ID
SCHEMA_ID := { _package_name: STRING, _class_name: STRING, _type: _data | _event, _hash: UUID }
Field |
Optional |
Description |
---|---|---|
_package_name |
no |
Package name (namespace) for the described class |
_class_name |
no |
Name of the described class |
_type |
no |
Class type: data or event |
_hash |
yes |
Hash (uuid) to distinguish different versions of a class |
SCHEMA_CLASS
SCHEMA_CLASS := { _schema_id: SCHEMA_ID, _values: { EACH_ATTR_NAME: SCHEMA_PROPERTY | SCHEMA_METHOD }, _subtypes: { EACH_ATTR_NAME: qmfProperty | qmfMethod } }
Field |
Optional |
Description |
---|---|---|
_schema_id |
no |
Identifier for this schema class |
_values |
no |
Map of schema attribute names and either their property or method descriptions. The subtype defines whether an attribute is a property or a method. |
_subtypes |
no |
Map of subtype names ('qmfProperty' or 'qmfMethod') for each attribute |
SCHEMA_PROPERTY
SCHEMA_PROPERTY := { _type: QMF_TYPE, _access: RO | RC | RW, _unit: STRING, _min: NUMBER, _max: NUMBER, _maxlen: NUMBER, _dir: I | O | IO, _desc: STRING, _references: SCHEMA_ID, _subtype: QMF_SUBTYPE }
Field |
Optional |
Description |
---|---|---|
_type |
no |
The QMF data type of this property |
_access |
yes |
The remote access rules for this property: |
_unit |
yes |
Annotation. Units of measure for numeric values |
_min |
yes |
Minimum numeric value |
_max |
yes |
Maximum numeric value |
_maxlen |
yes |
Maximum length of a variable length value (in octets) |
_dir |
yes |
Used only for method arguments. Direction of transfer: |
_desc |
yes |
Annotation. Description of the property |
_references |
yes |
If the type is a reference to another managed object, this field may be used to specify the required class for that object |
_subtype |
yes |
May be used to further specify the meaning of the value of this field. For example, a number may actually be a timestamp or a duration. A string may be a reference to another object, or a URL. |
QMF_TYPE
QMF_TYPE := TYPE_VOID | TYPE_BOOL | TYPE_INT | TYPE_FLOAT | TYPE_STRING | TYPE_MAP | TYPE_LIST | TYPE_UUID
QMF_SUBTYPE
QMF_SUBTYPE := reference | url | timestamp | duration
SCHEMA_METHOD
SCHEMA_METHOD := { _desc: STRING, _arguments: { EACH_ARG_NAME: SCHEMA_PROPERTY } }
Field |
Optional |
Description |
---|---|---|
_desc |
yes |
Annotation. Description of this method |
_arguments |
no |
Map of argument names and SCHEMA_PROPERTY data to describe them |
QMF_METHOD_CALL
QMF_METHOD_CALL := { _object_id: OBJECT_ID, _method_name: STRING, _arguments: { EACH_KEY: VALUE }, _subtypes: { EACH_KEY: STRING } }
Field |
Optional |
Description |
---|---|---|
_object_id |
yes |
The identity of the managed object receiving the method call. If not supplied, this method applies generally to the agent. |
_method_name |
no |
The name of the method |
_arguments |
yes |
The input arguments, if any |
_subtypes |
yes |
Subtype information for the input arguments, if any |
QMF_METHOD_RESULT
QMF_METHOD_RESULT := { _arguments: { EACH_KEY: VALUE }, _subtypes: { EACH_KEY: STRING } }
Field |
Optional |
Description |
---|---|---|
_arguments |
yes |
Output arguments from a successful method call, if any |
_subtypes |
yes |
Subtype information for the output arguments, if any |
QMF_DATA
QMF_DATA := { _schema_id: SCHEMA_ID, _object_id: OBJECT_ID, _values: { EACH_KEY: VALUE }, _subtypes: { EACH_KEY: STRING } }
Field |
Optional |
Description |
---|---|---|
_schema_id |
yes |
If this data is "described", this field references the schema class that describes the data. |
_object_id |
yes |
If this data is "managed", this field provides the identifier that can be used to address this managed object. |
_values |
no |
The map of values keyed by their property names |
_subtypes |
yes |
Per-property subtypes that may be used to provide more information about the meaning of a value than its QMF_TYPE |
OBJECT_ID
OBJECT_ID := { _agent_name: STRING, _agent_epoch: NUMBER, _object_name: STRING }
Field |
Optional |
Description |
---|---|---|
_agent_name |
yes |
Name of the agent that is managing the referenced data |
_agent_epoch |
yes |
Numeric epoch of the agent process. This number is managed by the agent and is incremented each time the agent process starts. This field is only present for transient object IDs that must not be the same for a given object across an agent restart. Persistent object IDs must not include this field. |
_object_name |
no |
Name of the data that uniquely identifies the data within the context of the agent (or the context of the schema class, if provided) |
QMF_QUERY
QMF_QUERY := { _what: QMF_CONTENT_TYPE, _where: QMF_QUERY_PREDICATE _id: STRING | SCHEMA_ID | OBJECT_ID }
Field |
Optional |
Description |
---|---|---|
_what |
no |
Identifies the kind of data being queried |
_where |
yes |
Query predicate to limit the number of results of the query |
_id |
yes |
(Alternative to _where) This identifier indicates which single item the query is looking for. If the content-type in _what is: |
QMF_CONTENT_TYPE
QMF_CONTENT_TYPE := { _agent: VOID } := { _schema_id: VOID } := { _schema: VOID } := { _object_id: VOID | SCHEMA_ID } := { _object: VOID | SCHEMA_ID }
QMF_QUERY_PREDICATE
QMF_SUBSCRIBE
QMF_SUBSCRIBE := { _query: QMF_QUERY, _duration: NUMBER, _interval: NUMBER }
Field |
Optional |
Description |
---|---|---|
_query |
no |
The query that defines the set of data being subscribed to |
_duration |
yes |
The requested time (in seconds) after which this subscription will be automatically canceled. If a subscribe_refresh_indication is received by the agent running this query, this time interval will start over. |
_interval |
yes |
The request time (in milliseconds) between periodic updates of data in this subscription. The agent may place a minimum on this interval. |
QMF_SUBSCRIPTION
QMF_SUBSCRIPTION := { _subscription_id: STRING, _duration: NUMBER, _interval: NUMBER, }
Field |
Optional |
Description |
---|---|---|
_subscription_id |
yes |
Assigned by the Agent when replying to a successful subscription request. Must be supplied by the Console when sending a subscription refresh or cancel to the Agent for this subscription. |
_duration |
no |
The time (in seconds) after which this subscription will be automatically canceled. |
_interval |
no |
The time (in milliseconds) between periodic updates of data in this subscription. |
QMF_SUBSCRIPTION_ID
QMF_SUBSCRIPTION_ID := { _subscription_id: STRING}
Field |
Optional |
Description |
---|---|---|
_subscription_id |
no |
Supplied by the Console when sending a subscription refresh or cancel to the Agent for this subscription. |