You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Proposal for "Next Generation" QMF API

Goals

  • Simplify QMF API
  • Use new QPID Messaging API
  • Implement QMF protocol via QPID Map messages
  • Improve thread safety by minimizing the callback notification mechanism.
  • Move to a work-queue based event model.

Component Addressing

QMF uses AMQP messaging as the means for communications between Console and Agent components. Thus instances of Agents and Consoles must have addresses which uniquely identify them within the AMQP messaging domain.

QMF uses AMQP string types to represent Agent and Console addresses. These strings are assigned by the application, and their contents are opaque to the QMF implementation.

A QMF address is composed of two parts - an optional domain string, and a mandatory name string. The domain string is used to construct the name of the AMQP exchange to which the component's name string will be bound. If not supplied, the value of the domain defaults to "default". Both Agents and Components must belong to the same domain in order to communicate.

When a Console or Agent is instantiated, it will create a receiving endpoint (source) on the AMQP message domain. The endpoint's address will be sent as the reply_to field for all messages originating from that Console or Agent. The address of the endpoint is created from the value of the domain and name strings, using the following format:

    "qmf.<domain-string>.direct/<name-string>"

Data Model

Representation of Managment Data

The QMF data model supports two methods for representing management data:

  1. arbitrarily structured data
  2. data with a formally defined structure

Arbitrarily structured data is the simplest method for representing data under QMF. It consists of a set of named data values. Each data value is represented by a primitive AMQP data type. The data is accessed using the data's name as a key. QMF represents arbitrarily structured data as a map of AMQP data types indexed by a name string.

Data that has a formally defined structure extends the data representation by associating the data with a Schema. The Schema describes the structure of all instances of data that are based on that Schema. This is akin to a record type in database design.

Both types of data representations can be managed by an agent. Managed data includes:

  1. an object identifier
  2. object lifecycle state

An object identifier uniquely addresses a data object within the domain of its managing agent. QMF represents the object identifier as an opaque string. An object identifier can be assigned to the object by the agent when the object is instantiated. Alternatively, a schema can define an object identifier by defining an ordered list of names of data items. In this case, the object identifier string is built by concatenating the string representations of the value of each named data item. This approach is similar to defining index fields within a database record.

For example, assume a managed object with the following map of data item values:

{"field1" : "foo",
 "field2" : "bar",
 "field3" : 42,
 "field4" : "baz"}

and assume the data item list defined by the managed object's schema is:

["field1", "field3"]

The identifier for this data object would be:

"foo42"

QmfData Class

QMF defines the QmfData class to represent an atomic unit of managment data. The QmfData class defines a collection of named data values. Optionally, a string tag, or "sub-type" may be associated with each data item. This tag may be used by an application to annotate the data item.

When representing formally defined data, a QmfData instance is assigned a Schema.

When representing managed data, a QmfData instance is assigned an object identifier (either explicitly, or via the Schema).

class QmfData:
      <constructor>( _values=map of "name"=<AMQP Type> pairs, 
                     _subtypes=optional map of "name"=<AMQP String> pairs for subtype information,
                     _desc=optional AMQP string containing a human-readable description for this data object.
                     _object_id=optional AMQP string that uniquely identifies this QmfData instance.
                     _schema=optional <class SchemaClass> reference
                     _const=False )
      <constructor>( _map=map representation of QmfData, as generated by mapEncode() method, 
                     _schema=optional <class SchemaClass> reference
                     _const=False)
      .isManaged(): returns True if object identifier string assigned, else False.
      .isDescribed(): returns True if schema is associated with this instance
      .getDesc(): return the description of this object if present, else None
      .getValue(name): return the value of the named data item, returns None if named property is not present.
      .hasValue(name): returns True if the named data item is present in the map, else False.
      .setValue(name, value, subType=None): set the value of the named data item. Creates a new item if the named data does not exist. Raises an exception if _const is True. 
      .getSubType(name): returns the sub type description string, else None if not present. 
      .setSubType(name, subType): set or modify the subtype associated with name.
      .get_object_id(): returns the object id string associated with the object instance, or None if no id assigned.
      .get_schema_class_id: returns the identifier of the Schema that describes the structure of the data, or None if no schema.
      .mapEncode(): returns a map representation of the QmfData instance, suitable for use by the constructor.

QMF uses a map encoding to represent a QmfData class in an AMQP message. The map encoding of an instance of the QmfData Class is made up of the following elements:

Index

Optional

Type

Description

"_values"

N

map

Map containing all the "name"=<AMQP Type> pairs for this object.

"_subtype"

Y

map

Map containing all "name"=<AMQP String> subtype entries for this object.

"_desc"

Y

string

Description of this data object.

"_object_id"

Y

string

Unique identifier for this data object.

"_schema_id"

Y

map

Map containing the SchemaClassId for this object.

QmfEvent Class

QMF supports event management functionality. An event is a notification that is sent by an Agent to alert Console(s) of a change in some aspect of the system under management. Like QMF Data, Events may be formally defined by a Schema or not. Unlike QMF Data, Events are not manageable entities - they have no lifecycle. Events simply indicate a point in time where some interesting action occurred.

An instance of an event is represented by the QmfEvent class.

An AMQP timestamp value is associated with each QmfEvent instance. It indicates the moment in time the event occurred. This timestamp is mandatory.

class QmfEvent(QmfData):
      <constructor>( timestamp, 
                     _values=map of "name"=<AMQP Type> pairs,
                     _subtypes=optional map of "name"=<AMQP String> pairs for subtype information,
                     _desc=optional AMQP string containing a human-readable description for this data object.
                     _schema=optional <class SchemaEventClass> )
       <constructor>( _map= map encoding of a QmfEvent instance, 
                     _schema=optional <class SchemaEventClass> )
      .getTimestamp(): return a timestamp indicating when the Event occurred.

The map encoding of an instance of the QmfEvent class extends the map of the parent class with the following class properites:

Index

Optional

Type

Description

"_timestamp"

N

AMQP Timestamp

Time the event occurred.

Schema

Schemas are used by QMF to describe the structure of management data and events. The use of Schemas is optional.

Schema Types

There are two classes (or types) of Schema - those that describe data objects, and those that describe event objects.

SchemaTypeData:
SchemaTypeEvent:

These types may be represented by the strings "_data" and "_event",
respectively.

Schema Identifier

Schema are identified by a combination of their package name and class
name. A hash value over the body of the schema provides a revision
identifier. The class SchemaClassId represents this Schema
identifier.

class SchemaClassId:
      <constructor>( package=<package-name-str>, 
                     class=<class-name-str>,
                     type=<SchemaTypeData|SchemaTypeEvent>)
                     hash=<hash-str, format="%08x-%08x-%08x-%08x">, 
      .getPackageName(): returns <package-name-str>
      .getClassName(): returns <class-name-str>
      .getHashString(): returns <hash-str, "%08x-%08x-%08x-%08x">
      .getType(): returns SchemaTypeObject or SchemaTypeEvent
      .mapEncode(): returns a map encoding of the instance.

If the hash value is not supplied, then the value of the hash string will be set to None. This will be the case when a SchemaClass is being dynamically constructed, and a proper hash is not yet available.

The map encoding of a SchemaClassId:

Index

Optional

Type

Description

"_package_name"

N

string

The name of the associated package.

"_class_name"

N

string

The name of the class within the package.

"_type"

N

string

The type of schema, either "data" or "event".

"_hash_str"

Y

string

The MD5 hash of the schema, in the format "%08x-%08x-%08x-%08x"

Schema For Describing The Properties of a Data Item

The SchemaProperty class describes a single data item in a QmfData object. A SchemaProperty is a list of named attributes and their values. QMF defines a set of primitive attributes. An application can extend this set of attributes with application-specific attributes.

QMF reserved attribute names all start with the underscore character ("_"). Do not create an application-specific attribute with a name starting with an underscore.

Once instantiated, the SchemaProperty is immutable.

class SchemaProperty:
      <constructor>( name=<name-value>, 
                     type=<type-value>,
                     ...)
      .getType(): AMQP typecode for encoding/decoding the property data
      .getAccess(): "RC"=read/create, "RW"=read/write, "RO"=read only (default)
      .isOptional(): True if this property is optional
      .getUnit(): string describing units (optional)
      .getMin(): minimum value (optional)
      .getMax(): maximum value (optional)
      .getMaxLen(): maximum length for string values (optional)
      .getDesc(): optional string description of this Property
      .getDirection(): "I"=input, "O"=output, or "IO"=input/output
                       (required for method arguments, otherwise
                       optional)
      .getSubtype(): string indicating the formal application type 
                     for the data, example: "URL", "Telephone number",
                     etc.
      .isPolled(): True if changes to the data cannot be practically 
                   monitored by an Agent.  Such a data item can only
                   be queried or polled - not published on change.
      .getReference(): if type==objId, name (str) of referenced class
                       (optional) 
      .isParentRef(): True if this property references an object in
                       which this object is in a child-parent
                       relationship.
      .getAttribute("name"): get the value of the attribute named
          "name". This method can be used to retrieve
          application-specific attributes.  "name" should start with
          the prefix "x-"
      .mapEncode(): returns a map encoding of the instance.

The map encoding of a SchemaProperty's body:

Index

Optional

Type

Description

"_qmf_type"

N

integer

The QMF type code indicating the value's data type.

"_access"

N

string

The access allowed to this property, default "RO"

"_optional"

N

boolean

True if this property is optional, default False

"_unit"

Y

string

Description of the units used to express this property.

"_min"

Y

integer

The minimum allowed value for this property

"_max"

Y

integer

The maximun allowed value for this property

"_maxlen"

Y

integer

For string types, this is the maximum length in bytes required to represent the longest permittable instance of this string.

"_desc"

Y

string

Human-readable description of this property.

"_dir"

Y

string

Direction for an argument when passed to a Method call: "I", "O", "IO", default value: "I"

"_subtype"

Y

string

Type information for use by the application.

"_polled"

Y

boolean

True if this property can only be queried/polled. Default False.

"_reference"

Y

string

unknown?

"_parent_ref"

Y

boolean

True if this property references an object in which this object is in a child-parent relationship. Default False

"x-"<varies>

Y

Any AMQP type

An application-defined attribute.

Schema For Describing Method Calls

The SchemaMethod class describes a method call its parameter list. The parameter list is represented by an unordered map of SchemaProperty entries indexed by parameter name.

class SchemaMethod:
      <constructor>( [args=<map of "name":<SchemaProperty> entries>],
                     desc="description of the method")
      .getDesc(): return human-readable description of the method.
      .getArgumentCount(): return the number of arguments
      .getArguments(): returns a copy of the map of "name":<SchemaProperty>
      .getArgument("name"): return the SchemaProperty for the parameter "name"

      .addArgument("name", SchemaProperty): adds an additional argument to
                  the parameter list.
      .mapEncode(): returns a map encoding of the SchemaMethod instance

The map encoding of a SchemaMethod:

Index

Optional

Type

Description

"_name"

N

string

The name of the method.

"_arguments"

Y

map

Map of "name":<SchemaProperty> values, one for each argument to the method.

Note that the "dir" SchemaProperty attribute applies to each
argument. If "dir" is not specified, it is assumed to be "I".

Schema for Data Objects and Events

The structure of QmfData objects is formally defined by the class SchemaObjectClass.

The structure of QmfEvent objects is formally defined by the class SchemaEventClass.

Both of these classes derive from the virtual base SchemaClass.

Agent applications may dynamically construct instances of these objects by adding properties and methods at run time. However, once the Schema is made public, it must be considered immutable, as the hash value must be constant once the Schema is in use.

QMF defines the following classes to represent data and event schema:

class SchemaClass(QmfData):
      <constructor>( classId=<class SchemaClassId>, 
                     _desc=optional AMQP string containing a human-readable description for this schema)
      .getClassId(): return the SchemaClassId that identifies this Schema instance.
      .generateHash(): generate a hash over the body of the schema, and return a string representation of the hash in format  "%08x-%08x-%08x-%08x"

The map encoding of an instance of the SchemaClass class extends the map of its parent class with elements for the following class properties.

Index

Optional

Type

Description

"_schema_id"

N

map

Map containing the SchemaClassId for this object.

class SchemaObjectClass(SchemaClass):
      <constructor>( classId=<class SchemaClassId>, 
                     _desc=optional AMQP string containing a human-readable description for this schema,
                     _props=map of "name"=<SchemaProperty> instances,
                     _meths=map of "name"=<SchemaMethod> instances,
                     _id_names=(optional) ordered list of "name"s used to identify the properties whose values are used to construct the object identifier)

      .get_id_names(): return an ordered list of names of the values that are used to construct the key for identifying unique instances of this class.  Returns None if no list defined.
      .getPropertyCount(): return the count of SchemaProperty's in this instance.
      .getProperties(): return a map of "name":<SchemaProperty> entries for each value in the object.
      .getProperty("name"): return the SchemaProperty associated with "name", else None if "name" value does not exist.

      .getMethodCount(): return the count SchemaMethod's in this instance.
      .getMethods(): return a map of "name":<SchemaMethod> entries for each method associated with the object.
      .getMethod("name"): return the SchemaMethod associated with the "name", else None if "name" does not exist or is not a method.

      .addProperty("name", SchemaProperty): add a new property.
      .addMethod("name", SchemaMethod): add a new method.
      .set_id_names([name-list]): set the value of the order list of names to use when constructing the object identifier.

The map encoding of an instance of the SchemaObjectClass class extends the map of its parent class with elements for the following class properties.

Index

Optional

Type

Description

"_primary_key"

Y

list

Order list of property names used to construct the primary key for objects defined by this schema

class SchemaEventClass(SchemaClass):
      <constructor>( classId=<class SchemaClassId>,
                     _desc=optional AMQP string containing a human-readable description for this event,
                     _props=map of "name":SchemaProperty instances)
      .getPropertyCount(): return the number of properties.
      .getProperties(): return a map of "name":<SchemaProperty> entries for each property in the event.
      .getProperty("name"): return the SchemaProperty associated with "name".

      .addProperty("name", SchemaProperty): add a new property.

The map encoding of a SchemaEventClass instance uses the same format as the map for the SchemaClass.

Data Management

QMF allows data managment via the Query, Subscription, and Method Call actions.

Queries

A Query is a mechanism for interrogating the management database. A Query represents a selector which is sent to an Agent. The Agent applies the Query against its management database, and returns those objects which meet the constraints described in the query.

Queries are expressed using the following map object:

{"query": {"what":<target>}
          {"where":<predicate>}
          ...future selectors....
}

Where:

The value of the "what" key indicates the set of desired objects to match against. The value of the "where" key is an optional predicate to use as the match criteria. If the "where" key is not present, then the query matches against all members of the target set. The agent processes a received query, and returns a list of instances of target types that match the query.

<target> is implemented as a map with a single element in the format:

{"<target name string>": <optional map of target qualifiers>}

QMF defines the following values for <target name string>:

Target

Description

"schema_package"

Query against the set of known packages. Returns a list of package name strings.

"schema_id"

Query against the set of SchemaClass objects, return a list of SchemaClassId instances for the objects that match.

"schema"

Query against the set of SchemaClass objects, and return a list of matched SchemaClass instances.

"object_id"

Query against the set of managed QmfData objects, return a list of name strings for all matching instances.

"object"

Query against the set of managed objects, return a list of matching QmfData instances.

"agent"

Query against all agents within the current QMF domain, return a list of name strings for each matching agent. Used only by the "agent-locate" message.

more tbd ...

...

The value of the <target name string> map entry is ignored for now, its use is TBD.

<predicate> describes a logical match expression that is applied to the target. The expression can be built up from the following syntax:

  binary_cmp=["eq", "ne", "lt", "le", "gt", "ge", "re_match"]
  unary_cmp=["exists"]

  compare_exp = [binary_cmp, <name>, <value>] | 
                [unary_cmp, <name>]

  n_ary_logic = ["and", "or", "not"]

  logic_exp = [n_ary_logic, <expression>, <expression>, ... ] 

  expression = compare_exp | logic_exp

In implementation, predicate expressions are expressed as nested maps. Each map
element is indexed by the operator keyword. The value of the element
is an n-entry ordered list of the operands to the operator. The
format of a single operator expression would be:

    {operator: [operand1, operand2, ...]}

Examples:

     the conditional:
    
         "name" == "tross" or ("name" == "jross" 
                                and "address" == "1313 Spudboy Lane"
                                and "town" == "Utopia")

     would be expressed in a python map as:
  
         {"or": [{"eq": ["name", "tross"]},
                 {"and": [{"eq": ["name", "jross"]},
                          {"eq": ["address", "1313 Spudboy Lane"]},
                          {"eq": ["town", "Utopia"]}]
                 }
                ]
         }


     the conditional:

         !(("name" matches regular expression "?ross") and 
           (("age" property is present) or ("status" is not "sleepy")))

     would be:

        {"not": [{"and": [{"re_match": ["name", "?ross"]},
                          {"or": [{"exists": ["age"]},
                                  {"ne": ["status", "sleepy"]}]
                          }
                         ]
                }]
        }

The valid set of <name> values in an expression is determined by the <target> supplied in the "what" qualifier. QMF reserves a set of <name> values it recognizes. The tables below list the set of name strings allowed for each type of query, and what these names evaluate to on the target.

Target "schema_package" valid names

Description

"_package_name"

Evaluates to the schema's package name string.

Target "schema_id" and "schema" valid names

Description

"_package_name"

Evaluates to the schema's package name string.

"_class_name"

Evaluates to the schema's class name string.

"_type"

Evaluates to the schema's type string ("_data" or "_event").

"_hash_str"

Evaluates to the schema's hash string value.

"_schema_id"

Evaluates to the schema's full identifier (SchemaClassId).

<property-name>

Name of a property defined by the schema. Evaluates to the name string, or None if the property is not defined.

<method-name>

Name of a method defined by the schema. Evaluates to the name string, or None if the method is not defined.

Target "agent_info" valid names

Description

"_name"

Evaluates to the agent's name string.

Target "object_id" and "object" valid names

Description

"_package_name"

If schema assigned, evaluates to the schema's package name string, else None.

"_class_name"

If schema assigned, evaluates to the schema's class name string, else None.

"_hash_str"

If schema assigned, evaluates to the schema's hash string value, else None.

"_schema_id"

If schema assigned, evaluates to the schema's full identifier (SchemaClassId), else None.

"_object_id"

Evaluates to the identifying name string.

"_update_ts"

Evaluates to the last update timestamp.

"_create_ts"

Evaluates to the creation timestamp.

"_delete_ts"

Evaluates to the deletion timestamp.

<value-name>

Specifies the name of a data item in the QmfData data object. Evaluates to the value of the data item.

QMF reserved <name> values all start with the underscore character "_". Do not create value or method names starting with an underscore.

The QmfQuery class represents a query:

class QmfQuery:
      <constructor>(_target=<target map>,
                    _predicate=(optional)<predicate map>)
      .get_target(): return target map
      .get_predicate(): return predicate map
      .evaluate(QmfData): evaluate query against a QmfData instance.  Returns True if query matches the instance, else false.
      .mapEncode(): returns a map encoding of the QmfQuery instance

The map encoding of a SchemaMethod:

Index

Optional

Type

Description

"what"

N

map

The target map.

"where"

Y

map

The predicate map.

Example Queries

With the above syntax, QMF queries can be constructed using AMQP maps
and lists.

For example, a query for all known schema identifiers:

    {"query": {"what": {"schema_id":None}}}

Note that the absence of a "where" clause acts as a "match all"
directive. This query will return a list of all known SchemaId's.

A query for all schema identifiers whose package names are equal to
the string "myPackage":

    {"query": {"where": {"eq": ["_package_name", "myPackage"]},
               "what": {"schema_id": None}
              }
    }

Query for all SchemaClass objects that match a given package and class
name:

    {"query": {"what": {"schema": None}
               "where": {"and": [{"eq": ["_package_name", "myPackage"]},
                                 {"eq": ["_class_name", "someClass"]}]}
              }
    }

Query all managed objects belonging to the "myPackage" package, of the
class "myClass", whose primary key matches a given regular
expression. Return a list of matching object identifiers:

    {"query": {"what": {"object_id": {"_package_name": "myPackage,
                                      "_class_name": "myClass"}},
               "where": {"re_match": ["_primary_key", "foo*"]}
              }
    }

Query for all QmfData objects from a particular schema, whose
"temperature" property is greater or equal to 50:

    {"query": {"what": {"object": {"_package_name": "aPackage,
                                   "_class_name": "someClass"}},
               "where": {"ge": ["temperature", 50]}
              }
    }

In the previous example, the agent will convert the value 50 to a
type compatible with the type given by the "temperature" property's
schema in order to do the comparison.

Query for all objects that match the given schema, which have a
property named "state" which does not match the regular expression
"$Error*", or whose "owner" property is not set to "Cartman".

    {"query": {"what": {"object": {"_package_name": "aPackage,
                                   "_class_name": "someClass"}},
               "where": {"or": [{"not": [{"re_match": ["state", "$Error"]}]},
                                {"ne": ["owner", "Cartman"]}]
                        }
              }
    }

Subscriptions

A subscription is a mechanism for monitoring management data for
changes in its state. A Console creates a subscription with an Agent
based on a Query. The Query specifies the set of management data that
is to be monitored. When the Agent detects changes to the selected
set, a notification is sent to the subscribing Console(s). A
Subscription is represented by the SubscriptionId class. A Console
must cancel the subscription when the console no longer needs to
monitor the data.

class SubscriptionId:
      ?tbd?

Invoking Methods

A managed object's methods provide a mechanisms for a Console application to
perform a "remote procedure call" against the object. The method
actually executes on the Agent, and the result is passed back to the
Console on completion.

The value(s) returned to the Console when the method call completes
are represented by the MethodResult class. The MethodResult class
indicates whether the method call succeeded or not, and, on success,
provides access to all data returned by the method call.

Should a method call result in a failure, this failure is indicated by
the presence of a QmfData object in the MethodResult. This object is
described as an "exception", and contains a description of the reason
for the failure. There are no returned parameters when a method call
fails.

A successful method invokation is indicated by the absence of an
exception in the MethodResult. In addition, a map of returned
parameter values may be present. The structure of the returned
parameters is described by the SchemaMethod for the method.
The map contains only those parameters that the SchemaMethod marks
with an "output" direction.

class MethodResult:
      <constructor>( QmfData <exception> | <map of properties> )
      .getException(): returns exception data if method fails.
      .getArguments(): returns a map of "name"=<value> pairs
                       of all returned arguments.
      .getArgument(<name>): returns value of argument named "name".

Console Application Model

This section describes the API that is specific to Console components.

A QMF console component is represented by a Console class. This class is the topmost object of the console application's object model.

A Console is composed of the following objects:

  • a connection to the AMQP bus
  • a queue of inbound work items
  • a collection of all known schemas
  • a list of all known remote Agents
  • a cache of known data object proxies

The connection to the AMQP bus is used to communicate with remote Agents. The queue is used as a source for notifications coming from remote Agents.

QmfConsoleData Class

The QmfData class is subclassed to provide a Console specific representation of management data.

The Console application represents a managed data object by the QmfConsoleData class. The Console has "read only" access to the data values in the data object via this class. The Console can also invoke the methods defined by the object via this class. The actual data stored in this object is cached from the Agent. In order to update the cached values, the Console invokes the instance's refresh() method.

Note that the refresh() and invoke_method() methods require communication with the remote Agent. As such, they may block. For these two methods, the Console has the option of blocking in the call until the call completes. Optionally, the Console can receive a notification asynchronously when the operation is complete. See below for more detail regarding synchronous and asynchronous API calls.

class QmfConsoleData(QmfData):
      .get_timestamps(): returns a list of timestamps describing the
                        lifecycle of the object.  All timestamps are
                        represented by the AMQP timestamp type.
                        [0] = time of last update from Agent,
                        [1] = creation timestamp 
                        [2] = deletion timestamp, or zero if not deleted.
      .get_create_time(): returns the creation timestamp
      .get_update_time(): returns the update timestamp
      .get_delete_time(): returns the deletion timestamp, or zero if not yet deleted.
      .is_deleted(): True if deletion timestamp not zero.

      .refresh([reply-handle | timeout]): request that the Agent
                    update the value of this object's contents.
      .invoke_method(name, inArgs{}, [[reply-handle] | [timeout]]): 
                          invoke the named method.

Asychronous Event Model.

The original QMF API defined a set of callback methods that a Console
application needed to provide in order to process asynchronous QMF
events. Each asynchonous event defined its own callback method.

The new API replaces this callback model with a work-queue approach.
All asynchronous events are represented by a WorkItem object. When
a QMF event occurs it is translated into a WorkItem object and placed
in a FIFO queue. It is left to the console application to drain
this queue as needed.

This new API does require the console application to provide a single
callback. The callback is used to notify the console application that
WorkItem object(s) are pending on the work queue. This callback is
invoked by QMF when the work queue transitions from the empty to the
non-empty state. To avoid any potential threading issues, the console
application is not allowed to call any QMF API from within the
callback context. The purpose of the callback is to allow the console
application to schedule itself to drain the work queue at the next
available opportunity.

For example, a console application may be designed using a select()
loop. The application waits in the select() for any of a number
of different descriptors to become ready. In this case, the callback
could be written to simply make one of the descriptors ready, and then
return. This would cause the application to exit the wait state, and
start processing pending events.

The callback is represented by the Notifier virtual base class. This
base class contains a single method. A console application derives a
custom handler from this class, and makes it available to the Console
object.

class Notifier:
    .indication():  Called when the internal work queue becomes
    non-empty due to the arrival of one or more WorkItems. This method
    will be called by the internal QMF management thread - it is
    illegal to invoke any QMF APIs from within this callback.  The
    purpose of this callback is to indicate that the application
    should schedule itself to process the work items.  

The WorkItem class represents a single notification event that is read
from the work queue:

class WorkItem:
    #
    # Enumeration of the types of WorkItems produced by the Console
    #
    AGENT_ADDED = 1
    AGENT_DELETED = 2
    NEW_PACKAGE = 3
    NEW_CLASS = 4
    OBJECT_UPDATE = 5
    EVENT_RECEIVED = 7
    AGENT_HEARTBEAT = 8

    .getType(): Identifies the type of work item by returning one of
    the above type codes. 

    .getHandle(): return the handle for an asynchronous operation, if present.

    .getParams(): Returns the data payload of the work item.  The type
    of this object is determined by the type of the workitem (?TBD?). 

Local representation of a remote Agent.

The console application maintains a list of all known remote Agents.
Each Agent is represented by the Agent class:

class Agent:
      .get_name(): returns the identifying name string of the agent.  This name is used to send AMQP messages directly to this agent.
      .isActive(): returns True if the agent is alive (heartbeats have not timed out)
      .invoke_method(name, inArgs{}, [[reply-handle] | [timeout]]): 
                          invoke the named method against the agent.
      ?tbd?

The Console Object.

The Console class is the top-level object used by a console application. All QMF console functionality is made available by this object. A console application must instatiate one of these objects.

As noted below, some Console methods require contacting a remote Agent. For these methods, the caller has the option to either block for a (non-infinite) timeout waiting for a reply, or to allow the method to complete asynchonously. When the asynchronous approach is used, the caller must provide a unique handle that identifies the request. When the method eventually completes, a WorkItem will be placed on the work queue. The WorkItem will contain the handle that was provided to the corresponding method call.

All blocking calls are considered thread safe - it is possible to have a multi-threaded implementation have multiple blocking calls in flight simultaineously.

If a name is supplied, it must be unique across all Consoles attached to the AMQP bus under the given domain. If no name is supplied, a unique name will be synthesized in the format: "qmfc-<hostname>.<pid>"

class Console:
      <constructor>(name=<name-str>, 
                    domain=(optional) domain string for console's AMQP address,
                    notifier=<class Notifier>,
                    reply_timeout=<default for all blocking calls>,
                    agent_timeout=<default timeout for agent heartbeat>,
                    subscription_duration=<default lifetime of a subscription>)

      .destroy(timeout=None): Must be called to release Console's resources.

      .addConnection(QPID Connection): Connect the console to the AMQP cloud.

      .removeConnection(conn): Remove the AMQP connection from the
          console.  Un-does the addConnection() operation, and
          releases any agents associated with the connection.  All
          blocking methods are unblocked and given a failure status.
          All outstanding asynchronous operations are cancelled
          without producing WorkItems.

      .getAddress():
          Get the AMQP address this Console is listening to (type str).

      .findAgent( name string, [timeout] ): Query for the presence of a specific agent in the QMF domain. Returns a class Agent if the agent is present.  If the agent is not already known to the console, this call will send a query for the agent and block (with default timeout override) waiting for a response.

      .enableAgentDiscovery( [Query] ): Called to enable the asynchronous
          Agent Discovery process. Once enabled, AGENT_ADDED and AGENT_DELETED work items
          can arrive on the WorkQueue.  If a query is supplied, it
          will be used to filter agent notifications.

      .disableAgentDiscovery(): Called to disable the async Agent
      Discovery process enabled by calling enableAgentDiscovery().  

      .getWorkItemCount(): Returns the count of pending WorkItems that
      can be retrieved. 

      .getNextWorkItem([timeout=0]): Obtains the next pending work
      item, or None if none available. 

      .releaseWorkItem(wi): Releases a WorkItem instance obtained by
      getNextWorkItem(). Called when the application has finished
      processing the WorkItem. 

      .getAgents(): Returns a list of available agents (class Agent)

      .getAgent( name string ): Return the class Agent for the named agent, if known. 

      .getPackages( [class Agent] ): Returns a list of the names of
          all known packages.  If an optional Agent is provided, then
          only those packages available from that Agent are returned.

      .getClasses( [class Agent] ):  Returns a list of SchemaClassIds
          for all available Schema.  If an optional Agent is provided,
          then the returned SchemaClassIds are limited to those
          Schema known to the given Agent.

      .getSchema( class SchemaClassId [, class Agent]): Return a list
          of all available class SchemaClass across all known agents.
          If an optional Agent is provided, restrict the returned
          schema to those supported by that Agent.

      .getObjects( _SchemaClassId= | _package=, _class= |
                    _object_identifier=,
                   [timeout=],
                   [list-of-class-Agent] ): perform a blocking query
           for QmfConsoleObjects.  Returns a list (possibly empty) of matching
           objects. The selector for the query may be either:
           * class SchemaClassId - all objects whose schema match the
                    schema identified by _SchemaClassId parameter.
           * package/class name - all objects whose schema are
                    contained by the named package and class.
           * the object identified by _object_identifier
           This method will block until all known agents reply, or the
                    timeout expires. Once the timeout expires, all
                    data retrieved to date is returned.  
           If a list of agents is supplied, then the query is sent to
                    only those agents.  


      .createSubscription( class Query [, duration=<secs> [, list of agents] ): creates a
            subscription using the given Query.  If a list of agents
            is provided, the Query will apply only to those agents.
            Otherwise it will apply to all active agents, including
            those discovered during the lifetime of the subscription.
            The duration argument can be used to override the
            console's default subscription lifetime for this
            subscription.  Returns a class SubscriptionId.

      .refreshSubscription( SubscriptionId [, duration=<secs>] ):
      (re)activates a subscription.  Uses the console default duration
      unless the duration is explicitly specified.

      .cancelSubscription( SubscriptionId ): terminates the
      subscription. 

Example Console Application

The following pseudo-code performs a blocking query for a particular agent.

logging.info( "Starting Connection" )
conn = Connection("localhost")
conn.connect()

logging.info( "Starting Console" )
myConsole = Console()
myConsole.addConnection( conn )

logging.info( "Finding Agent" )
myAgent = myConsole.findAgent( "com.aCompany.Examples.anAgent", _timeout=5 )

if myAgent:
   logging.info( "Agent Found: %s" % myAgent )
else:
   logging.info( "No Agent Found!")

logging.info( "Removing connection" )
myConsole.removeConnection( conn )

logging.info( "Destroying console:" )
myConsole.destroy( _timeout=10 )

The following pseudo-code performs a non-blocking query for all
agents. It completes when at least one agent is found.

class MyNotifier(Notifier):
    def __init__(self, context):
        self._myContext = context
        self.WorkAvailable = False

    def indication(self):
        print("Indication received! context=%d" % self._myContext)
        self.WorkAvailable = True

noteMe = MyNotifier( 668 )

logging.info( "Starting Connection" )
conn = Connection("localhost")
conn.connect()

myConsole = Console(notifier=noteMe)
myConsole.addConnection( conn )

myConsole.enableAgentDiscovery()
logging.info("Waiting...")


while not noteMe.WorkAvailable:
    print("No work yet...sleeping!")
    time.sleep(1)


print("Work available = %d items!" % myConsole.getWorkItemCount())
wi = myConsole.getNextWorkitem(timeout=0)
while wi:
    print("work item %d:%s" % (wi.getType(), str(wi.getParams())))
    wi = myConsole.getNextWorkitem(timeout=0)


logging.info( "Removing connection" )
myConsole.remove_connection( conn )

logging.info( "Destroying console:" )
myConsole.destroy( 10 )

Agent Application Model

This section describes the API that is specific to Agent components.

A QMF agent component is represented by a instance of the Agent class. This class is the topmost object of the agent application's object model. Associated with a particular agent are:

  • the set of objects managed by that agent
  • the set of schema that describes the structured objects owned by the agent
  • a collection of consoles that are interfacing with the agent

The Agent class communicates with the application using the same work-queue model as the console. The agent maintains a work-queue of pending requests. Each pending request is associated with a handle. When the application is done servicing the work request, it passes the response to the agent along with the handle associated with the originating request.

QmfAgentData Class

The Agent manages the data it represents by the QmfAgentData class - a derivative of the QmfData class. The Agent is responsible for managing the values of the properties within the object, as well as servicing the object's method calls. Unlike the Console, the Agent has full control of the state of the object.

class QmfAgentData(QmfData):
      .destroy(): mark the object as deleted by setting the deletion
                  timestamp to the current time.
      .setValue(name, value): update the value of the property.
      .incValue(name, delta): add the delta to the property
      .decValue(name, delta): subtract the delta from the property
      ?tbd?

An agent can support one of two different models for managing its database of QmfAgentData objects: internal or external store.

Internal Object Store

An agent that implements internal object store gives full responsibility for managing its data objects to the QMF infrastructure. In this model, the application passes a reference for each managed object to the QMF agent. The agent manages the set of objects internally, directly accessing the contents of the object in order to service console requests.

With this model, the application's complexity is reduced. The application needs to instantiate the object and register it with the agent. The application also maintains a reference to the object, as the application is responsible for updating the object's properties as necessary.

However, the application must still service method calls. The agent notifies the application when a method call has been requested by a console. The application services the method call, passing the result of the method back to the agent. The agent then relays the response to the originating console.

The application may decide to delete an object instance. The application does this by invoking the destroy() method on the object. This notifies the agent, which will mark the object as deleted in its database. Once the application invokes the destroy() method on an object, it must no longer access the object. The agent will clean up the object at a later point in time.

Internal object store is the default model for agent object managment.

Data Consistency

The internal object store requires sharing of the managed data between the agent and the application. The application is responsible for keeping the data up to date, while the agent is responsible for providing the data to client consoles. It is likely that these components may be implemented in separate execution contexts. This raises the possibility that a data item could be in the process of being written to by the application at the same moment the agent attempts to read it. This could result in invalid data being read.

To prevent this from occuring, the QmfAgentObject class provides accessors for all data in the object. These accessors provide atomic access to the underlying data. Therefore, both the agent and the application code must use these accessors to manipulate a shared object's data.

External Object Store

An alternate agent implementation allows the application to take full responsibility for managing the objects. With this model, all instances of managed objects exist external to the agent. When a console requests an action against an object, that action is transferred from the agent to the application. The application then must process the request, and send the result to the agent. The agent then sends a reply to the requesting console.

The model gives full control of the managed objects to the application, but usually requires more application development work.

Agent Class

The base class for the agent object is the Agent class. This base
class represents a single agent implementing internal store.

class Agent:
      <constructor>( name=<name-string>,
                     domain=(optional) domain string for agent's AMQP address,
                     notifier=class Notifier,
                     heartbeat_interval=30,
                     max_msg_size=65535)
      .get_name(): return the name string of the agent.
      .setConnection( QPID Connection ): connect the agent to the AMQP cloud.
      .registerObjectClass( class SchemaObjectClass ): Register a
            schema for an object class with the agent.  The agent must
            have a registered schema for an object class before it can
            handle objects of that class.
      .registerEventClass( class SchemaEventClass ) : Register a
            schema for an event class with the agent.  The agent must
            have a registered schema for an event class before it can
            handle events of that class.
      .raiseEvent( class QmfEvent ): Cause the agent to raise the
            given event.
      .addObject( class QmfAgentData ):
            passes a reference to an instance of a managed QMF object
            to the agent. The object's name must uniquely identify this
            object among all objects known to this agent.
      .getWorkItemCount(): Returns the count of pending WorkItems that
            can be retrieved.
      .getNextWorkItem([timeout=0]): Obtains the next pending work
            item, or None if none available. 
      .releaseWorkItem(wi): Releases a WorkItem instance obtained by
            getNextWorkItem(). Called when the application has finished
            processing the WorkItem. 
      .methodResponse( handle=<handle from WorkItem>,
                       [output argument list],
                       result=<status code>,
                       exception=<QmfData> ): Indicate to the agent
            that the application has completed processing a method
            request. A result code of zero indicates success.  If the
            result code is non-zero, exception may optionally be set to a
            QmfData object that describes the failure.  On success, zero or
            more output arguments may be supplied as defined by the method's
            schema. 

AgentExternal Class

The AgentExternal class must be used by those applications that
implement the external store model. The AgentExternal class extends
the Agent class by adding interfaces that notify the application when
it needs to service a request for management operations from the
agent.

class AgentExternal(Agent):
      <constructor>(name=<name-string>,
                    domain=(optional) domain string for agent's AMQP address,
                    notifier= class Notifier,
                    heartbeat_interval=30,
                    max_msg_size=65535)
      .allocObjectId( name="object name"): indicate to QMF that the named object is available to be managed.  Once this method returns, the agent will service requests from consoles referencing this data.
      .freeObjectId( name="object name" ): indicate to QMF that the named object is no longer available to be managed.
      .queryResponse( handle=<handle from WorkItem>,
                      class QmfAgentObject): send a managed object in 
            reply to a received query. Note that ownership of the
            object instance is returned to the caller on return from
            this call. 
      .queryComplete( handle=<handle from WorkItem>, 
                      result=<status code> ):  Indicate to the agent
            that the application has completed processing a query request.
            Zero or more calls to the queryResponse() method should be
            invoked before calling queryComplete().  If the query should
            fail - for example, due to authentication error - the result
            should be set to a non-zero error code ?TBD?.
      .subscriptionResponse( handle=<handle from WorkItem>,
                             result=<status code>,
                             subscription_handle=<application context>):
            Indicate the status of a subscription request.  If result
            is zero, the subscription is accepted by the application,
            and an subscription handle is provided.  This handle must
            be passed to the application when the agent unsubscribes.

Asychronous Event Model.

The Agent uses the same notification driven work queue model as the
Console. In the Agent case, the WorkItem supports the following set
of work types:

  • METHOD_CALL
  • QUERY
  • SUBSCRIBE
  • UNSUBSCRIBE

In the case of an internal store agent implementation, only the
METHOD_CALL work item is generated. An external store agent must support
all work item types.

  METHOD_CALL parameters: ( name=<method name>, 
                            [argument list],
                            object identifier,
                            user_id=<authenticated id of the user> )

The METHOD_CALL WorkItem describes a method call that must be serviced by the
application on behalf of this agent. On completion of the
method call, the application must invoke the agent's
methodResponse() method.

  QUERY parameters: ( class Query, 
                      user_id=<authenticated id of the user> )

The QUERY WorkItem describes a query that the application must
service. The application should call the queryResponse() method for
each object that satisfies the query. When complete, the application
must call the queryComplete() method. If a failure occurs, the
application should indicate the error to the agent by calling the
queryComplete() method with a description of the error.

  SUBSCRIBE parameters: ( class Query, 
                          user_id=<authenticated id of the user> )

The SUBSCRIBE WorkItem provides a query that the application should
monitor until the subscription is cancelled. On receipt of this
WorkItem, the application should call the subscriptionResponse() agent
method to acknowledge the response. Whenever the matching objects are
updated, the application should call queryResponse() for each updated
object, followed by a call to queryComplete() when done. The
subscription remains in effect until an UNSUBSCRIBE WorkItem for the
subscription is received.

  UNSUBSCRIBE parameters: ( <handle associated with subscription> )

Alerts the application that the corresponding subscription has been
cancelled. The application should no longer supply query updates
against the subscribed query.

Revision History

11/20/2009 - First Revision
11/24/2009 - Added agent classes
12/01/2009 - Cleaned up Schema api and added map definitions.
12/04/2009 - Removed distinction between properties and statistics.
Object identification now based on Primary Key List.
Added more map definitions.
12/11/2009 - Formally define query implementation.
12/14/2009 - Update query implementation based on feedback.

Todo List

  • validation of described object vs its schema
  • verify that schema's primary keys are present in the defined properties
  • make schemas "const" once register with the agent
  • No labels