Broker Management via QMF

As part of the effort to bring more commonality between the Java and C++ brokers I have been attempting to implement QMF management in the Java Broker.  Currently the Java Broker can process the majority of the QMF Management commands (although it will not accept agents connecting to it).  Ultimatlely the aim must be to allow management and configuration of the brokers using the same set of tools.

In implementing QMF management I have noted a number of questions - some general and related to the nature of QMF, some more specific to the particular set of entities and methods in the management schema for the Qpid (C++) Broker, and some simply relating to the deifnition of individual properties or statistics.  I have also noted where I think additions to the existing schema would be useful.

Currently, even if fully implemengted, management through QMF would not be sufficient to duplicate the existing management functions available in the Java Broker (e.g. through the JMX console).

General Comments on the Schema

  • Why are there no methods to create entities? Although one can use AMQP commands to accomplish this task it means you have to switch between two distinct command sets to manage the broker; it also means you cannot simply piggyback the QMF management over other transports. (further, for AMQP 1-0 we are removing "management" commands from the core protocol)

GS: See QPID-2317; I'd like using this to be as simple as sending a correctly formatted/encoded message to a special address. I believe with QMFv2 that will indeed be the pattern. It is a useful addition as the new APIs in c++ and python aim to be be protocol independent and thus do not expose the 0-10 declare methods.

  • There are no tools for inspecting the contents of messages within the broker. Where a message is stuck on the queue, it is useful to be able to inspect its properties and/or content.

GS: You could do this by browsing the queue (though there is no support then for moving or modifying a message). A scheme that avoids duplicating too much of the standard messaging behaviour would be good.

RG: the problem with that is that it requires you to be on an AMQP 0-10 connection.  I would like it to be that we can conduct QMF over other protocols, namely AMQP0-9 or JMX.  I understand your concern about duplicating functionality... as you already mentioned, moving can't be done easily anyway, but I see this inspection somewhat differently to browsing... and (in particular) without some form of server side selector I don't think the C++ broker could do what I am talking about (pick an arbitrary message from the queue and inspect it).

  • Values with High/Low watermarks... the historic low for these will almost invariably be 0, so what is the point?
  • Absolute times are specified in nanoseconds... this is "unusual" and it's pretty much impossible to get an accurate absolute time in nanoseconds. Milliseconds for absolute time and nanoseconds for deltas would make more sense.
  • The model and methods are not designed to cope with a broker which can support multiple virtual hosts.
  • It would be nice to have the ability to configure dynamic event configuration... i.e. the ability to say "publish an event to this address every time this condition is met". You can build this on top of the statistic updates, but it requires you to have a fairly frequent management publish period and a process actively monitoring the output.

* GS: I'm very much in favour of aligning the two brokers. We would however need to come up with a mechanism for allowing the c++ broker to retain backwards compatibility with a schema as currently defined. E.g. support for multiple schemas?

  • The java broker exposes an interface for conducting live User Management, eg add/delete/view users, set password, set management access rights (read only, read/write, or admin which is read/write plus access to the more sensitive management mbeans such as UserManagement itself). It would be good to expose this type of management via QMF as well.
  • The Java broker exposes an interface for viewing and adjusting its logging levels while running. This too would be useful to expose via QMF.

General comments on QMF

  • have a generic way to be able to create new entities,
  • have a way to modify the mutable properties of existing entities.
  • when sending schema, where a particular value is of a restricted domain (e.g. has min and/or max values; or takes a value from a restricted enumerated set) then that restricted domain is communicated.
  • add ability to "reset" statistic counters... This would actually make low watermark counts useful.
  • add ability to query for objects based on criteria (e.g. get me all exchanges whose vhostRef is X)

Comments on Individual Schema Classes

System

Properties

General: what is this supposed to represent - the machine the broker is running on, or the process that is contains the broker?

Name

Type

Description

C++

Java

Notes

systemId

uuid

 

 

Y

Is this a qpid specific property?

osName

sstr

Operating System Name

 

Y

 

nodeName

sstr

Node Name

 

Y

What exactly is this supposed to be... hostname?

release

sstr

 

 

Y

What exactly is this supposed to be... release of the OS?

version

sstr

 

 

Y

What's the difference between this and release?

machine

sstr

 

 

Y

What is this?

Broker

Properties

Name

Type

Description

C++

Java

Notes

systemRef

objId

System ID

 

Y

 

port

uint16

TCP Port for AMQP Service

 

Y

A broker may be listening on more than one port - I suggest that we want to have a new entity to represent listening ports/transports

workerThreads

uint16

Thread pool size

 

Y

 

maxConns

uint16

Maximum allowed connections

 

N

Java broker currently doesn't support a connection limit

connBacklog

uint16

Connection backlog limit for listening socket

 

N

What is this? It "defines the maximum length to which the queue of pending connections for sockfd may grow (from man page for the listen() system call"

stagingThreshold

uint32

Broker stages messages over this size to disk

 

N

Java Broker does not currently support stagingThe 'staging' functionality is poorly conceived on the c++ broker and needs reviewed; it is really just about handling very large messages by not requiring that the full contents ever be held in memory

mgmtPubInterval

uint16

Interval for management broadcasts

 

Y

 

version

sstr

Running software version

 

Y

 

dataDir

sstr

Persistent configuration storage location

 

Y

Java Broker currently displays the value of $QPID_HOME here

Statistics

Name

Type

Description

C++

Java

Notes

uptime

deltaTime

 

 

Y

Why have this as a statistic that is going to constantly change? start time as a property would do equally well

Sggested Additions
  • number of current / high watermark of connections

Methods

Signature

Description

C++

Java

Notes

( uint32 sequence, lstr body ) echo ( uint32 sequence, lstr body)

Request a response to test the path to the management broker

 

Y

 

void connect ( sstr host, uint32 port, bool durable, sstr authMechanism, sstr username, sstr password, sstr transport)

Establish a connection to another broker

 

Y

Doesn't allow a vhost to be specified, nor is transport defined (is ssl a transport?) yes, ssl would be a transport as would e.g. rdma. Perhaps a URL would be simpler?

void queueMoveMessages ( sstr srcQueue, sstr destQueue, uint32 qty)

Move messages from one queue to another

 

N

Queues are local to vhosts, not brokers... since you may have queues with the same name on two separate vhosts on the same broker - this method cannot be implemented on a broker which supports multiple vhosts. Should also add an arguments map to allow filters to be applied

Suggested Additions
  • ability to create virtual hosts
  • shutdown/restart broker?
  • reload (security) configuration.

Agent

The Java Broker does not currently support the Agent class

Properties

Name

Type

Description

C++

Java

Notes

connectionRef

objId

 

 

N

 

label

sstr

Label for agent

 

N

 

registeredTo

objId

Broker agent is registered to

 

N

 

systemId

uuid

Identifier of system where agent resides

 

N

 

brokerBank

uint32

Assigned object-id broker bank

 

N

 

agentBank

uint32

Assigned object-id agent bank

 

N

 

Vhost

Properties

Name

Type

Description

C++

Java

Notes

brokerRef

objId

 

 

Y

 

name

sstr

 

 

Y

 

federationTag

sstr

 

 

Y

 

Statistics

Suggested Additions
  • number (and high watermark) of queues/exchanges
  • number (and high watermark) of connections

Methods

Suggested Additions
  • create queue
  • create exchange
  • create binding
  • delete queue
  • delete exchange
  • delete this vhost

Queue

Properties

Name

Type

Description

C++

Java

Notes

vhostRef

objId

 

 

Y

 

name

sstr

 

 

Y

 

durable

bool

 

 

Y

 

autoDelete

bool

 

 

Y

 

exclusive

bool

 

 

Y

 

arguments

map

Arguments supplied in queue.declare

 

Y

 

altExchange

objId

 

 

Y

 

Statistics

Name

Type

Description

C++

Java

Notes

msgTotalEnqueues

count64

Total messages enqueued

 

Y

 

msgTotalDequeues

count64

Total messages dequeued

 

Y

 

msgTxnEnqueues

count64

Transactional messages enqueued

 

 

 

msgTxnDequeues

count64

Transactional messages dequeued

 

 

 

msgPersistEnqueues

count64

Persistent messages enqueued

 

Y

 

msgPersistDequeues

count64

Persistent messages dequeued

 

Y

 

msgDepth

count32

Current size of queue in messages

 

Y

Is this with or without unacked messages (i.e. does it include all messages for which the dequeue has not yet been committed)?It includes unacked messages
Also would seem to make sense to provide high watermark for queue depth (bytes and msg)GS: +1 on high watermark for queue depth; would be very useful

byteDepth

count32

Current size of queue in bytes

 

Y

 

byteTotalEnqueues

count64

Total messages enqueued

 

Y

 

byteTotalDequeues

count64

Total messages dequeued

 

Y

 

byteTxnEnqueues

count64

Transactional messages enqueued

 

 

 

byteTxnDequeues

count64

Transactional messages dequeued

 

 

 

bytePersistEnqueues

count64

Persistent messages enqueued

 

Y

 

bytePersistDequeues

count64

Persistent messages dequeued

 

Y

 

consumerCount

hilo32

Current consumers on queue

 

Y

 

bindingCount

hilo32

Current bindings

 

Y

 

unackedMessages

hilo32

Messages consumed but not yet acked

 

 

It would seem more useful on a per subscription basis (so we can see which subscriptions are holding on to messages without acking them).GS: I agree

messageLatency

mmaTime

Broker latency through this queue

 

 

This generates a number of statistics (Max/Min/Avg/Samples) each of which is said to be measured in nanoseconds. What does Samples mean in this context?this isn't actually implemented on c++ yet

Suggested Additions
  • age of oldest message on the queue (and high watermark),
  • high watermark for message size
  • high watermark for queue depth (bytes)
  • high watermark for message count
  • messages/bytes expired due to TTL

Methods

Signature

Description

C++

Java

Notes

void purge ( uint32 request)

Discard all or some messages on a queue

 

Y

Would be nice to add an arguments map so that filters could be supplied. Also making request an IO parameter so you can return how many messages were purged. Finally are the messages supposed to be purged from the head, the tail or randomly throughout the queue?

Suggested Additions:
  • mechanisms to move / copy messages to another queue (with filters to allow specifying arbitrary messages, not first X msg)
  • rename this queue
  • delete this queue
  • mechanism to inspect messages on the queue

Exchange

Properties

Name

Type

Description

C++

Java

Notes

vhostRef

objId

 

 

Y

 

name

sstr

 

 

Y

 

type

sstr

 

 

Y

 

durable

bool

 

 

Y

 

autoDelete

bool

 

 

Y

 

altExchange

objId

 

 

Y

 

arguments

map

Arguments supplied in exchange.declare

 

 

 

Statistics

Name

Type

Description

C++

Java

Notes

producerCount

hilo32

Current producers on exchange

N

N

How is this supposed to be calculated? RGem: This is not actually implemented on the C++ broker either

bindingCount

hilo32

Current bindings

 

Y

 

msgReceives

count64

Total messages received

 

Y

 

msgDrops

count64

Total messages dropped (no matching key)

 

Y

 

msgRoutes

count64

Total routed messages

 

Y

 

byteReceives

count64

Total bytes received

 

Y

 

byteDrops

count64

Total bytes dropped (no matching key)

 

Y

 

byteRoutes

count64

Total routed bytes

 

Y

 

Methods

Suggested Additions
  • rename
  • set AlternateExchange
  • delete

Binding

Properties

Name

Type

Description

C++

Java

Notes

exchangeRef

objId

 

 

Y

 

queueRef

objId

 

 

Y

 

bindingKey

sstr

 

 

Y

 

arguments

map

 

 

Y

 

origin

sstr

 

 

Y

definition would be good... why is this not in "arguments" (since that is how it is set)?

Statistics

Name

Type

Description

C++

Java

Notes

msgMatched

count64

 

 

Y

 

Methods

Suggested Additions
  • delete

Subscription

Properties

Name

Type

Description

C++

Java

Notes

sessionRef

objId

 

 

Y

 

queueRef

objId

 

 

Y

 

name

sstr

 

 

Y

 

browsing

bool

 

 

Y

is this supposed to be a pre/non acquire indicator? Is this  true only when pre-acquiring and accept-mode is none?This is true of the acquire-mode was not-acquired (for 0-10). It is independent of the accept-mode which is exposed via the acknowledged property

acknowledged

bool

 

 

Y

is this simply accept-mode = explicit?yes

exclusive

bool

 

 

Y

 

creditMode

sstr

WINDOW or CREDIT

 

Y

 

arguments

map

 

 

Y

 

Statistics

Name

Type

Description

C++

Java

Notes

delivered

count64

Messages delivered

 

Y

 
Suggested Additions
  • available credit
  • unacknowledged msg/bytes
  • delivered bytes

Methods

Suggested Additions
  • delete

Connection

Properties

Name

Type

Description

C++

Java

Notes

vhostRef

objId

 

 

Y

 

address

sstr

 

 

Y

Is this the local or remote address?  Whichever, we should also have the other one. It is the remote address

incoming

bool

 

 

Y

 

SystemConnection

bool

Infrastucture/ Inter-system connection (Cluster, Federation, ...)

 

Y

 

federationLink

bool

Is this a federation link

 

Y

 

authIdentity

sstr

authId of connection if authentication enabled

 

Y

 

remoteProcessName

sstr

Name of executable running as remote client

 

 

These are a subset of the clientProperties sent by the client - should we not just show the client Props?

remotePid

uint32

Process ID of remote client

 

 

 

remoteParentPid

uint32

Parent Process ID of remote client

 

 

 
Suggested Additions
  • vhost
  • server/client properties maps
  • protocol version

Statistics

Name

Type

Description

C++

Java

Notes

closing

bool

This client is closing by management request

 

 

Why is this a statistic???

framesFromClient

count64

 

 

 

 

framesToClient

count64

 

 

 

 

bytesFromClient

count64

 

 

 

 

bytesToClient

count64

 

 

 

 
Suggested Additions
  • attached sessions (with high watermark)

Methods

Signature

Description

C++

Java

Notes

void close ( )

 

 

 

 

Properties

Name

Type

Description

C++

Java

Notes

vhostRef

objId

 

 

Y

 

host

sstr

 

 

Y

 

port

uint16

 

 

Y

 

transport

sstr

 

 

Y

What are the valid transports? at present, tcp, rdma and ssl(which is over tcp)

durable

bool

 

 

Y

 
Suggested Additions
  • remote vhost

Statistics

Name

Type

Description

C++

Java

Notes

state

sstr

Operational state of the link

 

 

Are the valid states defined anywhere? c++ broker currently includes the following states: WAITING, CONNECTING, OPERATIONAL, FAILED, CLOSED, PASSIVE (from Link.h)

lastError

sstr

Reason link is not operational

 

 

 

Methods

Signature

Description

C++

Java

Notes

void close ( )

 

 

 

 

void bridge ( bool durable, sstr src, sstr dest, sstr key, sstr tag, sstr excludes, bool srcIsQueue, bool srcIsLocal, bool dynamic, uint16 sync)

Bridge messages over the link

 

Y

 

Bridge

Properties

Name

Type

Description

C++

Java

Notes

linkRef

objId

 

 

Y

 

channelId

uint16

 

 

Y

 

durable

bool

 

 

Y

 

src

sstr

 

 

Y

 

dest

sstr

 

 

Y

 

key

sstr

 

 

Y

 

srcIsQueue

bool

 

 

Y

 

srcIsLocal

bool

 

 

Y

 

tag

sstr

 

 

 

what is this and what does it do? It is used for loop detection and is the value to be used for qpid.trace.id on the subscription queue

excludes

sstr

 

 


what is this and what does it do? Also used for loop detection and is the value to be used for qpid.trace.excludes on the subscription queue

dynamic

bool

 

 

Y

 

sync

uint16

 

 

 

 

Methods

Signature

Description

C++

Java

Notes

void close ( )

 

 

 

 

Session

Properties

Name

Type

Description

C++

Java

Notes

vhostRef

objId

 

 

Y

 

name

sstr

 

 

Y

 

channelId

uint16

 

 

Y

 

connectionRef

objId

 

 

Y

 

detachedLifespan

uint32

 

 

 

 

attached

bool

 

 

Y

 

expireTime

absTime

 

 

 

 

maxClientRate

uint32

 

 

 

 
Suggested Additions
  • transactional mode

Statistics

Name

Type

Description

C++

Java

Notes

framesOutstanding

count32

 

 

 

 

TxnStarts

count64

Total transactions started

 

 

Are these supposed to be tx or dtx transactions? or either

TxnCommits

count64

Total transactions committed

 

 

 

TxnRejects

count64

Total transactions rejected

 

 

 

TxnCount

count32

Current pending transactions

 

 

 

clientCredit

count32

Client message credit

 

 

 
Suggested Additions:
  • subscription count/watermark
  • unacked messages/bytes

Methods

Signature

Description

C++

Java

Notes

void solicitAck ( )

 

 

 

 

void detach ( )

 

 

 

 

void resetLifespan ( )

 

 

 

 

void close ( )

 

 

 

 
  • No labels