...

The ConsumerTarget is the broker-side representation of a consuming client. Due to multi-queue consumers a ConsumerTarget has one or more Consumers associated with one queue each.

The responsibility of the Queue is to notify the Consumers A Queue MUST notify at least one interested Consumer when there is work for them (i.e., messages the consumer is interested in).to be done.

A Consumer MUST  The Consumers responsibility is to notify the Queue when it is ready to do some work and when the time has come to pull messages of the queue and process them (i.e. send them to the consuming client).work. When notified by a Queue of available work. A Consumer MUST either notify the Queue that it is no longer interested OR try to pull a message of said Queue.

Thread Model

Consumers are always invoked from the consuming connection's IO-Thread whereas the Queue might be invoked from different threads (producing connection's IO-Thread, Housekeeping thread for held or TTLed messages, a consuming connection's IO-Thread in case for message reject).

The interface between Consumers and Queues are

• AbstractQueue#setNotifyWorkDesired
• QueueConsumer#notifyWork
• AbstractQueue#deliverSingleMessage

These methods MUST be thread-safe and SHOULD be lock free.

Simple Flow

1. Message arrives on the Queue
2. The Queue notifies some interested Consumers that there is work to be done
3. The Consumers notify their ConsumerTarget that they would like to do work
4. The ConsumerTargets notify their Session that they would like to do work
5. The Sessions notify their Connections that they would like to do work
6. The Connections schedule themselves
7. The Scheduler kicks off a IO-Thread to process the work of a Connection
8. The Connection iterates over its Sessions that want to do work
9. The Sessions iterate over its ConsumerTargets that want to do work
10. The ConsumerTargets iterate over its Consumers that want to do work
11. The Consumer tries to pulls a message from the Queue
12. If successful the message is put on the IO-buffer to be sent down the wire
    

...

This list contains all acquiring Consumers that indicated to the Queue that they currently are not interested in doing any work (i.e., taking messages). This typically happens when a Consumer/Connection is suspended due to FlowControl/TCP backpressure. The main purpose of this list is to avoid spurious wake-ups of Consumers which we know are not going to do any work.

The "Interested" List

This is the default list for acquiring Consumers. It signifies that they are ready to process messages. When a new message arrives on the Queue it will notify Consumers from this list. It will only notify a single interested Consumer to avoid spurious wake-ups.

The "Notified" List

Once an acquiring Consumer is notified that there is work to do it is moved from the "Interested" list to the "Notified" list. The QCM expects such a Consumer to either indicate that it is no longer interested (e.g., it became suspended in the meantime and therefore will not do the work we expected it to) or call AbstractQueue#deliverSingleMessage. The Consumer should remain in the "Notified" list and continue to call deliverSingleMessage until deliverSingleMessage cannot deliver a message to it any more in which case it is moved to the back to of the "Interested" list. This is to decrease latency due to wake-ups when there continues to be work done (i.e., there is a steady stream of messages). Appending it to the end of the Interested list ensures some level of fairness. Note that this is not perfect. It is possible that a consumer is notified but at the time it tries to pull a message of the Queue there is no longer a message available and the Consumer is returned to the end of the Interested list without having done work. The assumption is that while this may happen it is to avoid unnecessary wakeupsunlikely to always happen to the same consumer leading to a kind of "asymptotic fairness".

Handling Consumer Priorities

When deciding which Consumer to notify the QCM should take consumer priorities into account. To do this in a performant way it maintains a QueueConsumerNodeList per consumer priority in a list of PriorityConsumerListPairs. This ensures that iteration of the Interested list happens in the right order and lookup of consumers with higher priority can be performed efficiently.