0.6 Java Client Dispatcher Changes

Change Details

The change summary page only covers the idea that extracting the Dispatcher class from the AMQSession class is a goal. The page does not adequately explain how this will address the ordering issue highlighted in QPID-1871. The chief concerns here are to minimise complexity and increase readability/understandability of the client code base.

The goal of this page is to identify the changes that will be required and to show that the change will address the current rollback issue whilst addressing the chief concerns above. In addition the page should detail how the change can be safely executed, given the know issues with locking in the client.

New Dispatcher Interface

The extraction of the Dispatcher from AMQSession will improve readability as we will not have such a large monolithic class file that is currently AMQSession.

/** 
 * Dispatcher is responsible for delivering messages from the IO layer to registered consumers.
 * The dispatcher provides asynchronous dispatch via Consumer MessageListeners and
 * synchronous delivery by placing incoming messages in a Consumers recieveQueue.
 */
public interface DispatcherInterface<C extends BasicMessageConsumer>
{
        /** Start this Dispatcher if required */
        public void startDispatcherIfNecessary(boolean initiallyStopped);
        /** Close this Dispatcher and release any resources held. Cannot be re-opened */
        public void close();
        /** Register a consumer to receive messages from this Dispatcher. */
        public void registerConsumer(C consumer);
        /** Unregister a consumer from this Dispatcher @return C removed consumer */
        public C unregisterConsumer(C consumer);
        /** Rollback the received message state in this Dispatcher */
        public void rollback();
        /** Process the received message from the IO Layer */
        public void dispatchMessage(UnprocessedMessage message);
}

Message Dispatch

The extraction of this class will wrap the current _queue object that receives incoming messages from the the IO Layer. As a result a new interface method will be required for the AMQSession to add the incoming messages to _queue. This is the only interaction that AMQSession has with _queue so this is a simple refactor.

        /** Process the received message from the IO Layer */
        public void dispatchMessage(UnprocessedMessage message);

Ownership of Consumer list

The list of consumers in _consumers is used by both AMQSession and AMQSession.Dispatcher. Here there are two approaches presented for discussion as a clean refactoring is not possible. The introduction of _removedConsumers, which does not behave as the local comments describe, gives two approaches to the refactor. Whilst the need for _removedConsumers is unclear its removal is not being considered here, the focus is to address the current rollback issues.

Dispatcher maintains own copy

Additionally methods will be required to update the _consumers list used for message dispatching. The _consumer list is used for more than just dispatching. AMQSession also uses the list of consumers during failover for resubscription. At this time refactoring how failover operates is not prudent.
So it is proposed that the Dispatcher maintains its own list of consumers which is updated via this interface.

        /** Register a consumer to receive messages from this Dispatcher. */
        public void registerConsumer(C consumer);
        /** Unregister a consumer from this Dispatcher @return C removed consumer */
        public C unregisterConsumer(C consumer);

This interface can easily be called from AMQSession as there is a single put call made from the consumeFromQueue() method. Removal is performed during deregisterConsumers() and a call to keep the dispatcher in sync not unreasonable. This method does highlight an additional data structure that is updated by AMQSession but only used by the Dispatcher: _removedConsumers. The list of removed consumers can be maintained through the unregisterConsumer() method this simplifying AMQSession.
The remaining point of shared use of _consumers is in resubscribeConsumers() and is called as part of failover. The _consumer list is cleared and all consumers re-registered.

Dispatcher

        /** Clear all consumers from this Dispatcher as used by failover @return List<C> removed consumers */
        public List<C> clearAllConsumer();

This approach is allows the resubscribeConsumers() method to clear all the consumers in the Dispatcher ahead of it re-registering. We only need to perform this clear because the client does not reuse the consumerTag when it re-registers the client. New tags are used in the registration with the broker.

AMQSession maintains canonical list

An alternative would be to define an internal interface between the Dispatcher and AMQSession.

        /** Get the current consumer for the given consumerTag. */
        public C getConsumerForTag(int consumerTag);
        /** Get all the consumers for this Session. */ 
        public Collection<C> getAllConsumersRegistered();

This would allow AMQSession to maintain the list of consumers but the list of _removedConsumers would need to be made available to the Dispatcher class. I do not intend to change how this is used as the comments and the usage of this data structure do not align. In deregisterConsumer() it states:

            // Consumers that are closed in a transaction must be stored
            // so that messages they have received can be acknowledged on commit
            if (_transacted)
            {
                _removedConsumers.add(consumer);
            }

However, the only usage of _removedConsumers is in the Dispatcher rollback(), which does not send acknowledgements, yet the comments in both sections of code state that is the use of this structure.

                for (int i = 0; i < _removedConsumers.size(); i++)
                {
                    // Sends acknowledgement to server
                    _removedConsumers.get(i).rollback();
                    _removedConsumers.remove(i);
                }

Looking at it it appears that this is used to ensure that any prefetched messages are rolled back. However, there will only be messages to rollback if the consumer was closed by an error. I believe that the correct course of action is actually to ensure that when all consumers are closed/become invalid/are stopped that any prefetched messages are correctly released. However, this is an additional change out of the scope of this refactoring. If it is desired that AMQSession alone should maintain a list of consumers then the Dispatcher interface could simply be modified to maintain the functionality of _removedConsumers.

        /** Record a consumer that has been removed in Dispatcher for rollback purposes. */
        public void recordRemovedConsumer(C consumer);

Rollback Changes

The Dispatcher Interface calls for a rollback() method and this is where part of the proposed changes will occur.

        /** Rollback the received message state in this Dispatcher */
        public void rollback();

The current code in AMQSession rollback() will be moved to the new Dispatcher class. The existing locking, transactional checks, channel suspension and clearing of session state will be left for AMQSession to handle.

                releaseForRollback();

                sendRollback();

The extraction of releaseForRollback() is where the protocol dependent components will initially be presented. The initial refactoring will maintain the protocol dependent differences however, after the work is completed it is expected that the only difference will be that 0-10 allows for release of a range of deliveryTags while prior to that individual rejects must be sent.

Current Dispatcher Rollback pseudo code

Currently the act of calling session.rollback() performs the following tasks:

# In AMQSession.rollback
 Whilst under the _suspensionLock
  - Check that we are transacted
  - Suspend the session, stopping all new message delivery from broker
  - Perform releaseForRollback()
   # 0-8 
    - reject all Delivered Messages
    - Call dispatcher.rollback()
   # 0-10
    - startDispatcherIfNecessary
    - Place a message on the IO to Dispatcher _queue
    - wait for this message to be processed by the dispatcher thread
    - Call dispatcher.rollback()
    - release and then clear all recorded message tags in _txRangeSet

# In AMQSession.Dispatcher.rollback()
 Whilst under Dispatcher._lock
  - Set the connection Stopped(Stop Dispatcher Thread running)
  - For all Consumers, perform rollback
  - For all browsers, clear prefetch 
  - For all removed consumers perform rollback
  - Set the connection stop/start state back to the value before we started

During this time the Dispatcher thread will be dong the following

# If Disptacher is already running 0-8 & 0-10
   - block for next message on _queue 
   - remove message from _queue and start to dispatch
   # For Dispatchable = UnprocessedMessages
    Whilst under Dispatcher._lock
     - wait until the connection is not stopped
     - If the message is not a CloseConsumerMessage and the deliveryTag is <= current _rollbackMark 
       - reject message
     - Otherwise 
       Whilst under AMQSession._messageDeliveryLock
         - notifyConsumer of the message        
     
   # For Dispatchable = syncDispatchQueue
    - signal queue processed to this point
  
# if Dispatcher is not running 0-8
  Dispatcher is not started, so _queue is not processed. However, Dispatcher is always started if we
  have the possibility of receiving messages so there is no potential to have messages stuck on _queue.

# if Dispatcher is not running 0-10
  Not sure this is possible as the logic should be the same as for 0-8, however as there is an explicit
  startDispatcherIfNecessary call there is the assumption that it may not be running so lets track what 
  happens in that case.
  - Dispatcher thread starts and waits for next message on _queue
  - remove message (which will be a syncDispatchQueue Dispatchable) from _queue and start to dispatch
  - Signal that we have processed it.
  - block on _queue.take() as _queue will be empty.

The current problem is in the Dispatcher thread, when it is already running when the rollback is called and the _queue has messages for dispatch. The Dispatcher Thread can remove a message and then become blocked whilst the connection is stopped. Then after the other thread has performed the rollback the Dispatcher Thread then proceeds to reject the message it has.

Proposed Dispatcher Rollback pseudo code

# In AMQSession.rollback
 Whilst under the _suspensionLock
  - Check that we are transacted
  - Suspend the session
  - Call dispatcher.rollback()
   # 0-10
    - release and then clear all recorded message tags in _txRangeSet

# In AMQSession.Dispatcher.rollback()
 - If we are the Dispatcher Thread
   - Perform actual rollback()
 - otherwise
   - Request rollback be completed by placing service message in FlowControlBlockingQueue
   - wait for completion

# If Disptacher is already running 0-8 & 0-10
   - block for next message on _queue 
   - remove message from _queue and start to dispatch
   # For Dispatchable = UnprocessedMessages
    Whilst under Dispatcher._lock
     - wait until the connection is not stopped
     - If the message is not a CloseConsumerMessage and the deliveryTag is <= current _rollbackMark 
       - reject message
     - Otherwise 
       Whilst under AMQSession._messageDeliveryLock
         - notifyConsumer of the message        
     
   # For Dispatchable = syncDispatchQueue
    - signal queue processed to this point

   # For Dispatchable = RollbackService Message
    Whilst under the _lock
     - Set the connection Stopped(Stop Dispatcher Thread running)
     # For 0-8
       - Release/Reject all delivered messages  
     - For all Consumers, perform rollback
     - For all browsers, clear prefetch 
     - For all removed consumers perform rollback
     - Release/Reject all messages still in _queue
     - Set the connection stop/start state back to the value before we started

The above change to the Dispatcher is dependant on a change to the FlowControllingBlockingQueue to allow service requests to be injected. This is how this is proposed

    public Object take() throws InterruptedException
    {
        Object o = _queue.poll();
        if(o == null)
        {
            synchronized(this)
            {
                while((o = _queue.poll())==null)
                {
                    wait();
                }
            }
        }
     ...

    public void addServiceRequest(ServiceRequest o)
    {
        synchronized(this)
        {
            _serviceQueue.add(o);

            notifyAll();
        }
    }    

    public Object take() throws InterruptedException
    {
        Object o = _queue.poll();
        if(o == null)
        {
            synchronized(this)
            {
                while(((o = _serviceQueue.poll())==null) && 
                      ((o = _queue.poll())==null))
                {
                    wait();
                }
            }
        }
        // Return early so we do not upset the FlowControlCounts.
        if (o instanceof ServiceRequest)
        {
            return o;
        }
        
     ...

Child pages