Client Failover Behaviour

This is a draft of expected Failover Behaviour

Qpid client failover basic principles.

When connection to broker is lost due to network failure a Qpid client should be able to re-establish connection to a Qpid broker if failover policy is not switched off by specifying "nofailover" as a failover option in a connection URL.

The failover functionality on Qpid client should be based on principle "stop the world". When connection is lost and failover is started the Qpid Client should not allow an invocation of JMS operations which requires sending or receiving data over the network (such as producer.send(), connection.createSession(), consumer#receive etc). Such operations should be blocked until failover functionality restores the connectivity with any of the supported failover methods ('singlebroker', 'roundrobin', 'failover_exchange').

The failover reconnect feature should keep trying to reconnect to the Broker(s) in the background until connection is restored or the application calls Connection.close() or all failover method reconnection attempts are exhausted. Each failover method defines their own reconnection options and behavior.

How to configure failover with connection URL is depicted in Connection URL Format (This document describes all existing failover methods and their configuration options).

On restoring connectivity blocked JMS operations should be allowed to finish. If the failover functionality cannot re-establish the connection a JMSException should be thrown within any JMS operation requiring transferring data over the network.

When the client connection is recreated, existing Sessions, Producers, Consumers will be refreshed transparently to allow message processing to continue, with certain caveats described further below.

Failover notification via Exception Listener

In case if failover happens (or not, in the case of NoFailover method) and a Connection has registered ExceptionListener a special JMS exception (ConnectionLostException) needs to be sent into ExceptionListener to notify user that network failure happens. The JMS client application code can then decide what approach to take; call connection.close() etc, or take advantage of their configured failover reconnect feature.

If the client has no Exception Listener, they will not receive this notification. Exceptions indicating failover occured will only be thrown from other synchronous JMS methods as required by their functionality, e.g. commit() and acknowledge() (see below for further details).

Auto Acknowledge

In Auto Acknowledge mode, by default the last message received by the application may fail to be acknowledged if the connection gets closed during onMessage, or before the call to receive() completes the acknowledgement.

In the receive() case, any such failure should be propagated as a JMSException through the method call, and in onMessage such failure has to be notified through the ExceptionListener if there is one.

The spec allows for redelivery of this message:

4.4.14 Duplicate Delivery of Messages
...

When a client uses the AUTO_ACKNOWLEDGE mode it is not in direct control of message acknowledgment. Since such clients cannot know for certain if a particular message has been acknowledged, they must be prepared for re-delivery of the last consumed message. This can be caused by the client completing its work just prior to a failure that prevents the message acknowledgment from occurring. Only a session’s last consumed message is
subject to this ambiguity.

Any call to recover() performed following failover should be successfull, as the failover occurrence was already notified through the ExceptionListener if there was one, and the request to recover() would result in cleaning the Session and resuming message delivery with the first message sent by the new broker.

Dups Ok Acknowledge

Duplicates are allowed in this mode, therefor any application using it should be prepared to accept any number of duplicates and thus failover can be performed silently (other than previously mentioned Connection level Exception Listener notification that failover has occurred).

Client Acknowledge

A Client Ack Session should be considered 'dirty' if any unacknowledged messages have been received by the application. When the Session is refreshed during failover, if the Session is dirty then any unacknowledged messages previously received on the Session before failover can no longer be acknowledged and must be considered 'stale'.

All messages held by the client prior to failover (unacknowledged messages given to the application, and prefetched messages) should be discarded by the client as they can no longer be acknowledged, and record retained as to whether the Session was dirty when failover occurred. Only messages given to the client after failover will now be available to the application. When the next call to message.acknowledge() is performed, recover() should be called implicitly to clean the Session and an exception should be thrown (which one TBC, but thrown in addition to previous notification through the Exception Listener that failover has occurred) to indicate that we were unable to complete the acknowledgement process. If none of the new messages given to the client by the broker have been received by the application, this recover() call could be a no-op other than marking the Session clean, otherwise it may have to perform a full recover against the broker.

Any call to recover() performed following failover should be successfull, as the failover occurence was already notified through the ExceptionListener if there was one, and the request to recover() would result in cleaning the Session and resuming message delivery with the first message sent by the new broker.

Session Transacted

A Transacted Session should be considered 'dirty' if any uncommited send/receive operations exist. When the Session is refreshed during failover, if the Session is dirty then any uncommitted send/receive work previously conducted on the Session before failover must be considered 'stale'. When the next call to commit() is made, the Session will automatically be rolled back and TransactionRolledBackException thrown to notify the application of the situation, allowing it to simply replay its transaction and continue.

Any call to rollback() performed following failover should be successfull, as the failover occurence was already notified through the ExceptionListener if there was one, and the request to rollback() would result in cleaning the Session and resuming message delivery with the first message sent by the new broker.

Queue Browsers

If failover occurred while iterating through QueueBrowser enumerations a sub-class of NotSuchElementException should be thrown by default.

Temporary Queues

On successful failover, it is expected that a Qpid client should restore all temporary queues (by redeclaring the queues with the same name+attributes) created before failover.

Link Reliability Options

Where a Link Reliability option is specified on an Address, it must be in conformance with the Acknowledge Mode being used by the Session. E.g, if requesting at-least-once link behaviour for a destination on a No-Acknowledge Session, an exception should be thrown as this combination is contradictory. The JMS Session Acknowledge Mode set in the code should take precedence.