Page History

...

The BDB HA clustering features the ability for a node with an "out of date" store to re-join the cluster; however if the node is too far behind, the whole store must be retransmitted 9although (although again BDB HA provides facilities for this: http://docs.oracle.com/cd/E17277_02/html/ReplicationGuide/logfile-restore.html )

Design Questions

• How to manage two-node groups? Need an external process which can set the "primary" on the remaining node.
• How to configure the durability guarantees - should we offer a relaxation to "WRITE_NO_SYNC" on the primary if it is also committing to the network? (Note WRITE_NO_SYNC will survive broker failure, but not machine failure).
• Dynamic management of groups through management - the ability to add/remove node from groups?

Other Considerations

Interop with the C++ Broker

A new effort around HA clustering is currently being undertaken on the C++ Broker.  This effort is concentrating on the ability to replicate queues through the use of special "browsers" an using QMF to inform replicas of configuration state changes. Ultimately this approach may lead to a mechanism the Java Broker also should support in order to be more interopable with the C++ codebase; however for our current requirements it is insufficient as it does not support transactional atomicity; and recovery after a node failure requires a full replay of the store.

HA Clustering for Transient Messages / non-BDB stores

While BDB offers an HA solution, this does not help those who are focused on transient messages or wish to use a non-BDB store. For these cases we need an alternative solutions. One proposal would be to write a "replicating facade" the would wrap any existing store and serialize and replicate transactions to other nodes in a cluster. (Note that in order to be able to perform "recovery" from a restarted node it would also require the persistence of "last processed transaction" and "outstanding uncommited transactions"; and the nodes would need to keep a replay buffer of in-doubt transactions which could be replayed to failed nodes. A process would also need to be put inplace for leadership election - though something like Apache Zookeeper might help us in all this.