Cluster Design Note

The Big Idea behind the cluster design is to separate clustering from broker logic as much as possible by having the cluster regard the broker to a large extent as a "black box". The cluster's job is to sort all the frames arriving at all the nodes into a single ordered list (using virtual synchrony). Since every node then processes the same list in the same order, we get the same results everywhere.

There are some areas where this assumption doesn't hold: in c++ messages are dequeued according to the writability of outbound connections, and message TTL and producer flow control both involve decisions based on the local clock, which we can't assume is synced across the cluster. So cluster nodes multicast extra "control frames" to ensure that everybody agrees on the outcome of these kinds of indeterminate decisions. This involves some abstract classes into the broker code so the cluster can alter the behavior.

The big upside of this is minimizing multicast communication. We multicast all the input frames (which has to be done anyway) and then everybody deduces the outcome following the same rules without extra multicast traffic (except in the cases mentioned above) The downside is it serializes all the work and makes it harder to take advantage of multi-CPU architectures. Ideas in that area are in progress.

Reliable Broker design note attached in OpenOffice and HTML format.