Running a Qpidd cluster

There are several pre-requisites to running a qpidd cluster:

Install and configure openais/corosync

Qpid clustering uses a multicast protocol provided by the corosync (formerly called openais) library. Install whichever is available on your OS.
E.g. in fedora10: yum install corosync.

The configuration file is /etc/ais/openais.conf or /etc/corosync.conf. You will need to edit the default file created when you installed

Here is an example, with places marked that you will
change. ( Below, I will describe how to change the file. )

# Please read the openais.conf.5 manual page

totem {
        version: 2
        secauth: off
        threads: 0
        interface {
                ringnumber: 0
                ## You must change this address ##
                bindnetaddr: 20.0.100.0
                mcastaddr: 226.94.32.36
                mcastport: 5405
        }
}

logging {
        debug: off
	timestamp: on
	to_file: yes
	logfile: /tmp/aisexec.log
}

amf {
     	mode: disabled
}

You must sent the bindnetaddr entry in the configuration file to the network address of your network interface. This must be a real network interface, not the loopback address 127.0.0.1

You can find your network interface by running ifconfig. This will list the address and the mask, e.g.

inet addr:20.0.20.32  Bcast:20.0.20.255  Mask:255.255.255.0

The bindnetaddr is the logical AND of the inet addr and mask values, in the example above 20.0.20.0

Open your firewall

In the above example file, I use mcastport 5405.
This implies that your firewall must allow UDP
protocol over port 5405.

Use the ais group

The installation of openAIS/corosync on your system will create a new
group called "ais". The user that starts the qpidd processes of the cluster
must have "ais" as its primary group.

Starting a Cluster

To be a member of a cluster you must pass the --cluster-name argument to qpidd. This is the only required option to join a  cluster, other options can be set as for a normal qpidd.

For example to start a cluster of 3 brokers on the current host
Here is an example of starting a cluster of 3 members, all on the current host but with different ports and different log files:

qpidd -p5672 --cluster-name=MY_CLUSTER --log-output=cluster0.log -d --no-data-dir --auth=no
qpidd -p5673 --cluster-name=MY_CLUSTER --log-output=cluster0.log -d --no-data-dir --auth=no
qpidd -p5674 --cluster-name=MY_CLUSTER --log-output=cluster0.log -d --no-data-dir --auth=no

In a deployed system, cluster members will normally be on different hosts but for development its useful to be able to create a cluster on a single host.

SELinux conflicts

Developers will often start openais/corosync as a service like this:

service openais start

But will then will start a cluster-broker without using the service script like this:

/usr/sbin/qpidd --cluster-name my_cluster ...

If SELinux is in enforcing mode this may cause qpidd to hang due because of the different SELinux contexts.

The solution for developers is to make selinux permissive:

To check what mode selinux is running:

# getenforce

To change the mode:

# setenforce permissive

Note that in a deployed system both openais/corosync and qpidd should be started as services, in which case there is no problem with SELinux running in enforcing mode.

Troubleshooting checklist.

If you have trouble starting your cluster, make sure that:

1. Your firewall allows UDP on the openais/corosync mcastport
2. Your effective group is "ais"
3. Your firewall allows TCP on the ports used by qpidd.
4. If you're starting openais as a service but running qpidd directly, ensure selinux is in permissive mode