Clustering and High Availability in Servicemix
ServiceMix supports both High availability(HA) and clustering of containers. Clustering in ServiceMix provides
- load balancing of messages across servers/machines
- high availability (HA) of services and fault tolerance (if a machine crashes, messages are automatically redelivered to another machine
- durability (persist messages to disk/database) to ensure they survive catastrophic hardware failure
- local JMS transactions or full XA transaction support
Configuring a cluster
To run ServiceMix in a cluster, enable a JMS Flow - see Configuring ServiceMix.
Component deployment happens in the same way as a normal ServiceMix JBI container (both for POJO and archive Component deployment) but all the containers in the cluster are notified of a deployment, and the JMS Flow will handle automatic routing (and failover) of MessageExchange(s) between the members of the cluster.
Note | ||
---|---|---|
| ||
In the cluster mode all ServiceMix instance must have a unique name into whole cluster. You could see Configuration for more information. |
. As you can configure HA without clustering its helpful to deal with them as two separate tasks.
High Availability in ServiceMix
Within the scope of this document, high availability (HA) can be defined as two distinct ServiceMix container instances configured in a master/slave configuration. In ALL cases, the master is ACTIVE and the slave is in STANDBY mode waiting for a failover event to trigger it to take over.
Info | ||
---|---|---|
| ||
ACTIVE - components are listening for and servicing requests STANDBY - administrative ports may be open (e.g. JMX) but the process is not listening for or servicing requests |
As ServiceMix leverages ActiveMQ to provide HA functionality, there are a few choices in terms of how to configure master/slave via ActiveMQ:
- Shared Filesystem Master-Slave - With shared database master/slave two containers use the same physical data store for container state. Care should be taken to ensure that the file system supports file level locking as this is the mechanism used to elect the master. If the master process exits, the database lock is released and the slave acquires it and it then becomes the master.
- JDBC Master-Slave - In JDBC master/slave, the master will lock a table in the backend database. The failover event in this case is that the lock is released from the database.
- Pure Master-Slave - With pure master/slave either a shared database or a shared filesystem can be used. The master replicates all state changes to the slave so there is additional overhead incurred. The failover trigger in pure master/slave is that the slave loses its network connection to its master. Due to the additional overhead and maintenance involved, options 1) and 2) above are more desirable.
(See An Introduction to ActiveMQ Master/Slave for more information)
Clustering in ServiceMix
Within the scope of this document, a cluster can be defined as two or more ACTIVE ServiceMix container instances that are networked together. A cluster provides the following benefits:
- Connectivity to other ServiceMix containers forming a network of containers. Containers can be dynamically added and removed from the network.
- Remote component awareness where each container is aware of its peer container's components. Networked containers listen for remote component registration/deregistration events and once aware can route requests to those components.
- Load balancing where a cluster provides the ability to balance requests in a couple of ways including:
- Installation of the same component on multiple containers to provide increased capacity and additional HA capabilities. If one container fails the same component in another container can still service the request.
- Partition workload among container instances whereby different containers can handle different tasks and flows can span multiple containers.
Example: Configuring HA
When learning about ServiceMix HA configuration, a good starting point is to configure HA on a single host. In this scenario lets assume we have container1_host1(master) and container2_host1(slave).
To configure our example using a shared filesystem master/slave", the steps involved including the following:
1) Install two ServiceMix instances (servicemix1 & servicemix2).
2) Ensure <servicemix2> is an exact replica of <servicemix1>. The exact same files should be installed in the <servicemix2>/hotdeploy
directory and <servicemix2>/conf
.
3) Edit the <servicemix1>/conf/activemq.xml
file and configure the ActiveMQ peristence adapter:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<!-- We use 0.0.0.0 per AMQ-2094 -->
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616"/>
</amq:transportConnectors>
<amq:persistenceAdapter>
<amq:amqPersistenceAdapter directory="file:<shared>/data/amq"/>
</amq:persistenceAdapter>
|
4) Edit <servicemix2>/conf/activemq.xml
and verify the ActiveMQ peristence adapter configuration.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<amq:transportConnectors>
<amq:transportConnector uri="tcp://0.0.0.0:61616"/>
</amq:transportConnectors>
<amq:persistenceAdapter>
<amq:amqPersistenceAdapter directory="file:<shared>/data/amq"/>
</amq:persistenceAdapter>
|
Info | |||||||||
---|---|---|---|---|---|---|---|---|---|
| |||||||||
|
5) When using the same host you MAY need to change the JMX ports to avoid port conflicts by editing <servicemix2>/conf/servicemix.properties
file to change the rmi.port
.
Panel |
---|
rmi.port = 1098 |
How to Test the Configuration
Below are the steps to testing the configuration:
1) Start a ServiceMix instance by running ServiceMix in the root of the install directory.
servicemix1_home $ ./bin/servicemix
Info |
---|
As this instance is started first it acquires a lock on the persistent store and becomes the master |
2) Start the slave container
servicemix2_home $ ./bin/servicemix
Info |
---|
You should see that the startup for the slave is different to the master. It waits to acquire a lock on the shared database. |
3) Ctrl-c the master instance. You should see the slave fail over.
This configuration has some advantages and some disadvantages:
Advantages:
1) You don't need to configure a failover URL for your clients
Disadvantages:
1) Not resilient to a hardware failure unless you are using a third party clustering software.
Extending the HA Example to Separate Hosts
This example can be extended such that the master and slave are on separate hosts. Both the shared filesystem master/slave and the JDBC master/slave approaches still have the requirement that both instances access the SAME presistent store. With shared filesystem master/slave, this can be provided via a SAN and for JDBC master/slave both instances need to be able to access the same database instance.
You pay for this flexibility with additional configuration overhead. Specifically you may need to introduce failover URLs into your deployed component configurations.
Example:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
failover://(tcp://host_servicemix1:61616,tcp://host_servicemix2:61616)?initialReconnectDelay=100
|
You may also need to provide your clients with both HA host addresses (or insulate clients from this details via a hardware load balancer). Clients need to be aware that only one instance will be ACTIVE at a time.
See the ActiveMQ Failover Transport Reference for more details on failover URLs.
Clustering Configuration
To create a network of ServiceMix containers, you need to establish network connections between each of the containers in the network. So if we extend the HA example above and add two more container processes such that we have:
Panel |
---|
smix1_host1(master) and smix2_host1(slave) |
Then we need to establish a network connection between the two active containers (namely smix1_host1 and smix1_host2).
As Servicemix leverages ActiveMQ to provide clustering functionality, you have a few choices in terms of how you configure the *NetworkConnector
*s:
1) Static Network Connectors
With a static configuration, each NetworkConnector
in the cluster is wired up explicitly via
<servicemix>/conf/activemq.xml
.
Here's an example of a static discovery configuration:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<!-- We are setting our brokerName to be unique for this container -->
<amq:broker id="broker" brokerName="host1_broker1" depends-on="jmxServer">
....
<networkConnectors>
<networkConnector name="host1_to_host2" uri="static://(tcp://host2:61616)"/>
<!-- Here's what it would like for a three container network -->
<!-- (Note its not necessary to list our own hostname in the uri list) -->
<!-- networkConnector name="host1_to_host2_host3" uri="static://(tcp://host2:61616,tcp://host3:61616)"/ -->
</networkConnectors>
</amq:broker>
|
Info | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
Edit the
|
2) Multicast
If you have multicast enabled on your network and you have multicast configured in the <servicemix>/conf/activemq.xml
, then when the two containers are started they should detect each other and transparently connect to one another.
Here's an example of a multicast discovery configuration:
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
<networkConnectors>
<!-- by default just auto discover the other brokers -->
<networkConnector name="default-nc" uri="multicast://default"/>
</networkConnectors>
|
Info | ||
---|---|---|
| ||
|
Diagram
Below is an example of a remote flow in a clustered configuration.
...
Panel | ||||||
---|---|---|---|---|---|---|
| ||||||