Introduction

Currently Cloud Stack has only one active master DB setup. Going forward it should support multiple Master/Slave cluster setup to enable High Availability of the Database.

Scope

To Achieve Active/Active cluster set up (Multiple masters and slaves).

Options Available

1. MariaDB
2. Percona Xtra DB (uses Galera cluster)
3. MHA provided by SkySql
4. Mysql Active/Active setup with connector specific configuration to read from slaves in case of master goes down

Results

MariaDB

• It does not support any DB HA out of the box, we can use Galera with it but the same can be used with Mysql directly

Percona Xtra DB

• It has the following limitations
  http://www.percona.com/doc/percona-xtradb-cluster/limitation.html

MHA

• It is very hard to install this and also need to write our own scripts to switch between master and slave either using VIP concept or some other way of LB

Mysql Active/Active Setup

We can use Mysql's 2-way replication (Master-Master replication setup) along with connector's configuration to read/write data from one of the slave if master goes down.

• It worked well with 2 nodes(master - master) set up for both fail over and fall back
• Need to verify the same for more than 2 nodes set up using chain replication with first 2 nodes as cyclic pointers for master/slaves and remaining are chain replication strategy starting from 2nd node.
• Also need to solve Communication link failure exception that is coming when master is going down.
• Admin guide steps to do the chain way of replication for multi node setup
• The connection will be reset if auto commit is true other wise not

Conclusion

• We are good to go with this approach.

Changes in Code Base (wrt mysql's active/active setup)

• Need to add extra properties related to connector parameters in db.properties
• Code changes are needed in com.cloud.utils.db.Transaction class to add the new properties to the url construction of cloud databases.

Proposed Solution

• Added the following properties to db.properties
  

#High Availability And Cluster Properties
db.ha.enabled=false (change it to true to enable db ha)
db.ha.loadBalanceStrategy=com.cloud.utils.db.StaticStrategy

#cloud stack Database
db.cloud.slaves=localhost,localhost (Comma separated list of slave hosts)
db.cloud.autoReconnect=true
db.cloud.failOverReadOnly=false
db.cloud.reconnectAtTxEnd=true
db.cloud.autoReconnectForPools=true
db.cloud.secondsBeforeRetryMaster=3600
db.cloud.queriesBeforeRetryMaster=5000
db.cloud.initialTimeout=3600

#usage Database
db.usage.slaves=localhost,localhost (Comma separated list of slave hosts)
db.usage.autoReconnect=true
db.usage.failOverReadOnly=false
db.usage.reconnectAtTxEnd=true
db.usage.autoReconnectForPools=true
db.usage.secondsBeforeRetryMaster=3600
db.usage.queriesBeforeRetryMaster=5000
db.usage.initialTimeout=3600

• We will rely on Mysql's Connector properties to achieve Master-Master DB HA
  • Look at the following link to know the parameters used to achieve HA using mysql connector
    http://dev.mysql.com/doc/refman/5.0/en/connector-j-reference-configuration-properties.html
• We need to have at least 2 nodes which are connected through 2-way replication and should be consecutive in connector's url
  • look at the following urlss to setup Mysql replication
    http://dev.mysql.com/doc/refman/4.1/en/replication-howto.html
    https://wikis.oracle.com/display/CommSuite/MySQL+High+Availability+and+Replication+Information+For+Calendar+Server
    

Limitations/Things DBHA is not supported

• Currently Monitoring of Slave by admin is not possible
• Monitoring events are not integrated with MS

Appendix

• We can achieve multiple nodes set up(2 master max and remaining would be slaves) with chain way of replication configuration
  

another way is circular chaining where all can act as master

• this is tested against mysql version 5.5.x

• Currently started testing against 5.1.x