Page History

...

CloudStack HA checks for failures in the hypervisor host.   Upon detecting a hypervisor host failure, VMs on that specific hypervisor host is restarted.   This detection must distinguish between a connection lost to a hypervisor host and an actual failure of a hypervisor host.   Differences between the capabilities of hypervisor types and even versions within the same hypervisor type can cause problems in proper detection. 

CloudStack HA achieves this by performing an application ping between CloudStack and the hypervisor host.   When the application ping falls behind or is unable to be performed, CloudStack sends a CheckHealthCommand to the hypervisor host.   If this command cannot be performed or takes a long time to perform, CloudStack HA then invokes host investigators to check on the status of the hypervisor host in question.

Currently, two types of host investigators have been implemented: network ping investigation and hypervisor specific investigation.   Network ping investigation sends a PingTestCommand to a neighboring hypervisor host which carries out a network ping and network arping on the IP address of the hypervisor host in question. If the hypervisor host responds to the network ping or arping, the host is consider to be alive and no HA is performed.  If there is no response, the investigator returns that it cannot detect the status of the host. Note that it cannot return that the host is down because network ping and arping responses may be blocked and does not accurately indicate that the host is down.

If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation.   For VMware, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage.  If the timestamp cannot be written, the hypervisor host self-fences by rebooting itself.  For these two hypervisors, CloudStack HA sends a CheckOnHostCommand to a neighboring hypervisor host that shares the same storage. The neighbor then checks on the heartbeat file on shared storage and see if the heartbeat is no longer being written.   If the heartbeat is still being written, the host reports that the host in question is still alive. If the heartbeat file’s timestamp is lagging behind, after an acceptable timeout value, the host reports that the host in question is down and HA is started on the VMs on that host.

...