Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

CloudStack HA contains three different phases: Detection, Restart, and Manual Intervention.   Detection deals with different ways to detect the possibilities of VM and hypervisor host failure.  Restart is the actual process to restart the failed VMs.  System administrators can manually intervene when Restart is unable to safely restart a VM.

...

If the network ping investigation returns that it cannot detect the status of the host, CloudStack HA then relies on the hypervisor specific investigation. For VMware, there is no such investigation as the hypervisor host handles its own HA. For XenServer and KVM, CloudStack HA deploys a monitoring script that writes the current timestamp on to a heartbeat file on shared storage.  If the timestamp cannot be written, the hypervisor host self-fences by rebooting itself.   For these two hypervisors, CloudStack HA sends a CheckOnHostCommand to a neighboring hypervisor host that shares the same storage. The neighbor then checks on the heartbeat file on shared storage and see if the heartbeat is no longer being written. If the heartbeat is still being written, the host reports that the host in question is still alive. If the heartbeat file’s timestamp is lagging behind, after an acceptable timeout value, the host reports that the host in question is down and HA is started on the VMs on that host.

...

As part of the CloudStack HA design, CloudStack HA must not restart a VM if it cannot be sure that it can be done safely.  In those cases, the VM stays in a transitional state.  In those cases, the system administrator can manually intervene by checking to make sure the VM or the host the VM is shutdown, and then calling CloudStack to stop the VM with forced stop flag set to true.  When called in this manner, CloudStack attempts to stop the VM but will continue to release resources even if any of the operations to the hypervisor host cannot be completed.  After After doing this, CloudStack HA will stop any attempt to restart the VM and it can be restarted by the administrator. 

...