Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following is a flow chart for how HighAvailabilityManager works through its process.

Image Removed

Investigator

The need for Investigator rises from the fact that the technologies for the environments CloudStack is deployed into change rapidly. So depending on what new technology is available, Investigators can be written quickly and plugged into the HA process to help determine if a VM is Running or Stopped. The following are keys to the implementation of an Investigator.

...

As part of the CloudStack HA design, CloudStack HA must not restart a VM if it cannot be sure that it can be done safely.  In those cases, the VM stays in a transitional state.  In those cases, the system administrator can manually intervene by checking to make sure the VM or the host the VM is shutdown, and then calling CloudStack to stop the VM with forced stop flag set to true.  When called in this manner, CloudStack attempts to stop the VM but will continue to release resources even if any of the operations to the hypervisor host cannot be completed.  After doing this, CloudStack HA will stop any attempt to restart the VM and it can be restarted by the administrator. 

Putting it together

Image Added Image Added
The above sequence diagram shows the normal application ping and storage heartbeat being sent by the hypervisor 1 and 2. At some point, the application ping falls behind and the monitor informs CloudStack that the application ping has fallen behind. CloudStack then launches an investigation. On the first timeout, CloudStack was still able to contact the host so it leaves the host's status unchanged. On the second timeout, CloudStack was no longer able to contact the host, so it asks the investigator to check on the host. The first investigator, a PingTestInvestigator, was inconclusive because it timed out on the network ping. The second investigator, the XenServerInvestigator, finds that the storage heartbeat was no longer being written to. At this stage CloudStack hands off to HAManager to HA all of the VMs.Sequence Diagram