Out-of-band Management for CloudStack

• Restarting stalled/failed hosts
• Powering off under-utilised hosts
• Powering on hosts for provisioning and to increase capacity (for hosts already provisioned)
• Allowing system administrators to see the current power state of the host
  
  

Feature Specifications

Given the wide range of system management interfaces (e.g. IPMI, iLO, DRAC etc.), the power management service will separate the general power management functions from the implementation using plugins. This service uses a FSM (finite state machine) to transition a host's out-of-band management power state, following are the valid out-of-band management  power state of host:

• On: The host is powered on
• Off: The host is powered off
• Unknown: The host's power status cannot be determined
• Disabled: Out-of-band management is not enabled for the host

The FSM defaults to UNKNOWN and transitions based on power state retrieved. If the power status can not be retrieved (e.g. no provider available, not configured, unable to connect to the system management interface), the state remains UNKNOWN. The power management service will provide to the following operations to affect changes to this state machine: 

• ON: hard power on a host resulting in the ON state
• OFF: hard power off a host resulting in the OFF state
• CYCLE: hard power off and power of a host resulting in the ON State
• RESET: warm boot of a host resulting in the ON state
• SOFT: soft shutdown via ACPI of a host resulting in the OFF state

The Out-of-band Management service also needs to account for the following failure scenarios:

• The power status of the host is changed outside of CloudStack (e.g. a datacenter technician manually powers down a host)
• Connectivity to the host's system management interface is lost
• The credentials of the out-of-band management were changed without updating them in CloudStack, notify admin via alerts/email

The Management Server will regularly poll the system management interface of each host to refresh power state and verify credentials. A background sync thread will schedule STATUS check jobs and update out-of-band management host power state using a FSM. Each out-of-band management will owned by a management server who will be responsible for performing sync checks on them. For example, in case of multiple management server environment, all of them will share loads of performing such background sync checks. To protect against slow responses from the out-of-band management interface, all out-of-band management operations should be bounded by a timeout. The timeout will be configured globally with a per-cluster override. Finally, the power management FSM state transitions work as follows:

• ON->OFF: The out-of-band management  power state sync daemon detects that the host was turned off
• OFF->ON: The out-of-band management power state sync daemon detects that the host was turned on
• ON→UNKNOWN, OFF->UNKNOWN: The out-of-band management  power state sync daemon loses connectivity to the system management controller
• UNKNOWN->OFF, UNKNOWN->ON: The power state sync daemon gains connectivity to the system management controller
• *Any -> DISABLED: The out-of-band management is disabled for a host
• DISABLED→UNKNOWN: When out-of-band management is enabled from a disabled state for a host
• User triggered power management action
• All changes to a host's out-of-band power state would be broadcast on the event bus

The finite state machine is a state machine that governs how states change based on an action or event. The out-of-band management power state FSM is described in the following diagram:

Implementation

The Power Management Service will provide the following new root admin APIs:

• changeOutOfBandManagementPassword: Changes the password system management interface password on the host and in the Management Server database. If the change the password fails on either the host or in the database, the password will be set to the old password in both the Management Server database and on the host.
• configureOutOfBandManagement: Add or update a configuration about a host’s system management interface (e.g. host/ip address, port, credentials, driver, etc.)
• enableOutOfBandManagement/disableOutOfBandManagement: Toggle whether or not CloudStack should use the configured out-of-band management
• issueOutOfBandManagementPowerAction: Initiates the specified power management function (see above) to the system management interface. This API will support an optional operation timeout which would override the global and cluster-level settings.

The out-of-band management driver could be implemented in order to support any of the popular variants. In general, most servers have a IPMI 2.0 supported on their out-of-band management interface so the feature will include a ipmitool based driver. No Apache license-compatible reliable Java library was available so this driver shells out to run ipmitool with some arguments.

The following diagram shows high level design and architecture of out-of-band management for CloudStack:

Global settings

The outofbandmanagement.ipmitool.* settings are specific to the ipmitool driver. Changing these values don't require restarting management server, update values are picked up dynamically.

The outofbandmanagement.ipmitool.interface is the interface option that will be used while running the ipmitool driver (with a -I <the value provided to this option>). The default value implies to have ipmitool use the lanplus interface which is IPMI 2.0. The outofbandmanagement.ipmitool.retries option is the number of retries for each command execution (this is used with a -R <whatever values you provide> while running ipmitool). For more details run ipmitool -h or readhttp://linux.die.net/man/1/ipmitool. 

For any IPMI 2.0 compliant out-of-band management interface, these options/values should work out of the box. For example, on both iDRAC and iLO with IPMI2.0 enabled the feature should work out of box.

The outofbandmanagement.sync.interval is the amount of time after which the background thread starts a new thread to scan available hosts with out-of-band management enabled and tries to find and update power state of the hosts. Changing this setting will require restarting the management server as the threadpool is only initialized when mgmt server starts.

The outofbandmanagement.sync.poolsize is the number of max. no. of ipmitool background power state scanners that can run at a time. Based on the maximum no. of hosts you've, you can increase/decrease the value depending on how much stress your management server host can endure. For example, assuming an environment with the default global setting and say 5000 hosts will take at most (number of hosts * action.timeout / sync.poolsize) seconds or (5000 * 60 / 50)s, or 5000 seconds to complete on background sync scan per round. To ensure no overlapping scan attempts you should set the outofbandmanagement.sync.interval to 5000 or increase the sync.poolsize. Changing this setting will require restarting the management server as the threadpool is only initialized when mgmt server starts.

UI

A. Power state icons:

• Green: On state
• Red: Off state
• Orange: Unknown state
• Grey: Disable state

B. Out-of-band management power state in Host view and Host metrics view:

C. Out-of-band management power state in Host metrics view:

D. Out-of-band management operation icons and information tab on Host view page:

E. Out-of-band management configuration dialog box:

F. Out-of-band management issue power action dialog:

G: Out-of-band management change password dialog:

Testing

For testing the feature, a ipmi simulator can be used that is purely Python based: https://pypi.python.org/pypi/ipmisim

`ipmisim` was further used as an importable library to write Marvin tests. To run out-of-band management test:

$ nosetests --with-xunit --xunit-file=integration-test-results.xml --with-marvin --marvin-config=<cfg file> -s -a tags=advanced,required_hardware=true --zone=LangurCloud-basic --hypervisor=KVM  test/integration/component/test_outofbandmanagement.py