Enabling Storage XenMotion for XenServer

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-659

Branch

master, 4.2.0

Introduction

XenServer introduced support for Storage XenMotion in the latest version (6.1). Storage XenMotion allows VMs to be moved from one host to another, where the VMs are not located on storage shared between the two hosts. It provides the option to live migrate a VM's disks along with the VM itself. It is now possible to migrate a VM from one resource pool to another, or to migrate a VM whose disks are on local storage, or even to migrate a VM's disks from one storage repository to another, all while the VM is running.

CloudStack currently allows live migration of a virtual machine from one host to another only within a cluster. The intend of this feature is enable support for live migration of VMs across hosts that do not share storage. It also enables migration of VMs disk from one storage repository to another, but the VM stays on the same host.

Purpose

This document describes the specifications and design of the feature.

References

http://blogs.citrix.com/2012/08/24/storage_xenmotion/

Document History

Glossary

VM - Virtual Machine.
VDI - Virtual disk image.
SR - Storage repository http://docs.vmd.citrix.com/XenServer/4.1.0/1.0/en_gb/reference.html#id2550841.

Feature Specification

Summary of the what storage xenmotion will allow is given below.

Test Guidelines

1. Storage XenMotion is only supported on XenServer 6.1. Live migrate a virtual machine running on cluster of XenServer 6.1 servers to another cluster running XenServer 6.1 servers should be allowed.
2. Live migration of a virtual machine using local storage should be allowed to another host within the cluster or another cluster.
3. It should be possible to move the disks of a virtual machine from one storage pool to another storage pool within the cluster while the virtual machine stays on the same host and is always running.
4. All virtual disks are created with either a shared or a local disk offering. During storage xenmotion, a disk created on a shared storage pool should move to a shared pool. Similarly, a disk created on a local storage pool should move only to local storage pool.

Hypervisor support

Support for storage xenmotion on XenServer 6.1 will be enabled by this feature.

Negative usage scenarios

• Storage XenMotion involves moving the virtual disks of the VM and it takes time. While a storage xenmotion is in progress, operations like stopping, destroying a virtual machine will not be allowed.
• If storage xenmotion of a VM fails, the resources consumed on the destination host/cluster should be cleaned.

Supportability characteristics

Logging

Storage XenMotion logs all the successful operations to INFO, all exceptions/failures to ERROR, and all synchronization checks to DEBUG.

Debugging/Monitoring

In addition to looking at the management server logs, administrators can look up the xenserver logs for analysis

Use cases

It allows the system administrator to relocate the VMs Virtual Disk Image without shutting down the VM.

• A system administrator can move a virtual machine from cheap local storage to fast, resilient, array-backed storage.
• A system administrator can move the virtual disks of a VM between tiers of storage when it is limited by storage capacity.
• It enables system administrators to re-balance or move virtual machine between XenServer pools. For example, promoting a virtual machine from a development environment to a production environment.
• It enables system administrators to perform software maintenance. For example upgrading or updating standalone XenServer hosts without VM downtime.
• It enables system administrators to perform hardware maintenance. For example upgrading standalone XenServer host hardware without VM downtime.

Architecture and Design description

There are two primary scenarios that need to be supported.
Volume of an instance is migrated from one storage pool to another and the instance continues to run on the same host
To migrate a volume of a running instance, the administrator should be able to list the storage pools available for migration and initiate volume migration.

1. List storage pools available for migration of a volume
   1. For listing storage pools to which a given volume can be migrated a new api, findStoragePoolsForMigration will be added. The api will take volume id as a parameter and return a list of storage pools.
   2. It'll check if the volume is in the right state (Ready) and is attached to a running instance. The instance should be running on a host that supports storage motion.
   3. If a volume used local storage pool, it cannot be migrated and an empty list of storage pools is returned.
   4. It'll enumerate and call into the storage pool allocators to get the list of pools available based on the disk offering with which the volume was created. Note: no changes in storage pool allocators is expected.
   5. The list of storage pools available for migration of a volume is returned. The storage pools that have sufficient capacity and appropriate tags will be flagged as suitable for migration.
2. Migrate volume of a running instance to a given storage pool
   The existing migrateVolume api will be updated to allow migration of a volume, root or datadisk, attached to a running instance.
   1. Only Root Admin can issue a migrate volume api request.
   2. The volume should be in 'Ready' state and attached to an instance running on a host that supports storage motion.
   3. livemigrate parameter should be set to 'true' for migrating a volume while it is attached to an instance. If it isn't passed, the api call will fail.
   4. The volume is moved to 'Migrating' state.
   5. A 'MigrateVolumeCommand' is send to the resource for migrating the volume from on storage pool to another. Details of the destination storage pool are captured in the command.
   6. The resource sends a VDI.pool_migrate request to the host and waits for the operation to complete. Success or failure of the task is returned as an answer for the command.
   7. On success the db is updated with the new location of the volume and the volume is moved back to 'Ready' state.
   8. On failure the volume is updated to 'Ready' state and operation is failed.

Migration of an instance from one host to another and the two hosts do no share storage
To migrate an instance from one host ot another when they do not share storage, the administrator should be able to list the hosts available and issue a request for migration

1. List hosts available for migration of a instance
   1. findHostsForMigration api will be added to list the hosts available for migration of a given virtual machine.
   2. If the virtual machine is running on a host that doesn't support storage motion, only the hosts within the cluster will be listed. The hosts that have sufficient capacity and appropriate tags will be flagged as suitable in the response.
   3. If the virtual machine is running on a host that does support storage motion, the hosts outside the cluster will also be listed.
   4. The hosts that are listed; for migration of a vm with its volumes; should also support storage motion and running the same version of hypervisor.
   5. They should have storage pools attached to them so that a request for moving all the volumes of an instance can be satisfied. e.g if the instance has one volume on shared storage pool and one volume on local pool; the destination host should also have a shared and a local pool connected to it.
   6. The list of hosts available for migration of an instance is returned. The hosts that have appropriate tags and capacity are flagged as suitable in the response.
2. Migrate an instance
   If a request is placed to migrate a vm from one host to another (within or across cluster) and it requires storage motion
   1. Check if the source and destination host are running a hypervisor that supports storage migration.
   2. Get the list of storage pools that are available on the destination.
   3. Check if the disk offerings the virtual machine uses are shared or local and whether there are storage pools available on the destination for the same. If not fail the migration request.
   4. Set the state of the virtual machine and its disks to Migrating.
   5. Send a MigrateWithStorage command to the destination (host) agent. The command parameters has the list of storage pools that can be used for migrating disks of the virtual machine.
      1. The agent issues a Host.migrateReceive command to the destination host.
      2. The agent then opens a connection to the source host. Open Issues 1.
      3. The agent maps which vdis of the VM should go on to which SRs on the destination host and issues a Host.migrateSend command to the source host.
      4. Success or failure of the command is returned.

Integration with the storage framework

1. Migrate volume. Diagram below shows the integration with the storage framework.
   1. Volume service gets a request for migrating a volume to another storage pool.
   2. It dispatches a call (copyAsync) to the data motion service for migrating the volume after registering a completion callback.
   3. Data motion service checks which data motion strategy can handle migration of a volume to another pool.
   4. Data motion strategy (AncientDataMotionStrategy) issues a request to the hypervisor resource to migrate the volume to a given storage pool. AncientDataMotionStrategy::copyAsync will be updated to handle migration of a volume.
2. Migrate virtual machine with its volumes.
   1. Virtual machine manager puts the vm to be migrated in 'Migrating' state and calls the volume service passing a volume to data store map. The map tells which volume of the vm needs to be migrated to which storage pool.
   2. Volume service puts the volumes to be migrated in 'Migrating' state and calls the data motion service.
   3. Data motion service checks which data motion strategy can handle migration of a vm and its volume. A new data motion strategy 'XenServerStorageMotionStrategy' will be added for the same.
   4. XenServerStorageMotionStrategy issues a request to the hypervisor resource to migrate the vm with its volumes.

Database modifications

• hypervisor_capabilities table will updated to have a new column for storage_migration_supported. The default value for the column will be 0 (not supported).
• The entry for XenServer version 6.1.0 will be updated to hold a value of 1 for the column storage_migration_supported.

Web Services APIs

1. findHostsForMigration - This will be a new api that'll be added to list hosts to which a virtual machine can be migrated. It'll check if the virtual machine is running on a host that supports storage migration. If so, it'll also return the list of hosts, both within and from other clusters, to which the virtual machine can be migrated with its storage. Otherwise, it'll only return the list of hosts within the cluster. The hosts which have sufficient capacity and which have appropriate tags will be marked as suitable in the response object. Open issue 2.

   Parameter	Type	Optional/Required	Comments
   virtualmachineid	Uuid	Required	Id of the virtual machine to be migrated.

   
   

   Response Object

   Comment

   HostForMigrationResponse

   This will be a new response object. It will contain the details of the host which has been found available for migration. It will also contain two additional attributes, namely suitableformigration and requiresstoragemotion. 'suitableformigration' will be true if the host has enough capacity and the right tags on it. 'requiresstoragemotion' will be true if migrating the given virtual machine to it will require live migrating the volumes of the vm too. Open issue 3

   
   
2. migrateVirtualMachineWithVolume - This will be a new api that'll be added to allow migration of a virtual machine with its volume from one host to another. The behavior of the API will be as follows.
   • If migration of the virtual machine to the given host doesn't require storage motion, the api will fail.
   • migrateto is an optional parameter which tells which volume of the vm should be migrated to which storage pool. If the parameter isn't given, cloudstack management server will itself make a decision and pick an appropriate storage pool connected to the destination host for migrating the volumes of the virtual machine.
   • If migrateto is passed as a parameter, a mapping should be provided for every volume attached to the virtual machine being provided. Otherwise the api will fail.

     Parameter	Type	Optional/Required	Comments
     virtualmachineid	Uuid	Required	The virtual machine to be migrated
     hostid	Uuid	Required	The host to which the vm should be migrated
     migrateto	Map	Optional	A map of volume id to storage pool id. Specifies which volume of the vm should go on which storage pool.

     
     
3. findStoragePoolsForMigration - This is a new api that'll be added. It'll take volume id as a required parameter and return the list of storage pools to which the volume can be live migrated. The pools which have the appropriate tags on them and have enough space will be marked as suitable in the response object.

   Parameter	Type	Optional/Required	Comments
   id	Uuid	Required	Id of the volume to be migrated

   
   

   Response Object	Comment
   StoragePoolForMigrationResponse	The will be a new response object that will be returned by the api. It will contains the details of the storagepool that were found to be available for migration. It will also contain suitableformigration property. If true, the pool is suitable for storage motion of the given volume.

   
   
4. migrateVolume - Currently the api only allows moving a datadisk, which isn't attached to any instance, from one storage pool to another. This api will be updated to allow migration of a volume, root or datadisk, attached to a running instance.

   Parameter	Type	Optional/Required	Comments
   livemigrate	Boolean	Optional	This is a new parameter that'll be added to the api. If true, the volume will be live migrated to the given storage pool.

UI Flow

• Changes required in UI for live migrating a volume from one storage pool to another.
  • UI needs to be updated to allow migration of a volume even if it is attached to a running vm.
  • UI will be updated to make a findStoragePoolsForMigration call for volumes; root or datadisk; to get the list of storage pools available.
  • For live migrating a volume, migrateVolume call will be made with volumeid, storageid passed as a parameter. The livemigrate parameter should also be set to true.
• Changes required in UI for live migrating an instance along with its volume.
  • Currently UI makes a listHosts api call with virtualmachineid as a parameter to get the list of hosts to which a vm can be migrated.
  • UI will be instead updated to make a findHostsForMigration call.
  • If migration of an instance to a host requires storage motion; as indicated in the response object by the requiresStorageMotion field; UI will make a migrateVirtualMachineWithVolume api call. The virtual machine id and host id are passed as the parameters.

Open Issues

1. CloudStack keeps a connection pool per cluster. For storage xenmotion, agent needs to send a migrateReceive command first to the destination host and then a migrateSend command to the source host. For the entire operation, the session that was created during migrationReceive command needs to be kept active. If vm migration is across cluster, it would mean the agent for destination host will have to open a connection to the source host (on another cluster) and execute the migrateSend command. Is it the right approach? Proposal: Cloudstack will go ahead and send two commands, migrate receive to the destination followed by migrate send to the source. it will not open a connection from source (agent) to destination.
2. listHosts api is currently also used to list the hosts to which a vm may be migrated if vmid is passed as the parameter. It is probably not the right way to do it. findHostsForMigration is the new api that is being added. Should the ability the list the hosts for migration be removed from the listHosts api or kept in place for backward compatibility? Proposal: listHosts functionality has also been kept intact for backward compatibility.
3. Should the response object returned by findHostsForMigration be HostReponse or a new response object created for the same? Proposal: findHostsForMigration will return the HostResponse a new response object HostForMigrationResponse.
4. If HostResponse object is used as a host response object for findHostsForMigration, is it ok to add a 'requiresStorageMotion' field in the object. Proposal: a new 'requiresStorageMotion' field will be added to the response object A new response object HostForMigration will be returned by the api. It will contain the requiresstoragemotion field too.
5. Similarly StoragePoolResponse object is returned by the findStoragePoolsForMigration response. Is it ok to add a 'suitableForMigration' parameter in the object. Proposal: a new 'suitableForMigration' field will be added to the response object a new response object 'StoragePoolForMigrationResponse' will be returned by the api. It'll contain the 'suitableformigration' property.
6. In the findHostsForMigration api, if a virtual machine can be migrated with storage to another cluster; hosts in other clusters that are compatible will be listed. To get a list of suitable hosts from this list, host allocators need to be invoked to check for the rights tags and capacity. A new allocateto routine in host allocators, that takes a list of hosts and checks within them if they are suitable, will be added. Is it the right approach. Proposal: a new allocateto routine will be added to the host allocators that will take a list of hosts and will check within them if they are suitable.

VMware Resource Support

1. VmwareStorageMotionStrategy is introduced to support storage motion work specific for Vmware resources in CloudStack deployments.
2. MigrateVolumeCommand is implemented in Vmware resource to support volume migration across datastores with VM continuing to run on same host
3. MigrateWithStorageCommand is implemented in Vmware resource to support migration of VM along with it's storage within as well as across clusters in a cloudstack zone.
4. VMware doesn't support storage migration between 2 datastores unless both datastores are accessible to source host. This means storage migration is not support from a local datastore to another local datastore unless both local datastores are attached to same host where the VM is running.