Storage motion for Hyper-V

Bug Reference

CLOUDSTACK-6143

Branch

4.4, master

Introduction

Hyper-V 2012 R2 allows migration of volumes (virtual disks) of a virtual machine from one storage to another, while the virtual machine continues to run. It also allows live migration of a virtual machine and its volumes to another host and storage without any downtime.

The intend of this feature is to enable support of live migration of a virtual machines with its volumes across hosts and storage pools. It'll also migration of volumes across storage pools while the volume stays attached to a running virtual machine.

Purpose

This is a feature specification for supporting storage motion on Hyper-V.

References

• Storage motion in cloudstack
• http://blogs.technet.com/b/canitpro/archive/2013/03/21/step-by-step-completing-storage-live-migration-in-hyper-v.aspx
• http://msdn.microsoft.com/en-us/library/hh850079(v=vs.85).aspx

Document History

Glossary

Instance/vm - Virtual machine running on a hypervisor. The terms may be used interchangeably.

Live migration - Migrating a vm from one host to another without any downtime of the vm,

Volume live migration - Migrating a volume of a vm from one storage pool to another without any downtime of the vm.

Feature Specifications

Scenarios

Storage motion operations can only carried out by a CloudStack root administrator. Summary of the operations that storage motion will allow on Hyper-V

• Scenario 1: Live migrating volume of a virtual machine. Virtual machine stays running on the same host.
  
  • A volume deployed on a local storage cannot be migrated to another storage pool. The volume has been created with a 'Disk Offering' as local. It can only reside on a local storage pool. Since the volume is attached to a running vm and is on a local storage it cannot be live migrated to another local storage pool on another host.

  • A volume on a cluster wide shared storage can be migrated to a another cluster wide shared storage within the same CloudStack cluster.
    
    Following table summarizes the scenario

    	Destination Storage type = Local	Destination Storage type = Shared
    Source Storage type = Local	No	No
    Source Storage type = Shared	No	Yes

• Scenario 2: Live migrating a virtual machine with its volumes to another host and storage pool respectively.
  • A vm with its volumes on a local storage storage can be live migrated to another host which has local storage attached. The volume of the vm will be placed on the local storage on the destination host. The host may be within the cluster or in another cluster.
  • A vm with its volumes on a shared storage can be live migrated to another host which a shared storage pool attached to it. The host may be within the cluster or in another cluster.

  • A vm with volumes on both shared and local storage can be live migrated to another host if the destination host has access to both shared and local storage. Source volumes on shared storage will be migrated to shared storage accessible on the destination. Similarly, volumes on local storage will be migrated to local storage on the destination.
    
    Following table summarizes the scenario

    	Destination Storage type = Local	Destination Storage type = Shared
    Source Storage type = Local	Yes	No
    Source Storage type = Shared	No	Yes

    
    

• Unsupported Scenario: A volume on a local storage cannot be live migrated to a volume of shared storage. A volume on local storage is created by a disk offering of  type "Local". Migrating it to a shared storage will conflict with the disk offering. The same applies to not allowing migration of a volume from shared to local storage.

Test Guidelines

Functional

• For live migrating a volume from one storage pool to another, only the shared storage pools within the cluster should be listed.
• If storage tags were used in disk offerings, findStoragePoolsForMigration api should return the storage pools with matching tags as suitable. Other shared storage pools in the cluster should be marked unsuitable.
• Instance with one volume (only Root volume). Migrate volume of an instance from one shared storage pool to another within the cluster. At the end of the operation the volume should be on the destination pool and path updated in the db.
• Instance with multiple volumes (Root and Data volumes). Migrate one volume from one shared storage pool to another. At the end of the operation the volume should be on the destination pool and path updated in the db.
• Live migrating a virtual machine with its volumes, findHostsForMigration should return the list of hosts within the cluster and across cluster to which the vm can be migrated with volumes. The 'requiresStorageMotion' attribute should be set to true in the response.
• If host tags were used in service offering, findHostsForMigration api should return the hosts with matching tags as suitable. Other hosts should be marked as unsuitable.
• Migrate a vm from one host to another (migrateVirtualMachineWithVolume api) that do not have access to the same shared storage. Do not give a volume to pool mapping; that is, do not specify in the api which volume should be placed on which pool. The vm should be successfully live migrated with its volumes. The volumes created with shared disk offering should be placed on a shared storage pool and volumes with local disk offering placed on local storage pool.
• Validate that when a volume to pool mapping is given in migrateVirtualMachineWithVolume api, the volumes of a vm are placed on the given pool id.

Negative

• While migrating a virtual machine with a volume, gave an invalid volume to pool mapping; that is give a mapping for migrating a volume from shared to local or vice-verse. The operation should fail.
• Try live migrating a volume of a vm on a local storage. It should not be allowed.
• Live volume migration or live virtual machine migration with volume is a time consuming operation. When migration is going on, take the destination storage pool offline. Migration should fail and vm and its volume should be retained on the source host and storage pool respectively.

Support-ability Characteristics

Logs can be looked up at the following location

• Management server logs for decisions on which host and storage pools can be used. All the requests forwarded to the Hyper-V agent can also be seen here.
• The Hyper-V agent logs can be seen in windows event viewer on the respective hosts. The logs can be filtered by source "CloudStack Hyper-V Agent".
• Hyper-V specific logs can be viewed in the event viewer under "Application and Services Logs" >> Microsoft >> Windows >> "Hyper-V-VMMS".

Configuration Characteristics

• By default a Hyper-V host allows only 2 simultaneous storage migration requests. The configuration parameter may be configured on a hyper-v host to a higher or lower value; but it is recommended to not to do so.
• After a request is forwarded to a resource; the agent will wait for "wait" *2 (global config parameter) seconds for the request to complete. Storage motion can take a long time and may not complete in "wait" *2 seconds. The global config parameter "wait" can be configured if that is the case.

Users Communities

• Only a root admin can migrate a volume from one storage pool to another.
• Only a root admin can migration a virtual machine with its volume.

Supported Hypervisor

This feature is implemented for Windows Server 2012 R2 and Hyper-V 2012 R2 (Core).

Use cases

• Live migrate volumes of a virtual machine from one storage pool to another. It could be because
  • The storage pool has to be brought down for maintenance.
  • Move the volumes to a storage pool which gives better performance/IOs etc.
• Live migrating a virtual machine with its volume on local storage to another host within or across cluster. The administrator doesn't want to bring down the services running on that virtual machine and the host has to be brought down for maintenance.
• Live migrating a virtual machine with its volume on shared storage to another host in a different cluster.
  • The administrator doesn't want to bring down the services running on that virtual machine and the host on which the instance is running has to be brought down for maintenance. Additionally, the cluster is running low on available capacity.
  • Live migrating a virtual machine with its volume to other hosts in different cluster to better distribute the load across hosts/clusters.

Architecture and Design description

Please refer to storage motion in cloudstack for details on how storage motion is implemented. The details on what is required for enabling it for Hyper-V are given below.

Migrating volume

• Volume service gets a request for migrating a volume to another storage pool.
• It dispatches a call (copyAsync) to the data motion service for migrating the volume.
• Data motion service checks which data motion strategy can handle migration of a volume to another pool.
• Data motion strategy (AncientDataMotionStrategy) issues a request (MigrateVolumeCommand) to the hypervisor resource (HypervDirectConnectedResource) to migrate the volume to a given storage pool.
• HypervDirectConnectedResource serializes the request and forwards the command to the Hyper-V agent running on the host for migrating the volume.
• The Hyper-V agent handles the MigrateVolumeCommand command. The command holds the information on
  • Which volume has to be live migrated.
  • The vm to which the volume is attached.
  • The destination pool for the volume.
• The agent lists all the volumes of the vm and identifies which volume needs to be migrated. It sets the updated path; from pool details; for the volume and issues a wmi request for migrating the volume.
• On success or failure of the operation, a response is send to the management server.
• On receiving the answer from the agent; the data motion strategy checks for success or failure of the operation.
  • On success the volume path is updated and volume moved out of migrating state.
  • On failure, the volume is moved out of migrating state and the failure logged and reported.

Migrating virtual machine with volume

• Virtual machine manager puts the vm to be migrated in 'Migrating' state and calls the volume service passing a volume to data store map. The map tells which volume of the vm needs to be migrated to which storage pool.
• Volume service puts the volumes to be migrated in 'Migrating' state and calls the data motion service.
• Data motion service checks which data motion strategy can handle migration of a vm and its volume. 'HypervStorageMotionStrategy' (a new strategy that is implemented for the hypervisor) returns true that it can handle the migration request for the hypervisor and request is forwarded to it.
• HypervStorageMotionStrategy issues a request (MigrateWithStorageCommand) to the hypervisor resource (source host) to migrate the vm with its volumes. The resource forwards it to the Hyper-V agent.
• The Hyper-V agent handles the MigrateWithStorageCommand. The command holds the following information
  • The vm to be migrated.
  • The destination host for the vm.
  • Volume to pool mapping; that is; which volume of the vm needs to be placed on which destination pool.
• The agent sets the new volume path for each volume of a vm and issues a wmi request for migrating the vm with its volumes to the destination host.
• On success or failure of the operation, a response is send to the management server.
• On receiving the answer from the agent; the storage motion strategy checks for success or failure of the operation.
  • On success the volume path is updated and volume and vm moved out of migrating state.
  • On failure, the volume and vm is moved out of migrating state and the failure logged and reported.

Database modifications

In the hypervisor_capabilities table, the entry for Hyper-V 2012 R2 is updated to allow storage migration. In column, storage_migration_supported, the value is set to 1 to allow storage motion for Hyper-V.

Web Services APIs

No new apis are added for this feature. Existing apis; explained in Storage motion in cloudstack; will enable storage motion for Hyper-V too. List of apis used for storage motion

• findStoragePoolsForMigration. For listing the suitable storage pools to which a volume of a vm can be live migrated too.
• migrateVolume. For live migrating a volume of a vm to another storage pool. 'livemigrate' parameter to the api should be true.
• findHostsForMigration. For listing the suitable hosts to which a vm can be live migrated along with its volumes. In the response object; HostForMigrationResponse; 'requiresStorageMotion' attribute/flag is set to 'true' if migrating the vm the host requires storage motion.
• migrateVirtualMachineWithVolume. For live migrating a vm with its volumes to a host.

Since these are existing apis; they are not explained here. Please refer to Storage motion in cloudstack for api details.

UI flow

No new UI screens are introduced by this feature. Also, no changes need to be made in UI for adding support for storage motion in Hyper-V. The UI workflow looks as follows.

Migrating volume

• From the Storage option in admin UI, select the volume that is attached to a running vm and has to be live migrated.
• Select the migrate option. 'findStoragePoolsForMigration' api is invoked by the UI. This will give a list of shared pools available to which the volume can be migrated.
• Selecting one of the storage pools and selecting ok will issue the 'migrateVolume' api request to migrate the volume.

Migrate virtual machine with volume

• From the Instances option in the admin UI, select the instance that needs to be migrated (with its volumes).
• Select the migrate option. 'findHostsForMigration' api is invoked by the UI. This will give a list of hosts to which the vm can be migrated.
• If migrating to a host requires storage motion too, 'requiresStorageMotion' flag is set to true in the response and the UI shows it as "Storage migration required".
• Selecting a host; to which storage mition is required; will issue the 'migrateVirtualMachineWithVolume' api request.

IP Clearance

No IP dependencies are introduced by this feature.

Appendix

Appendix A:

Appendix B: