Mapping model for CloudStack zone and Vmware datacenter

Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-1963

Branch

master, 4.2.0

Introduction

In today's implementation, VMware DataCenter is invisible to CloudStack. CloudStack recognizes and manages cluster as resource container. There is no restriction over the DC or vCenter to which the cluster, that is being added to cloudstack zone, belongs. This implies following,

Multiple vCenter instances can exist in same cloudstack zone.
Multiple DCs can exist in same cloudstack zone.
Cannot associate N1kv to cloudstack zone because N1kv operates over single DC but cloudstack zone might have multiple DCs underneath (see above point) and also CloudStack doesn't track DC.
A primary storage of the scope zone is not inherently possible because datastore is an object scoped at DC, if a zone contains more than 1 DC this is not supported.
Live migration of VM across clusters in a zone is not inherently possible if 2 clusters are in different DC's.
Multiple cloudstack deployments in same DC enforced 1:1 mapping of ASA instance with cluster & 1:1 mapping of N1kv VSM instance with cluster.
vDS - zone wide traffic label override is required in case of a zone with multiple DCs. vDS specific namespace issues across DCs in vCenter.

To address these issues, Cloudstack should explicitly manage DCs. Treat CloudStack zone as capacity-planning and operating unit, in case of deployment with multiple DCs use multiple CloudStack zones configured one per DC and share the same network infrastructure, they only need to carefully allocate resources like IPs/VLANs etc. to multiple CloudStack zones to make it happen.

Purpose

This document describes the specifications and design of new data model for cloudstack zone in VMware environment.

References

1 http://www.vmware.com/pdf/vsphere5/r51/vsphere-51-configuration-maximums.pdf
2 http://www.vmware.com/pdf/vsphere5/r50/vsphere-50-configuration-maximums.pdf
3 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf

Document History

Author	Description	Date
Sateesh Chodapuneedi	Initial Revision	04/09/2013
Sateesh Chodapuneedi	Proposed Feature Specification	04/21/2013

Glossary

DC - VMware vCenter Data Center.
zone - CloudStack zone.
N1kv - Cisco Nexus 1000v Distributed Virtual Switch
ASA1kv - Cisco Adaptive Security Appliance 1000v
vDS - VMware vNetwork Distributed Virtual Switch.
vSwitch - VMware vNetwork Standard Virtual Switch
dvPortgroup - VMware vNetwork Distributed Virtual Portgroup
Portgroup - VMware vNetwork Standard Virtual Portgroup

Feature Specification

With this feature, Vmware datacenter will be put under management of CloudStack zone.

In previous releases, Vmware datacenter is hidden behind a CloudStack cluster, we need an explicit mapping policy for Vmware datacenter both for zone-wide storage and zone-wide network resources. This model allows a datastore/dvSwitch, scoped at DC, to be managed/used as zone level resource which has use cases like zone wide primary storage, storage live migration across clusters within a zone, orchestration of virtual network spanning across clusters in a zone.

1:1 relationship between vSphere DC to CloudStack zone is a natural mapping from resource point of view, whether it's storage (datastore) or network (distributed switches etc.). Within the zone/DC, it's possible to have multiple vSphere clusters (within the same DC). And this will have the implication that CloudStack zone size (how many hosts within a zone) will be limited to what vSphere allows. See 1, 2 & 3 in References section.

Test Guidelines

All VMware test cases need to be covered, as new mapping of DC to CloudStack zone is a change in critical path of VMware hypervisor plugin.
Test upgrade to new product (with this feature) when the deployment already has multiple zones, where clusters in each zone encompasses one or more of following,
1. Single DC
2. Multiple DCs
3. Multiple vCenters
After upgrade, test adding new cluster to existing/new zone, where the cluster belongs to a DC/vCenter that is,
1. Already part of zone
2. Not part of zone
Test creation of new zone

Hypervisor support

VMware ESXi 4.1 or later.

Negative usage scenarios

A DC should not be allowed to be part of multiple CloudStack deployments.
In a CloudStack deployment, a DC should be associated with only one CloudStack zone.

Supportability characteristics

Logging

VMware hypervisor plugin logs all the successful operations to INFO, all exceptions/failures to ERROR, and all synchronization checks to DEBUG.

Debugging/Monitoring

In addition to looking at the management server logs, administrators can look up the vcenter logs for analysis.

Use cases

It allows the system administrator to configure cloudstack zone per DC to manage all the the resources in DC.

All DC wide features would be available across cloudstack zone consistently.
Zone wide virtual network orchestration is supported using distributed virtual switch (VMware DVS, Nexus 1000v DVS) in a DC. A virtual machine can move across clusters in zone while its dvNics are connected to dvPortgroup of dvSwitch in the DC.
A DC wide datastore could be used as zone wide primary storage facilitating virtual disk accessibility across clusters in zone.

Architecture and Design description

Vmware datacenter will be put under management of CloudStack zone.
In this explicit model, CloudStack will understand the relationship between a Vmware datacenter and resources underneath it, so when resources like a Vmware cluster or dvSwitch that is added to CloudStack, the underlying Vmware resource relationship between these objects will become constraint information for CloudStack to use. For example, a Vmware cluster can't be added to a CloudStack zone if it's DC is not under the CloudStack zone, also an ASA1kv instance cannot be added to a Vmware cluster if its DC is not in the zone. The model itself will not limit how many Vmware clusters that a N1kv VSM instance or ASA1kv or vDS will manage, as long as N1kv or ASA1kv or vDS and its managed cluster are within the same DC.
CloudStack should be able to add/manage following DC level resources in scope of zone.
1. Clusters
2. Datastores for zone wide storage
3. vDS/N1kv/ASA1kv (in future, see "Out of scope" section)
Upgrade to latest product (with this feature) does not enforce migration of existing CloudStack zones. This means that if an existing zone has clusters from multiple DCs/vCenters, it will be supported as-is. But zone wide features would not be supported on such zone. To quote some, zone wide primary storage, storage live migration across clusters in a zone.
During upgrade all zones with clusters from single DC would not be marked as legacy, otherwise marked as legacy.
Expansion of legacy zone is allowed. Expansion means adding more resources (like cluster).
A zone created after upgrade would be allowed to have only single DC.
New API would be added to associate a zone with a DC. Validation of DC name should happen during this operation.
New API would be added to dis-associate a zone from a DC.
Following constraint checks would be done while adding DC level resource (listed above) to zone. These constraints would not be applicable to legacy zones.
1. Only clusters of the zone's associated DC can be added to a zone.
  1. Retrieve name of DC which encompasses this cluster. Compare the retrieved DC name with name of DC (being) associated with this zone.
2. DC is not associated with any other zone already.
  1. Check guid, of new DC being added, doesn't already exist in table 'cloud'.'zone_vmware_data_center_map'. The guid is a string that encapsulates MOR of DC and vCenter host name/ip. By tracking both vCenter host name/ip and MOR of DC, it's possible to distinguish DC's across multiple vCenters.
3. DC is not associated with any other CloudStack deployment.
  1. Check custom property over VMware's DC object if this DC is already associated with a zone of a CloudStack deployment. Custom property of type boolean can represent this, 'cloudstackzone'
While adding resource to cloudstack zone, approaches to apply constraint checks are,
1. Track relationship between resources (DC <> Cluster <> DataStore <-> DVS etc.) in CloudStack database - suffices within single CloudStack deployment. Cannot detect if multiple CloudStack deployments are managing same resource (DC, DataStore etc.)
2. Use custom property on VMware object as flag. This works across CloudStack deployments but proper cleanup is required upon removal.
During discovery process, VmwareServerDiscoverer might have to retrieve vCenter host/ip & credentials from database given zone_id, because they are made optional parameters because this data is already available in database when the zone is associated with first cluster. Construct connection url from this data.
While re-loading resource make sure connection url is defined correctly, could be retrieved from vmware_data_center table (guid).
Associate DC to zone changes,
1. Validate DC
2. Validate zone
3. Check if DC is associated with any other zone in current deployment.
4. Check if DC's custom property 'cloudstackzone' is set to "true" or "false". If "true" fail the operation.
5. Update database to add information of DC to table 'cloud'.'vmware_data_center'
6. Update table 'cloud'.'zone_vmware_data_center_map' to associate DC with zone
7. Set custom property over VMware DC object. If not already present, add boolean property 'cloudstackzone' with value 'true'. If the property is already present, just set it to 'true'. Now this DC cannot be associated with any other cloudstack zone.
Dis-associate zone from DC
1. Check if all resources (clusters, N1kv etc.) of DC in zone are released, otherwise fail the operation.
2. Set custom property over VMware DC object 'cloudstackzone' to 'false'
AddCluster changes
1. No changes for legacy zone.
2. All constraint checks would be done while attempting to add cluster to zone. DC information would be retrieved from 'cloud'.'vmware_data_center' table. DC to zone mapping information can be referred from 'cloud'.'zone_vmware_data_center_map'. Constraint checks would be added to VmwareServerDiscoverer.
3. Update vmware DC properties in cloud.vmware_data_center table (Say DC name changes in vCenter)
4. While adding VMware cluster to zone, check if a DC is already associated with that zone. If not, cluster add operation would fail. To achieve this, check if zone_vmware_data_center_map does have a row for the zone in question. If no such row exists, that means this zone is not yet associated with any DC.
DeleteCluster changes
1. While deleting check if this is last cluster in the zone. If so, dis-association of DC to zone should be done. Hence do following,
  1. Cleanup tables cloud.zone_dc_map and cloud.vmware_dc for specific zone by removing entries by zone_id (if zone_id in row matches data_center.id then remove the row)
  2. Set the zone's corresponding DC object's property to 'false'. Now this DC can be associated with any other cloudstack zone.

Database modifications

'cloud'.'vmware_data_center' table to track all DC specific information along with details of resources & vCenter.
1. Columns in this table are id, uuid, DC name, guid, vCenter host, username & password. The 'guid' field encapsulates 'vCenter host/ip' and DC mor. E.g. 'Datacenter:datacenter-101@vcenter.abc.com'
2. id is primary key field of this table
3. Schema definition
  1. CREATE TABLE `cloud`.`vmware_data_center` (
    `id` bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
    `uuid` varchar(255) UNIQUE,
    `name` varchar(255) NOT NULL COMMENT 'Name of VMware datacenter',
    `guid` varchar(255) NOT NULL UNIQUE COMMENT 'String encapsulating VMware datacenter and vCenter',
    `vcenter_host` varchar(255) NOT NULL COMMENT 'vCenter host containing this VMware datacenter',
    `username` varchar(255) NOT NULL COMMENT 'Name of vCenter host user',
    `password` varchar(255) NOT NULL COMMENT 'Password of vCenter host user',
    PRIMARY KEY (`id`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
'cloud'.'vmware_data_center_zone_map' table will be added to persist the mapping of zone with DC.
1. Columns in this table are id, vmware_data_center_id, zone_id
2. Foreign key id from vmware_data_center table
3. Schema definition
  1. CREATE TABLE `cloud`.`vmware_data_center_zone_map` (
    `id` bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
    `zone_id` bigint unsigned NOT NULL UNIQUE COMMENT 'id of CloudStack zone',
    `vmware_data_center_id` bigint unsigned NOT NULL UNIQUE COMMENT 'id of VMware datacenter',
    PRIMARY KEY (`id`),
    CONSTRAINT `fk_vmware_data_center_zone_map__vmware_data_center_id` FOREIGN KEY (`vmware_data_center_id`) REFERENCES `vmware_data_center`(`id`) ON DELETE CASCADE
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
'cloud'.'legacy_zones' table to track legacy zones in cloudstack deployment. During CloudStack upgrade, this table is populated.
1. Columns in this table are id, zone id
  1. id is primary key field of this table
  2. Foreign key zone_id refers id from data_center table
2. Schema definition
  1. CREATE TABLE `cloud`.`legacy_zones` (
    `id` bigint unsigned NOT NULL AUTO_INCREMENT COMMENT 'id',
    `zone_id` bigint unsigned NOT NULL UNIQUE COMMENT 'id of CloudStack zone',
    PRIMARY KEY (`id`),
    CONSTRAINT `fk_legacy_zones__zone_id` FOREIGN KEY (`zone_id`) REFERENCES `data_center`(`id`) ON DELETE CASCADE
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;

Web Services APIs

addCluster will be modified
1. Interface perspective -
  1. No change
  2. If the zone is legacy, then hide Datacenter/DC text field in UI. Hence cluster url becomes http://vcenterhost/cluster
addVmwareDc -
1. Maps a cloudstack zone with specified VMware DC.
2. Throws RemoteException if vCenter is not reachable or any other server side Exception caught.
3. Upon failure in parameter validation throw InvalidParameterValueException
4. Parameters are,
  1. zoneId - required
  2. name - required
  3. vcenter - required
  4. username - required
  5. password - required
removeVmwareDc -
1. Removes and unmaps the VMWare DC that is added to specified zone.
2. If no association exists already, then throw CloudRuntimeException.
3. If no such zone exists, throw InvalidParameterValueException
4. If zone is have one or more clusters / elements, then throw ResourceInUseException - Zone should be empty (no clusters or other elements like Nexus 1000v VSM instance)
5. Parameter are,
  1. zoneId - required
listVmwareDcs -
1. Lists VMware DCs associated with specified zone.
2. If no association exists already, then returns empty list
3. Parameter is,
  1. zoneId - required

UI Flow

Changes required in UI
1. Add VMware DC to zone
2. Remove VMware DC from zone
3. Add Cluster form in zone wizard
4. Add Cluster wizard
5. Changes to support legacy zones.

Open Issues

Considering a model where a CloudStack deployment will have some zones with multiple DCs and some zones with single DC. This comes from the fact that existing CloudStack deployment has zones with multiple DCs. But after upgrade to the newest version, customer would like to have existing zones to operate as-is. But all new CloudStack zones would be having single DC. In this model, the existing zone with multiple DCs is termed as Legacy zone and it will not support the zone wide features are requires a zone to have only 1 DC. Also operations on legacy zone should be seamless (like adding a cluster etc.) as they were before upgrade. This means the code base doesn't apply any new logic, that affects the functionality, over legacy zone. For example, constraint checks that happen while adding a cluster to new zone should not be done if the zone is legacy zone.
New features like zone wide primary storage doesn't go well with zones having multiple hypervisors. How upgrade handled for such zones?

Impact on other areas/features

Impact of this feature on zone wide primary storage support - vmware.
Impact of this feature on storage live migration - vmware.
Cisco Nexus 1000v support needs changes. 1 cluster to 1 Nexus mapping to be removed. Allow Nexus 1000v VSM instance to be associated/dis-associated with zone.
Impact of this feature on ASA feature needs check.
Impact of this feature on refactoring in Vmsync.

Upgrade

During CloudStack upgrade operation, 'cloud'.'legacy_zones' table is populated. This table will legacy zones in cloudstack deployment.
Migration support
1. Existing CloudStack deployments with clusters from multiple DCs/vCenters require consolidation of clusters into single DC.
2. Following are options to migrate VM across datacenters
  1. Cold migrate each of the instance to this new cluster in chosen DC.
  2. Hot migrate the VMs by using disconnecting all hosts from source cluster followed by deletion of source cluster from source DC and adding all hosts to cluster in target DC.
  3. Hot migrate the VMs after suspending the VMs. Need to power on (mean resume) VMs after migration is complete. This needs the virtual disks of VM to be placed on shared storage before triggering the migration operation.
3. Pre-requisite checker tool - This tool should go through existing CloudStack deployment and VMware deployment to check if prerequisites are met. Following are list of prerequisites would be convered during upgrade to new version of product with this feature built-in,
  1. A CloudStack zone should encompass only 1 VMware DC
  2. The VMware DC encompassed by CloudStack zone should not be shared by multiple CloudStack deployments.
Guidelines of CloudStack version upgrade
1. Migration would be offline operation before CloudStack version upgrade operation.
2. Prerequisite checker tool would be run as part of CloudStack upgrade
3. Choice of migration procedure is above option 1.b.(ii) - migration of host across DC by disconnecting from cluster in source DC, followed by connecting it to target cluster in another DC. Following are detailed steps,
  1. Handle the case of clusters from multiple vCenter
    1. To be investigated.
  2. Handle the case of clusters from single vCenter but different DCs
    1. Handle the case of deployment using distributed virtual switch (Nexus 1000v or vDS)
      1. Deploy Nexus 1000v or vDS on target DC with same credentials (user/password)
      2. Ensure if 'cloud'.'virtual_supervisor_module' table has the new IP of Nexus 1000v VSM instance.
      3. Create all dvPortgroups required by VMs on the host that is moving across DCs. Make sure all properties of dvPortGroups are in sync, e.g. shaping policy should be same as that of original dvPortGroup in source DC. External switch configuration for trunking/routes is assumed to be taken care of by admin.
    2. Decide a DC as the final DC that contains all clusters.
    3. Create a cluster, in target DC, with same properties as that of original cluster
    4. Unmanage the cluster in CloudStack
    5. Disconnect each host in source cluster
    6. Remove each host from source cluster
    7. Add each host to target cluster
    8. Create all custom properties
      1. Each VM will have custom properties like cloud.nic.mask. This will be lost during above migration operations. Hence need to re-create after adding to target cluster
      2. Each template (user VM and system VM templates) will have custom properties like cloud.uuid. This need to be restored.
    9. Add the target cluster to CloudStack
    10. Make sure that guid field of 'cloud'.'host' table has the valid reference because when host is added to target cluster, it will acquire new unique reference.
    11. Make sure the field 'cloud'.'cluster'.'name' in database reflect new DC name in cluster name.
    12. Make sure the field 'cloud'.'cluster_details'.'url' in database reflect new DC name in cluster name.

Assumptions

Sharing of resources (datastores, virtual switches like vDS, N1kv and ASA1kv etc.) with other entity (cloudstack or non-cloudstack entity) is not allowed.
Cloudstack zone doesn't contain multiple hypervisors. A cloudstack zone will have only 1 hypervisor.

Out of scope

Support zone level Cisco Nexus 1000v VSM instance.
Support Zone level Cisco ASA 1000v instance.

Space shortcuts

Child pages