Bug Reference

https://issues.apache.org/jira/browse/CLOUDSTACK-1963

Branch

master, 4.2.0

Introduction

In today's implementation, VMware DataCenter is invisible to CloudStack. CloudStack recognizes and manages cluster as resource container. There is no restriction on which DC or which vCenter instance the cluster, that is being added to cloudstack zone belongs to. This implies following,

1. Multiple vCenter instances can exist in same cloudstack zone.
2. Multiple DCs can exist in same cloudstack zone.
3. Cannot associate N1kv to cloudstack zone because N1kv operates over single DC but cloudstack zone might have multiple DCs underneath (see above point) and also CloudStack doesn't track DC.
4. A primary storage of the scope zone is not inherently possible because datastore is an object scoped at DC, if a zone contains more than 1 DC this is not supported.
5. Live migration of VM across clusters in a zone is not inherently possible if 2 clusters are in different DC's.
6. Multiple cloudstack deployments in same DC enforced 1:1 mapping of ASA instance with cluster & 1:1 mapping of N1kv VSM instance with cluster.
7. vDS - zone wide traffic label override is required in case of a zone with multiple DCs. vDS specific namespace issues across DCs in vCenter.

To address these issues, Cloudstack should explicitly manage vSphere DCs. Treat CloudStack zone as capacity-planning and operating unit, in case of deployment with multiple DCs use multiple CloudStack zones configured per DC and share the same network infrastructure, they only need to carefully allocate resources like IPs/VLANs etc. to multiple CloudStack zones to make it happen.

Purpose

This document describes the specifications and design of new data model for cloudstack zone for VMware environment.

References

1 http://www.vmware.com/pdf/vsphere5/r51/vsphere-51-configuration-maximums.pdf
2 http://www.vmware.com/pdf/vsphere5/r50/vsphere-50-configuration-maximums.pdf
3 http://www.vmware.com/pdf/vsphere4/r41/vsp_41_config_max.pdf

Document History

Glossary

DC - VMware vCenter Data Center.
N1kv - Cisco Nexus 1000v Distributed Virtual Switch
vDS - VMware vNetwork Distributed Virtual Switch.
vSwitch - VMware vNetwork Standard Virtual Switch
dvPortgroup - VMware vNetwork Distributed Virtual Portgroup
Portgroup - VMware vNetwork Standard Virtual Portgroup

Feature Specification

With this feature, Vmware datacenter will be put under management of CloudStack zone.

In previous releases, Vmware datacenter is hidden behind a CloudStack cluster, we need an explicit mapping policy for Vmware datacenter both for zone-wide storage and zone-wide network resources. This model allows a datastore/dvSwitch, scoped at DC, to be managed/used as zone level resource which has use cases like zone wide primary storage, storage live migration across clusters within a zone, orchestration of virtual network spanning across clusters in a zone.

1:1 relationship between vSphere DC to CloudStack zone is a natural mapping from resource point of view, whether it's storage (datastore) or network (distributed switches etc.). Within the zone/DC, it's possible to have multiple vSphere clusters (within the same DC). And this will have the implication that CloudStack zone size (how many hosts within a zone) will be limited to what vSphere allows. See 1, 2 & 3 in References section.

Test Guidelines

1. All VMware test cases need to be covered, as new mapping of DC to CloudStack zone is a change in critical path of VMware hypervisor plugin.

Hypervisor support

• VMware ESXi 4.1 or later.

Negative usage scenarios

1. A DC should not be allowed to be part of multiple CloudStack deployments.
2. In a CloudStack deployment, a DC should be associated with only one CloudStack zone.

Supportability characteristics

Logging

VMware hypervisor plugin logs all the successful operations to INFO, all exceptions/failures to ERROR, and all synchronization checks to DEBUG.

Debugging/Monitoring

In addition to looking at the management server logs, administrators can look up the vcenter logs for analysis.

Use cases

It allows the system administrator to configure cloudstack zone per DC to manage all the the resources in DC.

• All DC wide features would be available across cloudstack zone consistently.
• Zone wide virtual network orchestration is supported using distributed virtual switch (VMware DVS, Nexus 1000v DVS) in a DC. A virtual machine can move across clusters in zone while its dvNics are connected to dvPortgroup of dvSwitch in the DC.
• A DC wide datastore could be used as zone wide primary storage facilitating virtual disk accessibility across clusters in zone.

Architecture and Design description

• Vmware datacenter will be put under management of CloudStack zone.
• In this explicit model, CloudStack will understand the relationship between a Vmware datacenter and resources underneath it, so when resources like a Vmware cluster or dvSwitch that is added to CloudStack, the underlying Vmware resource relationship between these objects will become constraint information for CloudStack to use. For example, a Vmware cluster can't be added to a CloudStack zone if it's Vmware DC is not under the CloudStack zone, also an ASA instance cannot be added to a Vmware cluster that its Vmware DC is not in the zone. The model itself will not limit how many Vmware clusters that a N1kv VSM instance or ASA or vDS will manage, as long as N1kv or ASA or vDS and its managed cluster are within the same Vmware DC.

• CloudStack should be able to add/manage following DC level resources in scope of zone.

1. Clusters
2. Datastores for zone wide storage
3. Distributed virtual switches for zone wide virtual networks.
   1. Nexus 1000v DVS
   2. VMware DVS

• Constraint checks

1. Only clusters of the associated DC can be added to a zone.
   1. Retrieve name of DC which encompasses this cluster. Compare the retrieved DC name with name of DC (being) associated with this zone.
2. Only N1kv VSM of the associated DC can be added to a zone.
   1. N1kv's vCenter server connection's properties would need to be checked to retrieve the association of N1kv with a vCenter and a DC. Compare the retrieved vCenter IP and DC name with vCenter IP and DC name of current zone
3. DC is not associated with any other zone already.
   1. Check guid, of new DC being added, doesn't already exist in table 'cloud'.'data_center_zone_map'. The guid is a string that encapsulates MOR of DC and vCenter host name/ip. By tracking both vCenter host name/ip and MOR of DC, it's possible to distinguish DC's across multiple vCenters.
4. DC is not associated with any other cloudstack deployment.
   1. Check custom property over VMware's DC object if this DC is already associated with a zone of a cloudstack deployment. Custom property of type boolean can represent this, 'cloudstackzone'

• While adding resource to cloudstack zone, approaches to apply constraint checks are,

1. Track relationship between resources (DC <-> Cluster <> DataStore <> DVS <> N1kv <-> ASA etc.) in cloudstack database - suffices within single CloudStack deployment. Cannot detect if multiple CloudStack deployments are managing same resource (DC, DataStore etc.)
2. Use custom property on VMware object as flag. This works across cloustack deployments but proper cleanup is required upon removal.

• During discovery process, VmwareServerDiscoverer might have to retrieve vCenter host/ip & credentials from database given zone_id, because they are made optional parameters because this data is already available in database when the zone is associated with first cluster. Construct connection url from this data.

• While re-loading resource make sure connection url is defined correctly, could be retrieved from vmware_dc table (guid).

• AddCluster changes

1. If a row matches zone_dc_map.zone id and if DC name is not specified, DC name can be retrieved from vmware_dc table and used. If DC name is specified, validation of DC name should happen.
2. In VmwareServerDiscoverer, implement constraint checks while adding a cluster as mentioned in design section.
3. Update vmware DC properties in cloud.vmware_dc table (Say DC name changes in vCenter)
4. While adding VMware cluster to zone, check if zone_dc_map is empty or not. If empty, while adding this cluster we need to associate corresponding DC to zone. Hence do following,
   1. Update tables cloud.zone_dc_map and cloud.vmware_dc to associate DC with zone
   2. If N1kv instance is specified, update table cloud.virtual_supervisor_module and cloud.zone_vsm_map to assocaite N1kv VSM instance with zone
   3. Set custom property over VMware DC object. If not already present, add boolean property 'cloudstackzone' with value 'true'. If the property is already present, just set it to 'true'. Now this DC cannot be associated with any other cloudstack zone.

• DeleteCluster changes

1. While deleting check if this is last cluster in the zone. If so, dis-association of DC to zone should be done. Hence do following,
   1. Cleanup tables cloud.zone_dc_map and cloud.vmware_dc for specific zone by removing entries by zone_id (if zone_id in row matches data_center.id then remove the row)
   2. Set the zone's corresponding DC object's property to 'false'. Now this DC can be associated with any other cloudstack zone.

Database modifications

1. 'cloud'.'vmware_dc' table to track all DC specific information along with details of resources & vCenter.
   1. Columns in this table are id, vCenter host/ip, DC name, DC Mor, guid
   2. id is primary key field of this table
   3. Implement Dao & VO for this table.
2. 'cloud'.'zone_dc_map' table will be added to persist the mapping of zone to DC.
   1. Columns in this table are id, vmware_dc_id, zone_id
   2. Foreign key id from vmware_dc table
   3. Foreign key id from data_center table
   4. Implement Dao & VO for this table.
3. 'cloud'.'cluster_vsm_map' would be removed.
4. 'cloud'.'zone_vsm_map' would be added. Also a zone can be associated with more than 1 VSM instances.
   1. Columns in this table are id, zone_id & vsm_id where id is Primary key of this table.
   2. Foreign key zone_id from data_center table.
   3. Foreign key vsm_id from virtual_supervisor_module table.
   4. Implement Dao & VO classes for this table.

Web Services APIs

1. AddCluster will be modified
   1. Interface perspective - Make parameters vCenter host/ip, credentials and DC name optional based on zone id's presence in zone_dc_map.

UI Flow

• Changes required in UI for live migrating a volume from one storage pool to another.
  1. Add Cluster form in zone wizard
  2. Add Cluster wizard
  3. Add NexusVSM instance to zone
  4. Add ASA instance to zone

Open Issues

1. Choosing to NOT allowing single DC to be managed by multiple cloudstack deployments. Name space collision issues within DC will be resolved by this.
2. Sharing of resources with other entity (cloudstack or non-cloudstack entity)
3. DC release criteria from association with zone - deletion of last host or last cluster?

Impact on other areas/features

1. Impact of this feature on zone wide primary storage support - vmware.
2. Impact of this feature on storage live migration - vmware.
3. Cisco Nexus 1000v support needs changes. 1 cluster to 1 Nexus mapping to be removed. Allow Nexus 1000v VSM instance to be associated/dis-associated with zone.
4. Impact of this feature on ASA feature needs check.
5. Impact of this feature on refactoring in Vmsync.

Upgrade

1. Migration support
   1. Existing CloudStack Deployments - Existing cloudstack deployments with multiple DC's in a zone
      1. Decide a DC as the final DC that contains all clusters.
      2. Move all clusters to the chosen DC.
      3. For each cluster create cluster with same name in target DC.
      4. Edit the field 'cloud'.'cluster'.'name' in database to reflect new DC name in cluster name.
      5. Edit the field 'cloud'.'cluster_details'.'url' in database to reflect new DC name in cluster name.
      6. Cold migrate each of the instance to this new cluster in chosen DC.
   2. Fresh CloudStack Deployments - Existing VMware deployments with clusters to be managed by cloudstack spread in multiple DC's
      1. Move all clusters meant for single zone into 1 DC
      2. For each cluster create cluster with same name in target DC.
      3. Put each host, in that cluster, in maintenance mode
      4. Remove each host in maintenance mode from the cluster
      5. Add each host to new cluster in target DC
      6. Get the host out of maintenance mode