Page History

Bug Reference

CLOUDSTACK-681

Branch

master, 4.2.0

Introduction

Dedicating pod, cluster or host to a specific domain/account means that the domain/account will have sole access to the dedicated pod, cluster or hosts such that scalability, security and manageability within a domain/account can be improved. The resources which belong to that tenant, will be placed into that dedicated pod, cluster or host.

Current Scenario

• Currently in CloudStack, zones can be reserved for specific domains. Only users in that domain or its subdomain may create guests in that zone. 
• Dedicated Hosts and HA Hosts:  if one of the dedicated hosts fail then the VMs are HAed onto a specific host(s) that is dedicated for purposes of HA.
• Domains/Accounts cannot have private pod, private cluster or private host

Purpose

Dedicating a zone might be very expensive offering for several end-users whereas dedicating a pod/cluster/host may be more economical. This feature will allow root admin to dedicate resources to a specific domain/account  that needs private infrastructure for additional security or performance guarantees.

This document describes the specifications and design of this feature.

Also see https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+for+VMs+on+hardware+dedicated+to+a+specific+account

References

• PM document: https://cwiki.apache.org/confluence/display/CLOUDSTACK/Private+Host%2C+Cluster%2C+Pod

Preliminaries

1. Explicitly Dedicated Resources: Resources dedicated to an account/domain during configuration time

2. Implicitly Dedicated Resources: Resources which are in common pool that any account can pick during runtime.3. Shared Resources: All the non-dedicated resources.

4. SO dedication flag: a new Parameter in Service Offering. if ON, Use Implicitly Dedicated Resources

5. DVM dedication flag: a new Parameter in deployVirtualMachine API, if ON, VM is deployed on explicitly dedicated resources 

Feature Specifications

Requirements

For Implicit Dedication, the options available are:

3. Strict Implicit dedication: VM with this requirement will be deployed on the host having VMs of same account. 

4. Preferred Implicit dedication: VM with this requirement will be deployed on host having VMs of same account if possible, otherwise can be place in shared resources

For Explicit Dedication,

5. Explicit dedication, a new type will be added in Affinity Groups which will indicate deployment on explicitly dedicated resources. 

Feature Specifications

Requirements

• Root Admins must be able to explicitly dedicate a zone, pod or cluster or host to a specific domain or account
• When deploying a VM, end user can optionally specify it to be placed on a dedicated resource
• Explicit dedication can be for a domain or account and can be a zone/pod/cluster/host; for example, if a cluster is explicitly dedicated to a domain, any account in that domain can
• Root Admins must be able to explicitly dedicate a zone, pod or cluster or host to a specific domain or account
• When deploying a VM, end user can optionally specify it to be placed on a dedicated resource
• Explicit dedication can be for a domain or account and can be a zone/pod/cluster/host; for example, if a cluster is explicitly dedicated to a domain, any account in that domain can share the hosts (as long as they DeployVM with dedication flag ondeploy a VM requesting dedication), but sharing is limited to accounts within that domain only. As another example, if a cluster is explicitly dedicated to an account, only that account’s VM can use those hosts in the dedicated cluster (as long as they DeployVM with dedication flag ondeploy a VM requesting dedication).
• AlternativelyOptionally, a host or cluster or pod or zone can be "implicitly" dedicated - this flag's description will be clarified below
• Strict Implicit dedication, when requested, means, Implicit dedication can be for a zone/pod/cluster/host, but not associated with any domain or account; however, during VM deployment, once chosen, a host will not be shared across multiple accounts – as an example, here is a reason: for deployment of certain types of applications, such as desktops, due to licensing reasons, no host can be shared between different accounts. However, pre-allocating specific (number of) hosts to each account is not desired as this will create forecasting issues/sub-optimal utilization
• Preferred Implicit Dedication, when requested, means VM would prefer to share a host with another VM of this account, if possible, but could be placed with another account's VM, if needed

• A service offering can optionally have an "implicitly" dedicated flag on - this flag's description will be clarified below in the table
• Admin must be able to live migrate of such a VM to a resource not owned by the account, but an alert must be generated

• Non-requirement: Must be able to dedicate a primary storage to an account
• Non-requirement: for the domain/sub-domain admins to manage the resource - the root admin will remain the owner of the resources
• Non-requirement: In general, CloudStack must detect and provide warning of inappropriate location for a VM (such as a dedicated VM on a non-dedicated host) - this is not a specific requirement of this feature, but more of a generic requirement

• Table 1:Deploying VM with Explicit Dedication

  VM requesting dedication

  VM NOT requesting dedication

• Fail the request if DeployVM dedication Flag is specified and SO has implicit dedication ON
• When you search implicitly dedicated “area” for a host (second table below), if there is already a host for this account, pick it up (instead of finding a new one every time)
• When you delete the last VM from a host in the implicit area, remove the association of the account with the host (i.e. make it implicit-free host as before)
• Table 1:Deploying VM with DeployVM dedication flag

  	DeployVM dedication Flag specified	DeployVM dedication Flag Not specified
  Explicitly Dedicated Resources	Pick from Explicitly dedicated else fail	Pick from shared resources else fail
  NO Explicitly Dedicated Resources	Fail the request	Pick from shared resources else fail

• Table 2: Deploying VM with SO dedication flag

  	SO dedication Flag =  ON	SO dedication Flag =  OFF
  YES - Implicitly Dedicated Resources	Pick from implicit dedicated resources else FAIL	Pick from shared resources else fail
  NO - Implicitly Dedicated Resources	Fail the request	Pick from shared resources else fail

 Logs

 Logs

1. Ensure proper Ensure proper logs are maintained into vmops.log and api.log

Test Guidelines

1. Dedicated resources can only be used if service offering dedication flag is ON or DeployVM dedication flag is ON but not both.

...

User Permissions

1. Only Root Admin can dedicate zone, pod, cluster or host to specific domain or an account.
2. Only Root admin can implicitly dedicate a zone, pod, cluster or a host.
3. If a user does not belong to a domain which has explicitly dedicated resources, he cannot access the pod, cluster or host dedicated to that domain/account.
4. Users belonging to domain/account having dedicated resources, can access them but should not be allowed to modify.
5. At this time, there is no requirement for the domain/sub-domain admins to manage the resource - the root admin will remain the owner of the resources
6. Only Root Admin can add a service offering with "isdedicated" option enabled. 
7. User/admin can enable the DeplovVM dedication flag. 

Use Cases: 

Use Cases: 

Dedicating Resources:

...

Domain level accessibility:

Let D1 domain has SD1, SD2, SD3 sub-domains. A1 is the admin account, U2 is normal user account.

Here z1, z2, ... are zones ,  p1,p1, ... are pods, c1, c11, c2 ,.... are clusters, H1, H2, ... are Hosts 

z1 - (p1-c1,c11,c111),(p2-c2,c22),(p3-c3, c33)
z2 - (p4-c4)

1. Root Admin should be able to dedicate a zone, pod, cluster or host to any domain or sub-domain.
2. Once a Zone is dedicated to a domain,  its pods and clusters will be dedicated to that domain eg: pod: p1 is dedicated to domain D1, clusters:c1, c11, c111 will be automatically dedicated to D1 by default.
3. Users in Sub-Domains SD1, SD2, SD3 should be able to deploy vm in parent domain's clusters c1, c11, c111 or pod p1. 
4. After dedicating pod p1 to domain D1, if further cluster c11 (in pod p1) is dedicated to SD3, then D1 or SD1 or SD2 should not be able to access c11. (Can SD3 use SD2, SD1 or D1's resources, verify)
5. If another pod p2 is dedicated to SD1, then SD11, SD12 or D2 should not be able to access pod p2.
6. Before dedicating a pod to a domain , check whether its zone is dedicated or not.
7. Child Domain can access pod/cluster/host dedicated to parent domain, vice-versa is not (TBD).

Account level accessibility:

1. Once a pod/cluster/host is dedicated to an account, only users in that account can access it.
2. No other user from different accounts  in the same domain or different domain can access the resources. 

Deletion of Account/domain

1. Deleting an account will delete all the VMs, snapshots, templates, etc. of that account, and also removes the dedication of host, cluster or pod to that account.
2. Deleting a domain, will remove the dedication from the hosts, clusters or pods (if dedicated).   

VM Deployment

1. If dedicated resources get exhausted for a domain/account, VM deployment will not fail unless shared resources has no free empty host, provided Implicit dedication flag: ON and service offering flag: ON.
2. VMs that belong to two different offerings can be on the same host as long as they belong to the same account/domain and have isdedicated flag ON. For e.g. If an instance is deployed by account user and : 
            a. If that account has dedicated resources, service offering flag "isdedicated" checked, then VM will be deployed on the dedicated host having VMs of same account or on the host which is empty.
            b. If that account has NO dedicated resources, service offering flag "isdedicated" checked then VM will be deployed on the host which is empty and that host will become dedicated to this account.
3. The dedicated VM of other accounts (e.g. A2 or A3 ) of same domain or other domain, cannot use above host, but can use an empty host or host having vms of same account(A2 or A3). 
4. If the service offering flag if OFF, the VM  will be deployed as CloudStack is doing now but should not use the host marked "dedicated for domain/account x".
5. If no such host exists, VM operation should fail. 
6. If the dedication is removed and host has NO dedicated VMs, then host will be available for all the accounts.

Host Tags with isDedicated flag and dedicated resources

1. If Host Tag is provided and isDedicated flag is true and NO dedicated Resources (Host) 
   1. if Host has dedicated vms, place the vm in the that host
   2. if Host has no dedicated vms, fail the request.
2. If host tag is provided and isDedicated flag is false and NO dedicated Resources 
   1. if Host has dedicated vms,  fail the request
   2. if Host has no dedicated vms, place the vm in that host as cloudstack is doing now
3. If host tag is provided and isDedicated flag is true and have explicitly Dedicated Resources (dedicated Host)
   1. Search for tagged as well as dedicated host, if found place the vm, if not found,  fail the request. 

Migration of VMs

1. If VM to be migrated is non-dedicated
   1. if destination host has dedicated vms, fail the request
   2. if the destination host is explicitly dedicated, fail the request.
   3. if destination host has no dedicated vms or is not explicitly dedicated, migrate it to the destination host.
2. If VM to be migrated is dedicated
   1. if destination host has dedicated vms, migrate it to that host.
   2. if destination host is explicitly dedicated to the account owning VM, migrate it to the host
   3. if destination host is empty, migrate it to the host. Now host is implicitly dedicated to the account.
   4. if destination host has non-dedicated vms or is not explictly dedicated, fail the request

Architecture and Design description

This feature will come as a separate plugin which will use the PluggableService to add the DedicatedResources feature APIs into CloudStack, the DedicatedResourceManager will be responsible for thread scheduling and the DedicatedResourcePlanner which will implement a deployment planner interface.

Dedication will be achieved as:

1. Admin adds Pod/cluster/host
2. Admin dedicates the pod/cluster/host to a domain/account using new dedication APIs 
3. Admin enables the pod/cluster/host

Dedication will be used when:

1. If a request is placed to deploy a VM with Service Offering flag "isdedicated" ON.
   1. Check if the account/domain has dedicated resources:
      1. If Yes, Place the VM in dedicated resources.
      2. If No, Place the VM in new free empty host, make that host dedicated to that account.

Note: If Service Offering Flag is OFF, non-dedicated (shared) resources will be used.

1. Resources under dedicated resource (to a domain) can be further dedicated, if dedication is done for sub-domain or accounts under domain or subdomain.
2. A resource for eg. a zone, cannot be dedicated if resources under this zone is dedicated to domain/sub-domain/account not belonging to the domain requesting dedication.
3. Suppose some hosts in a cluster are dedicated to Account A1. If admin wants to dedicate that cluster to same account A1, then dedicated hosts will be automatically released before the dedication of that cluster.   
4. Users in Sub-Domains SD1, SD2, SD3 should be able to deploy vm in parent domain's clusters c1, c11, c111 or pod p1. 
5. After dedicating pod p1 to domain D1, if further cluster c11 (in pod p1) is dedicated to SD3, then D1 or SD1 or SD2 should not be able to access c11. (Can SD3 use SD2, SD1 or D1's resources, verify)
6. If another pod p2 is dedicated to SD1, then SD11, SD12 or D2 should not be able to access pod p2.
7. Before dedicating a pod to a domain , check whether its zone is dedicated or not.
8. Child Domain can access pod/cluster/host dedicated to parent domain, vice-versa is not.

Account level accessibility:

1. Once a zone/pod/cluster/host is dedicated to an account, only users in that account can access it.
2. No other user from different accounts  in the same domain or different domain can access the resources. 
3. No resources under a dedicated resource can be further dedicated.

VM Deployment Use Cases:

The following figure graphically illustrates the allocation of hosts for VM deployment.

Host 1 is explicitly dedicated to domain D1, Host 2 is shared empty host, and Host 3 is a shared host. 

SID-VM  : VM with Strict Implicit Dedication required  

PID-VM  : VM with Preferred Implicit Dedication  

ED-VM   : VM with Explicit Dedication required 

      VM   : VM with no dedication required

Image Added
Use Case 1:  User U3 of Account A2 (Red color),  deploys a virtual machine VM1 with SID,

                    Host 2 is chosen from the shared host as it is emplty. "Host 2" is now dedicated to account  A2

Use Case 2:  User U1 of Account A1, deploys a virtual machine VM2 with ED affinity type ,  

                    Host 1 is chosen form the pool of explicitly dedicated resources (In this case, only Host 1 is explicitly dedicated)

Use Case 3: User U1 of Account A1,  deploys a virtual machine VM3 with NO dedication,

                   Host 3  is chosen from the shared pool,  (In this case, only Host 3 is a shared Host)

Use Case 4: User U1 of Account A1 (Green color), deploys a virtual machine VM1 with  SID,

                   Request Failed, No Host available

Use Case 5: User U4 of account A2,  deploys a virtual machine VM2 with  ED affinity type,

                   Host 1 is chosen form the pool of explicitly dedicated resources

Use Case 6:  User U5 of Account A3 (Blue Color),  deploys a virtual machine VM1 with SID,

                    Request Failed, No Host available

Use Case 7:  User U5 of Account A3, deploys a virtual machine VM2 with SID, 

                    Request Failed, No Host available.

Use Case 8:  User U5 of Account A3, deploys a virtual machine VM3 with NO Dedication, 

                    Host 3 is selected.

Use Case 9: User U3 of Account A2, deploys a VM with PID.

                   Host 2 is selected, VM with SID is OK if VM come from the same account

Use Case 10:User U3 of Account A2, deploys a VM with NO dedication .

                    Host2/Host3 can be chosen.

Use Case 11: User U5 of Account A3 deploys a VM with PID.

                    Search for Host having VMs of same account. If not available can go to shared host.

                     Host 3 is selected. 

Use Case 12: Once All the VMs in Host 2 is deleted, Host will get available for all users.

Use Case 13: Host 1 will remain dedicated to Domain D1 unless root admin changes dedication of this host.

Host Tags with Explicit and Implicit Dedication

1. If Host Tag is provided and Implicit Dedication is required, Host which is tagged as well as implicitly dedicated, will be allocated. If not found, fail the request. 
2. If host tag is provided and NO dedication is required, Host which is tagged but not implicitly or explicitly dedicated, will be allocated.  If not found, fail the request. 
3. If host tag is provided and Explicit dedication is required, Host which is tagged as well as explicitly dedicated, will be allocated.  If not found, fail the request. 

Migration of VMs

Migration is a Root Admin functionality, migrating VMs across dedicated hosts generates Alerts.

The following scenarios generate alerts:

...

Explicit Dedication :

...

VM is being migrated between :

• dedicated source host to non-dedicated host.
• non-dedicated source host to dedicated host.
• source host and destination hosts are dedicated to different accounts/domains.

...

Implicit Dedication :

...

• VM is deployed using strict implicit planner and moved to a destination host which has VMs of other account or VMs deployed using planner other than strict.
• VM is deployed using preferred implicit planner and moved to a destination host which has VM of other account or VMs deployed using a service offering other than ImplicitDedication 
• VM is not deployed using ImplicitDedication but moved between dedicated hosts. 

Deletion of Account/domain

1. Deleting an account will delete all the VMs, snapshots, templates, etc. of that account, and also removes the dedication of host, cluster or pod to that account.
2. Deleting a domain, will remove the dedication from the hosts, clusters or pods (if dedicated).  

Architecture and Design description

This feature will come as a separate plugin which will use the PluggableService to add the DedicatedResources feature APIs into CloudStack, the DedicatedResourceManager will be responsible for thread scheduling and the ExplicitDedicationProcessor will implement AffinityGroupProcessor adapter and  ImplicitDedicationPlanner will implement a deployment planner interface.

Explicit Dedication Design

1. New Admin APIs to dedicate Zones/Pods/Clusters/Hosts to a domain or account (these APIs will come in a separate plugin)
2. list affinity types: Add a new type: explicit dedication
3. Affinity group of type ExplicitDedication is created when the root admin dedicates a resource to any  account/domain.
4. User can associate above affinity group to VM during deployment.
5. A new ExplicitDedicationProcessor plugin that implements AffinityGroupProcessor adapter, will set deployment plan scope to the correct resource level (For AffinityGroupProcessor adapter see: https://cwiki.apache.org/confluence/display/CLOUDSTACK/FS+-+Affinity-Anti-affinity+groups).
6. The Deployment Planning Manager will do the following:
   1.  Processors: Call the ExplicitDedicationProcessor to process the dedication request based on the explicitly dedicated resources for the user.
   2.  Planners: Call Deployment Planner to drill down to the right set of clusters to look for placement based on explicitly dedicated clusters/hosts inside them. 
   3.  Allocators: Call Allocators to match the requirement to explicit dedication.

New Addition: ExplicitDedicationProcessor

This will implement the process to set the scope of deployment plan to the correct resource level for planners to look into. This processor will implement AffinityGroupProcessor adapter.

Non-dedicated Resources will be added in Avoid List.

For Example:

              Zone Z has Pods P1,  P2.

              Pod P1 has Clusters C1, C2  and P2 has Cluster C3

              Clusters

              C1 -> h1 (dedicated A), h2 (dedicated B)

              C2 -> h3 (dedicated A)

              C3 -> h4, h5(dedicated B) 

Case 1: Deploy VM  for account A with explicit dedication, processors should set the scope of plan to Pod P1 since both dedicated hosts are in Pod P1 even if in different clusters. Also set the Pod P2 in avoid set

Case 2: Deploy VM for account B with explicit dedication, processors should set the scope of plan to Zone Z  since the dedicated hosts are in Pod P1 and P2 , with parent being the zone itself. Also set the Cluster C2  in avoid set.

System VMs and Virtual Routers:

System VMs and Virtual Router do not adhere to explicit dedication as they are owned by the system account and can be deployed on any host.

Implicit Dedication Design

1. Admin should be able to choose a planner while creating a Service Offering.
2. In Service Offering the list of planners will be shown, which include the new planner for Implicit Dedication: ImplicitDedicationPlanner
3. Along with selecting a planner, admin may choose to configure the planner being used by the service offering. This is done by passing name-value parameters to the createServiceOffering command.
4. The name-value parameters may be used by the planner when the service offering is selected for deploying the vm.
5. Admin can set the name-value parameter for 
   1. Strict Implicit Dedication as  "ImplicitDedicationMode = Strict" 
   2. Preferred Implicit Dedication as "ImplicitDedicationMode = Preferred"
6. These parameters will get stored in reference to the service_offering_id.
7. ImplicitDedicationPlanner takes the above parameters as input and  set the scope of the plan of the VM to choose implicitly dedicated resources.

New Planner: ImplicitDedicationPlanner

ImplicitDedicationPlanner will extend firstfitplanner and search only in the dedicated resources.

Updated API: createServiceOffering

createServiceOffering API will take in name-value parameters. In the api they can be passed by specifing the 'serviceofferingdetails' parameter. For implicit dedication the valid name-value pairs are

1. Strict Implicit Dedication as "ImplicitDedicationMode = Strict"
2. Preferred Implicit Dedication as "ImplicitDedicationMode = Preferred"

   Code Block
   title Example
   For example: A service offering may be created as follows using the api. http://<ip>:8080/client/api?command=createServiceOffering&serviceofferingdetails[0].ImplicitDedicationMode=Preffered&issystem=false&name=Preferred&displaytext=Preferred&storageType=shared&cpuNumber=1&cpuSpeed=500&memory=512&deploymentplanner=ImplicitDedicationPlanner&offerha=false&limitcpuuse=false&isvolatile=false

Change in the Existing Private Zone functionality:

Zones can be dedicated to domains while creation. Users in that domain can deployVm only in that dedicated zone.

Change:

1. Zones dedicated to a domain at the creation level will be considered as explicit dedication of zones. 
2. All the Zones will be listed, deployment will only happen if any of the resource in that zone will be dedicated to that user's account or domain.
   

To achieve the functionality of private zone, dedicate the zone.

• During VM deployment through the VM wizard UI, when the user chooses the dedicated zone, the dedicated affinity group will be preselected for him and non-editable in the wizard.
• A user can deploy vms in a zone dedicated to him with/without ExplicitDedication affinity group. 

System VMs and Virtual Router:

The presence of system vms and virtual routers on a host makes it unsuitable for strict implicit dedication, as the planner cannot implicitly dedicate the host since it has vms of another account (the default system account).

However the same host could be used for implicit dedication in preferred mode.

API Changes

...

Following new APIs will be added:

• Dedicating Resources to an account/domain(Explicit or Implicit) 
  • dedicateZone
  • dedicatePod
  • dedicateCluster
  • dedicateHost

      These are new admin APIs to dedicate a zone/pod/cluster/host for an account/domain.

      Parameters include:
       a)    PodId ZoneId/PodId/ClusterId/HostId
       b)    DomainId
       c)      AccountIdAccountId 

• Updating Release dedication of resources ** releaseDedicatedZone
  • updateDedicatedPodreleaseDedicatedPod
  • updateDedicatedClusterreleaseDedicatedCluster
  • updateDedicatedHostreleaseDedicatedHost

      These are new admin APIs to update the dedicated remove the dedication of zone/pod/cluster/host for an account/domain.

       Parameters include:
       a)    PodId ZoneId/PodId/ClusterId/HostId
       b)    isPublic (if true, dedication is removed)HostId

• Listing Dedicated resources per account/domain** listDedicatedZones
  • listDedicatedPods
  • listDedicatedClusters
  • listDedicatedHosts

     These are new admin APIs to list dedicated pods/clusters/hosts for an account/domain.

     Parameters include:
       a)    PodId zoneId/PodId/ClusterId/HostId
       b)    DomainId
       c)    AccountId

Existing API modification:

• createServiceOffering:  Request Parameter Addition:

...

Parameter Name

...

Description

...

Required

...

isdedicated

...

 if true, Instance will be deployed on the host dedicated to the account 

...

Note: List Dedicated resources APIs will list all the dedicated resources wrt the resource(zones/pods/clusters/hosts) if no parameter is provided. 

DB Changes:

1. New Table : dedicated_resources (id, uuid, zone_id, pod_id, cluster_id, host_id, domain_id, account_id)
      || Field || Type || Null || Key || Default || Extra ||

   id	bigint(20) unsigned	NO	PRI	NULL	auto_increment
   uuid	varchar(40)	YES	UNI	NULL
   data_center_id	bigint(20) unsigned	NO	MUL	NULL
   pod_id	bigint(20) unsigned	NO	MUL	NULL
   cluster_id	bigint(20) unsigned	NO	MUL	NULL
   host_id	bigint(20) unsigned	NO	MUL	NULL
   domain_id	bigint(20) unsigned	YES	MUL	NULL
   account_id	bigint(20) unsigned	YES	MUL	NULL

2. service_offering table: Introduce a column “isdedicated” in  service_offering table. Default value should be 0.

UI flow

UI flow

• A resource can be dedicated at the time of adding it to CloudStack or at any point of time after it has been added.
• Dedicate <Resource> and Release<Resource> are the new action buttons added.

DRS? 

• This is applicable only for placement operations through CloudStack. This implementation is to only support scenarios where the HV does not do HA or DRSAdd a check option in "Add Compute Offering" for is_dedicated flag.