This spec addresses the creation of a feature that allows support for new disk controllers to be added without having to extend CloudStack's source code.

1 Problem description

Currently, CloudStack only supports IDE and SCSI (buslogic, lsisas1068, lsilogic, pvscsi) disk controllers for VMware, even though VMware also offers support for SATA (ahci) and NVME controllers. For KVM clusters, IDE, SCSI (virtio-scsi), SATA and Virtio are supported, but KVM also supports other subtypes of IDE (piix3, piix4, ich6), SCSI (buslogic, lsilogic, lsisas1068, lsisas1078, vmpvscsi, virtio-transitional, virtio-non-transitional, ncr53c90, am53c974, dc390) and Virtio (virtio, virtio-transitional, virtio-non-transitional) controllers. 

For VMware, the supported disk controllers are defined in com.cloud.hypervisor.vmware.mo.DiskControllerType. For KVM, the bus types are defined in com.cloud.hypervisor.kvm.resource.LibvirtVMDef.DiskDef.DiskBus, but the controllers used for each bus type are fixed. This current hard-coded approach is not interesting, because we have to change the code for every use case that appears, and operators have to wait for releases to start using new controllers.

2 Proposed changes

To address this problem, this spec introduces disk controller mappings, which allows adding support for new disk controllers without having to extend CloudStack. The initial implementation of this feature will be only for VMware, but it will be agnostic and flexible enough so that it can also be implemented for KVM in the future.

For this, we will create a table to store the necessary information about the disk controllers. By default, this table will map all the currently available disk controllers in VMware (IDE, SCSI subtypes, SATA and NVME). As a new disk controller gets supported by the hypervisor, operators will be able to add a new entry to the table to start using it immediately or to wait until a CloudStack upgrade maps the controller. It will be possible to use the mapped disk controllers for instances through the global setting vmware.root.disk.controller, and through the rootDiskController and dataDiskController details.

To identify the controller in the code we will use the column controller_reference. For VMware, we need the disk controller's class in order to configure the instance and perform validations. Thus, for VMware mappings, this column will contain the disk controller's classpath (e. g. com.vmware.vim25.VirtualLsiLogicController) so that we can obtain it through reflection. For KVM, we only need a string identifying the disk controller's model (e. g. lsilogic). In addition to this column, we will also use other columns to perform validations and apply restrictions.

The supported mappings will be loaded from the database when VmwareManagerImpl is configured, and stored in a static variable in VmwareHelper. We will adapt VMware's following workflows to use these controllers instead of using the hardcoded ones: (i) VM creation, (ii) VM start, (iii) VM import and (iv) disk attachment. Also, as some of the adapted methods are used in the SSVM as well, we will send these mappings to the system VM through the SecStorageSetupCommand after the management server connects to the SSVM.

2.1 Database model

A table named disk_controller_mapping will be created in the cloud schema with the following columns:`

2.2 VM start

The relevant parts from the current VM start process are summarized in the following workflow.

Below is a detailed description of this workflow.

1. CloudStack compares the settings rootDiskController and dataDiskController with the ENUMs in com.cloud.hypervisor.vmware.mo.DiskControllerType and creates a pair of strings representing the ENUMs.
2. If the virtual machine exists:
   1. If the VM does not have a snapshot, all attached virtual disks get detached.
   2. The controllers required by rootDiskController and dataDiskController are validated.
      1. If both of them require a SCSI controller, rootDiskController gets chosen.
      2. If only one of them requires a SCSI controller, that type gets chosen.
      3. If neither of them require a SCSI controller, do not create any controllers (skip to step 3).
   3. CloudStack verifies if the SCSI controllers exist.
      1. If they do not exist or the existing controller does not belong to the required subtype:
         1. All existing disk controllers are removed from the VM.
         2. A VirtualMachineConfigSpec is instantiated.
         3. 4 instances of the disk controller are created, and each one of them is associated with a new VirtualDeviceConfigSpec that gets added to the VirtualMachineConfigSpec.
         4. com.cloud.hypervisor.vmware.mo.VirtualMachineMO#configureVm is called to configure the instance by providing the VirtualMachineConfigSpec.
3. If the virtual machine does not exist, the VM creation workflow is executed, with the pair from step 1 being provided as a parameter.
4. If the VM does not use a deploy-as-is template, ideUnitNumber and scsiUnitNumber are initialized at 0; otherwise, they are initialized to the next available IDE and SCSI unit number respectively. The next available unit number is obtained by finding the first IDE/SCSI controller, finding all the disks associated with it and verifying what device numbers they are using.
5. CloudStack configures the instance's disks. For each disk:
   1. The disk's controller type is identified.
      1. If the machine uses a deploy-as-is template, CloudStack verifies if the disk is already associated to a controller by comparing its bus' name with hard-coded values representing the disk controller type.
      2. If the type matches IDE, the equivalent string is used.If the type matches SCSI or does not match anything, CloudStack iterates through all the instance's devices and uses the string representing the found SCSI controllers' subtype.
      3. If the machine does not use a deploy-as-is template:
         1. If the volume is a root disk, choose the string obtained from rootDiskController.
         2. If the volume is a data disk, choose the string obtained from dataDiskController.
   2. CloudStack finds which controller the disk should be associated with.
      1. If the disk uses IDE:
         1. CloudStack calculates which IDE controller the disk should be associated with (deciding it if should be associated with the first or the second controller, for instance) by diving ideUnitNumber by the maximum number of IDE controllers (this is wrong, but it works since the maximum number of devices an IDE controller can have is the same).
         2. CloudStack iterates through all devices until it finds the nth IDE controller (calculated in the previous step) to obtain its key.
         3. If it is a data disk, CloudStack verifies if there are already 4 disks attached to the VM, and throws an exception if so.
      2. If the disk uses SCSI:
         1. If scsiUnitNumber corresponds to a reserved device, it is incremented.
         2. CloudStack calculates which SCSI controller the disk should be associated with by diving scsiUnitNumber by the maximum number of devices a SCSI controller can have.
         3. CloudStack iterates through all devices until it finds the nth SCSI controller (calculated in the previous step).
         4. CloudStack verifies if the controller belongs to the correct subtype, if it has not reached the maximum amount of associated devices, and if its bus number does not exceed the maximum amount of SCSI controllers. If all conditions are met, it chooses that controller's key.
         5. If a key has not been chosen in the previous step, there are SCSI controllers belonging to another subtype. Therefore, CloudStack iterates through all the instance's devices verifying their class to find the subtype and repeats steps 5.b.ii.3 and 5.b.ii.4.
   3. If the VM does not have a snapshot:
      1. Based on the required controller type's maximum amount of devices and ideUnitNumber/scsiUnitNumber, CloudStack calculates which device number the disk should be associated with.
      2. The VirtualDevice representing the disk is created and associated with the chosen controller's key (5.b) and the calculated device number (5.c.i).
      3. A VirtualDeviceConfigSpec is added to the VM configuration spec instructing VMware to configure the disk.
      4. If the disk uses IDE, ideUnitNumber is incremented; otherwise, scsiUnitNumber is incremented.

In order to implement the proposed changes, we will need to change some parts of this process. Most of the changes will be small adjustments so that the code uses the values stored in the database instead of the hard-coded values. However, we will also reorder some parts of the workflow.

The verification of the disk controllers required by the settings rootDiskController and dataDiskController will be moved to the start of the workflow so that it gets done right after obtaining from the database the mappings which they point to. If both settings require different controllers that share the same bus type, this process makes the data disks use the controllers required by the root disk instead so that the number of volumes the instance can have is maximized. This change will be done to avoid repeating code, as this is also needed in the VM creation workflow, and to organize the code better, as the processes that analyze these two settings will be centered at the start of the workflow.

Moreover, during the configuration of the VM's disks, the check of whether the VM has a snapshot will be placed before the identification of which controller the disk should be associated with. This will be done to organize the code as well. As the disk does not get associated with the controller if the VM has a snapshot, there is no point in identifying it beforehand.

The new workflow is summarized in the flowchart below. 

Here is a detailed description of the new process:

1. CloudStack will search for mappings with names matching the configurations rootDiskController and dataDiskController, and store their VmwareDiskControllerMappings in a pair. If there is no corresponding mapping for one of these configurations, an exception will be thrown.
2. If the virtual machine exists:
   1. If the VM does not have a snapshot, detach all virtual disks.
   2. Based on the mappings obtained from the settings, verify what disk controllers the instance should have:
      1. If the controllers are OS recommended, obtain the recommended controllers before performing any validation.
         1. After obtaining the recommended controllers, if both root and data disk controllers share the same bus, prioritize using the root disk controller type for the data disks.
      2. If the machine is a system VM, we will choose the root disk's controller type.
      3. Otherwise, we will choose both controller types.
   3. If the required controllers exist, skip to step 5.
   4. Verify if the new controllers support the amount of existing disks. If they do not support, throw an exception.
   5. Remove all existing disk controllers from the VM.
   6. Instantiate a VirtualMachineConfigSpec.
   7. For each of the chosen controller types:
      1. If the controller is an IDE, skip to the next iteration.
      2. Decide how many controllers to create:
         1. If the machine is a system VM, create a single disk controller.
         2. If the machine is not a system VM, create the maximum possible amount of the controller.
      3. Repeat until the chosen amount has been created:
         1. Instantiate the VirtualDeviceConfigSpec
         2. Instantiate the disk controller type's class.
         3. If the disk controller is a subtype of SCSI, configure it to not share the bus.
         4. Associate the disk controller with the VirtualDeviceConfigSpec.
         5. Add the VirtualDeviceConfigSpec to the VirtualMachineConfigSpec.
   8. Call com.cloud.hypervisor.vmware.mo.VirtualMachineMO#configureVm to configure the instance by providing the VirtualMachineConfigSpec.
3. If the virtual machine does not exist, call com.cloud.hypervisor.vmware.mo.VmwareHypervisorHost#createBlankVm to begin the VM creation workflow, providing the pair from step 1.
4. Create a hash-map to store the current unit number for each required controller type (the keys can be controller_reference):
   1. If the VM does not use a deploy-as-is template, initialize all values as 0.
   2. If the VM uses a deploy-as-is template, for each of the required types, iterate through all the the instance's devices to find the first controller belonging to the required type. Then, verify what device nodes are being used. Initialize the value to the next available device node.
5. Configure the instance's disks. For each disk:
   1. If the disk an ISO, execute the ISO configuration workflow and go to the next iteration.
   2. If the disk is a root disk and the VM uses a deploy-as-is template, configure the disk through com.cloud.hypervisor.vmware.resource.VmwareResource#resizeRootDiskOnVMStart and go to the next iteration.
   3. Identify what disk controller type to use for the disk:
      1. If the machine uses a deploy-as-is template, iterate through all existing disk controllers and obtain their equivalent mappings. Verify if the disk is already associated to a controller by comparing its bus' name with the mappings' bus name.
         1.  If a mapping matches, choose it.
         2. If no mapping matches, choose the mapping for any existing disk controller that is available, prioritizing types that are not IDE.
      2. If the machine does not use a deploy-as-is template:
         1. If the volume is a root disk, choose the mapping obtained from rootDiskController.
         2. If the volume is a data disk, choose the mapping obtained from dataDiskController.
   4. If the VM does not have a snapshot:
      1. Find which controller the disk should be associated with:
         1. Calculate which controller the disk should be associated with by dividing the corresponding entry in the hash-map by the maximum number of devices (max_device_count) the chosen type can have.
         2. Iterate through all devices to find the nth (calculated in the previous step) disk controller belonging to the required type.
         3. Verify that the controller has not reached the maximum amount of associated devices (max_device_count), and that its bus number does not exceed the maximum amount of that controller type (max_controller_count). If these conditions are met, choose that controller's key.
      2. Calculate which device number the disk should be associated with (currentUnitNumber mod max_device_count).
      3. Create the VirtualDevice representing the disk and associate it with the controller's key and the calculated device number.
      4. Configure the disk's equivalent VirtualDeviceConfigSpec (in deviceConfigSpecArray[i]).
      5. Increment the appropriate entry in the hash-map for the controller type being used.
      6. If the hash-map entry for SCSI corresponds to a reserved device, increment it again.

2.3 VM creation

The VM creation process can only begin from step 3 in the VM start workflow when the instance does not exist. This process receives, from the previous workflow, a pair of strings that identify the root and data disk controllers. 

The relevant parts are described below.

1. A VirtualMachineConfigSpec is instantiated.
2. CloudStack verifies if either the root or data disks require SCSI controllers:
   1. If neither of them require, this workflow ends.
   2. If only one requires, that subtype gets chosen.
   3. If both of them require, the root disk's subtype is chosen.
3. A VirtualDeviceConfigSpecs instructing VMware to add a disk controller is created. If the machine is a system VMs, this step is executed a single time, and 4 times otherwise:
   1. A VirtualDeviceConfigSpec is instantiated.
   2. To identify the controller's class, the chosen string gets compared with the ENUMs from com.cloud.hypervisor.vmware.mo.DiskControllerType.
   3. The disk controller is instantiated.
   4. If the disk controller is a subtype of SCSI, it gets configured to not share the bus.
   5. The disk controller is associated with the VirtualDeviceConfigSpec.
   6. The VirtualDeviceConfigSpec is added to the VirtualMachineConfigSpec.
4. A virtual machine creation task is sent to VMware to create an instance using the VirtualMachineConfigSpec. 

First, we will adjust the VM start process to provide a pair of VmwareDiskControllerMappings that identify the root and data disks controllers. This pair will already contain the chosen disk controllers instead of the controllers specified by the instance settings without any conversion. Then, the workflow will adapted.

1. Instantiate the VirtualMachineConfigSpec.
2. Execute the same steps as in 2.b from the VM start process to verify what disk controllers the instance should have.
3. Execute the same steps as in 2.g from the VM start process in order to configure the VirtualMachineConfigSpec instructing VMware to create the required controllers.
4. Send a virtual machine creation task to VMware to create an instance using the VirtualMachineConfigSpec.

2.4 VM import

The relevant parts of the virtual machine import process are the following: 

1. A GetUnmanagedInstancesCommand is executed to fetch all unmanaged instances from VMware.
2. CloudStack searches the GetUnmanagedInstancesAnswer for the UnmanagedInstanceTO object containing the instance's data; if no instance is found, the workflow ends here.
3. A hash-map is created to store the instance's details.
4. CloudStack verifies the UnmanagedInstanceTO's disks (a list of UnmanagedInstanceTO.Disks). If there is a root disk, it inserts the disk's controller attribute into the hash-map with the key rootDiskController. If there are data disks, it inserts the first data disk's controller attribute into the hash-map with the key dataDiskController.
5. The hash-map is provided as a parameter to com.cloud.vm.UserVmManagerImpl#importVM to import the instance.

Here, we only need to adjust GetUnmanagedInstancesCommand's response so that the UnmanagedInstanceTO.Disks' controller attribute contain the names mapped in the database. This can be done by changing com.cloud.hypervisor.vmware.util.VmwareHelper#getUnmanageInstanceDisks, which creates the list containing the disks: instead of using com.cloud.hypervisor.vmware.mo.DiskControllerType#getType to obtain the disk controller's name based on the device's class, we will consult the available mappings to find if there is an entry mapping to the class; if there is, the controller attribute will be set to the mapping's name field; otherwise, it will be set to the string none in order to preserve the current behavior.

2.5 Disk attachment

The relevant parts of the disk attachment process are below.

1. VmwareStorageProcessor receives an AttachCommand containing a hash-map with the VM's rootDiskController and dataDiskController settings.
2. CloudStack chooses which disk controller to use:
   1. If dataDiskController is defined, it gets used.
   2. If dataDiskController is not defined, LSI Logic is used.
3. CloudStack finds which controller the disk should be associated with.
   1. If the required controller is an IDE:
      1. CloudStack iterates through all the instance's devices to verify how many disks over IDE there are, and throws an exception if the maximum amount has been reached.
      2. CloudStack finds which controller the disk should be associated with by dividing the current total number of disks by the maximum number of IDE controllers.
      3. CloudStack iterates through all devices until it finds the nth IDE controller.
      4. CloudStack counts how many VirtualDisks are associated with that controller. If the maximum number of devices has been reached, an exception is thrown; otherwise, this count corresponds to the device number the disk should be associated with.
   2. If the required controller is a SCSI:
      1. To find the controller the disk should be associated with, CloudStack iterates through all the instance's devices, verifying if they match the required disk controller's class, if the maximum amount of associated devices has not been reached, and if its bus number does not exceed the maximum amount of SCSI controllers. If all conditions are met, the controller is chosen.
      2. CloudStack decides the device number to associate with the disk:
         1. CloudStack iterates through all devices and, if their controller's key matches the chosen controller's key, it stores their device number on a list.
         2. The first device number that is neither on the list, nor reserved is chosen. 
4. The VMDK adapter is adjusted:
   1. CloudStack finds the appropriate VMDK adapter type for the new disk controller based on hardcoded ENUMs.
   2. If there is not a corresponding VMDK adapter type, an exception is thrown.
   3. If the new VMDK adapter type differs from the current one, the VMDK file's ddb.adapterType property gets updated. 
5. A VirtualMachineConfigSpec is instantiated.
6. The disk is instantiated and associated with the chosen disk controller, device number, and a new VirtualDeviceConfigSpec that gets added to the VirtualMachineConfigSpec.
7. A virtual machine reconfiguration task is sent to VMware in order to configure the instance based on the VirtualMachineConfigSpec. 

We propose the following changes in this process:

1. VmwareStorageProcessor will receive an AttachCommand containing a pair with the VM's rootDiskController and dataDiskController settings.
2. Choose which disk controller to use:
   1. Execute the same process as in VM start's step 1 to get the disk controller mappings for rootDiskController and dataDiskController.
   2. If the controllers are OS recommended, obtain the recommended controllers.
      1. After obtaining the recommended controllers, if both root and data disk controllers share the same bus, prioritize using the root disk controller type for the data disks.
   3. If the disk is being attached as a root disk, choose the controller required for the root disk; otherwise, choose the controller required by the data disks.
3. Find which controller the disk should be associated with:
   1. Iterate through all the instance's devices, verifying if they match the required disk controller's class, if the maximum amount of associated devices has not been reached, and if its bus number does not exceed the maximum amount of that controller type. If all conditions are met, choose that controller. For SCSI devices, device number 7 will need to be considered in the second validation.
4. Find which device number the disk should be associated with:
   1. Iterate through all devices again, verifying if they are associated with the chosen disk controller. If a device is, store its device number in a list.
   2. If the controller is a subtype of SCSI, add device number 7 to the list.
   3. Based on the stored values, choose the minimum available device number.
5. Update the VMDK adapter (ddb.adapterType) to the new disk controller's vmdk_adapter_type column.
6. Instantiate a VirtualMachineConfigSpec.
7. Instantiate the disk, associate it with the chosen disk controller, device number and a new VirtualDeviceConfigSpec that gets added to the VirtualMachineConfigSpec.
8. Send a virtual machine reconfiguration task to VMware in order to configure the instance based on the VirtualMachineConfigSpec.

3 Future works

This spec addressed the creation of a flexible method of adding support for new disk controllers without having to extend CloudStack. This initial proposal is for environments using VMware; however, this feature can also be extended for KVM and XenServer in the future. Moreover, we can also create APIs to allow adding, removing and listing the controllers available for each hypervisor.