Support OVA files containing multiple disks

Bug Reference

CLOUDSTACK-4757

Branch

Work is in progress in this PR  https://github.com/apache/cloudstack/pull/2146

Tested the PR that was submitted by Citrix. Made architectural changes, bug fixes, enhancements and feature additions for it to be useful.

Introduction

OVA is an archive or an envelope specification that can contain a complete VM including multiple disks and other files such as ISOs. And CloudStack templates in case of VMware are in OVA format.

Today in CloudStack, we only support template creation based on OVA files that contain a single disk. If an Admin/User registers a template using OVA that contains more than 1 disk or iso files and launches an instance using this template, only the first disk is attached to the new instance and the rest of the disks and iso's are ignored. 

Purpose

The purpose of this document is to present functional specifications for improvements that are required to extend the current OVA support to include multiple disks.

Glossary

VM template – Represents a ROOT disk contained in an OVA along with other VM related information like Guest OS.

Datadisk Template - An abstraction for a DATA volume contained in an OVA

Iso Template - An abstraction for an ISO contained in an OVA

Goals

The current CloudStack behavior needs to be improved to support OVA files with multiple disks and/or an ISO.

In case of an instance launched from a template that is an OVA containing more than 1 disk 

• First, disk should be attached to the new instance as the ROOT disk
• Volumes should be created for the additional disks and attached to the instance as additional DATA disks
• Any ISO contained is mounted as CD-ROM and connected.

Scope

The scope of this proposal is limited to adding support for additional disks or ISO present in a simple OVA that is registered in CloudStack. It is not in the scope to fully support an OVA and hence CloudStack will ignore any other information that is encapsulated in the OVA which may include information about networks, multiple Virtual machines. The limitations on OVA are contained in the Limitations section.

Feature specifications

Currently, in CloudStack, template abstraction is that of a VM template that contains a single disk. To represent the additional disks present in an OVA we will introduce the concept of Datadisk templates.

Datadisk templates will be

• Used to create a data disk from the volume image that is present in the template
• Stored in vm_templates tables along with the current CS templates
• Used only when a user tries to deploy a VM from the corresponding VM template and never independently

Datadisk templates will provide flexibility to orchestrate, deploy and manage additional disks present in an OVA like any other Datadisk in CloudStack.

Similarly, ISO templates are introduced in Cloudstack.

Architecture

Work flow changes

Template registration

• No change in User experience during template registration
• Once template is downloaded and installed in the secondary storage it will be post processed
• Post processing will be handled by 2 new Agent Secondary Storage commands - GetDataDisksCommand and CreateDataDiskTemplateCommand
  • GetDataDisksCommand  (this used to runs on MS by mounting SS, as the vmware agent is running in MS context) **1
    • Will temporarily import the OVA into CloudStack to identify the ROOT disk and additional data disks present (if any). If present, path and size of each of the data disks will be returned
    • Identifying the ROOT disk (remaining disks will be considered as data disks)
      
      • OVA template custom property "cloud.ova.bootdisk" will be used to determine the ROOT disk. If this property is not set the first disk will be identified as the ROOT disk (again there is a check for this in the code, but surprisingly some non cloudstack ova seems to be working) The OVA template can come from non cloudstack sources so the custom cloudstack property check is removed, instead the right controller is configured thru cloudstack so that the resulting VM boots properly.
      • The value of parameter “cloud.ova.bootdisk” should be in format ”<controllertype><controllerId>:<devicenumber_on_this_controller>” e.g. “cloud.ova.bootdisk”=”scsi0:2”
        • Possible values for <controllertype> - "ide" or "scsi", <controllerId> - 0, 1, 2, 3, <devicenumber_on_this_controller> - 0 to 15
        • Custom property can be added to VM before creating OVA from the VM per VMware documentation. Refer -  Modifying Virtual machine custom property
    • For each of the additional disks, Datadisk and ISO template entries will be made in the DB
      • Entries will be made in vm_templates and vm_template_store_ref table
      • Datadisk Template will be referenced to the corresponding VM template by with DB column parent_template_id where parent_template_id points to the VM template
  • CreateDataDiskTemplateCommand
    • This agent command will be invoked for each of the templates returned by GetDataDisksCommand
    • As part of the Datadisk template creation following files will be created to represent the template
      • An OVF descriptor based on the disk properties returned by GetDataDisksCommand
      • The OVF descriptor will be cleaned up for each of the disk to carry information just for that disk.
      • An OVA .meta file to represent an OVA
      • template.properties file to encapsulate the properties of the Datadisk template
    • Datadisk, OVF, META and template.properties file will be present in a separate folder e.g. -
      • Say the Datadisk template id is 223 then it will be present in folder  - template/tmpl/2/223/
      • Sample contents of the Datadisk template folder will be
        • 223-2-9f6a1c42-472b-3bd1-9fac-1101a6228df9.ova.meta multipledisk.ovf multipledisk-disk1.vmdk template.properties
    • If the creation of any of the individual Datadisk templates fails then we will rollback the VM template and the associated data disks that were created.

Handling of ISO's

The other functionality currently missing in the import of OVA is the treatment of ISOs. The ISOs are simply ignored. The ISO files mentioned in the OVF will be mounted as CD-ROM that is to say that ISOs mentioned in the OVF should be backing up a CD-ROM device.

Listing templates

• All template i.e. both VM templates and Datadisk templates will be listed.
• Datadisk template can be differentiated from VM templates by the property templatetypeFollowing attributes will also be different for the two type of templates – bootable(set to false for Datadisk template), guestos(unset for Datadisk template)
  • Datadisk templates will have templatetype set to 'Datadisk'
  • VM templates will continue to have templatetype set to 'User'

Virtual Machine deployment

• If user tries to deploy a VM from a template containing multiple disks, user will have to choose a Datadisk offering for every Datadisk template chosen to be deployed along with the VM templateDuring VM create process, when CloudStack makes DB entry for the ROOT disk, DB entries will be made in volumes table for each of the Datadisk templates
  • If the actual size of the disk exceeds the size of the disk offering then VM deployment will fail
  • The UI is modified to let the user select disk offerings for each contained disk.
• During VM orchestration while the storage is being prepared, a volume will be created in primary storage for each of the additional disks
  • Each of the volumes will be migrated from secondary storage to the primary storage chosen by the deployment planner
  • When VM starts it will be booted with all the chosen disks and ISO

• The disks are resized to match the disk offering selected for the disk. 

Template deletion

• Deletion of individual Datadisk Templates will be allowed
• When a VM template is deleted, all the corresponding Datadisk templates will deleted

Template copy

• Copy template will not be supported on a individual Datadisk template
• When a VM template is copied to a zone, all the corresponding Datadisk templates will also be copied to the zone

Template update

• Users will be allowed to update Datadisk templates
  • Template attributes that can be updated - bootable, displayext, format, name, ostypeid, sortkey

Update template permissions

• Not supported for Datadisk Templates
• By default the Datadisk templates will be private and non-featured

Change in the existing behavior

• When a user downloads a CS template with multiple volumes, the entity that is downloaded will not be same as the one that was registered
  • The downloaded template will only contain the ROOT disk because when a OVA template is registered it is now consumed and split into 2 or more templates (VM Template and Datadisk Template(s))

Web Service API

DeployVirtualMachine API

New parameter – datadisktemplatetodiskofferinglist

• Mapping from Datadisk Template to Datadisk Offering
• If a VM is being deployed from an OVA template that contains additional disks, then during the deployment user will have to choose a Datadisk offering for every Datadisk template he/she chooses to deploy along with the VM template
• Mutually exclusive with the existing parameter diskofferingid
• Optional
• Usage - &disktemplatetodiskofferinglist[0].disktemplateid=2&disktemplatetodiskofferinglist[0].diskofferingid=3

UI Flow

Deploy VM wizard

• While listing all templates list only VM templates i.e. non Datadisk templatesOnce the user has selected the VM template allow user to pick between disk offering or Datadisk template with the help of radio buttons (user can choose to select neither)

  • VM templates can be distinguished from the Data disk templates by property ‘templatetype’. VM templates will have the type set to USER whereas Datadisk templates will have the type set to DATADISK.

• If user chooses disk offering then rest of the work flow will remain the same as today
• If user chooses Datadisk template then list all the Datadisk templates that belong to the VM template chosen in the first step (templates that have parentTemplateId parameter set to that of VM template)

DB Schema changes

• Add new column parent_template_id to vm_templates table

  • default value is null

  • for Datadisk templates, value will be Id of the VM template that the Datadisk template belongs to

  • for VM templates, value will always be null

Upgrade scenario

• In a setup that has been upgraded to 4.5, templates with multiple volumes
  • That were registered prior to upgrade will continue to operate as before. Any VM launched from these templates will only have the ROOT disk
  • That will be registered after upgrade will follow the new behavior. Any VM launched from these templates will have additional data disks attached to them along with the ROOT disk (i.e. if user chooses to)

Limitations

• CloudStack templates and uploaded volumes in case of VMware are in OVA format. For 4.10 release, we will be adding multiple disk OVA support only for CloudStack templates and not uploaded volumes.
• In 4.10, users will not be allowed to perform any independent operations on the Datadisk templates
  • User cannot use the Datadisk template to add a data disk volume after VM is created based on the template
  • Following operations will not be supported on the Datadisk templates - updateTemplatePermissions, extractTemplate, prepareTemplate (Future work)
• While deploying a Virtual Machine, user will not be able to provide custom disk offerings for datadisk templates
  • disktemplatetodiskofferinglist can only have disk offerings that are non-custom

Limitations placed by Cloudstack

Following are the limitations that are placed by current Cloudstack implementation and its limitation.

1. Cloudstack allows single disk controller for all data disk. In case the OVA contains multiple data disk only one type of disk controller will be applicable for all of them.
2. Cloudstack allows iscsi disks to be resized. Due to this limitation, if datadisk are ide for example, the resize to disk offering size will fail. And user will see actual disk sizes contained in OVA in the vCenter, not the ones selected by disk offerings.
3. The disk controller specified in the OVA file could have a wide range of types and sub-types. Cloudstack has only limited types available. In case the disk controller is not supported by cloudstack, there can be boot issues with the VM. 
4. If ISO is attached from CloudStack, it's not being picked up when exporting the VM. Therefore, in order to prepare proper template containing ISO, it must be attached from vSphere, connected, with the option to 'connect when power on' before being shut down.  
   
   

Limitations on OVA

There are some limitations on OVA template that Cloudstack will be able to handle. These limitations if found restricting can be further worked upon and removed.

Some assumptions on OVF files contained in the OVA packages are:

• The first disk is considered to be the boot disk, if it is an iso instead the VM creation will fail. If not it can result in boot issues.
• All standard parmeters like size should be in there for all the vmdks and ISOs contained in OVA
• An ovf file is considered to be in proper format and with valid information.
• It is also assumed that the imported OVA file is compatible with vCenter
• Multi-disk OVA file that bundles a single VM with multiple disks and an optional ISO
• The disks should be bundled in the OVA with only relative paths in the OVF file (href attribute)
• The disk offering size utilized for data disks needs to be equal or larger to original disk size in the OVA
  
• There will not be any empty disk in the OVF file, all disks will point to a valid File resource
• Since Cloudstack treats each disk and iso file as separate unit, the specifications contained in InstallSection or StartupSections of OVF are ignored.
• Cloudstack has its own mapping for operating systems and the information contained in OperatingSystemSection of OVF is ignored
• The files are in uncompressed format, 'ovf:compression' attribute on File will be ignored
• Files are not chunked and 'ovf:chunkSize' attribute is ignored
• Any manifest and certificate files will be ignored
• Disk format vmdk for data disk
• Essentially we will only be looking at disk information in the ovf and other spec will be ignored
• If there is a ISO in the ovf package then it will be mounted as CD-ROM
• The network section is ignored and network is instrumented by Cloudstack.
• There's no error handling in the UI when copying additional disks and ISOs, it has been observed that some templates appears as ready but are not deployable, due to a copy failure. The admin needs to monitor logs and check the template on storage when installed.