...

• CloudStack assumes that the operator has configured the GPU devices correctly on the host. The role of CloudStack is just to discover the devices on the host and assign/unassign them to the Instance.
• Admins will need to build custom templates with GPU tools/drivers installed to have their users effectively use Instances with GPUs.
• Apart from this, the following VM Instance operations for GPUs, are unsupported (as of now):
  • VM Instance snapshot with memory for GPU enabled VM Instance is not supported.
  • Live Migration of GPU enabled VM Instance is not supported. However, cold VM Instance migration would be possible by stoping stopping the VMInstance, optionally migrating its disks and starting it on another  compatible GPU-enabled host. We have added framework level support for live migration of VMs Instances with vGPUs, but we are unable to verify due to unavailability of hardware.
  • Dynamic scaling of VMs Instances with GPU attached

High-Level Overview 

...

• Libvirt Domain XML Transformer (libvirt-vm-xml-transformer.sh)

• VM Instance Start hook (libvirt-vm-state-change.sh)

• VM Instance Stop hook (libvirt-vm-state-change.sh)

...

• Transformer hook:
  • transform
  • libvirt xml

• Start hook:
  • start
  • vm Instance name

• Stop hook:
  • stop
  • vm Instance name

Only the response for the transformer hook is used. The response for the start and stop hooks is ignored.

...

• service_offering
  • vgpu_profile_id - The vGPU profile to be used for the guest VM Instance
  • gpu_count - number of gpu devices to attach to the guest VM Instance
  • gpu_display - Whether to use the gpu cards as display on the guest vm Instance

We have added the following new APIs:

...

The configuration settings changes below, are incorporated.

Usage Instructions

As part of the GPU integration, we need to configure the GPU on the host and install or upgrade the CloudStack packages with GPU support.

...

Ensure the PCI Device is Not in Use by the Host

1. Disable the nouveau kernel module and add the following line to the top of the /etc/modprobe.d/50-denylist.conf file: 
   blacklist nouveau
2. Check if vfio-pci is loaded and used for the GPU devices:
   $ lspci -Check if vfio-pci is loaded and used for the GPU devices:
   $ lspci -k | grep -A3 -E "3D controller|VGA"
   00:02.0 VGA compatible controller: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] (rev 0c)
           Subsystem: Dell Device 0b19
           Kernel driver in use: i915
           Kernel modules: i915, xe
   --
   01:00.0 3D controller: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] (rev a1)
           Subsystem: Dell Device 0b19
           Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
3. Prevent the GPU devices from being used by the host:
   $ lspci -nn | grep -E "3D controller|VGA"
   00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] [8086:46a6] (rev 0c)
   01:00.0 3D controller [0302]: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] [10de:25a0] (rev a1)
4. Update grub configuration file /etc/default/grub. In this example, we are preventing the NVIDIA GPU with "vendor ID:Device ID" 10de:25a0 from being used by the host:
   GRUB_CMDLINE_LINUX_DEFAULT=".... pci-stub.ids=10de:25a0"
5. Update grub configuration file:
   $ sudo update-grub
   or
   $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
6. Reboot the host:
   $ sudo reboot

Install GPU drivers

1. Download the GPU drivers from your vendor. For nvidia, login to https://www.nvidia.com/en-us/drivers/vgpu-software-driver/ and download the drivers for your host & GPU Card.

Note: This is only required to create vGPUs. If only full passthrough is required, this can be ignored.

2. Check if GPU Devices are Discovered on the Host

...