Summary

As of now, CloudStack has limited support for GPUs. It supports only Xenserver and some workarounds are possible with KVM but it has a lot of limitations. This feature allows the operator to discover the GPU devices on the host and create a Compute Offering with GPU support based on the available GPU devices on the host. Once the operator has created the Compute offering, it can be used by users to launch Instances with GPU devices.

Limitations

CloudStack assumes that the operator has configured the GPU devices correctly on the host. The role of CloudStack is just to discover the devices on the host and assign/unassign them to the Instance.
Admins will need to build custom templates with GPU tools/drivers installed to have their users effectively use Instances with GPUs.
Apart from this, the following Instance operations for GPUs, are unsupported (as of now):
- Instance snapshot with memory for GPU enabled Instance is not supported.
- Live Migration of GPU enabled Instance is not supported. However, cold Instance migration would be possible by stopping the Instance, optionally migrating its disks and starting it on another compatible GPU-enabled host. We have added framework level support for live migration of Instances with vGPUs, but we are unable to verify due to unavailability of hardware.
- Dynamic scaling of Instances with GPU attached

High-Level Overview

As part of the GPU integration with KVM in CloudStack, these are the features we have integrated with GPU support for KVM:

GPU Discovery and Inventory Management
GPU Classification and Mapping
GPU Limits and Placement
Flexible Hypervisor GPU Hooks

Key capabilities include:

Support for passthrough and vGPU
Auto-discovery of GPU devices on the host
GPU inventory management and allocation
Host allocation

1. GPU Discovery and Inventory Management

For GPU discovery, the agent executes a script to discover the GPU devices on the host and sends the discovered devices to the management server. The management server uses the response from the agent to update the status of the GPU devices on the host. If a discovered GPU device doesn't exist in the database, it will be created. If a discovered GPU device already exists in the database, it will be updated with the new status. The administrator can also trigger the discovery of GPU devices on the host from the host details page. The details of the discovered GPU devices are stored in the database. The administrator can also view the details of the discovered GPU devices on the Host Details page and update them as per the requirements.

2. GPU Classification and Mapping

For GPU classification and mapping, we have created two new resource types:

GPU Card - Represents the physical GPU devices on the host.
vGPU Profile - A GPU Card can have multiple vGPU profiles. Each vGPU profile represents the type of vGPU profile that can be passed through to the guest instance (e.g., passthrough, GRID V100-1Q, GRID V100-4C, etc.).

The management server uses the discovered GPU devices to map them to the vGPU Profile based on information from the GPU inventory from the CloudStack agent.

If the name of the vGPU profile is passthrough , then the PCI device is passed through to the guest Instance as a PCI device using vfio-pci.

Some GPU devices allow partitioning of the GPU devices into multiple virtual functions (VF) or mediated devices (mdev). The possible partitions of the GPU devices are the names of the vGPU profile in CloudStack.

The vGPU profiles also contain metadata such as video memory, memory, maximum vGPUs per physical GPU, and resolution. If the GPU Card and vGPU Profile don't exist, then the management server will create a new GPU Card and vGPU Profile.

The required profile and the number of GPUs per instance can be specified in the Compute Offering.

When an instance is launched, the management server will check if the GPU devices with the required profile and count are available on the host. If the GPU devices are not available, the management server will not allow the creation of a new instance. If the GPU devices are available, the management server will allocate the required number of GPU devices to the instance.

3. GPU Limits and Placement

Administrators can specify tenant limits as the maximum number of GPU devices allowed on a per account/project/domain basis. The default account, domain, and project limits can be controlled through the global settings below:

max.account.gpus - The default maximum number of GPU devices that can be used for an account.
max.domain.gpus - The default maximum number of GPU devices that can be used for a domain.
max.project.gpus - The default maximum number of GPU devices that can be used for a project.

There is also the question of whether to consider the GPU devices as allocated to a stopped instance or not. For this, we have a new global setting called gpu.detach.on.stop which is set to false by default. If this is set to true, the GPU devices will be detached from the instance when it is stopped. This global setting can be configured per domain.

This is useful for cases when the operator doesn't want to bill users for GPUs on a stopped Instance and also allows the operator to use the GPU devices for other purposes.

We also track GPU capacity. This is also used to display the GPU capacity on the root admin dashboard. The capacity is calculated using the GPU devices on the host.

Note: Both passthrough and a device with a vGPU profile are counted as 1 GPU device. For example, if an instance is launched with 2 GPUs, then the count for the account/project/domain will be increased by 2 irrespective of the vGPU profile. If a GPU device has vGPU profiles created, it doesn’t include passthrough in the total count.

4. Flexible Hypervisor GPU Hooks

Discovery Hook

We have created a new hypervisor hook for discovery of GPU devices on the host. This script is called by the CloudStack agent to discover the GPU devices on the host. The script is located at /usr/share/cloudstack-agent/scripts/gpudiscovery.sh. The script will output a JSON like the one below with the GPU devices. This script can be updated by the administrator, but the output format should remain the same.

{
    "gpus": [
    {
      "pci_address": "01:00.0",
      "vendor_id": "10de",
      "device_id": "25a0",
      "vendor": "NVIDIA Corporation",
      "device": "GA107M [GeForce RTX 3050 Ti Mobile]",
      "driver": "nvidia",
      "pci_class": "3D controller [0302]",
      "iommu_group": "16",
      "pci_root": "0000:01:00.0",
      "numa_node": -1,
      "sriov_totalvfs": 0,
      "sriov_numvfs": 0,
      "full_passthrough": {
        "enabled": 1,
        "libvirt_address": {
          "domain": "0x0000",
          "bus": "0x01",
          "slot": "0x00",
          "function": "0x0"
        },
        "used_by_vm": null
      },
      "vgpu_instances": [],
      "vf_instances": []
    }
  ]
}

Domain XML Transformer Hook

While CloudStack already supports Groovy scripts, we have added support for shell scripts:

Libvirt Domain XML Transformer (libvirt-vm-xml-transformer.sh)

Instance Start hook (libvirt-vm-state-change.sh)

Instance Stop hook (libvirt-vm-state-change.sh)

The shell scripts can be placed in the /etc/cloudstack/agent/hooks directory.

The shell scripts can be used to execute any other script. The scripts take in the following arguments for each hook:

Transformer hook:
- transform
- libvirt xml

Start hook:
- start
- Instance name

Stop hook:
- stop
- Instance name

Only the response for the transformer hook is used. The response for the start and stop hooks is ignored.

API & Schema Changes

We have added the following new tables:

gpu_card - This table contains the metadata of the physical GPU Card
vgpu_profile - This table contains the metadata for vGPU profiles which are allowed to be created by the vendor for a GPU Card.
gpu_device - This table contains information about the discovered gpu devices on a host and links them with GPU card & vgpu profile

Updates to exisiting tables:

service_offering
- vgpu_profile_id - The vGPU profile to be used for the guest Instance
- gpu_count - number of gpu devices to attach to the guest Instance
- gpu_display - Whether to use the gpu cards as display on the guest Instance

We have added the following new APIs:

GPU Card

createGpuCard - Allow creation of a GPU Card to store metadata
updateGpuCard - Update GPU card information
listGpuCards - List GPU cards
deleteGpuCard - Remvove GPU Card

vGPU Profiles

createVgpuProfile - Creates a vGPU profile for a GPU Card
updateVgpuProfile - Update the metadata for vGPU profile
listVgpuProfiles - List vGPU profiles
deleteVgpuProfile - Deletes a vGPU profile

GPU Devices

createGpuDevice - Allow creating of a custom GPU devices on a host
updateGpuDevice - Update GPU Device details
listGpuDevices - List GPU Devices
unmanageGpuDevice - Disables a GPU devices to be used from CloudStack
discoverGpuDevices - Discover GPU Devices on a host
manageGpuDevice - Enables a GPU device to be used from CloudStack
deleteGpuDevice - Deletes the GPU Device from CloudStack. Discovering the GPU devices may add the deleted GPU device again to the list.

Configuration Settings

The configuration settings changes below, are incorporated.

Configuration	Description	Default Value	Scope
gpu.detach.on.stop	Whether to detach GPU devices from Instance on stop or keep them allocated	false	Domain
max.account.gpus	The default maximum number of GPU devices that can be used for an account	20	Global
max.domain.gpus	The default maximum number of GPU devices that can be used for a domain	20	Global
max.project.gpus	The default maximum number of GPU devices that can be used for a project	20	Global

Usage Instructions

As part of the GPU integration, we need to configure the GPU on the host and install or upgrade the CloudStack packages with GPU support.

Configure GPUs on the host
Check if GPU devices are discovered on the host. If not, trigger the discovery.
Create a Compute Offering. Select the GPU Card, vGPU profile, and GPU count.
Launch a GPU-enabled Instance

1. Configure GPU on Host

For passthrough, we need to enable IOMMU and ensure vfio-pci is loaded and used for the GPU devices. For vGPU, follow the steps below to enable IOMMU and then follow the instructions here to create a vGPU profile.

Enable IOMMU on Host

Before proceeding, make sure that IOMMU is enabled in BIOS. It's called "Intel VT-d" for Intel processors and "AMD IOMMU" for AMD processors.

Edit the GRUB configuration file:
$ sudo vi /etc/default/grub
Add the "intel_iommu=on" for Intel or "amd_iommu=on" for AMD to the GRUB configuration file /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
Update the GRUB configuration file. Depending on the distribution, the command may be different:
$ sudo update-grub
# or
$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Reboot the host:
$ sudo reboot
Check if IOMMU is enabled on the host:
$ grep IOMMU /var/log/dmesg
# or
$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-default [...] intel_iommu=on [...]

Ensure the PCI Device is Not in Use by the Host

Disable the nouveau kernel module and add the following line to the top of the /etc/modprobe.d/50-denylist.conf file:
blacklist nouveau
Check if vfio-pci is loaded and used for the GPU devices:
$ lspci -k | grep -A3 -E "3D controller|VGA"
00:02.0 VGA compatible controller: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] (rev 0c)
Subsystem: Dell Device 0b19
Kernel driver in use: i915
Kernel modules: i915, xe
--
01:00.0 3D controller: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] (rev a1)
Subsystem: Dell Device 0b19
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
Prevent the GPU devices from being used by the host:
$ lspci -nn | grep -E "3D controller|VGA"
00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] [8086:46a6] (rev 0c)
01:00.0 3D controller [0302]: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] [10de:25a0] (rev a1)
Update grub configuration file /etc/default/grub. In this example, we are preventing the NVIDIA GPU with "vendor ID:Device ID" 10de:25a0 from being used by the host:
GRUB_CMDLINE_LINUX_DEFAULT=".... pci-stub.ids=10de:25a0"
Update grub configuration file:
$ sudo update-grub
or
$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
Reboot the host:
$ sudo reboot

Install GPU drivers

Download the GPU drivers from your vendor. For nvidia, login to https://www.nvidia.com/en-us/drivers/vgpu-software-driver/ and download the drivers for your host & GPU Card.

Note: This is only required to create vGPUs. If only full passthrough is required, this can be ignored.

2. Check if GPU Devices are Discovered on the Host

Open the GPU tab on the Host Details page and verify if the GPU devices are discovered on the host.
If the GPU devices are not discovered, trigger the discovery using the "Discover" button.

If the GPU devices are still empty, execute the following script on the host which CloudStack uses to discover the GPU devices for debugging. The output should be a JSON file with the GPU devices:
$ sudo /usr/share/cloudstack-agent/scripts/gpudiscovery.sh
Example JSON output:

{
    "gpus": [
    {
      "pci_address": "01:00.0",
      "vendor_id": "10de",
      "device_id": "25a0",
      "vendor": "NVIDIA Corporation",
      "device": "GA107M [GeForce RTX 3050 Ti Mobile]",
      "driver": "nvidia",
      "pci_class": "3D controller [0302]",
      "iommu_group": "16",
      "pci_root": "0000:01:00.0",
      "numa_node": -1,
      "sriov_totalvfs": 0,
      "sriov_numvfs": 0,
      "full_passthrough": {
        "enabled": 1,
        "libvirt_address": {
          "domain": "0x0000",
          "bus": "0x01",
          "slot": "0x00",
          "function": "0x0"
        },
        "used_by_vm": null
      },
      "vgpu_instances": [],
      "vf_instances": []
    }
  ]
}

The Host Details page should look something like below:

3. Create a Compute Offering with GPU

Once the GPU devices are discovered on the host, we can create a Compute Offering.

Create a Compute Offering. Select the GPU Card, vGPU profile, and GPU count. The GPU card and vGPU profile should match the GPU devices discovered on the host.
Create a Virtual Machine. Select the Compute Offering.
Check if the GPU devices are visible to the guest Instance.

4. Launch a GPU-enabled Instance

Once you have created the compute offering,

Launch a Virtual Machine with GPU Enabled offering. Make sure the selected template supports GPU drivers.
Once your instance is launched, it will have the required GPU devices attached to it. If there are no available GPU devices, it Instance launch will fail.

Space shortcuts

Child pages

Summary

Limitations

High-Level Overview

1. GPU Discovery and Inventory Management

2. GPU Classification and Mapping

3. GPU Limits and Placement

4. Flexible Hypervisor GPU Hooks

Discovery Hook

Domain XML Transformer Hook

API & Schema Changes

Configuration Settings

Usage Instructions

1. Configure GPU on Host

Enable IOMMU on Host

Ensure the PCI Device is Not in Use by the Host

Install GPU drivers

2. Check if GPU Devices are Discovered on the Host

3. Create a Compute Offering with GPU

4. Launch a GPU-enabled Instance

Space shortcuts

Child pages

Support for GPU with KVM hosts

Summary

Limitations

High-Level Overview

1. GPU Discovery and Inventory Management

2. GPU Classification and Mapping

3. GPU Limits and Placement

4. Flexible Hypervisor GPU Hooks

Discovery Hook

Domain XML Transformer Hook

API & Schema Changes

Configuration Settings

Usage Instructions

1. Configure GPU on Host

Enable IOMMU on Host

Ensure the PCI Device is Not in Use by the Host

Install GPU drivers

2. Check if GPU Devices are Discovered on the Host

3. Create a Compute Offering with GPU

4. Launch a GPU-enabled Instance