Versions Compared

Old Version 1

changes.mady.by.user Unknown User (vishesh)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (vishesh)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents
Summary

As of now, CloudStack has limited support for GPUs. It supports only Xenserver and some workarounds are possible with KVM but it has a lot of limitations.This feature allows the operator to discover the GPU devices on the host and create a Compute Offering with GPU support based on the available GPU devices on the host.Once the operator has created the Compute offering, it can be used by users to launch Instances with GPU devices. 

Limitations 

  • CloudStack assumes that the operator has configured the GPU devices correctly on the host. The role of CloudStack is just to discover the devices on the host and assign/unassign them to the VMInstance.
  • Admins will need to build custom templates with GPU tools/drivers installed to have their users effectively use VMs Instances with GPUs.
  • Apart from this, the following VM Instance operations for GPUs, are unsupported (as of now):
    • VM Instance snapshot with memory for GPU enabled VM Instance is not supported.
    • Live Migration of GPU enabled VM Instance is not supported. However, cold VM Instance migration would be possible by stoping stopping the VMInstance, optionally migrating its disks and starting it on another  compatible GPU-enabled host. We have added framework level support for live migration of VMs Instances with vGPUs, but we are unable to verify due to unavailability of hardware.
    • Dynamic scaling of VMs Instances with GPU attached

High-Level Overview 

...

  • Libvirt Domain XML Transformer (libvirt-vm-xml-transformer.sh)
  • VM Instance Start hook (libvirt-vm-state-change.sh)
  • VM Instance Stop hook (libvirt-vm-state-change.sh)

...

  • Transformer hook:
    • transform
    • libvirt xml
  • Start hook:
    • start
    • vm Instance name
  • Stop hook:
    • stop
    • vm Instance name

Only the response for the transformer hook is used. The response for the start and stop hooks is ignored.

Usage and Installation Instructions

API & Schema Changes

We have added the following new tables:

  • gpu_card - This table contains the metadata of the physical GPU Card
  • vgpu_profile - This table contains the metadata for vGPU profiles which are allowed to be created by the vendor for a GPU Card.
  • gpu_device - This table contains information about the discovered gpu devices on a host and links them with GPU card & vgpu profile

Updates to exisiting tables:

  • service_offering
    • vgpu_profile_id - The vGPU profile to be used for the guest Instance
    • gpu_count - number of gpu devices to attach to the guest Instance
    • gpu_display - Whether to use the gpu cards as display on the guest Instance

We have added the following new APIs:

GPU Card

  • createGpuCard - Allow creation of a GPU Card to store metadata

  • updateGpuCard - Update GPU card information
  • listGpuCards - List GPU cards
  • deleteGpuCard - Remvove GPU Card

vGPU Profiles

  • createVgpuProfile - Creates a vGPU profile for a GPU Card

  • updateVgpuProfile - Update the metadata for vGPU profile
  • listVgpuProfiles - List vGPU profiles
  • deleteVgpuProfile - Deletes a vGPU profile

GPU Devices

  • createGpuDevice - Allow creating of a custom GPU devices on a host
  • updateGpuDevice - Update GPU Device details
  • listGpuDevices - List GPU Devices

  • unmanageGpuDevice - Disables a GPU devices to be used from CloudStack

  • discoverGpuDevices - Discover GPU Devices on a host

  • manageGpuDevice - Enables a GPU device to be used from CloudStack

  • deleteGpuDevice - Deletes the GPU Device from CloudStack. Discovering the GPU devices may add the deleted GPU device again to the list.

Configuration Settings

The configuration settings changes below, are incorporated.

Configuration

Description

Default Value

Scope

gpu.detach.on.stop

Whether to detach GPU devices from Instance on stop or keep them allocated

false

Domain

max.account.gpus

The default maximum number of GPU devices that can be used for an account

20

Global

max.domain.gpus

The default maximum number of GPU devices that can be used for a domain

20

Global

max.project.gpus

The default maximum number of GPU devices that can be used for a project

20

Global

Usage Instructions

As part of the GPU integration, we need to As part of the GPU integration, we need to configure the GPU on the host and install or upgrade the CloudStack packages with GPU support.

...

  1. Edit the GRUB configuration file:
    $ sudo vi /etc/default/grub
  2. Add the "intel_iommu=on" for Intel or "amd_iommu=on" for AMD to the GRUB configuration file /etc/default/grub:
    GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
  3. Update the GRUB configuration file. Depending on the distribution, the command may be different:
    $ sudo update-grub
    # or
    $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  4. Reboot the host:
    $ sudo reboot
  5. Check if IOMMU is enabled on the host:
    $ grep IOMMU /var/log/dmesg
    # or
    $ cat /proc/cmdline
    BOOT_IMAGE=/boot/vmlinuz-default [...] intel_iommu=on [...]

Ensure the PCI Device is Not in Use by the HostNot in Use by the Host

  1. Disable the nouveau kernel module and add the following line to the top of the /etc/modprobe.d/50-denylist.conf file: 
    blacklist nouveau
  2. Check if vfio-pci is loaded and used for the GPU devices:
    $ lspci -k | grep -A3 -E "3D controller|VGA"
    00:02.0 VGA compatible controller: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] (rev 0c)
            Subsystem: Dell Device 0b19
            Kernel driver in use: i915
            Kernel modules: i915, xe
    --
    01:00.0 3D controller: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] (rev a1)
            Subsystem: Dell Device 0b19
            Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
  3. Prevent the GPU devices from being used by the host:
    $ lspci -nn | grep -E "3D controller|VGA"
    00:02.0 VGA compatible controller [0300]: Intel Corporation Alder Lake-P GT2 [Iris Xe Graphics] [8086:46a6] (rev 0c)
    01:00.0 3D controller [0302]: NVIDIA Corporation GA107M [GeForce RTX 3050 Ti Mobile] [10de:25a0] (rev a1)
  4. Update grub configuration file /etc/default/grub. In this example, we are preventing the NVIDIA GPU with "vendor ID:Device ID" 10de:25a0 from being used by the host:
    GRUB_CMDLINE_LINUX_DEFAULT=".... pci-stub.ids=10de:25a0"
  5. Update grub configuration file:
    $ sudo update-grub
    or
    $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  6. Reboot the host:
    $ sudo reboot

  7. $ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
  8. Reboot the host:
    $ sudo reboot

Install GPU drivers

  1. Download the GPU drivers from your vendor. For nvidia, login to https://www.nvidia.com/en-us/drivers/vgpu-software-driver/ and download the drivers for your host & GPU Card.

Note: This is only required to create vGPUs. If only full passthrough is required, this can be ignored.

2. Check if GPU Devices are Discovered on the Host

...

  1. Launch a Virtual Machine with GPU Enabled offering. Make sure the selected template supports GPU drivers.
  2. Once your instance is launched, it will have the required GPU devices attached to it. If there are no available GPU devices, it Instance launch will fail.

Image Added