1.0. Problem Description

Up until CloudStack version 4.20.0.0 there was no support for native VM backups, only volume backups; VM backups relied on either Veeam or Dell Networker. Version 4.20.0.0 introduced the NAS backup plugin, a native backup solution for KVM full VM backups. However, the current functionality is not adequate for all use cases, namely, the following flaws are present:

...

Motivated by the aforementioned points and to provide a more complete native backup plugin for the KVM hypervisor, this document will present a novel virtual machine native backup solution for Apache CloudStack and KVM.

1.1. Basic Definitions

In this section we present some basic definitions that will be used throughout this spec:

...

1. At the start, the VM will not have an active backup chain, as no backups have been taken, this state is illustrated in figure 1:

   Figure 1. Initial state.

2. Then, during the first backup, a backup delta will be created and the volume will be consolidated and copied to the secondary storage, forming the first (full) backup. Also, the VM will have an active backup chain. This state is illustrated in figure 2:

   Figure 2. After taking the first backup.

3. During the second backup, a new backup delta will be created, the previous one will be copied to the secondary storage, forming the second (incremental) backup. Also, the active backup chain size will be increased by one. Figure 3 illustrates the state during the copy from primary to secondary of the previous backup delta. After it is copied, it will be merged into the base volume.

   Figure 3. Taking a second backup.

4. Considering the state above, after taking another backup, the number of backups on the chain will have reached the value configured for backup.chain.delta, and thus the backup chain will end, and we will not have an active backup chain, neither will we have a backup delta, this state is illustrated in figure 4:

   Figure 4. After a backup chain was completed.

2.0. Proposed Changes

To address the described problems, we propose to create a new backup plugin that will use native features to support incremental VM backups with KVM and ACS when using file-based storage. Many ideas for the incremental backups will be the same as the ones in #9524. The main characteristics of the feature are outlined below, all of them will be further presented and explained in the next sections.

...

Allow users to download a single file from a backup. Where users inform a path inside one of the backed up volumes and ACS generates a download link for it.

2.1. Limitations

• This proposal will not contemplate non file-based storage backends, such as LVM and RBD (Ceph).
• This feature will be mutually exclusive with incremental volume snapshots. When taking a volume snapshot of a volume that is attached to a VM with VM backups, the incremental volume snapshot will fail. Conversely, if one of the VM's volumes has an incremental volume snapshot, the user will not be able to create a VM backup. The KVM incremental volume snapshot feature uses an API that is not compatible with virDomainSnapshotCreateXML (the API used for this feature).
• This feature will also be mutually exclusive with VM snapshots with memory. When reverting a backup, all internal snapshots that were taken after the creation of the backup being reverted will be lost, causing data loss. As the VM snapshot with memory uses internal snapshots and this feature uses external snapshots, we will not allow both to coexist to prevent data loss.

While this feature will be compatible with disk-only VM snapshots, if a VM snapshot is reverted while there is an active backup chain, the next backup may have to be a full backup. When reverting a backup, if the VM has any VM snapshots, those will need to be removed before reverting the backup. Otherwise, we would have to create a new tree for the VM's volumes, and keep many trees would add complexity to the implementation. This limitation might be addressed in the future if a significant use case is presented.

3.0. Backup Offerings

Currently, the only way to create new offerings is to import them, using an external ID listed by the backup provider. Since we are proposing a solution where ACS is the provider, we will not have an external ID to inform on the importBackupOffering. Thus, a new API called createBackupOffering will be created, which will define a new backup offering from scratch, this API will have the necessary parameters to configure the behavior of KBOSS backups.

...

4.0. VM Backup Creation

We will create three two new DB tables and a new view, nativeinternal_backup_pool_ref, nativeinternal_backup_store_ref, backup_details and nativeinternal_backup_view. The first will store metadata about backup deltas, which are on the primary storage; the second will store metadata about the backups, which are on the secondary storage; the third table will be a generic details table, that may be used by any backup provider, for this provider, a some information will be stored in it: Whether the backup is current, if it is the end of its chain, what is its parent and which image store it is in. Regarding the view, it is used to facilitate some processes in the code. When a backup delta is created, all three tables will be updated.

The nativeinternal_backup_pool_ref table will contain the following columns:

The nativeinternal_backup_store_ref table will contain the following columns:

...

The nativeinternal_backup_view will contain the following columns:

...

• BackingUp: When the backup process begins, the VM will transition from the Ready or Stopped to the BackingUp state, this state cannot be reached otherwise. During the backup process, the backup job will be queued alongside the other VM jobs; therefore, we will not have to worry about the VM being stopped/started during the backup, as each job is processed sequentially for each given VM.
• BackupError: If an exception which cannot be automatically recovered from happens (if the MGMT/agent service goes down, for example) or the job times out, the VM will go from the BackingUp to the BackupError state. While the VM is in this state, the VM may not be started, stopped, rebooted, migrated or snapshotted; its volumes cannot be attached, detached, migrated or snapshotted. An operator must read the logs and recover the VM manually from The finishBackupChain API may be called to try to fix this state.

The proposed VM backup creation workflow is summarized in the following diagram. Every step is repeated for all the volumes of the VM, except the virDomainSnapshotCreateXML step and the validation step, which are done for the VM as a whole.

...

The impacts this feature will have on the disk-only VM snapshot feature will be further discussed on the VM snapshot section.

4.0.1. Backup Compression

If the backup offering supports compression, after all the backup creation steps have finished (but before validation, if supported), we will launch a asynchronous job to start the backup compression. The compression will be broken into two steps: compress the backup into new files and substitute the original backup files. The reason to break the compression into two steps is so that while the first step is being done, the backup may still be restored; the restore will only be blocked during the last step, which is the fastest. These jobs will execute in the compute plane hosts instead of the SSVMs.

...

To allow operators to pause the compression processes, the backup.compression.task.enabled account/domain/global setting will be created, when set to false in the global scope, no compression jobs will be executed, regardless of the value in other scopes. When true in the global scope, you may disable the compression task for certain domains or accounts by setting the value to false in the domain/account.

4.0.2 Ending the backup chain

The API finishBackupChain will be added to allow users to force a backup chain on a VM to end. This API may be called on VMs that are on the BackupError state and it will do a best-effort attempt at normalizing the VM and returning it to its original state, as well as finishing the backup chain on the VM. The API hast the following parameter:

4.1. Backup Restoration

As with the backup creation, this process will be queued with the other VM jobs and serialized, so we will not have to worry about the VM starting during the restoration process. Furthermore, the current backup chain will be severed.

It is important to note that, when restoring a backup, the size of the VM's volumes will be restored to what it was when the backup was taken. This happens because each QCOW2 has its size defined within themselves, thus, when copying a backup from the secondary storage to substitute the VM's volumes, the size of the volumes will be updated as well.

4.1.1 Standard Restoration

The proposed standard VM backup restoration workflow is summarized in the following diagram. Every step, except the two initial verifications, is repeated for all the volumes of the VM.

...

After a backup restoration, all of the restored volumes will be full clone, no longer depending on a template.

4.1.2 Volume Restoration

We will also allow the user to restore a single volume that was part of the backup. A new volume will be created and attached to the target VM. The proposed volume backup restoration workflow is summarized in the following diagram.

...

Concept-wise the implementation is simple, but a new workflow must be created so that ACS allocates a volume to a primary storage before attaching it to the VM, so that we have a target for step 2.

4.1.3 Quick Restoration

The quick restoration process is similar to the two processes above, however, instead of converting the volume from the secondary storage, for each volume being reverted, we create a delta on the primary storage that has the backup in the secondary storage as a backing file. Then, we start the VM (if it is not running when quick restoring a single volume), and use Libvirt's virDomainBlockPull to consolidate the volumes while the VM runs. The diagram below describes the process for quick reverting a VM. The process for a single volume will be similar, except for the VM start, which will only happen if the VM is not running already.

...

The advantage of using the quick restore over the standard one is that the user will not have to wait for the backup copy into the primary storage to start using the VM. However, during the consolidation process, the volume performance might be affected, as any reads to old data must fetch the data from the secondary storage.

4.1.4 Create VM from backup

This feature will be compatible with KBOSS, as we mostly just need to restore the backup to the given VM.

4.2. Backup Deletion

The backup deletion process is somewhat similar to the KVM incremental snapshot deletion process (see #8907). It will also be added to the VM job queue, since it might have to merge the top delta file of the VM's volumes. The process is summarized on the diagram below.

...

• We set the backup as removed on the database.
• If it is the ancestor of any other active backup, we stop the process here. We will only delete the backup from storage when no other backup depends on it. The alternative would be merging the backup on secondary storage. However, since we are working with incremental backups, on average we expect that merging two backups will not reduce that much the storage occupied by them. On the other hand, implementing this merging process means that the SSVM will have to merge these backups, which might bring significant unnecessary overhead on them. Furthermore, if we are deleting a complete chain, the proposed process is much faster than merging all the backups and then removing the last one (a possible scenario with the alternative process).
• Else, we delete the backup from the secondary storage.
• If it has any other removed ancestors, we delete them from the secondary storage as well.
• If this is the current backup of the VM, as the last step, we commit the current backup delta into the base volume.
  
  

5.0. Backup Validation

The backup validation process will consist of three steps: wait for the VM to boot; execute a user-defined command; take a screenshot of the console of the VM. The first two steps can only be performed if the VM has the QEMU guest agent installed and it is configured to start on system startup. The main idea of the backup validation process is to verify that the VM is able to boot using the backup being validated. The validation steps that will be executed are defined in the native backup offering through the validationSteps parameter, by default, only the screenshot step is executed.

...

Furthermore, operators will be able to configure periodic validation of already validated backups. At the end of the validation process, if the backup is deemed valid, a hash of the backup will be stored. The next validations of that backups will only take the hash and compare it to the original. Since the backup was already marked as valid, as long as the hash of the backup does not change, it should remain valid. To configure the periodic backup validation process, operators may use the validationInterval parameter of the native backup offerings. Furthermore, a new VM setting with the same name will be created that overrides the configuration set in the backup offering. By default, only operators will be able to see this VM setting, as it will be added to the user.vm.denied.details list of settings.

...

• We set the backup state as Validating on the database.
• We allocate a VM for validation on the same account as the backup owner. By default, the account limits will not be affected by the validation VM. However, if backup.validation.dummy.vm.limit is set to true (account/domain/global setting), the account must have enough quota to allocate the VM and its volumes. Furthermore, the Usage events will always be created. This VM will be allocated with a single NIC which is disabled.
• We create temporary deltas on the primary storage that have the backup as backing files, the backup will stay on the secondary storage. These deltas exist so that any writes the VM makes during the process do not touch the backup. The deltas will be created on the primary storage.
• We start the VM using the normal deploy process.
• If the wait for boot step is configured, we start polling to see if it has booted, using the guest-sync request.
• If the script setting has been configured and the boot step was successful, we will execute the specified command, using the guest-exec request. If configured, we will inform the arguments as well.
• If the screenshot step is configured, we will wait for the configured time and then take a screenshot of the VM.
• We cleanup the leftover elements:
  1. Stop and remove the dummy VM.
  2. Remove the temporary deltas.
• If the results of the executed steps are all as expected we set the backup as Valid.
• If the results of the executed steps are not as expected, we:
  1. Set the backup as Invalid.
  2. Terminate the current backup chain. This behavior will be configurable through the backup.validation.terminate.invalid.chain account/cluster/global configuration. The default value is true.
  3. Send an alert email, where we explain that a backup is not valid, and the reason why. The alert email configurations will be used to send this email, similar to when a management goes down. In the future, with the email feature being revamped for the backup system, we could expand this so that users also receive alert emails regarding their backups.
• If for some reason we are not able to execute the validation process, such as if there was an error during the VM allocation, we set the backup validation status as UnableToValidate. In this case the job will be rescheduled to be executed at a later time. Regarding this, two new cluster/global configurations will be created: backup.validation.max.job.retries which specifies the number of retries, by default, 2; backup.validation.retry.interval which determines the minimum amount of time (in minutes) to retry a backup validation job, by default, 300.
  
  
  

6.0. File Extraction

Sometimes, users do not need to restore a complete backup of their volumes. Instead, they simply want a single file from the backup. To allow users to extract a single file from created backups, a new API extractFileFromBackup will be created, which, when called, will return a download link for the Zipped file. This API will have the following parameters:

...

Please note that the file extraction feature is not compatible with encrypted volumes.

7.0. Interactions with other features

This session will go over changes and/or interaction with other features.

7.1. Changes to the Disk-Only VM Snapshot Feature

During the disk-only VM snapshot revert, we must check if there was a backup taken after the snapshot being reverted. If there was, we must merge the backup delta with its parent. The procedure will be the same used in the backup creation process, when the backup being taken is not the first one. Namely this part:

...

Furthermore, all of the file-based KVM disk-only VM snapshot processes that merge deltas will also have to be tweaked to take into account that there might be a backup delta between two VM snapshot deltas.

7.2. Volume Snapshot Restoration

When reverting a volume snapshot, if the VM has an active backup chain, the next backup will be a full backup for the volume that was reverted. Furthermore, the old backup delta for that volume will be removed.

7.3. Volume Attach/Detach

7.3.1 Volume Attach

If a volume is attached after the VM already has backups, it will start to be contemplated on the next backups, even if the next backup will be incremental for all the other volumes, for the newly attached volume, it will be a full backup. When restoring the older backups, if the volume has any backup delta on its backing chain, it will be merged with the rest of the volume, to keep it clean, so that the next backup will be a full one (as the rest of the volumes will be as well).

Even if a VM is restored to a point where the volume was not attached, it will still stay attached and will not be affected by the restore operation.

7.3.2 Volume Detach

If a volume is detached after the VM already has backups and this volume has any backup delta on its backing chain, it will be merged with the rest of the volume, to keep it clean. When restoring the old backups, from when the VM still had that volume, a duplicate volume will be created and attached to the VM.

7.4. Migration

7.4.1 VM Migration

VM migration will not be affected.

7.4.2 Live Volume Migration

During the volume migration process, we will check if the volume is attached to a VM which has a ongoing backup chain. If it is, we will merge the backup delta with its parent, so that the volume is returned to only having either a single file (if full clone) or a top file with a template as a backing file (if linked clone). We do this because we cannot guarantee that a host will have access to the target primary storage to copy the backup delta. For example, for migration between two local storages, the origin host will not have direct access to the target storage, and thus cannot copy the delta by itself.

The next backup for that volume will be a full backup.

7.4.3 Cold Volume Migration

There will be no changes to the cold volume migration process. However, as it consolidates the volume, the next backup for that volume will be a full backup.

7.5. Secondary storage migration

The secondary storage migration process will be executed by the KVM compute plane hosts. The process will be as follows:

...

Processing each backup individually enables a safe procedure, lowering the chances of a timeout, and diminishing the possible problems with an interruption on the copy process.

8.0. Future works

• In the future, we could expand the backup feature to allow download of backups.
• When the email feature is revamped, we could expand the alert emails sent by this feature so that users would also receive alert emails regarding their backups.