1.0. Problem Description

Up until CloudStack version 4.20.0.0 there was no support for native VM backups, only volume backups; VM backups relied on either Veeam or Dell Networker. Version 4.20.0.0 introduced the NAS backup plugin, a native backup solution for KVM full VM backups. However, the current functionality is not adequate for all use cases, namely, the following flaws are present:

• Every backup is a full VM backup. #8907 and #9524 introduced the notion of incremental volume backups and incremental VM snapshots for the KVM plugin in ACS, respectively; thus, there should be support for incremental VM backups as well.

• The plugin forces the user to use additional NAS storages (NFS, CephFS, CIFS/Samba) in order to backup the VMs. As ACS already has the concept of secondary storages, which are storage areas separate from the primary storage, there is no need to add a new storage concept in order to perform backups, as backups can be copied to the secondary storage. Furthermore, we could extend the secondary storage selectors introduced in #7659 to allow backup storage selectors, thus giving control over where the backups will be saved to the operators. By avoiding extra configurations, we make the system more flexible and natural for cloud operators.

• The backups are stored without any compression. Using the qemu-img tool, we could compress QCOW2 files to save space.

• The backups are not validated neither during creation nor post-creation.

Motivated by the aforementioned points and to provide a more complete native backup plugin for the KVM hypervisor, this document will present a novel virtual machine native backup solution for Apache CloudStack and KVM.

1.1. Basic Definitions

In this section we present some basic definitions that will be used throughout this spec:

• Backing file: a read-only file that will be read when data is not found on the top file of the QCOW2 image.
• Delta: a file that stores data that was changed in comparison to its backing file(s). When a volume snapshot is taken, the current delta/file will become the backing file of a new delta, thus preserving the data on the file being snapshotted.
• Backing chain: a chain of backing files, with the top delta being the end of the chain, and the base delta being the start of the chain.
• Current delta: the current delta that is being written to. It may have a parent delta and siblings.
• Parent delta: the immediate predecessor of a given delta in its backing chain. A delta can only have one parent.
• Child delta: the immediate successor of a given delta in the backing chain. A delta might have multiple children. The current delta cannot have children.
• Sibling delta: a delta that is the child of a given delta's parent.
• Active backup chain: a backup chain that has been started but has not ended yet. A VM does not have an active backup chain if either it does not have any backups, or the last backup was the end of its chain (meaning the next one will be a full backup).
• Backup delta: a delta that was created as part of a backup process. Every volume should have at most one delta of this type, except during the start of the backup creation process, where there might be two backup deltas. This delta is always on the primary storage. This delta will only exist when a VM has an active backup chain.

To illustrate the active backup chain and backup delta definitions, let us follow the example below:

1. At the start, the VM will not have an active backup chain, as no backups have been taken, this state is illustrated in figure 1:

   Figure 1. Initial state.

2. Then, during the first backup, a backup delta will be created and the volume will be consolidated and copied to the secondary storage, forming the first (full) backup. Also, the VM will have an active backup chain. This state is illustrated in figure 2:

   Figure 2. After taking the first backup.

3. During the second backup, a new backup delta will be created, the previous one will be copied to the secondary storage, forming the second (incremental) backup. Also, the active backup chain size will be increased by one. Figure 3 illustrates the state during the copy from primary to secondary of the previous backup delta. After it is copied, it will be merged into the base volume.

   Figure 3. Taking a second backup.

4. Considering the state above, after taking another backup, the number of backups on the chain will have reached the value configured for backup.chain.delta, and thus the backup chain will end, and we will not have an active backup chain, neither will we have a backup delta, this state is illustrated in figure 4:

   Figure 4. After a backup chain was completed.

2.0. Proposed Changes

To address the described problems, we propose to create a new backup plugin that will use native features to support incremental VM backups with KVM and ACS when using file-based storage. Many ideas for the incremental backups will be the same as the ones in #9524. The main characteristics of the feature are outlined below, all of them will be further presented and explained in the next sections.

• Use the snapshot creation/deletion process implemented in #9524 to create incremental snapshots, then copy those snapshots into the secondary storage, transforming them into backups. This feature is designed to be compatible with the one implemented in #9524.
• Extend the secondary storage selectors feature, implemented in #7659, to allow creating rules that apply to backups. Therefore, operators of the cloud can introduce secondary storage areas in ACS that are dedicated to store backups.
• Allow users to choose to compress their backups.
• Implement backup validation, where a test VM will be created using the backups as backing files, to verify if the VM boots, we will also allow cloud users to configure scripts to check the backup integrity.
• Implement quick restore process, allowing users to restore backups quickly directly from the secondary storage.
• Implement volume restore, allowing users to restore only one volume.

Allow users to download a single file from a backup. Where users inform a path inside one of the backed up volumes and ACS generates a download link for it.

2.1. Limitations

• This proposal will not contemplate non file-based storage backends, such as LVM and RBD (Ceph).
• This feature will be mutually exclusive with incremental volume snapshots. When taking a volume snapshot of a volume that is attached to a VM with VM backups, the incremental volume snapshot will fail. Conversely, if one of the VM's volumes has an incremental volume snapshot, the user will not be able to create a VM backup. The KVM incremental volume snapshot feature uses an API that is not compatible with virDomainSnapshotCreateXML (the API used for this feature).
• This feature will also be mutually exclusive with VM snapshots with memory. When reverting a backup, all internal snapshots that were taken after the creation of the backup being reverted will be lost, causing data loss. As the VM snapshot with memory uses internal snapshots and this feature uses external snapshots, we will not allow both to coexist to prevent data loss.

While this feature will be compatible with disk-only VM snapshots, if a VM snapshot is reverted while there is an active backup chain, the next backup may have to be a full backup. When reverting a backup, if the VM has any VM snapshots, those will need to be removed before reverting the backup. Otherwise, we would have to create a new tree for the VM's volumes, and keep many trees would add complexity to the implementation. This limitation might be addressed in the future if a significant use case is presented.

3.0. Backup Offerings

Currently, the only way to create new offerings is to import them, using an external ID listed by the backup provider. Since we are proposing a solution where ACS is the provider, we will not have an external ID to inform on the importBackupOffering. Thus, a new API called createBackupOffering will be created, which will define a new backup offering from scratch, this API will have the necessary parameters to configure the behavior of KBOSS backups.

The createBackupOffering API will have the following parameters:

4.0. VM Backup Creation

We will create two new DB tables and a new view, internal_backup_pool_ref, internal_backup_store_ref, and internal_backup_view. The first will store metadata about backup deltas, which are on the primary storage; the second will store metadata about the backups, which are on the secondary storage; the third table will be a generic details table, that may be used by any backup provider, for this provider, a some information will be stored in it: Whether the backup is current, if it is the end of its chain, what is its parent and which image store it is in. Regarding the view, it is used to facilitate some processes in the code. When a backup delta is created, all three tables will be updated.

The internal_backup_pool_ref table will contain the following columns:

The internal_backup_store_ref table will contain the following columns:

The backup_details will be the same as all the other details tables:

The internal_backup_view will contain the following columns:

Furthermore a new configuration will be created: backup.chain.size, which will determine the max size of a backup chain. The default value will be 8. If cloud admins set it to 1, all the backups will be full backups. With values lower than 1, the backup chain will be unlimited, unless it is stopped by another process. Please note that unlimited backup chains have a higher chance of getting corrupted, as new backups will be dependant on all of the older ones. In any case, if the backup has validation enabled, the validation process will catch corrupt backups.

Two new VM states will be created:

• BackingUp: When the backup process begins, the VM will transition from the Ready or Stopped to the BackingUp state, this state cannot be reached otherwise. During the backup process, the backup job will be queued alongside the other VM jobs; therefore, we will not have to worry about the VM being stopped/started during the backup, as each job is processed sequentially for each given VM.
• BackupError: If an exception which cannot be automatically recovered from happens (if the MGMT/agent service goes down, for example) or the job times out, the VM will go from the BackingUp to the BackupError state. While the VM is in this state, the VM may not be started, stopped, rebooted, migrated or snapshotted; its volumes cannot be attached, detached, migrated or snapshotted. The finishBackupChain API may be called to try to fix this state.

The proposed VM backup creation workflow is summarized in the following diagram. Every step is repeated for all the volumes of the VM, except the virDomainSnapshotCreateXML step and the validation step, which are done for the VM as a whole.

• If the VM is running, we will call the virDomainSnapshotCreateXML API, informing all the VM's volumes with the snapshot key and the external value, and using the flags:
  1. VIR_DOMAIN_SNAPSHOT_CREATE_ATOMIC: to make the snapshot atomic across all the volumes;
  2. VIR_DOMAIN_SNAPSHOT_CREATE_DISK_ONLY: to make the snapshot disk-only;
  3. VIR_DOMAIN_SNAPSHOT_CREATE_NO_METADATA: to tell Libvirt not to save any metadata for the snapshot. This flag will be informed because we do not need Libvirt to save any metadata, all the other processes regarding the VM snapshots will be done using qemu-img.
  4. VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE: if quiesceVM is true, this flag will be informed as well to keep the VM frozen during the snapshot process, once the snapshot is done it will be already thawed.
• Otherwise (if the VM is not running), we will call qemu-img create for every volume of the VM, to create a delta on top of the current file.
• If any VM snapshot has been taken after the last backup on this backup chain:
  1. Convert all the deltas starting from the previous backup delta, until the previous top delta, to the secondary storage. For example, in a volume backing chain like this: A <-- B <-- C <-- D, with A being the volume base file, B being the delta created during the last backup, C being the delta created by the VM snapshot operation, and D being the delta created on the last step, we will convert B and C to the secondary storage, when converting the deltas, we already inform their correct backing files, in this example, the backing file for B would be the last backup, while the one for C would be B;
  2. Call qemu-img commit, merging all the converted deltas from the last step into one. Here we must inform the -b parameter, to let qemu-img know not to commit all the way into the volume's base file. In the example above, C would be commited to B.
  3. Delete the emptied files. In the example above, C.
• Else:
  1. Call qemu-img convert on the previous top delta. If the backup is not the start of a chain, we inform its backing file on the secondary storage.
• If it is not the first backup on the backup chain:
  • If the VM is running:
    1. Call Libvirt's virDomainBlockCommit API informing the previous backup delta as the top and its parent as the base.
  • Else:
    1. Merge previous backup delta with its parent delta, using qemu-img commit, to commit the backup delta to it;
    2. Rebase children to point to previous backup delta's parent. After the commit, the previous backup delta file will be empty and has to be removed from the chain;
    3. Remove previous backup delta file;
  • If a VM snapshot was created after the last backup:
    1. We update the VM snapshot path. The file pointed by the VM snapshot was the same as the previous backup delta. Thus, we need to update the VM snapshot path, as it has changed.
• If the backup will be the last of its backup chain, we merge the newly created delta with its parent. If the VM is running we use Libvirt's virDomainBlockCommit; else use qemu-img commit.
• We set the VM as ready, even if the backup will be validated and compressed later, all the work with the VM itself is done, and thus it can be returned to the ready state.
• If the backup offering supports backup compression, we validate the backup. The backup compression process will be explained in the backup compression section.
• If the backup offering supports backup validation, we validate the backup. The backup validation process will be explained in the backup validation section.

In order to let users choose whether the VM will be frozen for the snapshot process, a new parameter will be added to the createBackup and createBackupSchedule APIs, called quiesceVM. The VM will only be kept frozen during the initial snapshot creation process, during the rest of the backup it will be running.

If a VM snapshot is reverted while there is an active backup chain, the next backup may have to be a full backup:

• If the reverted snapshot was taken before the last backup, the current backup chain will be broken and next backup will have to be a full backup;
• If the snapshot being reverted was taken after the last backup on the current backup chain, then the next backup will not be affected, and thus will be a incremental backup.

The impacts this feature will have on the disk-only VM snapshot feature will be further discussed on the VM snapshot section.

4.0.1. Backup Compression

If the backup offering supports compression, after all the backup creation steps have finished (but before validation, if supported), we will launch a asynchronous job to start the backup compression. The compression will be broken into two steps: compress the backup into new files and substitute the original backup files. The reason to break the compression into two steps is so that while the first step is being done, the backup may still be restored; the restore will only be blocked during the last step, which is the fastest. These jobs will execute in the compute plane hosts instead of the SSVMs.

The host will have the better performance, meaning that the compression will be faster. However, if left unlimited, multiple concurrent compression processes could affect the host's performance, thus the guest VMs would be affected. On the other hand, the SSVM tends to be quite limited, and would easily get overloaded. Meaning that operators might need to upgrade their compute offerings, and also multiply the number of SSVMs in the environment (which also means more public IPs being used). Also, we need the compression process to not take too long, otherwise it might not end before the next backup.

To address the concerns over the host's resource usage, a few new configurations will be created:

• backup.compression.max.concurrent.compressions: Will determine the max number of concurrent compressions. The idea is to limit the concurrent load on the hosts. Defaults to 5. Values lower than 1 remove the limit, meaning that as many compressions as necessary will be done at the same time. This configuration will be at the cluster and global level. A new property on agent.properties will be created with the same name that when set will override the cluster/global level.
• backup.compression.rate.limit: Will limit the compression rate (in MB/s) lowering the stress on the host's resources. Defaults to 0; if a value lower than 1 is used, no limit will be applied to the compression rate. This configuration will be at the cluster and global level.
• backup.compression.coroutines: Will set the number of parallel coroutines for the compression process. This is translated to qemu-img '-m' parameter.

The compression process follows the following steps, each being its own job:

1. Compression start: Call qemu-img convert to create a copy of the backup and compress it, informing its backing file and the -c flag. This is the slowest part.
2. Compression finalization: Move the compressed file on top of the old uncompressed file.

In order to add more information for users regarding the backup compression, two new columns will be added to the backups table:

As the compression is a long running job, it is prone for timeouts or interruptions, thus, the backup.compression.max.job.retries and backup.compression.retry.interval configurations will be created, which will determine the number of retries and their interval, each job will be retried a number of times equal to the value of the backup.compression.max.job.retries configuration, by default, 2.

A new global and cluster-wide configuration (backup.compression.timeout) will be created to determine the timeout of the compression. Defaults to 8 hours. Values lower than 1 will disable the timeout. If the backup compression times out, it may be retried, if all the retries fail, the compression state will go into CompressionError, any leftover files will be deleted and the backup will be left uncompressed.

To allow operators to track the compression jobs, a new API (listBackupCompressionJobs) will be added, it will allow listing the current, scheduled and executed compression jobs. It will only be available for Root Admin accounts. It will have the following parameters:

To allow operators to pause the compression processes, the backup.compression.task.enabled account/domain/global setting will be created, when set to false in the global scope, no compression jobs will be executed, regardless of the value in other scopes. When true in the global scope, you may disable the compression task for certain domains or accounts by setting the value to false in the domain/account.

4.0.2 Ending the backup chain

The API finishBackupChain will be added to allow users to force a backup chain on a VM to end. This API may be called on VMs that are on the BackupError state and it will do a best-effort attempt at normalizing the VM and returning it to its original state, as well as finishing the backup chain on the VM. The API hast the following parameter:

4.1. Backup Restoration

As with the backup creation, this process will be queued with the other VM jobs and serialized, so we will not have to worry about the VM starting during the restoration process. Furthermore, the current backup chain will be severed.

It is important to note that, when restoring a backup, the size of the VM's volumes will be restored to what it was when the backup was taken. This happens because each QCOW2 has its size defined within themselves, thus, when copying a backup from the secondary storage to substitute the VM's volumes, the size of the volumes will be updated as well.

4.1.1 Standard Restoration

The proposed standard VM backup restoration workflow is summarized in the following diagram. Every step, except the two initial verifications, is repeated for all the volumes of the VM.

• If the VM is running, we throw an exception, as we cannot overwrite the VM's volumes while it is running without risk of corruption.
• If the VM has any VM snapshots, we throw an exception, as restoring the backup would delete the VM snapshots, causing data loss. The operator must remove the VM snapshots before restoring a backup.
• Delete the volume's backup delta file, if it has any.
• Use qemu-img convert to convert the backup from the secondary storage directly over the volume's base file.

After a backup restoration, all of the restored volumes will be full clone, no longer depending on a template.

4.1.2 Volume Restoration

We will also allow the user to restore a single volume that was part of the backup. A new volume will be created and attached to the target VM. The proposed volume backup restoration workflow is summarized in the following diagram.

• If the selected VM has any VM snapshots, we throw an exception, as VM snapshots on KVM do not support adding new volumes to the VM. The operator must remove the VM snapshots before restoring a backup.
• Duplicate the original volume's definition (basically its logical size and disk offering).
• Allocate the volume to a primary storage.
• Convert the backup from the secondary storage into the primary storage.
• Attach the volume on the VM.

Concept-wise the implementation is simple, but a new workflow must be created so that ACS allocates a volume to a primary storage before attaching it to the VM, so that we have a target for step 2.

4.1.3 Quick Restoration

The quick restoration process is similar to the two processes above, however, instead of converting the volume from the secondary storage, for each volume being reverted, we create a delta on the primary storage that has the backup in the secondary storage as a backing file. Then, we start the VM (if it is not running when quick restoring a single volume), and use Libvirt's virDomainBlockPull to consolidate the volumes while the VM runs. The diagram below describes the process for quick reverting a VM. The process for a single volume will be similar, except for the VM start, which will only happen if the VM is not running already.

• If the VM is running, we throw an exception, as we cannot overwrite the VM's volume while it is running without risk of corruption.
• If the VM has any VM snapshots, we throw an exception, as restoring the backup would delete the VM snapshots, causing data loss. .
• Delete the volume's files, both the backup delta and the base file.
• Use qemu-img create to create a new delta on the primary storage with the backup on the secondary storage as the backing file.
• Start the VM.
• Set the VM state as Consolidating on the DB. The VM is accessible normally while on this state, however, new operations, such as VM stop or migrate, are not allowed.
• Call virDomainBlockPull to pull all the necessary data from the volume's backing files. When the process finishes, the volume will not depend on any backing file anymore.
• Set the VM state as Running.

Since the process above involves a lot of communication between different components. The following (simplified) sequence diagram details a bit more the process:

The advantage of using the quick restore over the standard one is that the user will not have to wait for the backup copy into the primary storage to start using the VM. However, during the consolidation process, the volume performance might be affected, as any reads to old data must fetch the data from the secondary storage.

4.1.4 Create VM from backup

This feature will be compatible with KBOSS, as we mostly just need to restore the backup to the given VM.

4.2. Backup Deletion

The backup deletion process is somewhat similar to the KVM incremental snapshot deletion process (see #8907). It will also be added to the VM job queue, since it might have to merge the top delta file of the VM's volumes. The process is summarized on the diagram below.

• We set the backup as removed on the database.
• If it is the ancestor of any other active backup, we stop the process here. We will only delete the backup from storage when no other backup depends on it. The alternative would be merging the backup on secondary storage. However, since we are working with incremental backups, on average we expect that merging two backups will not reduce that much the storage occupied by them. On the other hand, implementing this merging process means that the SSVM will have to merge these backups, which might bring significant unnecessary overhead on them. Furthermore, if we are deleting a complete chain, the proposed process is much faster than merging all the backups and then removing the last one (a possible scenario with the alternative process).
• Else, we delete the backup from the secondary storage.
• If it has any other removed ancestors, we delete them from the secondary storage as well.
• If this is the current backup of the VM, as the last step, we commit the current backup delta into the base volume.
  
  

5.0. Backup Validation

The backup validation process will consist of three steps: wait for the VM to boot; execute a user-defined command; take a screenshot of the console of the VM. The first two steps can only be performed if the VM has the QEMU guest agent installed and it is configured to start on system startup. The main idea of the backup validation process is to verify that the VM is able to boot using the backup being validated. The validation steps that will be executed are defined in the backup offering through the validationSteps parameter, by default, only the screenshot step is executed.

The backup validation will be executed at the end of the backup creation process, as an asynchronous job. To allow operators to pause the validation processes, the backup.validation.task.enabled account/domain/global setting will be created, when set to false in the global scope, no validation jobs will be executed, regardless of the value in other scopes. When true in the global scope, you may disable the validation task for certain domains or accounts by setting the value to false in the domain/account.

Furthermore, operators will be able to configure periodic validation of already validated backups. At the end of the validation process, if the backup is deemed valid, a hash of the backup will be stored. The next validations of that backups will only take the hash and compare it to the original. Since the backup was already marked as valid, as long as the hash of the backup does not change, it should remain valid. To configure the periodic backup validation process, operators may use the validationInterval parameter of the backup offerings. Furthermore, a new VM setting with the same name will be created that overrides the configuration set in the backup offering. By default, only operators will be able to see this VM setting, as it will be added to the user.vm.denied.details list of settings.

The backup.validation.max.concurrent.operations configuration will be created to determine the max number of concurrent validations per host. The idea is to limit the concurrent load on the hosts. Defaults to 1. Values lower than 1 remove the limit, meaning that as many validations as necessary will be done at the same time. This configuration will be at the cluster and global level. A new property on agent.properties will be created with the same name that when set will override the cluster/global level.

Regarding the wait for boot step, the backup.validation.boot.default.timeout configuration will be created to specify the default time to wait for a validation VM to boot. By default it is set to 240 seconds. It will be at the account/domain/global scope. Furthermore, a VM setting called backupValidationBootTimeout will also be created for the same purpose, when set, it overrides the value on the backup.validation.boot.default.timeout configuration. By default, the VM setting will only be writable for operators.

Regarding the command execution step, 4 new VM settings will be created to configure how this step will be executed:

• backupValidationCommand: A command to be executed after the VM has started. This might be a script that does some validation and returns an output.
• backupValidationCommandArguments: A comma-separated list of arguments to be passed to the executed command.
• backupValidationCommandExpectedResult: The result expected to be returned by the executed command. If this is not set, we will only check if the exit value of the process is 0. If the command result is different than what has been set, the backup will be considered invalid. As Libvirt returns the value of the process as a Base64-encoded string, this setting should be set as a Base64-encoded string of the result that is expected of the command. This also helps avoid issues with transmitting special characters through the API.
• backupValidationCommandTimeout: A timeout to wait for the command to finish.

By default, backupValidationCommand, backupValidationCommandArguments and backupValidationCommandExpectedResult will be available for end-users to edit. While backupValidationCommandTimeout will be visible to end-users, but not editable.

As backupValidationCommand will only accept a single command or path to an executable script that must exist inside the user's VM, users are encouraged to create scripts inside their VM that validate what is needed and inform the script's fully qualified path on the setting.

Regarding the screenshot step, the backup.validation.screenshot.default.timeout configuration will be created to specify the default time to wait before taking a screenshot of the VM's console. By default it is set to 60 seconds. It will be at the account/domain/global scope. Furthermore, a VM setting called backupValidationScreenshotWait will also be created for the same purpose, when set, it overrides the value on the backup.validation.screenshot.default.timeout configuration. By default, the VM setting will only be writable for operators.

Moreover, to recover the screenshots, a new API will be created called downloadValidationScreenshot, this API will have a single parameter, backupId, which is the ID of the backup that should have its screenshot downloaded. Each backup can have at most a single validation screenshot.

The following diagram describes the backup validation process.

• We set the backup state as Validating on the database.
• We allocate a VM for validation on the same account as the backup owner. By default, the account limits will not be affected by the validation VM. However, if backup.validation.dummy.vm.limit is set to true (account/domain/global setting), the account must have enough quota to allocate the VM and its volumes. Furthermore, the Usage events will always be created. This VM will be allocated with a single NIC which is disabled.
• We create temporary deltas on the primary storage that have the backup as backing files, the backup will stay on the secondary storage. These deltas exist so that any writes the VM makes during the process do not touch the backup. The deltas will be created on the primary storage.
• We start the VM using the normal deploy process.
• If the wait for boot step is configured, we start polling to see if it has booted, using the guest-sync request.
• If the script setting has been configured and the boot step was successful, we will execute the specified command, using the guest-exec request. If configured, we will inform the arguments as well.
• If the screenshot step is configured, we will wait for the configured time and then take a screenshot of the VM.
• We cleanup the leftover elements:
  1. Stop and remove the dummy VM.
  2. Remove the temporary deltas.
• If the results of the executed steps are all as expected we set the backup as Valid.
• If the results of the executed steps are not as expected, we:
  1. Set the backup as Invalid.
  2. Terminate the current backup chain. This behavior will be configurable through the backup.validation.terminate.invalid.chain account/cluster/global configuration. The default value is true.
  3. Send an alert email, where we explain that a backup is not valid, and the reason why. The alert email configurations will be used to send this email, similar to when a management goes down. In the future, with the email feature being revamped for the backup system, we could expand this so that users also receive alert emails regarding their backups.
• If for some reason we are not able to execute the validation process, such as if there was an error during the VM allocation, we set the backup validation status as UnableToValidate. In this case the job will be rescheduled to be executed at a later time. Regarding this, two new cluster/global configurations will be created: backup.validation.max.job.retries which specifies the number of retries, by default, 2; backup.validation.retry.interval which determines the minimum amount of time (in minutes) to retry a backup validation job, by default, 300.
  
  
  

6.0. File Extraction

Sometimes, users do not need to restore a complete backup of their volumes. Instead, they simply want a single file from the backup. To allow users to extract a single file from created backups, a new API extractFileFromBackup will be created, which, when called, will return a download link for the Zipped file. This API will have the following parameters:

The API workflow is summarized below:

• Load the NBD kernel module with modprobe nbd max_part=16. The max_part option must be informed, so that the kernel may create device files for the partitions on the NBD device. We assume that no disk will have more than 16 partitions.
• Create a temporary delta with the backup as a backing file.
• Use qemu-nbd to connect to the newly created image.
• User blkid to list the partitions and FS of the connected device.
• If the partition we are looking for is in a LVM. We:
  1. Use pvs to locate the PV.
  2. Use lvdisplay to locate the LV.
• Mount the correct partition, using the informed FS, if it was not inform, try to derive it from the output of blkid.
• Zip the file in the path given by the user.
• Copy the file to a directory on the secondary storage.
• Unmount the partition.
• If the partition is in a LVM, we use vgchange -a n <lv_name> to deactivate the VG.
• Use qemu-nbd to disconnect from the image.
• Remove the temporary delta.
• Finally, we generate the download link for the copied file.

Please note that the file extraction feature is not compatible with encrypted volumes.

7.0. Interactions with other features

This session will go over changes and/or interaction with other features.

7.1. Changes to the Disk-Only VM Snapshot Feature

During the disk-only VM snapshot revert, we must check if there was a backup taken after the snapshot being reverted. If there was, we must merge the backup delta with its parent. The procedure will be the same used in the backup creation process, when the backup being taken is not the first one. Namely this part:

• If this is not the first backup on the backup chain:
  • If the VM is running:
    1. Call Libvirt's virDomainBlockCommit API informing the old backup delta as the top and its parent as the base.
  • Else:
    1. Merge old backup delta with its parent delta, using qemu-img commit, to commit the backup delta to it;
    2. Rebase children to point to old backup delta's parent. After the commit, the old backup delta file will be empty and has to be removed from the chain;
    3. Remove old backup delta file;
  • If a VM snapshot was created after the last backup:
    1. We update the VM snapshot path. The file pointed by the VM snapshot was the same as the old backup delta. Thus, we need to update the VM snapshot path, as it has changed.

However, if the backup delta is also the current volume's path, we do not need to go through the above procedure. We must only remove the current delta and its parent (which will be another delta).

In any case, if there was a backup after the snapshot being reverted, we must also mark its chain as ended.

Furthermore, all of the file-based KVM disk-only VM snapshot processes that merge deltas will also have to be tweaked to take into account that there might be a backup delta between two VM snapshot deltas.

7.2. Volume Snapshot Restoration

When reverting a volume snapshot, if the VM has an active backup chain, the next backup will be a full backup for the volume that was reverted. Furthermore, the old backup delta for that volume will be removed.

7.3. Volume Attach/Detach

7.3.1 Volume Attach

If a volume is attached after the VM already has backups, it will start to be contemplated on the next backups, even if the next backup will be incremental for all the other volumes, for the newly attached volume, it will be a full backup. When restoring the older backups, if the volume has any backup delta on its backing chain, it will be merged with the rest of the volume, to keep it clean, so that the next backup will be a full one (as the rest of the volumes will be as well).

Even if a VM is restored to a point where the volume was not attached, it will still stay attached and will not be affected by the restore operation.

7.3.2 Volume Detach

If a volume is detached after the VM already has backups and this volume has any backup delta on its backing chain, it will be merged with the rest of the volume, to keep it clean. When restoring the old backups, from when the VM still had that volume, a duplicate volume will be created and attached to the VM.

7.4. Migration

7.4.1 VM Migration

VM migration will not be affected.

7.4.2 Live Volume Migration

During the volume migration process, we will check if the volume is attached to a VM which has a ongoing backup chain. If it is, we will merge the backup delta with its parent, so that the volume is returned to only having either a single file (if full clone) or a top file with a template as a backing file (if linked clone). We do this because we cannot guarantee that a host will have access to the target primary storage to copy the backup delta. For example, for migration between two local storages, the origin host will not have direct access to the target storage, and thus cannot copy the delta by itself.

The next backup for that volume will be a full backup.

7.4.3 Cold Volume Migration

There will be no changes to the cold volume migration process. However, as it consolidates the volume, the next backup for that volume will be a full backup.

7.5. Secondary storage migration

The secondary storage migration process will be executed by the KVM compute plane hosts. The process will be as follows:

• Map the backup chains, so that they are processed from oldest to newest backup;
• For each backup:
  1. Copy it to the new location;
  2. If it has a backup parent, rebase it with its parent;
  3. If no backups depend on it on the original storage, delete it and its parents on the original storage;

Processing each backup individually enables a safe procedure, lowering the chances of a timeout, and diminishing the possible problems with an interruption on the copy process.

8.0. Future works

• In the future, we could expand the backup feature to allow download of backups.
• When the email feature is revamped, we could expand the alert emails sent by this feature so that users would also receive alert emails regarding their backups.