DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
Page History
1.0. Problem Description
Up until CloudStack version 4.20.0.0 there was no support for native VM backups, only volume backups; VM backups relied on either Veeam or Dell Networker. Version 4.20.0.0 introduced the NAS backup plugin, a native backup solution for KVM full VM backups. However, the current functionality is not adequate for all use cases, namely, the following flaws are present:
...
Motivated by the aforementioned points and to provide a more complete native backup plugin for the KVM hypervisor, this document will present a novel virtual machine native backup solution for Apache CloudStack and KVM.
1.1. Basic Definitions
In this section we present some basic definitions that will be used throughout this spec:
...
At the start, the VM will not have an active backup chain, as no backups have been taken, this state is illustrated in figure 1:
Figure 1. Initial state.Then, during the first backup, a backup delta will be created and the volume will be consolidated and copied to the secondary storage, forming the first (full) backup. Also, the VM will have an active backup chain. This state is illustrated in figure 2:
Figure 2. After taking the first backup.During the second backup, a new backup delta will be created, the previous one will be copied to the secondary storage, forming the second (incremental) backup. Also, the active backup chain size will be increased by one. Figure 3 illustrates the state during the copy from primary to secondary of the previous backup delta. After it is copied, it will be merged into the base volume.
Figure 3. Taking a second backup.Considering the state above, after taking another backup, the number of backups on the chain will have reached the value configured for
backup.chain.delta, and thus the backup chain will end, and we will not have an active backup chain, neither will we have a backup delta, this state is illustrated in figure 4:
Figure 4. After a backup chain was completed.
2.0. Proposed Changes
To address the described problems, we propose to create a new backup plugin that will use native features to support incremental VM backups with KVM and ACS when using file-based storage. Many ideas for the incremental backups will be the same as the ones in #9524. The main characteristics of the feature are outlined below, all of them will be further presented and explained in the next sections.
...
Allow users to download a single file from a backup. Where users inform a path inside one of the backed up volumes and ACS generates a download link for it.
2.1. Limitations
- This proposal will not contemplate non file-based storage backends, such as LVM and RBD (Ceph).
- This feature will be mutually exclusive with incremental volume snapshots. When taking a volume snapshot of a volume that is attached to a VM with VM backups, the incremental volume snapshot will fail. Conversely, if one of the VM's volumes has an incremental volume snapshot, the user will not be able to create a VM backup. The KVM incremental volume snapshot feature uses an API that is not compatible with
virDomainSnapshotCreateXML(the API used for this feature). - This feature will also be mutually exclusive with VM snapshots with memory. When reverting a backup, all internal snapshots that were taken after the creation of the backup being reverted will be lost, causing data loss. As the VM snapshot with memory uses internal snapshots and this feature uses external snapshots, we will not allow both to coexist to prevent data loss.
While this feature will be compatible with disk-only VM snapshots, if a VM snapshot is reverted while there is an active backup chain, the next backup may have to be a full backup. When reverting a backup, if the VM has any VM snapshots, those will need to be removed before reverting the backup. Otherwise, we would have to create a new tree for the VM's volumes, and keep many trees would add complexity to the implementation. This limitation might be addressed in the future if a significant use case is presented.
3.0. Backup Offerings
We propose to add native backup offerings as the way to configure the KNIB backup offerings natively. Three new APIs will be created: createNativeBackupOffering, deleteNativeBackupOffering, listNativeBackupOfferings; these APIs will allow the creation, removal, and listing of native backup offerings. These offerings will be imported into the standard backup framework flow for use.
Currently, the only way to create new offerings is to import them, using an external ID listed by the backup provider. Since we are proposing a solution where ACS is the provider, we will not have an external ID to inform on the importBackupOffering. Thus, a new API called createBackupOffering will be created, which will define a new backup offering from scratch, this API will have the necessary parameters to configure the behavior of KBOSS backups.
The createBackupOffering The createNativeBackupOffering API will have the following parameters:
| Parameter | Description | Default Value | Required |
|---|---|---|---|
name | Specifies the name of the offering | - | Yes |
description | The description of the backup offering | - | Yes |
compress | Specifies whether the offering supports backup compression | false | No |
validate | Specifies whether the offering supports backup validation | false | No |
validationSteps | A comma-separated list of which validation steps should be performed. |
| No |
validationInterval | Specifies the interval (in hours) between two validations of the same backup | false | No |
allowQuickRestore | Specifies whether the offering supports quick restore | false | No |
allowExtractFile | Specifies whether the offering supports file extraction from backups | false | No |
compressionlibrary | Compression library, for offerings that support compression. Accepted values are zstd and zlib. If the image only supports zlib, it will be used regardless of this parameter. | zstd | No |
backupchainsize | Backup chain size for backups created with this offering. | - | No |
zoneId | The |
...
| zone ID for the offering | - | Yes | |
userDrivenBackups | Whether users are allowed to create adhoc backups and backup schedules | - | Yes |
domainIds | the ID of the containing domain(s), null for public offerings | - | No |
4.0. VM Backup Creation
We will create two new DB tables and a new view, internal
| Parameter | Description | Required |
|---|---|---|
id | Identifier of the native backup offering. | Yes |
A native backup offering will only be removed if it is not currently imported.
The listNativeBackupOfferings API will have the following parameters:
| Parameter | Description | Required |
|---|---|---|
id | Identifier of the offering. | No |
compress | Lists only offerings that support backup compression. | No |
validate | Lists only offerings that support backup validation. | No |
allowQuickRestore | Lists only offerings that support quick restore. | No |
allowExtractFile | Lists only offerings that support file extraction from backups. | No |
showRemoved | Lists also offerings that have already been removed. | No |
name | Backup offering name. | No |
By default, it will list all offerings that have not been removed.
4.0. VM Backup Creation
We will create three new DB tables and a new view, native_backup_pool_ref, nativeinternal_backup_store_ref, backup_details and nativeinternal_backup_view. The first will store metadata about backup deltas, which are on the primary storage; the second will store metadata about the backups, which are on the secondary storage; the third table will be a generic details table, that may be used by any backup provider, for this provider, a some information will be stored in it: Whether the backup is current, if it is the end of its chain, what is its parent and which image store it is in. Regarding the view, it is used to facilitate some processes in the code. When a backup delta is created, all three tables will be updated.
The nativeinternal_backup_pool_ref table will contain the following columns:
| Name | Type | Description |
|---|---|---|
| id | bigint | Row ID |
| backup_id | bigint | The backup's ID. Foreign key that points to the backups table. |
| storage_pool_id | bigint | The storage's ID. Foreign key that points to the storage_pool table. |
| volume_id | bigint | The volume's ID. Foreign key that points to the volumes table. |
| backup_delta_path | varchar(255) | Path of the created delta |
| backup_parent_path | varchar(255) | Path of the created delta's parent |
The nativeinternal_backup_store_ref table will contain the following columns:
| Name | Type | Description |
|---|---|---|
| id | bigint | Row ID |
| backup_id | bigint | The backup's ID. Foreign key that points to the backups table. |
| volume_id | bigint | The volume's ID. Foreign key that points to the volumes table. |
| volume_size | bigint | The volume's size at the time of the backup |
| path | varchar(255) | Path of the backup |
The backup_details will be the same as all the other details tables:
| Name | Type | Description |
|---|---|---|
id | bigint | Row ID |
backup_id | bigint | The backup's ID. Foreign key that points to the backups table. |
name | varchar(255) | The detail name |
value | varchar(1024) | The detail value |
display | tinyint(1) unsigned |
The nativeinternal_backup_view will contain the following columns:
| Name | Type | Description |
|---|---|---|
id | bigint | Backup ID |
uuid | varchar(255) | Backup UUID |
vm_id | bigint | VM ID |
type | varchar(255) | Backup type |
date | datetime | Backup creation date |
status | varchar(32) | Backup status |
backup_offering_id | bigint | Backup offering ID |
image_store_id | text | Secondary storage ID where the backup is stored |
parent_id | text | Backup parent ID |
end_of_chain | text | If the backup is the end of its chain |
current | text | If the backup is current or not |
Furthermore a new configuration will be created: backup.chain.size, which will determine the max size of a backup chain. The default value will be 8. If cloud admins set it to 1, all the backups will be full backups. With values lower than 1, the backup chain will be unlimited, unless it is stopped by another process. Please note that unlimited backup chains have a higher chance of getting corrupted, as new backups will be dependant on all of the older ones. In any case, if the backup has validation enabled, the validation process will catch corrupt backups.
...
BackingUp: When the backup process begins, the VM will transition from theReadyorStoppedto theBackingUpstate, this state cannot be reached otherwise. During the backup process, the backup job will be queued alongside the other VM jobs; therefore, we will not have to worry about the VM being stopped/started during the backup, as each job is processed sequentially for each given VM.BackupError: If an exception which cannot be automatically recovered from happens (if the MGMT/agent service goes down, for example) or the job times out, the VM will go from theBackingUpto theBackupErrorstate. While the VM is in this state, the VM may not be started, stopped, rebooted, migrated or snapshotted; its volumes cannot be attached, detached, migrated or snapshotted. An operator must read the logs and recover the VM manually from ThefinishBackupChainAPI may be called to try to fix this state.
The proposed VM backup creation workflow is summarized in the following diagram. Every step is repeated for all the volumes of the VM, except the virDomainSnapshotCreateXML step and the validation step, which are done for the VM as a whole.
...
The impacts this feature will have on the disk-only VM snapshot feature will be further discussed on the VM snapshot section.
4.0.1. Backup Compression
If the backup offering supports compression, after all the backup creation steps have finished (but before validation, if supported), we will launch a asynchronous job to start the backup compression. The compression will be broken into two steps: compress the backup into new files and substitute the original backup files. The reason to break the compression into two steps is so that while the first step is being done, the backup may still be restored; the restore will only be blocked during the last step, which is the fastest. These jobs will execute in the compute plane hosts instead of the SSVMs.
...
In order to add more information for users regarding the backup compression, two new columns will be added to the backups table:
| Name | Type | Description |
|---|---|---|
| compression_status | varchar(55) | Compression status, which may be Uncompressed, Compressing, FinalizingCompression, Compressed or CompressionError |
| uncompressed_size | bigint | Old physical size of the backup, before the compression was done |
As the compression is a long running job, it is prone for timeouts or interruptions, thus, the backup.compression.max.job.retries and backup.compression.retry.interval configurations will be created, which will determine the number of retries and their interval, each job will be retried a number of times equal to the value of the backup.compression.max.job.retries configuration, by default, 2.
...
To allow operators to track the compression jobs, a new API (listBackupCompressionJobs) will be added, it will allow listing the current, scheduled and executed compression jobs. It will only be available for Root Admin accounts. It will have the following parameters:
| Parameter | Description |
|---|---|
id | List only the job with the specified ID |
backupid | List jobs associated with the specified backup |
hostid | List jobs associated with the specified host. When this parameter is provided, the executing parameter is implicit |
zoneid | List jobs associated with the specified zone |
type | List jobs of the specified type. Accepts Starting or Finalizing |
executing | List jobs that are currently executing |
scheduled | List jobs scheduled to run in the future |
To allow operators to pause the compression processes, the backup.compression.task.enabled account/domain/global setting will be created, when set to false in the global scope, no compression jobs will be executed, regardless of the value in other scopes. When true in the global scope, you may disable the compression task for certain domains or accounts by setting will be created, when set to false in the global scope, no compression jobs will be executed, regardless of the value in other scopes. When true in the global scope, you may disable the compression task for certain domains or accounts by setting the value to false in the domain/accountthe value to false in the domain/account.
4.0.2 Ending the backup chain
The API finishBackupChain will be added to allow users to force a backup chain on a VM to end. This API may be called on VMs that are on the BackupError state and it will do a best-effort attempt at normalizing the VM and returning it to its original state, as well as finishing the backup chain on the VM. The API hast the following parameter:
| Parameter | Description |
|---|---|
vmId | ID of the VM to finish the chain. |
4.1. Backup Restoration
As with the backup creation, this process will be queued with the other VM jobs and serialized, so we will not have to worry about the VM starting during the restoration process. Furthermore, the current backup chain will be severed.
It is important to note that, when restoring a backup, the size of the VM's volumes will be restored to what it was when the backup was taken. This happens because each QCOW2 has its size defined within themselves, thus, when copying a backup from the secondary storage to substitute the VM's volumes, the size of the volumes will be updated as well.
4.1.1 Standard Restoration
The proposed standard VM backup restoration workflow is summarized in the following diagram. Every step, except the two initial verifications, is repeated for all the volumes of the VM.
...
After a backup restoration, all of the restored volumes will be full clone, no longer depending on a template.
4.1.2 Volume Restoration
We will also allow the user to restore a single volume that was part of the backup. A new volume will be created and attached to the target VM. The proposed volume backup restoration workflow is summarized in the following diagram.
...
Concept-wise the implementation is simple, but a new workflow must be created so that ACS allocates a volume to a primary storage before attaching it to the VM, so that we have a target for step 2.
4.1.3 Quick Restoration
The quick restoration process is similar to the two processes above, however, instead of converting the volume from the secondary storage, for each volume being reverted, we create a delta on the primary storage that has the backup in the secondary storage as a backing file. Then, we start the VM (if it is not running when quick restoring a single volume), and use Libvirt's virDomainBlockPull to consolidate the volumes while the VM runs. The diagram below describes the process for quick reverting a VM. The process for a single volume will be similar, except for the VM start, which will only happen if the VM is not running already.
...
The advantage of using the quick restore over the standard one is that the user will not have to wait for the backup copy into the primary storage to start using the VM. However, during the consolidation process, the volume performance might be affected, as any reads to old data must fetch the data from the secondary storage.
4.1.4 Create VM from backup
This feature will be compatible with KNIBKBOSS, as we mostly just need to restore the backup to the given VM.
4.2. Backup Deletion
The backup deletion process is somewhat similar to the KVM incremental snapshot deletion process (see #8907). It will also be added to the VM job queue, since it might have to merge the top delta file of the VM's volumes. The process is summarized on the diagram below.
...
- We set the backup as removed on the database.
- If it is the ancestor of any other active backup, we stop the process here. We will only delete the backup from storage when no other backup depends on it. The alternative would be merging the backup on secondary storage. However, since we are working with incremental backups, on average we expect that merging two backups will not reduce that much the storage occupied by them. On the other hand, implementing this merging process means that the SSVM will have to merge these backups, which might bring significant unnecessary overhead on them. Furthermore, if we are deleting a complete chain, the proposed process is much faster than merging all the backups and then removing the last one (a possible scenario with the alternative process).
- Else, we delete the backup from the secondary storage.
- If it has any other removed ancestors, we delete them from the secondary storage as well.
- If this is the current backup of the VM, as the last step, we commit the current backup delta into the base volume.
5.0. Backup Validation
The backup validation process will consist of three steps: wait for the VM to boot; execute a user-defined command; take a screenshot of the console of the VM. The first two steps can only be performed if the VM has the QEMU guest agent installed and it is configured to start on system startup. The main idea of the backup validation process is to verify that the VM is able to boot using the backup being validated. The validation steps that will be executed are defined in the native backup offering through the validationSteps parameter, by default, only the screenshot step is executed.
...
Furthermore, operators will be able to configure periodic validation of already validated backups. At the end of the validation process, if the backup is deemed valid, a hash of the backup will be stored. The next validations of that backups will only take the hash and compare it to the original. Since the backup was already marked as valid, as long as the hash of the backup does not change, it should remain valid. To configure the periodic backup validation process, operators may use the validationInterval parameter of the native backup offerings. Furthermore, a new VM setting with the same name will be created that overrides the configuration set in the backup offering. By default, only operators will be able to see this VM setting, as it will be added to the user.vm.denied.details list of settings.
...
- We set the backup state as
Validatingon the database. - We allocate a VM for validation on the same account as the backup owner. By default, the account limits will not be affected by the validation VM. However, if
backup.validation.dummy.vm.limitis set to true (account/domain/global setting), the account must have enough quota to allocate the VM and its volumes. Furthermore, the Usage events will always be created. This VM will be allocated with a single NIC which is disabled. - We create temporary deltas on the primary storage that have the backup as backing files, the backup will stay on the secondary storage. These deltas exist so that any writes the VM makes during the process do not touch the backup. The deltas will be created on the primary storage.
- We start the VM using the normal deploy process.
- If the wait for boot step is configured, we start polling to see if it has booted, using the
guest-syncrequest. - If the script setting has been configured and the boot step was successful, we will execute the specified command, using the
guest-execrequest. If configured, we will inform the arguments as well. - If the screenshot step is configured, we will wait for the configured time and then take a screenshot of the VM.
- We cleanup the leftover elements:
- Stop and remove the dummy VM.
- Remove the temporary deltas.
- If the results of the executed steps are all as expected we set the backup as
Valid. - If the results of the executed steps are not as expected, we:
- Set the backup as
Invalid. - Terminate the current backup chain. This behavior will be configurable through the
backup.validation.terminate.invalid.chainaccount/cluster/global configuration. The default value istrue. - Send an alert email, where we explain that a backup is not valid, and the reason why. The alert email configurations will be used to send this email, similar to when a management goes down. In the future, with the email feature being revamped for the backup system, we could expand this so that users also receive alert emails regarding their backups.
- Set the backup as
- If for some reason we are not able to execute the validation process, such as if there was an error during the VM allocation, we set the backup validation status as
UnableToValidate. In this case the job will be rescheduled to be executed at a later time. Regarding this, two new cluster/global configurations will be created:backup.validation.max.job.retrieswhich specifies the number of retries, by default, 2;backup.validation.retry.intervalwhich determines the minimum amount of time (in minutes) to retry a backup validation job, by default, 300.
6.0. File Extraction
Sometimes, users do not need to restore a complete backup of their volumes. Instead, they simply want a single file from the backup. To allow users to extract a single file from created backups, a new API extractFileFromBackup will be created, which, when called, will return a download link for the Zipped file. This API will have the following parameters:
| Name | Description | Type | Required |
|---|---|---|---|
| backupId | the ID of the backup | Long | Yes |
| volumeId | the ID of the volume | Long | Yes |
| filePath | the path to the file/directory | Long | Yes |
| partition | Which partition is the file located. If not informed, every non boot partition will be searched | String | No |
| fileSystem | Which FS is being used in the given partition. If not informed, ACS will try to derive it | String | No |
The API workflow is summarized below:
...
Please note that the file extraction feature is not compatible with encrypted volumes.
7.0. Interactions with other features
This session will go over changes and/or interaction with other features.
7.1. Changes to the Disk-Only VM Snapshot Feature
During the disk-only VM snapshot revert, we must check if there was a backup taken after the snapshot being reverted. If there was, we must merge the backup delta with its parent. The procedure will be the same used in the backup creation process, when the backup being taken is not the first one. Namely this part:
...
Furthermore, all of the file-based KVM disk-only VM snapshot processes that merge deltas will also have to be tweaked to take into account that there might be a backup delta between two VM snapshot deltas.
7.2. Volume Snapshot Restoration
When reverting a volume snapshot, if the VM has an active backup chain, the next backup will be a full backup for the volume that was reverted. Furthermore, the old backup delta for that volume will be removed.
7.3. Volume Attach/Detach
7.3.1 Volume Attach
If a volume is attached after the VM already has backups, it will start to be contemplated on the next backups, even if the next backup will be incremental for all the other volumes, for the newly attached volume, it will be a full backup. When restoring the older backups, if the volume has any backup delta on its backing chain, it will be merged with the rest of the volume, to keep it clean, so that the next backup will be a full one (as the rest of the volumes will be as well).
Even if a VM is restored to a point where the volume was not attached, it will still stay attached and will not be affected by the restore operation.
7.3.2 Volume Detach
If a volume is detached after the VM already has backups and this volume has any backup delta on its backing chain, it will be merged with the rest of the volume, to keep it clean. When restoring the old backups, from when the VM still had that volume, a duplicate volume will be created and attached to the VM.
7.4. Migration
7.4.1 VM Migration
VM migration will not be affected.
7.4.2 Live Volume Migration
During the volume migration process, we will check if the volume is attached to a VM which has a ongoing backup chain. If it is, we will merge the backup delta with its parent, so that the volume is returned to only having either a single file (if full clone) or a top file with a template as a backing file (if linked clone). We do this because we cannot guarantee that a host will have access to the target primary storage to copy the backup delta. For example, for migration between two local storages, the origin host will not have direct access to the target storage, and thus cannot copy the delta by itself.
The next backup for that volume will be a full backup.
7.4.3 Cold Volume Migration
There will be no changes to the cold volume migration process. However, as it consolidates the volume, the next backup for that volume will be a full backup.
7.5. Secondary storage migration
The secondary storage migration process will be executed by the KVM compute plane hosts. The process will be as follows:
...
Processing each backup individually enables a safe procedure, lowering the chances of a timeout, and diminishing the possible problems with an interruption on the copy process.
8.0. Future works
- In the future, we could expand the backup feature to allow download of backups.
- When the email feature is revamped, we could expand the alert emails sent by this feature so that users would also receive alert emails regarding their backups.
Overview
Content Tools