Page History

...

This framework provides CloudStack users the ability to backup their guest VMs for recovery purposes, in case they suffer a hardware or software issue with their instances or the underlying infrastructure. This framework allows CloudStack to be integrated with Backup and Recovery providers, as pluggable solutions. Each provider should implement the backup and recovery methods provided by the framework in their own way. Previous draft: Backup & Recovery

Use Case

Users wish to be able to back up their guest instances for recovery purposes should they suffer a hardware or software issue.

The current mechanism which users leverage is the volume snapshot feature as this has the feature set closest to that of a backup regime. i.e. images are stored on alternate location (although this may actually be the same physical array) it can be scheduled and users can set how many backups can be kept.

The volume snapshot mechanism causes a VM snapshot to be taken, then the required volume transferred to secondary storage from a hypervisor host in the cluster within which the VM resides. In the case of VMware, the image is compressed into an OVA by the SSVM.

Problems

This transfer can be slow and at times unreliable. Also the user cannot snapshot other volumes attached to the VM during this time, nor perform a number of other BAU operations. Large transfers and concurrency issues can saturate available bandwidth on network or storage.

The requirements for the a feature to fix this problem are:

Must Have

• Users able to backup whole VM or individual volumes
• Users able to schedule their backups and keep n versions
• Users able to restore VM in-place (overwrite existing VM)
•  Ability for Cloud Operators to leverage hardware capabilities of their respective storage solutions.
•  Seamless operation of SAN assisted backup vs non-assisted backups

Highly Desirable

• Users able to restore to alternate location
• Users able to restore to original location when original VM has been deleted
• Support for commercial backup software; Rubrik, Veeam, CommVault
• Support for 3rd Party Backup Solutions (e.g. Amanda)
• Support for in-guest (client based) backup solutions.

Nice to Have

• Support for Grandfather, Father, Child backup.
• User option for storage location
• Multi-tier archiving of volumes

Feature Specification

High level component diagram

It is possible to register a Backup and Recovery provider per zone. The entry point for each API request is the BackupManager. The BackupManager invokes the corresponding BackupProvider registered in a zone, and it should communicate with the Backup and Recovery provider API Server. A high level representation of this is shown in the following diagram:

Image RemovedImage Added

...

APIs

The Backup and Recovery framework defines new functionalities which that should be implemented by each Backup and Recovery provider plugin as a pluggable solution.

...

• Register a Backup and Recovery provider in a zone: This is done via dynamic Zone scoped configurations: 'backup.framework.enabled' and 'backup.framework.provider.plugin'
  • 'backup.framework.enabled' is false by default
  • 'backup.framework.provider.plugin' is 'dummy' by default. For a zone we can set the Backup provider name is lowercase. Available values for the moment are: 'dummy' and 'veeam'
• List available Backup and Recovery providers in CloudStack: listBackupProviders API method. 
• List Backup and Recovery Provider Policies: listBackupPolicies API method, setting the parameter 'external=true' and zoneid=ZONE_ID It returns every backup policy existing in the ZONE_ID backup provider. The goal of this method is listing backup policies to allow administrators to import them into CloudStack, so users can assign them to their guest VMs.
• Import a Backup Policy: importBackupPolicy API method. Allows importing an external backup policy from the zone Backup and Recovery provider into CloudStack.
• Delete a Backup Policy from CloudStack: deleteBackupPolicy API method. Removes the previously imported backup policy

For users:

...

• • 'backup.framework.sync.interval': 300 by default. B&R framework background sync task internal in seconds that performs GC, syncs usage/stats and schedules backups etc.
• Veeam B&R provider settings: (all have zone scope, i.e. admin can override on per-zone basis)
  • backup.plugin.veeam.url: Veeam B&R server URL (default:  http://localhost:9399/api/)
  • backup.plugin.veeam.username: Veeam B&R username (default: administrator)
  • backup.plugin.veeam.password: Veeam B&R password (default: P@ssword123)
  • backup.plugin.veeam.validate.ssl: Whether to validate server certificate if Veeam B&R server is SSL/TLS enabled (default: false)
  • backup.plugin.veeam.request.timeout: Veeam B&R API request timeout in seconds (default: 300)

Backend Configuration related:

• listBackupProviders: lists available backup provider plugins
• listBackupProviderOfferings: lists external backup policy/offering from a provider
• importBackupProviderOfferings: allows importing of an external backup policy/offering to CloudStack as a backup offering
• listBackupOfferings: lists CloudStack's backup offerings (searching via keyword, and pagination supported)
• deleteBackupOffering: deletes a backup offering by its ID

Offering assignment based:

• assignVirtualMachineToBackupOffering: adds a VM to a backup offering (returns backup object)
• removeVirtualMachineFromBackupOffering: removes a VM from a backup offering (returns success object), a forced=true parameter can be passed as for Veeam B&R provider this is needed which also removes VM from the Veeam Job deletes any backups

Backup operational APIs:

• createBackup: creates an adhoc backup for a VM
• deleteBackup: deletes a VM backup (not support for per restore point for Veeam, it should use the removeVMFromBackupOffering API)
• listBackups: lists backups
• restoreBackup: restore a previous VM backup in-place of a stopped or destroyed VM
• restoreVolumeFromBackup: restore and attach a backed-up volume (of a VM backup) to a specified VM

User-defined backup schedule:

• createBackupSchedule: creates a backup schedule for a VM
• updateBackupSchedule: updates backup schedule
• listBackupSchedule: returns backup schedule of a VM if defined
  
• deleteBackupSchedule: deletes backup schedule of a VM

Notes:

• The backup schedule defines a CloudStack driven backup schedule where CloudStack will schedule/start adhoc backups based on provided interval/schedule for a VM.
• The backup schedule runs in addition to operator-defined settings for an external backup job/offering which may enforce its own schedule and retention limit (such as the number of backups to keep). 
• Constraints:
• VM can only have a single backup container/veeam job, i.e. a single backup offering
• VM with backup offerings/backups not allowed to attach/detach volumes (like VM snapshots)
• For disabled zone, disabled account or missing VM or missing backup offering, backup and backup schedules are skipped
• Only one backup schedule per VM (unlike recurring snapshots that allows for multiple schedules/policies)

...

Schema

This feature introduces changes in the schema, adding the following entities:

• Backup table: Represented in the table ' backups'
• Backup Policy: Represented in the table 'schedule table: backup_policy'schedule
• Backup Policy - VM mapping: Represented in the table 'backup_policy_vm_map'Offering: backup_offering
• Backup usage helper table: usage_backup 
• Changes in user_vm_view to allow listing of backup offering/name in listVirtualMachines response

Backup and Recovery Providers

...

A dummy Backup and Recovery provider is already created , it simply implements each of the methods above. for testing purposes.

Screenshots

Zone level settings for registering a Backup and Recovery provider

...