Page History

Problem Statement

Currently CloudStack does not offer a flexible pluggable framework for users to easily integrate and configure any 3rd-party object stores for such backup services as registering templates, taking snapshots, etc. Along with Edison's recent refactored storage subsystem 2.0, we are proposing to develop a storage backup object store plugin framework to allow CloudStack to systematically manage and configure various types of backup data stores from different vendors.

Storage DataStore Plugin Framework

In this new refactored backup datastore plugin framework, we have clearly defined those pluggable service interfaces, such as PrimaryDataStore, ImageDataStore, DataMotionStrategy, AutoScaleStrategy, etc, so that different storage providers can develop their vendor-specific plugins based on the well-defined contracts that can be seemlessly managed by CloudStack orchestration.

Benefits of Backup DataStore Plugin Framework

With this new plugin framework, we should be able would like to achieve following functionalities:

Support different object store

...

providers in a uniform and extensible fashion

so that we can easily add an S3-based Object Storage as a secondary storage in CloudStack for storing all artifacts currently stored in secondary storage - snapshots, ISOs and templates. Although current CloudStack has some existing business logic to handle S3, Swift, NFS secondary storages, the code to interact with these vendor-specific providers is closely mingled together with CloudStack orchestration code, specifically, TemplateManagerImpl, SnapshotManagerImpl, etc, using an ugly if-else control flow. With such a tight coupling, it becomes very hard to extend the framework to support new object store providers. By extracting out out ImageDataStore pluggble  pluggble service interface, we can wrap these vendor-specific logic inside each provider plugin, then CloudStack orchestration code can be greatly simplified and provide a uniform and extensible way to manage different object store vendors.

Enable region wide object backup using s3-like object store.

Currently CloudStack only allows registering a template to a specific zone by using NFS secondary storage. In order to enable this template to another zone, user has to invoke invoke copyTemplate to  to copy the template from one zone to another. With this new plugin framework, customers can plugin their own object store providers, like S3/Swift, etc and choose them as the backup store. CloudStack can provide an option in *registerTemplate* api to allow them to create either "perZone" or "regionWide" templat

Provide cache storage framework

For certain hypervisors, when moving data(either templates or snapshots, etc) between primary storage and backup object store, a cache storage is required. The cache storage acts as an intermediate storage, stores templates, volumes and snapshots temporarily. Current CloudStack uses admin provided NFS server as the only one cache storage and takes cache storage as zone-wide for granted.

It has its drawbacks: 

1. Admin can't add other type of storages as cache storage.
2. Deep coupled with cache storage in the operations related to template

...

1. and snapshot upload, which is unnecessary and introduce scalability issue on cache storage.

With cache storage framework, it will allow developer add other type of cache storage into CloudStack, and let admin decide how to use cache storage.

Provide scalable cache storage solution

The current NFS based cache storage solution is not scalable:

1. NFS cache storage is zone-wide, has its limit on how many concurrent I/O can happen on cache storage
2. CloudStack does support multiple NFS cache storage, but lacks auto scaling capability based on I/O load. Currently CloudStack only randomly selects one of NFS cache storages whenever cache storage is needed, there is no policy to choose cache storage from cache storage pools based on certain criteria. 

In order to scale cache storage, there are possible solutions:

1. cache storage doesn't need to be zone wide, it can be pod or cluster wide.
2. CloudStack can create its own cache storage VMs(the VM created by CloudStack which can present NFS export to CloudStack), then admin doesn't need to setup lot of NFS servers by himself, all the cache storages are handled by CloudStack itself.
3. Add anther type of storage as cache storage, such as Ceph.

Provide pluggable Data motion strategies to handle data transfer

There is only one way to move data between primary and secondary storage in current CloudStack code, which is moving data between primary storage and NFS secondary storage, then between NFS secondary storage and object storage. This part of code is highly coupled with current CloudStack storage model, which is hard to be extended.

The new pluggable data motion strategies will make developer easier to write a new strategies, the possible strategies will be:

1. Move data between primary storage and object storage directly, totally bypass NFS cache storage. This is possible for Vmware and KVM.
2. Move data between primary storage and object storage, but through a cache storage. This is for certain operations on XenServer.
3. Move data between primary storages, through hypervisor specific storage motion functionalities, such XenMotion, VMotion etc.

 Storage DataStore Plugin Framework

In this new refactored backup datastore plugin framework, we have clearly defined those pluggable service interfaces, such as PrimaryDataStore, ImageDataStore, DataMotionStrategy, AutoScaleStrategy, etc, so that different storage providers can develop their vendor-specific plugins based on the well-defined contracts that can be seemlessly managed by CloudStack orchestration.

DB Schema

This is the new DB model related to data store.

The following are deprecated tables:

• template_host_ref (replaced with template_store_ref)
• template_s3_ref (replaced with template_store_ref)
• template_swift_ref (replaced with template_store_ref)
• volume_host_ref (replaced with volume_store_ref)
• s3 (replaced with image_data_store)
• swift (replaced with image_data_store)

instead of currently copying templates from one zone to another.

Make SSVM optional and configurable

Support pluggable Data Motion Strategies to handle various protocol combinations.

Provide CloudStack framework support to launch NFS transfer VMs dynamically for data transfer.

Support pluggable Auto Scale strategies to load balance data streams.

API Changes

Following will be the changes in their behavior post introduction of Object Store plugin framework: snapshot commands - createSnapshot, listSnapshots, deleteSnapshot, createSnapshotPolicy, deleteSnapshotPolicies, listSnapshotPolicies - No changes
template/iso commands createTemplate, updateTemplate, listTemplates, updateTemplatePermissions, listTemplatePermissions, prepareTemplate No changes

image store commands

NOTE that we have following assumptions:

1. you can add multiple image stores from the same provider, but we prevent you from adding multiple image
   stores from different providers.
2. For NFS image store providers, it is always zone-wide. And For S3/Swift, it is always region-wide.

• listStorageProvidersCmd - List all enabled image store providers (with type=IMAGE)'
  Parameters: type (Primary or Image)
  Response: List of StorageProviderResponse
• addImageStoreCmd - This command will be used to handle adding image store from different providers, including NFS, S3, Swift, etc
  Parameters: url, zoneId, providerName, details
  Response: ImageStoreResponse
• listImageStoresCmd - List all CloudStack managed image stores.
  Parameters: id, storeName, protocol, provider, zoneId
  Response: list of ImageStoreResponse
• deleteImageStoreCmd - Delete an image store when there are no resources (snapshots,volumes,templates) residing on it.
  Parameters: id
  Response: SuccessResponse

Here are previous APIs to be deprecated. For backwards-compatibility, we will internally wire these old api to use new api logic, so it will still work with one change of response object to ImageStoreResponse, but it will be marked as deprecated in API doc.

• addSecondaryStorageCmd
• addS3Cmd
• addSwiftCmd
• listS3Cmd
• listSwiftCmd

snapshot commands

• CreateSnapshotCmd flow

template/iso commands

• No changes: updateTemplate, updateTemplatePermissions, listTemplatePermissions
• To be changed:
  registerTemplate - Would register the URI. Download work would be done by

...

• image store provider now.
  deleteTemplate - Would just unregister the template, physically deleting it from OS should happen through

...

• image store provider api.
  extractTemplate - would just give the

...

• image store url
  copyTemplate - would return success since the template would be available region wide.
  

...

• createTemplate - done through data motion service
  prepareTemplate - done through data motion service
  listTemplates - we create db view to perform listTemplate and listIso, to get rid of direct sqls in our code.

volume commands

• No changes: listVolumes
• To be changed:
  extractVolume - need to be done through image store provider.
  uploadVolume - similar to register template. User needs to put in just the

...

• image store url here.
  createVolume - when creating volume from snapshot, zone id needs to be mentioned.
  migrateVolume - should work as is using

...

• storage cache manager.
  attachVolume/detachVolume - done through cloud-framework-ipc.
  deleteVolume - done through cloud-framework-ipc

UI Impact

There are several places that CloudStack UI will be impacted by this feature:

1. Add Secondary Storage Flow
   Secondary storage in our UI refers to our current image store concept. So all the API invoked from that flow should use the new image store commands documented above. Also our UI should allow to create an image store by choosing one provider from list of backup providers from listStorageProvidersCmd. By choosing a provider, UI should automatically expand to add more fields to allow user to provide provider-specific parameters, like those extra parameters required for S3, etc.

1. Register Template Flow
   This UI should allow user to register a zone-wide or region-wide template

Besides these existing API changes, we will provide two new admin APIs to configure backup object store provider:

...

1. .