Page History

...

1. Move data between primary storage and object storage directly, totally bypass NFS cache storage. This is possible for Vmware and KVM.
2. Move data between primary storage and object storage, but through a cache storage. This is for certain operations on XenServer.
3. Move data between primary storages, through hypervisor specific storage motion functionalities, such XenMotion, VMotion etc.

 Storage DataStore Plugin Framework

In this new refactored backup datastore plugin framework, we have clearly defined those pluggable service interfaces, such as PrimaryDataStore, ImageDataStore, DataMotionStrategy, AutoScaleStrategy, etc, so that different storage providers can develop their vendor-specific plugins based on the well-defined contracts that can be seemlessly managed by CloudStack orchestration.

...

The following are deprecated tables:

• template_host_ref (replaced with template_store_ref)
• template_s3_ref (replaced with template_store_ref)
• template_swift_ref (replaced with template_store_ref)
• volume_host_ref (replaced with volume_store_ref)
• s3 (replaced with image_data_store)
• swift (replaced with image_data_store)

API Changes

Following will be the changes in their behavior post introduction of Object Store plugin framework:

image store commands

NOTE that we have following assumptions:

1. you can add multiple image stores from the same provider, but we prevent you from adding multiple image
   stores from different providers.
2. For NFS image store providers, it is always zone-wide. And For S3/Swift, it is always region-wide.

• listStorageProvidersCmd - List all enabled image store providers (with type=IMAGE)'
  Parameters: type (Primary or Image)
  Response: List of StorageProviderResponse
• addImageStoreCmd - This command will be used to handle adding image store from different providers, including NFS, S3, Swift, etc
  Parameters: url, zoneId, providerName, details
  Response: ImageStoreResponse
• listImageStoresCmd - List all CloudStack managed image stores.
  Parameters: id, storeName, protocol, provider, zoneId
  Response: list of ImageStoreResponse
• deleteImageStoreCmd - Delete an image store when there are no resources (snapshots,volumes,templates) residing on it.
  Parameters: id
  Response: SuccessResponse

Here are previous APIs to be deprecated. For backwards-compatibility, we will internally wire these old api to use new api logic, so it will still work with one change of response object to ImageStoreResponse, but it will be marked as deprecated in API doc.

• addSecondaryStorageCmd
• addS3Cmd
• addSwiftCmd
• listS3Cmd
• listSwiftCmd

snapshot commands

• CreateSnapshotCmd flow

template/iso commands

• No changes: createTemplate, updateTemplate, listTemplates, updateTemplatePermissions, listTemplatePermissions, prepareTemplate
• To be changed:
  registerTemplate - Would register the URI. Download work would be done by S3 image store provider now.
  deleteTemplate - Would just unregister the template, physically deleting it from OS should happen through S3 image store provider api.
  extractTemplate - would just give the S3 image store url
  copyTemplate - would return success since the template would be available region wide.
  createTemplate - done through data motion service
  prepareTemplate - done through data motion service
  listTemplates - we create db view to perform listTemplate and listIso, to get rid of direct sqls in our code.

volume commands

• No changes: attachVolume, detachVolume, deleteVolume, listVolumes
• To be changed:
  extractVolume - Do we put it into S3 now internally ?need to be done through image store provider.
  uploadVolume - similar to register template. User needs to put in just the S3 image store url here.
  createVolume - when creating volume from snapshot, zone id needs to be mentioned.
  migrateVolume - should work as is using scratch storage instead of nfs sec. storage.

Besides these existing API changes, we will provide two new admin APIs to configure backup object store provider:

• listBackupProviders – List CS managed backup providers and their capabilities.
• configureBackupProvider – choose a provider (like S3 or Swift) as backup provider.

Tasks:

• storage cache manager.
  attachVolume/detachVolume - done through cloud-framework-ipc.
  deleteVolume - done through cloud-framework-ipc

UI Impact

There are several places that CloudStack UI will be impacted by this feature:

1. Add Secondary Storage Flow
   Secondary storage in our UI refers to our current image store concept. So all the API invoked from that flow should use the new image store commands documented above. Also our UI should allow to create an image store by choosing one provider from list of backup providers from listStorageProvidersCmd. By choosing a provider, UI should automatically expand to add more fields to allow user to provide provider-specific parameters, like those extra parameters required for S3, etc.

1. Register Template Flow
   This UI should allow user to register a zone-wide or region-wide template.
2. Backup storage framework
   1. refactoring snapshot service
   2. refactoring template service
   3. refactoring data motion strategy plugin
3. Autoscaling cache storage framework
   1. cache storage plugin interface
   2. autoscaling strategy plugin
   3. NFS based cache storage implementation
   4. NFS transport VM based cache storage implementation
   5. NFS cache storage vs NFS transport VM based cache storage
4. image store plugins at the mgt server side
   1. classical NFS plugin
   2. S3 plugin
   3. swift plugin
5. image store plugins at the hypervisor side
   1. vmware
   2. kvm
   3. xenserver
6. API changes
7. end user APIs
admin APIS