...

This document describes the S3-backed secondary storage functional behavior and technical design in support of enhancement ticket <todo fill-in ticket number> CLOUDSTACK-509.

References

• Amazon Web Service
• Amazon S3 REST API
• Amazon Web Service (AWS) SDK for Java
• Riak CS

...

AWS: Amazon Web Service – an IaaS offering from amazon.com
S3: AWS ' Simple Storage Service. Highly available object storage exposed via a REST interface structured in terms of buckets and contained key/value pairs. It permits the storage objects up to 5 TB in size.
Riak CS: An S3-compatiable object storage solution built by Basho atop its highly available Riak key-value store.
Asset: A template, ISO, or snapshot stored in secondary storage

...

Back NFS secondary storage of templates, ISOs, and snapshots with an S3-compatible object store to synchronize immutable assets across zones. The initial integration

Limitations and Assumptions

The current implementation of this feature is based on the following limitations and assumptions:

• Zero to One S3-compatible object stores per CloudStack implementation
• S3-backed secondary storage can not used with Swift-backed secondary storage.
• All objects are stored in full redundancy mode. A future version may introduce the ability to specify the redundancy level.
• The synchronization process is single-threaded. Therefore, for synchronizations of multiple large files their may be a measurable latency dependent on the available bandwidth between the secondary storage virtual machines and the object store. A future relase release may provide a configurable upload thread pool for these operations to reduce this latency.

Limitations and Assumptions

• All objects are stored in full redundancy mode. A future version may introduce the ability to specify the redundancy level.
• Does not use multipart uploads

Currently, multipart uploads are not utilized by the integration in order to support the widest range of S3-compatible storage solutions. This limitation impacts the S3 integration operations as follows:

• Incomplete directory uploads: When uploading a directory of files (e.g. a template or exploded ISO), an interupted upload will result in a partial upload to the object store. However, synchonrization operations are driven by an entry in the database that is not inserted until the directory upload is completed successfully. In this scenario, the partial download will be present in the object store, but not pushed to other zones because it will not be present in the database.
• No progress reporting: Progress reporting of uploads is only provided via the multipart upload interface. Therefore, the logs only reflect when an upload has started and, sucessfully or unsuccessfully, ended.

As support for multipart uploads broadens across S3 implementations, this feature may be enhanced to utilize this capability. The S3Utils#putFile and S3Utils#putDirectory operations to explicitly isolate single and multi-object put operations – permitting the transperant implementation of multipart uploads without impact to client code.

...

The current implementation of S3-backed secondary storage synchronizes files from NFS volumes -based secondary storage to an object store. The primary reason for this approach is that there is no provision for global (i.e. cross zone) secondary storage. Currently. the design of the secondary storage resource management facilities are under re-design. This new facility will also need to work In order to support direct access to an object store, it would beneficial if a simple object store SPI was introduced that met the following requirements:

1. Stream Oriented: All interfaces must based on Input/Output stream rather than File. This higher-level abstraction would permit uploads and downloads to be re-directed to the object store rather than using intermediate writes to disk. Furthermore, a stream-based interface would also permit the use of diskless compute nodes.
2. Zone Optional: Remove the requirement that secondary storage and assets be associated with a zone.

When this architectural redesign is completed, then this feature can be modified to directly access the object store. Such an enhancement would greatly simplify the implementation and deployment of S3-backed secondary storage. OutputStreams raher than requiring underlying file system. When this architectural redesign is completed, then this feature can be modified to directly access the object store. Such an enhancement would greatly simplify the implementation and deployment of S3-backed secondary storage. It would also allow users of S3-backed secondary storage to realize significant cost savings by removing redundant storage capacity for NFS vlumes volumes in each zone.

Quality Risks/Test Guidelines

...

S3-backed secondary storage does not add any system-level requirements to CloudStack.

Deployment

The only pre-requisite following are the prerequisites for S3-backed secondary storage is a :

• A subscription to or configuration of an S3-compatible object store
• An empty bucket configured and associated access key and secret key with read/write access for the exclusive use of CloudStack

. A global configuration option, enable. s3.secondary.storageenable, is then enabled and following a restart of the management server(s), the administrator can configure the object store's connection parameters.

...

The design of this feature is based on the approach taken for the Swift integration. As such, this feature requires an NFS volume in each zone to search as a cache for assets. This approach was selected due to the architecture re-design work being performed around secondary storage, and the relative newness of the developer implementing the feature. As the new architecture stablizes, this decision will a lack of support for global secondary storage (i.e. secondary stroage that is not associated with a zone). If the secondary storage re-design permits global storage and integration points compatible with an object store, then this approach may be re-evaluated. Given the basic nature of the core S3 integration, much of the work for this implementation should be transferable to the new architecture.

Template and ISO upload to the object store is facilitated by a reaper thread (com.cloud.template.S3SyncTask) Template and ISO upload to the object store is facilitated by a thread which compares the contents of the NFS volume and the object store. Each element that is not present in the object store is uploaded to the object store. Currently, this process is single threaded whereby one assest is uploaded at a time. This approach simplifies the synchronization process, but may introduce measurable latency when a number of large files need to pushed to the object store.

...

Template, snapshot, and ISO download from the object store occur on-demand in each zone. Therefore, when an asset is request requested in a zone where it is not present in the zone's NFS secondary storage volume, there it will be downloaded from the object store. N.B. There may be measurable lag while the asset is downloaded from the object store.

The following UML class diagram depicts a static view of the classes impacted by the implementation of this feature (due to the image size, a separate download may be required to properly view it):

Third Party Libraries and Licensing

This feature adds a dependency to the AWS SDK for Java which is licensed under the Apache 2.0 license.

Database Modifications

The following ERD depicts the tables added or changed by implementation of this feature:

When a template is added to the system, a row is added to the vm_template table describing the particulars of the template (.e.g name, platform, URL, etc). For cross-zone templates, a row is added to the template_zone_ref table for each zone in the system (existing functionality). Once a template is uploaded to an S3-compatible object store, a row for the template is added to the template_s3_ref to indicate its presence in the object store. The reaper thread identifies templates to be uploaded to the object store by joining through the template_host_ref, template_s3_ref, and vm_template tables to identify templates to identify downloaded, non-system, non-host templates that are not present in the object store.

Web Services APIs

The following APIs support this feature:

...