...

Back NFS secondary storage of templates, ISOs, and snapshots with an S3-compatible object store to synchronize immutable assets across zones. The initial integration

Limitations and Assumptions

The current implementation of this feature is based on the following limitations and assumptions:

• Zero to One S3-compatible object stores per CloudStack implementation
• S3-backed secondary storage can not used with Swift-backed secondary storage.All objects are stored in full redundancy mode. A future version may introduce the ability to specify the redundancy level.
• The synchronization process is single-threaded. Therefore, for synchronizations of multiple large files their may be a measurable latency dependent on the available bandwidth between the secondary storage virtual machines and the object store. A future relase release may provide a configurable upload thread pool for these operations to reduce this latency.

Limitations and Assumptions

• All objects are stored in full redundancy mode. A future version may introduce the ability to specify the redundancy level.
• Does not use multipart uploads

Currently, multipart uploads are not utilized by the integration in Currently, multipart uploads are not utilized by the integration in order to support the widest range of S3-compatible storage solutions. This limitation impacts the S3 integration operations as follows:

• Incomplete directory uploads: When uploading a directory of files (e.g. a template or exploded ISO), an interupted upload will result in a partial upload to the object store. However, synchonrization operations are driven by an entry in the database that is not inserted until the directory upload is completed successfully. In this scenario, the partial download will be present in the object store, but not pushed to other zones because it will not be present in the database.
• No progress reporting: Progress reporting of uploads is only provided via the multipart upload interface. Therefore, the logs only reflect when an upload has started and, sucessfully or unsuccessfully, ended.

As support for multipart uploads broadens across S3 implementations, this feature may be enhanced to utilize this capability. The S3Utils#putFile and S3Utils#putDirectory operations to explicitly isolate single and multi-object put operations – permitting the transperant implementation of multipart uploads without impact to client code.

...

1. Stream Oriented: All interfaces must based on Input/Output stream rather than File. This higher-level abstraction would permit uploads and downloads to be re-directed to the object store rather than using intermediate writes to disk. Furthermore, a stream-based interface would also permit the use of diskless compute nodes.
2. Zone Optional: Remove the requirement that secondary storage and assets be associated with a zone.

...

The design of this feature is based on the approach taken for the Swift integration. As such, this feature requires an NFS volume in each zone as a cache for assets. This approach was selected due to a lack of support for global secondary storage (i.e. secondary stroage that is not associated with a zone). If the secondary storage re-design permits global storage and integration points compatible with an object store, then this approach may be re-evaluated.

Template and ISO upload to the object store is facilitated by a thread which compares the reaper thread (com.cloud.template.S3SyncTask) which compares the contents of the NFS volume and the object store. Each element that is not present in the object store is uploaded to the object store. Currently, this process is single threaded whereby one assest is uploaded at a time. This approach simplifies the synchronization process, but may introduce measurable latency when a number of large files need to pushed to the object store.

...

Template, snapshot, and ISO download from the object store occur on-demand in each zone. Therefore, when an asset is request requested in a zone where it is not present in the zone's NFS secondary storage volume, there it will be downloaded from the object store. N.B. There may be measurable lag while the asset is downloaded from the object store.

...

The following ERD depicts the tables added or changed by implementation of this feature:

When a template is added to the system, a row is added to the vm_template table describing the particulars of the template (.e.g name, platform, URL, etc). For cross-zone templates, a row is added to the template_zone_ref table for each zone in the system (existing functionality). Once a template is uploaded to an S3-compatible object store, a row for the template is added to the template_s3_ref to indicate its presence in the object store. The reaper thread identifies templates to be uploaded to the object store by joining through the template_host_ref, template_s3_ref, and vm_template tables to identify templates to identify downloaded, non-system, non-host templates that are not present in the object store.

Web Services APIs

The following APIs support this feature:

...