Introduction

Currently Cloudstack accepts only URL's to register a template/volume. If the user has a downloaded template/volume available, user has to host it either on dropbox etc. or run web server so as to get downloadable url. It is desirable to be able to upload a template/volume directly through standard web browser.

This feature enables the users to directly upload the template/volume to Cloudstack and eliminates the dependency on an external http server. This will be a complementary functionality and users can continue to register template/volume with URL as well.

Use cases

1. User wants to upload a template from the browser and launch a vm with it on Cloudstack
2. User wants to upload a volume from the browser and attach it to a vm on Cloudstack

Functional Requirements

• Support upload of template through https POST request with pre-generated URL
• Support upload of volume through https POST request with pre-generated URL
• Template/volume upload failure is irrecoverable, it needs to be reinitiated by requesting a new upload url
• Cleanup of partially uploaded templates/volumes in case of failure
• Provide upload status (percentage, bytes etc) using the list templates/volumes api call
• Global configuration parameter to limit the max upload size of template/volume
• Enforcement of account limits for templates/volumes 
• This feature is hypervisor agnostic and should work for all hypervisors 

Out of Scope

1. No unencrypted or http version of POST will be available

2. Upload progress from SSVM Apache web-server when POST request is active (http://creativejs.com/tutorials/advanced-uploading-techniques-part-1/)

3. Split upload and join in Apache so as to not hit browser timeouts

4. Ability to cancel in-progress uploads by admin

5. Multiple files for the same template/volume will not be supported for now

Design

Process Flow

The browser-based uploading process will have the following steps:

1. Management Server generates a unique pre-shared key (PSK) and shares it with the SSVM agent. 
2. New API request "getUploadParams" encoded the usual Cloudstack API way using api keys. Request parameters include format, hypervisor, md5 sum etc.
3. Management Server response to (b) returns post url to upload the file, params to send to the url while uploading the file and a SHA1 signature generated using PSK and params json.  
4. User should upload the file over https POST request to the url obtained in (c). The signature and params returned in (c) should be passed as is.
5. The Apache web server on the SSVM matches the url, decrypts and passes on to the SSVM java agent as plain HTTP. The SSVM agent checks the signature against the params using the key PSK and SHA1. Once the params are verified, it writes to the NFS storage to the location encoded in the URL. 
6. Once the upload completes, the MD5 checksum is compared (if available from step c).

Pre-shared key management

This section covers when and how the pre-shared key is generated by management server, how is it stored, how is it passed to SSVM

1. On start of management server, if the key doesn't exist, it auto-generates one using SHA1 and stores it in db
2. Key is saved in encrypted form (db encryption is enabled) in the configuration table as hidden configuration with name upload.secret.key
   
3. During SSVM start-up, the key is passed to it by management server 
4. SSVM stores this key in a file at /etc/cloudstack/agent/ms-psk
5. If the key sharing fails SSVM agent would shut itself down

Sequence Diagram

Management server and SSVM agent interaction

When the management server receives getUploadParams api call, it selects SSVM and secondary store in the zone requested by the user and returns a POST url to that SSVM. There can be multiple SSVMs in the same zone. Management servers decides which SSVM to handle this based on the load.

SSVM agent will keep track of the uploads assigned to it. If the template/volume is selected to be cross-zone, it will uploaded to one zone and then copied to other zones from it (already existing functionality).

Management server owns the lifecycle of volume/template (initially gets created with upload state as "NOT_UPLOADED"). It polls the SSVM agent (using an already existing agent command) to get the status and triggers appropriate state transition. Polling starts after the successful completion of the getUploadParams API. The polling 'interval' and 'timeout' are determined by global configuration parameters (add configuration details).

Failure at any point during the upload process to SSVM would result in upload state getting transitioned to "UPLOAD_ERROR" (refer to the state machine below). Also if the upload is not completed within 'timeout' configuration the state transitions to error. This will take care of SSVM agent failure/SSVM failure.

If the management server itself goes down during the upload, the polling is resumed by the new management server that takes ownership of the SSVM.

SSVM agent and SSVM apache interaction

The SSVM agent will run nio server and will listen for any connections. It will be using the localhost and port number(come from a configuration) to listen.

SSVM apache proxies the data from https to http url on localhost: port using ProxyPass.

All the http events and data will be proxied by apache to agent. only SSL validation happens at apache layer.

Upload failures

The below failures can happen while SSVM agent is receiving the template. When management server polls for the status of the template/volume, if any of these below errors happen, the agent returns appropriate information and management server updates the state accordingly.

• pre-validation errors (file upload hasn't started yet)
  • SSL handshake failed
  • params validation with PSK failed
  • max file size from the header crossed the limit
• post validation error 
  • file successfully uploaded
    • file uploaded is not of the type specified (for ex: if the template is said to be of type tar, the actual downloaded file is not if this type)
    • max upload limit crossed
  • partial file upload (http errors can be caught through appropriate event handlers: exception,timeout,closed (http://hc.apache.org/httpcomponents-core-ga/tutorial/html/nio.html))
    • user stopped/canceled the post request before completion
    • uploading the template failed due to a network issue
    • browser request timed out
    • browser crashed
    • SSVM is down/restarted
    • SSVM apache down/restarted
    • secondary storage is not accessible
    • there isn't enough space in secondary storage and hence agent can't write to it

Upload successful

If the volume/template upload is successfully completed (and installed in case of template) within the polling timeout. In this agent returns a success to management server during the status poll.

State machine 

Template/Volume go through below state machine state transitions, when it goes through the browser based upload.

States:

NOT_UPLOADED: getUploadParams API was successful and the template is registered. But, upload hasn't started yet

UPLOAD_IN_PROGRESS: User has initiated the POST request on SSVM Apache and the validation/upload is in progress

UPLOADED: SSVM agent successfully copied the template to secondary storage

UPLOAD_ERROR: Any failure in between user initiating the POST request and template being downloaded OR failure to complete within configured timeout

ABANDONED: the POST request isn't called in the stipulated time (check sequence diagram point 4). In this case the SSVM agent has not record of the upload (TBD: Do we really need to distinguish ERROR and ABANDONED?)

State Transitions:

• Transition1: This state transition happens, when the user initiates the post request
• Transition2: Upload is in progress
• Transition3: On successful completion upload/copy (and installation in case of template) to secondary storage
• Transition4: This can happen in multiple ways outlined in the failures section above. All these errors will result in the same state for the template/volume as there isn't any way to recover from them or the handling isn't different for these. Additional error information will be provided in the form of a message for respective errors in the agent logs.

• Transition5: The POST request hasn't been initiated and the timeout has reached

Cleanup

A GC thread will be running at regular intervals (configurable, provide details). It will pick up all volume/template with upload state as "UPLOAD_ERROR" and "ABANDONED" and send agent command to cleanup any partial data from secondary store.

Recovery mechanisms

There isn't any recovery or retry mechanism as this is a POST request. Once error happens, template/volume will remain in the error state and admin will be able to troubleshoot it based on the appropriate log messages in management server log, agent log, apache access/error log files. This will be cleaned by the garbage collector eventually. The user has to reinitiate the upload by calling getUploadParams again.

One time POST url and parallel uploads

The POST url returned by the management server to upload template/volume is expected to be used only once. If there is a download in-progress/error/success state for template/volume in the URL and the SSVM agent gets a new request for the same URL, the agent rejects this request saying there is another download in progress (409 conflict).

Open item: The limitation here is that if the template/volume cleanup happened and the template is cleaned, agent would start accepting the url again. This will be fixed later. 

API changes

A new API getUploadParams which takes all the params as the registertemplate/uploadvolume except for URL and which returns all the required params plus a post URL to be used to upload volume/template

• Request 
  • type: volume or template : required
  • params while uploading a volume
    • format: the format for the volume. Possible values include QCOW2, OVA, and VHD : required
    • name: the name of the volume : required
    • zoneid: the ID of the zone the volume is to be hosted on : required
    • account: an optional accountName. Must be used with domainId.
    • checksum: the MD5 checksum value of this volume
    • domainid: an optional domainId. If the account parameter is used, domainId must also be used.
    • imagestoreuuid: Image store uuid
    • projectid: Upload volume for the project
  • params while uploading a tempalte
    • displaytext: the display text of the template. This is usually used for display purposes: required
    • format: the format for the template. Possible values include QCOW2, RAW, and VHD : required
    • hypervisor: the target hypervisor for the template : required
    • name: the name of the template : required
    • ostypeid: the ID of the OS Type that best represents the OS of this template : required
    • zoneid: the ID of the zone the template is to be hosted on : required
    • account: an optional accountName. Must be used with domainId.
    • bits: 32 or 64 bits support. 64 by default
    • checksum: the MD5 checksum value of this template
    • details: Template details in key/value pairs.
    • domainid: an optional domainId. If the account parameter is used, domainId must also be used.
    • isdynamicallyscalable: true if template contains XS/VMWare tools inorder to support dynamic scaling of VM cpu/memory
    • isextractable: true if the template or its derivatives are extractable; default is false
    • isfeatured: true if this template is a featured template, false otherwise
    • ispublic: true if the template is available to all accounts; default is true
    • isrouting: true if the template type is routing i.e., if template is used to deploy router
    • passwordenabled: true if the template supports the password reset feature; default is false
    • projectid: Register template for the project
    • requireshvm: true if this template requires HVM
    • sshkeyenabled: true if the template supports the sshkey upload feature; default is false
    • templatetag: the tag for this template.

• Response
  • postURL: post url to upload the file to. Ex: "https://ssvmpublicip/upload/uuid/signature" (signature in the postURL is SHA1 key generated using PSK initially shared between SSVM agent and Management Server and the params in the response.) The URL also has the uuid to query the status using listTemplate/Volumes API subsequently
  • params: params to be sent in the post request in JSON format Ex: params:{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"}, "hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207", "format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"}

POST url to post the data on to SSVM. This post url is returned from the getUploadParams api call Ex: https://ssvmpublicip/upload/key 

• Request: (this is a POST request)
  • params: JSON string of params returned from previous getUploadParams api call
  • file: the location of the template/volume to be posted a multipart/form-data
• Response: 
  • '200 OK' on successful upload
  • '401 authorization error' incase the signature key validation or any other validation failed
  • '500 internal server error' incase the file upload fails
  • '409 conflict' incase there is already another upload with same params in-progress/success/error

At any point, the api user should be able to call the listtemplates/listvolumes api to get the installation status of the template.

If the template failed to upload(401 or 500 above), the user should request for a new upload url and upload it again. He shouldn't be reusing the same upload URL again.

Example

management server api: 

request:

http://managementip:8080/client?command=getUploadParams&type=template&response=json&sessionkey=OXgYX%2BINFOOcxQgI2yzEzuA3xCg%3D&name=lxc1&displayText=lxc1&zoneid=-1&format=TAR&isextractable=false&passwordEnabled=false&isdynamicallyscalable=false&osTypeId=1b510c30-3352-11e4-aaca-a5c7f57670d0&hypervisor=LXC&requireshvm=false&_=1410847402478 

response:

{ 
"postURL":"https://ssvmpublicip/upload/uuid",
payload:{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"}, "hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207", "format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"},
signature:de7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9
}

file upload:

request:

curl -X POST "https://ssvmpublicip/upload/uuid" -F "template=@templatelocation.tar" -F "signature=de7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9" -F "payload={'store':{'com.cloud.agent.api.to.NfsTO':'nfs://10.147.28.7/export/home/rajani/lxc/secondary'}, 'hvm':false,'description':'lxc1','maxDownloadSizeInBytes':53687091200,'id':207,'resourceType':'TEMPLATE','installPath':'template/tmpl/2/207', 'format':'TAR','accountId':2,'name':'207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9'}" -v

response:

200 OK

DB Changes

No schema changes. Some configuration items will be created.

Security

• The NFS Store url, path and some other information are exposed to the end user in the response to getUploadParams. This is done so that agent gets all the information required to download the template with the POST request. One option can be to encrypt the entire JSON and then send it across as part of response
• The Apache web server will only accept upload requests over HTTPS
• Proxy between apache and agent is internal to the SSVM. Hence no security threat here.
• SSVM PSK location can only be accessed by admin. Hence no security threat.

Performance

• To increase the number of parallel uploads, multiple SSVMs needs to be created. This will be handled automatically by the management server using the SSVM scale up mechanism (system.vm.auto.reserve.capacity, secstorage.capacity.standby and secstorage.session.max configuration variables). The more upload for SSVM results in more polling related agent commands going to the SSVM and that should exhaust the session.max limit and result in spawning new SSVM.
• TBD: number of parallel uploads a single SSVM can handle

UI Changes

A new UI to upload volume/template which should be able to accept "browse and upload" file. 

UI flow:

1. Browser UI requests the upload form by doing the getUploadParams api call. 
2. Management Server returns a upload url, payload json string which should be passed on to the upload url while doing the upload
3. UI shows an upload dialog with a way to browse and upload file.
4. User chooses a file and posts on the url.
5. The post request completes once the template is downloaded/errored. 
6. The user can close the dialog(Post request should still be active) and use list page to check the status of it.

Open Issues

1. SSL CA certificates - what kind of certificates will be used?
2. PSK length - for now it can be a fixed length, later on it can be made configurable.

3. Upload url should be active only for one call - Its partially handled now (see below for details). The limitation needs to be addressed.

4. Recommended configuration of SSVM and apache web server
5. Create a separate section to list out all the global config params
   

Bug Reference & Branch

CLOUDSTACK-....

template-upload branch off master