...

This feature enables the user to directly upload the template/volume to cloudstack and eliminates the dependency on an external http server.

This is hypervisor agnostic and should work for all hypervisors. Multiple files for the same template/volume will not be supported for now. It will use the current resource limits for templates/volumes and would error on reaching the limits. 

use cases

1. A user, who wants to upload a template and launch a vm with it on cludstack
2. A user, who want to upload a volume and attach it to a vm on cloudstack

...

1. Management Server generates a unique pre-shared key (PSK) on install using SHA1 and shares it with the SSVM agent. Nobody else has access to this key of course.
2. New API request "getUploadParams" encoded the usual CloudStack API way using api keys. Request parameters include format, hypervisor, md5 sum
3. Management Server response to (b) returns post url to upload the file, params to send to the file along with url while uploading the file and a SHA1 signature generated using PSK and a params json which is to be sent while uploading the file.  
4. API user makes a User should upload the file over https POST request over HTTPS to the URL obtained url obtained in (c). The signature and params returned in (c) should be passed as is.
5. The Apache web server on the SSVM matches the url, decrypts and passes on to the SSVM java agent as plain HTTP. The SSVM agent checks the signature against the params using the key PSK1PSK and SHA1. Once the params are verified, it writes to the NFS storage to the location encoded in the URL. 
6. Once the upload completes, the MD5 checksum is compared (if available from step c). 

...

• Response
  • postURL: post url to upload the file to. Ex: "https://ssvmpublicip/upload/keysignature" (key signature in the postURL would be is SHA1 key generated using PSK initially shared between SSVM agent and Management Server and the params in the response.)
  • params: params to be sent in the post request Ex: params:{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"}, "hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207", "format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"}

...

1. Browser UI requests the upload form by doing the getUploadParams api call. 
2. Management Server returns a upload url, params json string which should be passed on to the upload url while doing the upload
3. UI shows an upload dialog with a way to browse and upload file.
4. User chooses a file and posts on the url.
5. The post request completes once the template is downloaded/errored. 
6. The user can close the dialog(Post request should still be active) and use list page to check the status of it.

DB Changes

No db changes.

Major Tasks

...

.

...

...

Future

1. Upload progress to the user (http://creativejs.com/tutorials/advanced-uploading-techniques-part-1/ )
2. split upload and join in apache so as to not hit browser timeouts

3. post url should be active only for one time call - return proper error message on subsequent calls - how do we determine if its already used?

4. Admin to list all the uploads happening and cancel any if required?

5. Partial data upload and template cleanup (can be due to SSVM to agent restart or network error)

...

2. a new global param to limit the max upload size

Questions

1. Multipart form data - Can this go in multiple connections? - NO

2. SSVM storage issue. What should we tell the user? How can we notify the admin to recover from it?

3. Should there be an unencrypted HTTP version of the post?  No for now
4. No. of parallel uploads SSVM can handle? - a default value should be provided to tune the apache to handle those many parallel connections. These should be determined based on the CPU RAM of SSVM - a proper error message to the user on reaching this limit

...

check download template/volume code to see how the below scenarios are handled

1. multiple SSVMs. Which agent should handle the upload? 

2. cross zone template upload - file to be uploaded to one zone and then copied to all the other zones
3. system vm template upload - same as the current. will be differentiated using the filename.
4. default  default timeouts and configurable values - updated default values. There are no configurable values
5. multiple management servers with the one monitoring the download crashed (How is the template status updated? A. when the download hasnt started yet. B. upload in progress C. upload done) - Template status wont be updated unless the SSVM is restarted and is connected to new management server
6. How does the agent return template status for DownloadProgressCommand? (- in memory hashmap of in progress items. need to check of it recovers the status during a restart or when done)crash, its starts the wget download again.