Versions Compared

Old Version 40

changes.mady.by.user Murali Reddy

Saved on

compared with

New Version Current

changes.mady.by.user user-fce10

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migrated to Confluence 5.3

Table of Contents

Introduction

Currently CloudStack Cloudstack accepts only URL's to register a template/volume. If the user has a downloaded template/volume available, user has to host it either on dropbox etc. or run web server so as to get downloadable url. It is desirable to be able to upload a template/volume or template directly through standard we web browser.

This feature enables the users to directly upload the template/volume to CloudStack and Cloudstack and eliminates the dependency on an external http server. This will be a complementary functionality and users can continue to register template/volume with URL as well.

...

Use cases

  1. A user, who User wants to upload a template from the browser and launch a vm with it on CloudStackCloudstack
  2. User wants A user, who want to upload a volume from the browser and attach it to a vm on CloudStackCloudstack

Functional Requirements

This section will explain key requirement this feature shall address.

Design

This section will explain how the feature shall be implemented in CloudStack

process flow

The browser-based uploading process will have four main steps:

  • management server and SSVM will establish a pre-shared key between them, which shall be used for generating signature with SHA1
  • api call to get 'post url'
  • retrieve post url from the api response
  • post the volume/template to through http POST request to port url obtained from the 2nd step

Image Removed

  • Support upload of template through https POST request with pre-generated URL
  • Support upload of volume through https POST request with pre-generated URL
  • Template/volume upload failure is irrecoverable, it needs to be reinitiated by requesting a new upload url
  • Cleanup of partially uploaded templates/volumes in case of failure
  • Provide upload status using the list templates/volumes api call
  • Global configuration parameter to limit the max upload size of template/volume
  • Enforcement of account limits for templates/volumes 
  • This feature is hypervisor agnostic and should work for all hypervisors 

Out of Scope

  1. Support for Http/unencrypted upload

  2. Upload progress during POST request

  3. Splitting of template/volume during upload and merge back

  4. Ability to cancel in-progress uploads by admin

  5. Upload for template/volume with more than one file will not be supported

Process Flow

The browser-based uploading process will have the following steps:

Image Added

  1. Management Server generates a unique pre-shared key (PSK) and shares it with the SSVM agent. 
  2. New API request "getUploadParamsForVolume/Template" encoded the usual Cloudstack
  3. Management Server generates a unique pre-shared key (PSK) and shares it with the SSVM agent. 
    1. key is saved in encrypted form in the configuration tables as hidden configuration with name upload.secret.key
    2. on start of management server, if the key doesnt exist, it generates one using sha1 and stores it in db
    3. when the management server sends StartCommand to SSVM agent, it shares this key with it. 
    4. SSVM agent stores this key in a file at /etc/cloudstack/agent/ms-psk.
  4. New API request "getUploadParams" encoded the usual CloudStack API way using api keys. Request parameters include format, hypervisor, md5 sum etc.
  5. Management Server response to (b) returns post url to upload the file, params encrypted metadata to send to the url while uploading the volume/template file and a SHA1 signature generated using PSK and params json.  
  6. User should upload the file over https POST request to the url obtained in (c). The signature and params metadata returned in (c) should be passed as is.
  7. The Apache web server on the SSVM matches the url, decrypts and passes on to the SSVM java agent as plain HTTP. The SSVM agent checks the signature against the params decrypted metadata using the key PSK and SHA1. Once the params are data is verified, it writes to the NFS storage to the location encoded in the URL. 
  8. Once the upload completes, the MD5 checksum is compared (if available from step c). 

 

This is hypervisor agnostic and should work for all hypervisors. Multiple files for the same template/volume will not be supported for now. It will use the current resource limits for templates/volumes and would error on reaching the limits. 

 

To increase the number of parallel uploads, multiple SSVMs needs to be created. This will be handled automatically by the management server using the system.vm.auto.reserve.capacity, secstorage.capacity.standby and secstorage.session.max configuration variables.

API changes

A new API getUploadParams which takes all the params as the registertemplate/uploadvolume except for URL and  which returns all the required params plus a post URL to be used to uploadVolume/registerTemplate

API changes

New APIs getUploadParamsForVolume/getUploadParamsForTemplate to enable users to upload volume/template from a local or network file share. The API response has a POST URL to which the volume/template needs to be uploaded. These are available to regular and admin users of Cloudstack. The request and response parameters are described below. Required parameters are marked with 'required'. Some of the parameters are exclusive for admin user and are marked 'admin only'.

GetUploadParamsForVolume
  • Request
    • name: the name of the volume : required
    • format: the format of the volume to be uploaded. Possible values for volume include QCOW2, OVA and VHD. : required
    • zoneid: the UUID of the zone the volume is associated to : required
    • checksum: the MD5 checksum of volume to be uploaded. If specified this is used to validate the content of the uploaded volume for integrity.
    • account: an optional account name. Must be used with 'domainid' parameter below. Defaulted to account name of caller if not specified.
    • domainid: an optional domain to which the account belongs
    Request 
    • type: volume or template : required
    • params while uploading a volume
    • format: the format for the volume. Possible values include QCOW2, OVA, and VHD : required
    • name: the name of the volume : required
    • zoneid: the ID of the zone the volume is to be hosted on : required
    • account: an optional accountName. Must be used with domainId.
    • checksum: the MD5 checksum value of this volume
      • domainid: an optional domainId. If the account parameter is used, domainId 'domainid' must also be used.
    • imagestoreuuid: Image store uuid
    • projectid: Upload volume for the project
    • params while uploading a tempalte
      • displaytext: the display text of the template. This is usually used for display purposes: required
      • format: the format for the template. Possible values include QCOW2, RAW, and VHD : required
      • hypervisor: the target hypervisor for the template : required
      • name: the name of the template : required
      • ostypeid: the ID of the OS Type that best represents the OS of this template : required
      • zoneid: the ID of the zone the template is to be hosted on : required
      • account: an optional accountName. Must be used with domainId.
      • bits: 32 or 64 bits support. 64 by default
      • checksum: the MD5 checksum value of this template
      • details: Template details in key/value pairs.
      • domainid: an optional domainId. If the account parameter is used, domainId must also be used.
      • isdynamicallyscalable: true if template contains XS/VMWare tools inorder to support dynamic scaling of VM cpu/memory
      • isextractable: true if the template or its derivatives are extractable; default is false
      • isfeatured: true if this template is a featured template, false otherwise
      • ispublic: true if the template is available to all accounts; default is true
      • isrouting: true if the template type is routing i.e., if template is used to deploy router
      • passwordenabled: true if the template supports the password reset feature; default is false
      • projectid: Register template for the project
      • requireshvm: true if this template requires HVM
      • sshkeyenabled: true if the template supports the sshkey upload feature; default is false
      • templatetag: the tag for this template.
  • Response
    • postURL: post url to upload the file to. Ex: "https://ssvmpublicip/upload/signature" (signature in the postURL is SHA1 key generated using PSK initially shared between SSVM agent and Management Server and the params in the response.)
    • params: params to be sent in the post request Ex: params:{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"}, "hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207", "format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"}

POST url to post the data on to SSVM. This post url is returned from the getUploadParams api call Ex: https://ssvmpublicip/upload/key 

  • Request: (this is a POST request)
    • params: json string of params returned from previous getUploadParams api call file: file to upload
    • file: the location of the template/volume to be posted a multipart/form-data
  • Response: 
    • '200 OK' on successful upload
    • '401 Authorization error' incase the signature key validation or any other validation failed
    • '500 internal server' incase the file upload fails

At any point while this post is active, the api user should be able to call the listtemplates/listvolumes api to get the upload status of it. 

DB Changes

No db changes.

Sequence Diagram

Image Removed

  1. failure at any point during the form upload to ssvm apache would return error to the user and DOWNLOAD_ERROR when the management server pings for download status.
  2. polling starts after the success of getUploadParams
  3. current polling interval is 10000 ms 
  4. currently it polls thrice to check the status. We may need to increase this. (is the current value 5 or 3?)

Agent Commands

  1. DownloadProgressCommand will be used the check the download progress (from Management server to Agent)
  2. DownloadCommand will be used to start the download(SSVM Apache to SSVM Agent)

template/volume upload task life cycle management

This section shall describe how the request to upload a volume is handled by management server.

  • how does management server decide which SSVM to pick
  • how does management server decide which NFS server to store
  • if there is enough load on the SSVM, will management server try launch new SSVM
  • how does the admin the outstanding browser based uploads
  • how does the use know the state of browser based upload
  • how does the management server know download complete
  • how does the management server know the % of download
  • how does the management server know the failed upload
  • where does the management server persist the state & percentage of upload

fault scenarios & recovering failed uploads 

This section shall describe how the management server shall detect a failed volume/template upload.

  • how does the management server detect failed tasks
  • when will it take action
  • what it will perform to cleanup the storage.

State machine 

Template/Volume go through below state machine state transitions, when it goes through the browser based upload.

Image Removed

States:

NOT_UPLOADED: getUploadParams API was successful and the template is registered. But, upload hasnt started yet

UPLOAD_IN_PROGRESS: User has initiated the POST request SSVM Apache and the validation/download is in progress

UPLOADED: SSVM Agent successfully Downloaded the template to secondary storage

UPLOAD_ERROR: Any Failure in between user initiating a post request and template being downloaded

ABANDONED: the post request isnt called in the stipulated time (check sequence diagram point 4)

State Transitions:

  • This state transition happens, when the user initiates the post request
  • Post request is active SSVM agent is downloading validation or downloading the template but, not done yet
  • on successful completion of template installation to secondary storage
  • This can happen in multiple ways:
    • user stopped/canceled the post request before completion
    • SSL handshake failed
    • params validation with PSK failed
    • uploading the template failed due to a network issue
    • secondary storage is not accessible
    • there isn't enough space in secondary storage and hence agent cant write to it.
      All these errors have the same state as there isnt any way to recover from them or the handling isnt different for these. Additional error information will be provided in the form of a message for respective errors. Admin will be able to troubleshoot it based on the appropriate log messages in management server log, agent log, apache access/error log files.
  • The post request hasn't happened and the timeout has reached

recovery mechanisms

Security

 

UI Changes

A new register template and upload volume UI which should be able to accept "browse and upload" file. 

UI flow:

  1. Browser UI requests the upload form by doing the getUploadParams api call. 
  2. Management Server returns a upload url, params json string which should be passed on to the upload url while doing the upload
  3. UI shows an upload dialog with a way to browse and upload file.
  4. User chooses a file and posts on the url.
  5. The post request completes once the template is downloaded/errored. 
  6. The user can close the dialog(Post request should still be active) and use list page to check the status of it.

Future Enhancements

  1. Upload progress to the user (http://creativejs.com/tutorials/advanced-uploading-techniques-part-1/ )
  2. split upload and join in apache so as to not hit browser timeouts

  3. post url should be active only for one time call - return proper error message on subsequent calls - how do we determine if its already used?

  4. Admin to list all the uploads happening and cancel any if required?

  5. Partial data upload and template cleanup (can be due to SSVM to agent restart or network error)
  6. a new global param to limit the max upload size

Assumptions & Out-of-Scope

  1. Multipart form data - Can this go in multiple connections? - NO

  2. Should there be an unencrypted HTTP version of the post?  No for now

  3. No. of parallel uploads SSVM can handle? - a default value should be provided to tune the apache to handle those many parallel connections. These should be determined based on the CPU RAM of SSVM - a proper error message to the user on reaching this limit

  4. cross zone template upload - file to be uploaded to one zone and then copied to all the other zones
  5. system vm template upload - same as the current. will be differentiated using the filename.
  6. default timeouts and configurable values - updated default values. There are no configurable values
  7. multiple management servers with the one monitoring the download crashed (How is the template status updated? A. when the download hasnt started yet. B. upload in progress C. upload done) - Template status wont be updated unless the SSVM is restarted and is connected to new management server
  8. How does the agent return template status for DownloadProgressCommand? - in memory hashmap of in progress items. during a restart or crash, its starts the wget download again.

Open Issues

  1. SSVM storage issue. What should we tell the user? How can we notify the admin to recover from it?

  2. SSL CA certificates - what kind of certificates will be used?
  3. multiple SSVMs. Which agent should handle the upload?

Bug Reference & Branch

CLOUDSTACK-....

    • Defaulted to domain of the caller if not specified.
    • projectid: the UUID of the project if the volume needs to be associated with one
    • imagestoreuuid: the UUID of the storage pool where the uploaded volume gets stored. This can be obtained using listImageStores API call (https://cloudstack.apache.org/docs/api/apidocs-4.4/root_admin/listImageStores.html)
    • diskofferingid: the UUID of the disk offering. This must be a custom disk offering as the volume size is not known before the actual upload. If not specified the default custom disk offering is used.
  • Response
    • uuid: Unique UUID to identify the volume. This is used to query the status of volume after successful completion of upload
    • postURL: POST url to upload the file to; for e.g. "https://1-2-3-4.xyz.com/upload/uuid", where 1-2-3-4 is derived from SSVM public IP 1.2.3.4, and xyz.com is the domain name registered with Cloudstack. For more details refer to section "POST URL" below.
    • metadata: encrypted data to be sent in the POST request. This is used to transfer some internal meta-data required for upload
    • expires: the timestamp after which the signature expires
    • signature: signature is SHA1 key generated using PSK based on 'postURL', 'metadata' and 'expires' in the response. This is used to validate that the actual POST request to upload data is a genuine one
GetUploadParamsForTemplate
  • Request
    • name: the name of the template : required
    • format: the format of the template to be uploaded. Possible values for template include QCOW2, RAW and VHD : required
    • zoneid: the UUID of the zone the template is associated to : required
    • displaytext: the display text of the template. This is used for setting a suitable name for display purposes. : required
    • hypervisor: the target hypervisor for the template : required
    • ostypeid: the UUID of the OS type that best represents the OS of this template : required
    • checksum: the MD5 checksum of template to be uploaded. If specified this is used to validate the content of the uploaded template for integrity.
    • account: an optional account name. Must be used with 'domainid' parameter below. Defaulted to account name of caller if not specified.
    • domainid: an optional domain to which the account belongs. If the account parameter is used, 'domainid' must also be used. Defaulted to domain of the caller if not specified.
    • projectid: the UUID of the project if the template needs to be associated with one
    • bits: specifies if template supports 32 or 64 bit. Default is 64 bit.
    • details: additional template details in key/value pairs
    • isdynamicallyscalable: true if template contains XS/VMWare tools in order to support dynamic scaling of VM cpu/memory
    • isextractable: true if the template or its derivatives are extractable. Default is false
    • isfeatured: true if the template is a featured template. Default is false. : admin only
    • ispublic: true if the template is available to all users. Default is true. Regular users are allowed to create public templates based on global config 'allow.public.user.templates'
    • isrouting: true if the template type is routing i.e. if template is used to deploy router : admin only
    • passwordenabled: true if the template supports the password reset feature. Default is false.
    • requireshvm: true if this template requires hardware assisted virtualization support
    • sshkeyenabled: true if the template supports the sshkey upload feature. Default is false.
    • templatetag: the tag for this template. Used to deploy VMs on hosts with this tag. : admin only
  • Response
    • id: Unique UUID to identify the template. This is used to query the status of template after successful completion of upload
    • postURL: POST url to upload the file to; for e.g. "https://1-2-3-4.xyz.com/upload/uuid", where 1-2-3-4 is derived from SSVM public IP 1.2.3.4, and xyz.com is the domain name registered with Cloudstack. For more details refer to section "POST URL" below.
    • metadata: encrypted data to be sent in the POST request. This is used to transfer some internal meta-data required for upload
    • expires: the timestamp after which the signature expires
    • signature: signature is SHA1 key generated using PSK based on 'postURL', 'metadata' and 'expires' in the response. This is used to validate that the actual POST request to upload data is a genuine one
GetUploadParamsForVolume API call

http://managementip:8080/client/api?command=getUploadParamsForVolume&response=json&sessionkey=TW1GLzPclNGgKtoYN5Xznbw8Nds%3D&name=windows&zoneId=c2bcad2f-1eb1-45ba-bebe-1c21873831b7&format=VHD&apiKey=miVr6X7u6bN_sdahOBpjNejPgEsT35eXq-jB8CG20YI3yaxXcgpyuaIRmFI_EJTVwZ0nUkkJbPmY3y2bciKwFQ&signature=Lxx1DM40AjcXU%2FcaiK8RAP0O1hU%3D

Volume response

{ 
postURL:https://1-2-3-4.xyz.com/upload/C7D351D2-F167-4CC8-A9FF-3BECB0A625C4,
metadata:TKPFeuz2nHmE/kcREEu24mnj1MrLdzOeJIHXR9HLIGgk56bkRJHaD0RRL2lds1rKKhrro4/PuleEh4YhRinhxaAmPpU4e55eprG8gTCX0ItyFAtlZViVdKXMew5Dfp4Qg8W9I1/IsDJd2Kas9/ftDQLiemAlPt0uS7Ou6asOCpifnBaKvhM4UGEjHSnni1KhBzjgEyDW3Y42HKJSSv58Sgmxl9LCewBX8vtn9tXKr+j4afj7Jlh7DFhyo9HOPC5ogR4hPBKqP7xF9tHxAyq6YqfBzsng3Xwe+Pb8TU1kFHg1l2DM4tY6ooW2h8lOhWUkrJu4hOAOeTeRtCjW3H452NKoeA1M8pKWuqMo5zRMti2u2hNZs0YY2yOy8oWMMG+lG0hvIlajqEU=,
signature:de7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9,
expires: 2014-10-17T12:00:00+0530,
id:C7D351D2-F167-4CC8-A9FF-3BECB0A625C4
}

GetUploadParamsForTemplate API call

http://managementip:8080/client/api?command=getUploadParamsForTemplate&type=template&response=json&name=centos 64 64bit&displayText=centos 64 64bit&zoneid=-1&format=VHD&isextractable=false&passwordEnabled=false&isdynamicallyscalable=false&osTypeId=1b510c30-3352-11e4-aaca-a5c7f57670d0&hypervisor=XenServer&requireshvm=false&apiKey=miVr6X7u6bN_sdahOBpjNejPgEsT35eXq-jB8CG20YI3yaxXcgpyuaIRmFI_EJTVwZ0nUkkJbPmY3y2bciKwFQ&signature=Lxx1DM40AjcXU%2FcaiK8RAP0O1hU%3D

Template response

{ 
postURL:https://1-2-3-4.xyz.com/upload/DD0A9FC6-C17E-4180-963C-870B9D03A80A,
metadata:TKPFeuz2nHmE/kcREEu24mnj1MrLdzOeJIHXR9HLIGgk56bkRJHaD0RRL2lds1rKKhrro4/PuleEh4YhRinhxaAmPpU4e55eprG8gTCX0ItyFAtlZViVdKXMew5Dfp4Qg8W9I1/IsDJd2Kas9/ftDQLiemAlPt0uS7Ou6asOCpifnBaKvhM4UGEjHSnni1KhBzjgEyDW3Y42HKJSSv58Sgmxl9LCewBX8vtn9tXKr+j4afj7Jlh7DFhyo9HOPC5ogR4hPBKqP7xF9tHxAyq6YqfBzsng3Xwe+Pb8TU1kFHg1l2DM4tY6ooW2h8lOhWUkrJu4hOAOeTeRtCjW3H452NKoeA1M8pKWuqMo5zRMti2u2hNZs0YY2yOy8oWMMG+lG0hvIlajqEU=,
signature:de7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9,
expires: 2014-10-17T12:00:00+0530,
id:DD0A9FC6-C17E-4180-963C-870B9D03A80A
}

POST URL to post the data on to SSVM. This post url is returned from the getUploadParamsForVolume/Template api calls

Ex: https://1-2-3-4.xyz.com/upload/uuid

As seen in the example, the URL has a domain name (in this case xyz.com) and a prefix (in this case 1-2-3-4). The reason for having a domain name is that SSL certs can only issued for a domain and not any IP address. The domain name to be used is configured using the global config "secstorage.ssl.cert.domain". The prefix is to identify the SSVM based on its public IP address. So if the public IP is 1.2.3.4 then the prefix is 1-2-3-4. This approach for generating the URL is already present in Cloudstack. Refer (1) and (2) for more details.

  • As can be seen from the description of 'signature' above, the user should't tamper with the values of postURL, metadata, expires while making the POST request. Otherwise it would result in an upfront failure during POST request.
  • Request: (this is a POST request)
    • Custom HEADER field to be sent
      • X-metadata: encrypted data returned from getUploadParams API call: required
      • X-signature: signature returned from getUploadParams API call: required
      • X-expires: expires returned from getUploadParams API call: required
    • request parameters
      • file: the location of the template/volume to be posted as multipart/form-data: required
  • Response:
    • '200 OK' on successful upload
    • '400 Bad Request' if any fields are empty or the signature validation failed or the request expired or metadata cannot be decoded or the post request is already used
    • '500 internal server error' incase the file upload fails due to some runtime exception
Volume upload request

curl -X POST "https://1-2-3-4.xyz.com/upload/C7D351D2-F167-4CC8-A9FF-3BECB0A625C4"  -H "X-signature:de7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9" -H "X-metadata:TKPFeuz2nHmE/kcREEu24mnj1MrLdzOeJIHXR9HLIGgk56bkRJHaD0RRL2lds1rKKhrro4/PuleEh4YhRinhxaAmPpU4e55eprG8gTCX0ItyFAtlZViVdKXMew5Dfp4Qg8W9I1/IsDJd2Kas9/ftDQLiemAlPt0uS7Ou6asOCpifnBaKvhM4UGEjHSnni1KhBzjgEyDW3Y42HKJSSv58Sgmxl9LCewBX8vtn9tXKr+j4afj7Jlh7DFhyo9HOPC5ogR4hPBKqP7xF9tHxAyq6YqfBzsng3Xwe+Pb8TU1kFHg1l2DM4tY6ooW2h8lOhWUkrJu4hOAOeTeRtCjW3H452NKoeA1M8pKWuqMo5zRMti2u2hNZs0YY2yOy8oWMMG+lG0hvIlajqEU=" -H "X-expires:2014-10-17T12:00:00+0530" -F "file=@volume.vhd" -v

Volume response

200 OK

upload successful.

Template upload request

curl -X POST "https://1-2-3-4.xyz.com/upload/DD0A9FC6-C17E-4180-963C-870B9D03A80A"  -H "X-signature:de7c9b85b8b78aa6bc8a7a36f70a90701c9db4d9" -H "X-metadata:TKPFeuz2nHmE/kcREEu24mnj1MrLdzOeJIHXR9HLIGgk56bkRJHaD0RRL2lds1rKKhrro4/PuleEh4YhRinhxaAmPpU4e55eprG8gTCX0ItyFAtlZViVdKXMew5Dfp4Qg8W9I1/IsDJd2Kas9/ftDQLiemAlPt0uS7Ou6asOCpifnBaKvhM4UGEjHSnni1KhBzjgEyDW3Y42HKJSSv58Sgmxl9LCewBX8vtn9tXKr+j4afj7Jlh7DFhyo9HOPC5ogR4hPBKqP7xF9tHxAyq6YqfBzsng3Xwe+Pb8TU1kFHg1l2DM4tY6ooW2h8lOhWUkrJu4hOAOeTeRtCjW3H452NKoeA1M8pKWuqMo5zRMti2u2hNZs0YY2yOy8oWMMG+lG0hvIlajqEU=" -H "X-expires:2014-10-17T12:00:00+0530" -F "file=@templatelocation.vhd" -v

Template response

200 OK

 upload successful.

Query status of uploaded template/volume

User should be able to query the status of uploaded template/volume by calling listtemplates/listvolumes API with unique uuid obtained as part of the getUploadParams response (present in POST URL as mentioned above). The polling for status should start on successful completion of the POST request.

Volume status check request

http://managementip:8080/client/api?command=listVolumes&id=C7D351D2-F167-4CC8-A9FF-3BECB0A625C4&response=json&apiKey=miVr6X7u6bN_sdahOBpjNejPgEsT35eXq-jB8CG20YI3yaxXcgpyuaIRmFI_EJTVwZ0nUkkJbPmY3y2bciKwFQ&signature=Lxx1DM40AjcXU%2FcaiK8RAP0O1hU%3D

Volume status response

{ "listvolumesresponse" : { "count":1 ,"volume" : [  {"id":"C7D351D2-F167-4CC8-A9FF-3BECB0A625C4","name":"windows","zoneid":"c2bcad2f-1eb1-45ba-bebe-1c21873831b7","zonename":"zone","type":"DATADISK","provisioningtype":"thin","size":0,"created":"2014-10-17T15:24:06+0530","state":"Uploading","account":"admin","domainid":"fad51d88-449b-11e4-9cf2-2d5524b62490","domain":"ROOT","storagetype":"shared","hypervisor":"XenServer","diskofferingid":"a1bb6667-780c-47ad-91cb-b98aed589c18","diskofferingname":"Custom","diskofferingdisplaytext":"Custom Disk","destroyed":false,"isextractable":true,"status":"1% Uploaded","tags":[],"displayvolume":true,"quiescevm":false} ] } }

Template status check request

http://managementip:8080/client/api?command=listTemplates&response=json&templatefilter=self&id=DD0A9FC6-C17E-4180-963C-870B9D03A80A&apiKey=miVr6X7u6bN_sdahOBpjNejPgEsT35eXq-jB8CG20YI3yaxXcgpyuaIRmFI_EJTVwZ0nUkkJbPmY3y2bciKwFQ&signature=Lxx1DM40AjcXU%2FcaiK8RAP0O1hU%3D

Template status response

{ "listtemplatesresponse" : { "count":1 ,"template" : [  {"id":"DD0A9FC6-C17E-4180-963C-870B9D03A80A","name":"centos 64 64bit","displaytext":"centos 64 64bit","ispublic":false,"created":"2014-10-17T10:54:40+0530","isready":false,"passwordenabled":false,"format":"tar","isfeatured":false,"crossZones":false,"ostypeid":"1b510c30-3352-11e4-aaca-a5c7f57670d0","ostypename":"CentOS 6.2 (64-bit)","account":"admin","zoneid":"c2bcad2f-1eb1-45ba-bebe-1c21873831b7","zonename":"zone","status":"8% Downloaded","templatetype":"USER","hypervisor":"XenServer","domain":"ROOT","domainid":"fad51d88-449b-11e4-9cf2-2d5524b62490","isextractable":true,"details":{"hypervisortoolsversion":"xenserver61"},"tags":[],"sshkeyenabled":false,"isdynamicallyscalable":false} ] } }

Failed uploads

If the template/volume fails to upload (HTTP code other than 200 OK), the user should request for a new upload url and upload it again. The same upload URL shouldn't be reused.

Design

Pre-shared key management

This section covers when and how the pre-shared key is generated by management server, how is it stored, how is it passed to SSVM

  1. On start of management server, if the key doesn't exist, it auto-generates one using SHA1 and stores it in db
  2. Key is saved in encrypted form (if db encryption is enabled) in the configuration table as hidden configuration with name upload.post.secret.key
  3. During SSVM start-up, the key is passed on to it by management server 
  4. SSVM stores this key in a file at /etc/cloudstack/agent/ms-psk
  5. If the key sharing fails SSVM agent would shut itself down
  6. Updating the PSK would involve the following:
    1. Delete the upload.secret.key config from configuration table in db (manual step)
    2. Restart the MS, so that it gets regenerated again (manual step)
    3. Destroy all the existing SSVMs (manual step)
    4. When the new SSVMs starts up the key is copied to it
    5. There will be a window when the key is not in sync between management server and SSVMs

Signature generation and verification

The signature returned from the management server in the response of getUploadParams will be validated on the SSVM Agent.

  1. The signature is generated using SHA1 and the PSK on the metadata, expires and postURL
  2. Management server and SSVM machines should be in time sync for the verification to work
  3. On the SSVM agent, it first checks if the expires has crossed
  4. It computes hash on metadata, expires and postURL using SHA1 and PSK and rejects the request if they do not match.
  5. The postURL, expires and metadata returned from getUploadParamsForVolume/Template shouldn't be tampered by the user. Else, signature validation would fail.

Encryption and decryption of 'metadata'

The PSK will also be used to encrypt and decrypt the metadata sent from the management server in the api response and received on the SSVM agent in the api request.

  1. jasypt library with PBEWITHSHA1ANDDESEDE algorithm and the initial PSK will be used to encrypt and decrypt the metadata.
  2. Management server encrypts the data using the method in #1 and returns the encrypted string in the response
  3. SSVM Agent, after receiving the metadata, decrypts it using the method in #1 and then does signature validation.

Unencrypted metadata

The unencrypted metadata is a JSON string having all the necessary information to store the uploaded volume/template to secondary store. For a template it looks something like below.

{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"},"hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207","format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"}

Sequence Diagram

Image Added

Management server and SSVM agent interaction

When the management server receives getUploadParams api call, it selects SSVM and secondary store in the zone requested by the user and returns a POST url to that SSVM. There can be multiple SSVMs in the same zone. Management servers decides which SSVM to handle this based on the load.

SSVM agent will keep track of all upload status assigned to it and sends it to management server (based on management server initiated polling). If the template/volume is selected to be cross-zone, it will uploaded to one zone and then copied to other zones from it (already existing functionality).

Management server owns the lifecycle of volume/template (initially gets created with upload state as "NOT_UPLOADED"). It polls the SSVM agent (using an already existing agent command) to get the status and triggers appropriate state transitions. Polling starts after the successful completion of the getUploadParams API. The polling 'interval' and 'timeout' are determined by global configuration parameters.

Failure at any point during the upload process to SSVM would result in upload state getting transitioned to "UPLOAD_ERROR" (refer to the state machine below). Also if the upload is not completed within 'timeout' configuration the state transitions to "UPLOAD_ERROR". This will take care of SSVM agent failure/SSVM failure scenarios.

If the management server itself goes down during the upload, the polling is resumed by the new management server that takes ownership of the SSVM. The 'timeout' is always taken in account from the time template entry gets created in db.

SSVM selection logic

TBD - check existing code and reuse

SSVM agent and SSVM apache interaction

The SSVM agent will listen for any connections from Apache web server (using netty server). It will be using the localhost and port number (8210) to listen.

SSVM apache proxies the data from https to http url on localhost:port using ProxyPass.

All the http events and data will be proxied by apache to agent. SSL validation only happens at apache layer.

Upload lifecycle

Upload failures

The below failures can happen while SSVM agent is receiving the template. When management server polls for the status of the template/volume, if any of these below errors happen, the agent returns appropriate information and management server transitions the state accordingly.

  • pre-validation errors (file upload hasn't started yet)
    • SSL handshake failed
    • params validation with PSK failed
    • max file size from the header crossed the limit
  • post validation error 
    • file successfully uploaded
      • file uploaded is not of the type specified (for ex: if the template is said to be of type tar, the actual downloaded file is not if this type)
      • max upload limit crossed
    • partial file upload (http errors can be caught through appropriate event handlers: exception,timeout,closed)
      • user stopped/canceled the post request before completion
      • uploading the template failed due to a network issue
      • browser request timed out
      • browser crashed
      • SSVM is down/restarted
      • SSVM apache down/restarted
      • secondary storage is not accessible
      • there isn't enough space in secondary storage and hence agent can't write to it

Upload success

The process is considered success If the volume/template upload is successfully completes (and installed in case of template) within the polling timeout. In this case agent returns a success status to management server during the polling.

State machine 

Template/Volume go through below state machine transitions, when it goes through the browser based upload.

Image Added

States:

NOT_UPLOADED: getUploadParams API was successful and the template is registered. But, upload hasn't started yet (agent has no data about this upload)

UPLOAD_IN_PROGRESS: User has initiated the POST request on SSVM Apache and the validation/upload is in progress

UPLOADED: SSVM agent successfully copied the template/volume to secondary storage

UPLOAD_ERROR: Any failure in between user initiating the POST request and template being downloaded OR failure to complete within configured timeout

ABANDONED: the POST request isn't initiated within the configured timeout. In this case the SSVM agent has no record of the upload (TBD: Do we really need to distinguish ERROR and ABANDONED?)

State Transitions:

  • Transition1: This state transition happens, when the user initiates the post request
  • Transition2: Upload is in progress
  • Transition3: On successful completion upload/copy (and installation in case of template) to secondary storage
  • Transition4: This can happen in multiple ways outlined in the failures section above. All these errors will result in the same state for the template/volume as there isn't any way to recover from them or the handling isn't different for these. Additional error information will be provided in the form of a message for respective errors in the agent logs.
  • Transition5: The POST request hasn't been initiated and the timeout has reached

Cleanup

A cleanup thread will be running at regular intervals (configurable, provide details). It will pick up all volume/template with upload state as "UPLOAD_ERROR" and "ABANDONED" and send agent command to cleanup any partial data from secondary store. The cleanup will be a best-effort approach.

Recovery mechanisms

There isn't any recovery or retry mechanism as this is a POST request. Once error happens user gets notified with a clear error message as part of the response. The template/volume will remain in the error state and admin will be able to troubleshoot it based on the appropriate log messages in management server log, agent log, apache access/error log files. This failed entries will eventually get be cleaned by the cleanup process. The user has to reinitiate the upload by calling getUploadParams API again.

One time POST url and parallel uploads

The POST url returned by the management server to upload template/volume is expected to be used only once. If there is a download in-progress/error/success state for template/volume in the URL and the SSVM agent gets a new request for the same URL, the agent rejects this request saying there is another download in progress (409 conflict).

Open item: The limitation here is that if the template/volume cleanup happened and the template is cleaned, agent would start accepting the url again. This will be fixed later. 

DB Changes

No schema changes. Some configuration items will be created.

Configurations

The following new configurations will be added

  • Global configuration to limit the max upload size of template - max.template.iso.size (existing)

  • Global configuration to limit the max upload size of volume - storage.max.volume.upload.size (existing)

  • hidden configuration to save sha1 PSK on the management server - upload.post.secret.key
  • management server upload polling interval and polling timeout - upload.monitoring.interval, upload.operation.timeout
  • time interval at which the GC or cleanup thread should run - upload.post.gc.interval

Security

  • The NFS Store url, path and some other information are exposed to the end user in the response to getUploadParams. This is done so that agent gets all the information required to upload the template with the POST request. The entire JSON will be encrypted using PSK and then sent across as part of getUploadParams response
  • The Apache web server will only accept upload requests over HTTPS
  • Proxy between apache and agent is internal to the SSVM. Hence no security threat here.
  • SSVM PSK location can only be accessed by admin. Hence no security threat.

Performance

  • To increase the number of parallel uploads, multiple SSVMs needs to be created. This will be handled automatically by the management server using the SSVM scale up mechanism (system.vm.auto.reserve.capacity, secstorage.capacity.standby and secstorage.session.max configuration variables). More uploads for a SSVM results in more polling related agent commands going to the SSVM and that should exhaust the session.max limit and result in spawning new SSVM. The SSVM selector logic then should pick up an SSVM during the request.
  • TBD: number of parallel uploads a single SSVM can handle

Troubleshooting/logs

On the SSVM all the upload server related logs will be from a thread with name format nioEventLoopGroup-%d-%d.

On the Management Server, the upload monitor related logs will be from a thread with name format Upload-Monitor-%1

log4j xml can be tuned both on management server and SSVM to get logs at different levels.

UI Changes

UI to upload volume/template which should be able to accept "browse and upload" file. Existing UI for volume/template upload will be extended with the browser-based upload functionality.

UI flow for Templates:

  1. User navigates to the Templates page then clicks on "Register template" (existing screen)
  2. Register template form pops up (existing screen)
  3. User has now the choice to provide URL of the template (existing functionality) or Upload File from Browser (new functionality)
  4. Selects File Upload from Browser then specifies the template file (Browse...) to be uploaded
  5. User fills in other existing form fields
  6. User clicks OK to submit the form
  7. Browser UI requests an unique upload URL using the getUploadParamsForTemplate api call
  8. Management server returns an upload url, encrypted metadata string which should be passed on to the upload url while doing the upload
  9. Browser starts uploading using POST request to the upload URL
  10. The post request completes once the template is uploaded or there was an error uploading.
  11. The user can close the dialog (POST request should still be active) and use list templates API result to check the status of it

 

UI flow for Volume/ISO:

  1. User navigates to the Templates page then switches to ISO using the "Select view" selector, then the user clicks on "Register ISO" (existing screen)
  2. Register ISO form pops up (existing screen)
  3. User has now the choice to provide URL of the volume/ISO (existing functionality) or Upload File from Browser (new functionality)
  4. Selects File Upload from Browser then specifies the volume/ISO file (Browse...) to be uploaded
  5. User fills in other existing form fields
  6. User clicks OK to submit the form
  7. Browser UI requests an unique upload URL using the getUploadParamsForVolume api call
  8. Management server returns an upload url, encrypted metadata string which should be passed on to the upload url while doing the upload
  9. Browser starts uploading using POST request to the upload URL
  10. The post request completes once the template is uploaded or there was an error uploading.
  11. The user can close the dialog (POST request should still be active) and use listvolumes API result to check the status of it

Testing

Automation/marvin cases

https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=commit;h=db7964fb11842ef008464e42e715838cbe46e72d

Manual Test Cases

Browser-based Template / Volume upload Test Plan

Open Issues

  1. PSK length - for now it can be a fixed length, later on it can be made configurable.

  2. Upload url should be active only for one call - Its partially handled now. The limitation needs to be addressed.

  3. Recommended configuration of SSVM and apache web server

Reference

  1. Procedure to Replace realhostip.com with Your Own Domain Name
  2. Implementation details and troubleshooting - uploading custom domain certificate instead of using realhostip.com

Bug Reference & Branch

CLOUDSTACK-7924

volume-upload branch off master

https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=log;h=refs/heads/volume-uploadtemplate-upload branch off master