• Created by Unknown User (chiradeep), last updated by user-fce10 on Sep 24, 2014 7 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 13 Next »

Bug Reference

CLOUDSTACK-....

Branch

template-upload branch off master

Introduction

To register a template/upload volume, one needs to host it some where as cloudstack only accepts a url. This eliminates that and enables the user to quickly upload a template/volume to cloudstack.

use cases

  1. Anyone who wants to quickly upload a template/volume and test cloudstack without having the need to host it.

Design

  1. Browser based upload
    1. Management Server generates a unique pre-shared key (PSK1) on install and shares it with the SSVM agent. Nobody else has access to this key of course.
    2. Browser UI requests the upload form
    3. MS sends down a form with standard fields pre-populated such as the upload url, url expiry time, account uuid, content-type AND a signature calculated over these fields. The signature is calculated using the pre-shared key PSK1. The user can be asked for inputs such as the filename on her local computer, format, hypervisor and the MD5 sum of the file. The form action is to POST to the SSVM's public IP
    4. The browser POSTS the form as enctype="multipart/form-data" over HTTPS to the SSVM's public IP

    5. The Apache web server on the SSVM matches the url, decrypts and passes on to the SSVM java agent as plain HTTP. The SSVM agent checks the signature against the form fields using the key PSK1. Once the fields are verified, it writes the  to the NFS storage to the location encoded in the URL. 

    6. Once the upload completes, the MD5 checksum is compared (if available from step c). If it matches, the agent informs the MS that the upload is successful and the template is registered. The MS returns the template identifier to the agent and the agent returns this id
  2. API-based upload
    1. Preshared key as before
    2. New API request "GetUploadParams" encoded the usual CloudStack API way using api keys. Request parameters include format, hypervisor, md5 sum
    3. MS response to (b) is a json document which includes an inner json structure detailing the form parameters as in (c) above. Outer json contains the upload URL and the pre-calculated signature (using PSK1) over the inner json.
    4. API user makes a PUT request over HTTPS to the URL obtained in (c)
    5. as before
    6. as before.

Sequence Diagram

 

  1. failure at any point during the form upload to ssvm apache would return error to the user and DOWNLOAD_ERROR when the management server pings for download status.
  2. polling starts after the success of GetUploadParams
  3. current polling interval is 10000 ms 
  4. currently it polls thrice to check the status. We may need to increase this. (is the current value 5 or 3?)

Agent Commands

  1. DownloadProgressCommand will be used the check the download progress (from Management server to Agent)
  2. DownloadCommand will be used to start the download(SSVM Apache to SSVM Agent)

State Diagram

Template/Volume states

States:

NOT_DOWNLOADED: GetUploadParams API was successful and the template is registered. But, upload hasnt started yet

DOWNLOAD_IN_PROGRESS: User has initiated the POST request SSVM Apache and the validation/download is in progress

DOWNLOADED: SSVM Agent successfully Downloaded the template to secondary storage

DOWNLOAD_ERROR: Any Failure in between user initiating a post request and template being downloaded

ABANDONED: the post request isnt called in the stipulated time (check sequence diagram point 4)

State Transitions:
  • 1: This state transition happens, when the user initiates the post request
  • 2: Post request is active SSVM agent is downloading validation or downloading the template but, not done yet
  • 3: on successful completion of template installation to secondary storage
  • 4: This can happen in multiple ways:
    • user stopped/canceled the post request before completion
    • SSL handshake failed
    • params validation with PSK failed
    • uploading the template failed due to a network issue
    • secondary storage is not accessible
    • there isn't enough space in secondary storage and hence agent cant write to it.
      All these errors have the same state as there isnt any way to recover from them or the handling isnt different for these. Additional error information will be provided in the form of a message for respective errors.
  • 5: The post request hasn't happened and the timeout has reached

API changes

  1. new API GetUploadParams which takes all the params as the registertemplate/uploadvolume except for URL and  which returns all the required params plus a post URL to be used to uploadVolume/registerTemplate

    1. volume upload api

      Parameter NameDescriptionRequired
      typevolume or template uploadtrue
      formatthe format for the volume. Possible values include QCOW2, OVA, and VHD.true
      namethe name of the volumetrue
      zoneidthe ID of the zone the volume is to be hosted ontrue
      accountan optional accountName. Must be used with domainId.false
      checksumthe MD5 checksum value of this volumefalse
      domainidan optional domainId. If the account parameter is used, domainId must also be used.false
      imagestoreuuidImage store uuidfalse
      projectidUpload volume for the projectfalse
    2. template upload api

       

      Parameter NameDescriptionRequired
      typevolume or template uploadtrue
      displaytextthe display text of the template. This is usually used for display purposes.true
      formatthe format for the template. Possible values include QCOW2, RAW, and VHD.true
      hypervisorthe target hypervisor for the templatetrue
      namethe name of the templatetrue
      ostypeidthe ID of the OS Type that best represents the OS of this template.true
      zoneidthe ID of the zone the template is to be hosted ontrue
      accountan optional accountName. Must be used with domainId.false
      bits32 or 64 bits support. 64 by defaultfalse
      checksumthe MD5 checksum value of this templatefalse
      detailsTemplate details in key/value pairs.false
      domainidan optional domainId. If the account parameter is used, domainId must also be used.false
      isdynamicallyscalabletrue if template contains XS/VMWare tools inorder to support dynamic scaling of VM cpu/memoryfalse
      isextractabletrue if the template or its derivatives are extractable; default is falsefalse
      isfeaturedtrue if this template is a featured template, false otherwisefalse
      ispublictrue if the template is available to all accounts; default is truefalse
      isroutingtrue if the template type is routing i.e., if template is used to deploy routerfalse
      passwordenabledtrue if the template supports the password reset feature; default is falsefalse
      projectidRegister template for the projectfalse
      requireshvmtrue if this template requires HVMfalse
      sshkeyenabledtrue if the template supports the sshkey upload feature; default is falsefalse
      templatetagthe tag for this template.false

       

      A sample request and response would look below
      request params: 

      name=lxc1
      displayText=lxc1
      zoneid=-1
      format=TAR
      isextractable=false
      passwordEnabled=false
      isdynamicallyscalable=false
      osTypeId=1b510c30-3352-11e4-aaca-a5c7f57670d0
      hypervisor=LXC
      requireshvm=false


      response params: { "postURL":"https://ssvmpublicip/upload/key",params:{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"}, "hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207", "format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"}}

      key in the postURL would be generated using PSK initially shared between agent and Management Server.
  2. POST url to post the data on to SSVM. url: https://ssvmpublicip/upload/key params: params returned from previous call file: file to upload
    response: 200 OK on successful upload, 401 Authorization error, 500 internal server incase the file upload fails.

 

UI Changes

  1. A new register template and upload volume UI which should be able to accept "browse and upload" file. 

DB Changes

No db changes.

Major Tasks

  1. A new SSVM template is required.
  2. setting up https on SSVM, opening the relevant ports doing the SSL handshake
    1. http://www.symantec.com/connect/articles/apache-2-ssltls-step-step-part-1
  3. initial keystore exchange and authentication using the keystore
  4. understanding template code (smile)

Future

  1. Upload progress to the user (http://creativejs.com/tutorials/advanced-uploading-techniques-part-1/ )
  2. split upload and join in apache so as to not hit browser timeouts

  3. post url should be active only for one time call - return proper error message on subsequent calls - how do we determine if its already used?

  4. Admin to list all the uploads happening and cancel any if required?

  5. Partial data upload and template cleanup (can be due to SSVM to agent restart or network error)

Open Questions

  1. Multipart form data - Can this go in multiple connections?

  2. SSVM storage issue. What should we tell the user? How can we notify the admin to recover from it?

  3. Should there be an unencrypted HTTP version of the post?  No for now

  4. No. of parallel uploads SSVM can handle? a default value should be provided to tune the apache to handle those many parallel connections. These should be determined based on the CPU RAM of SSVM - a proper error message to the user on reaching this limit

Needs current code investigation

check download template/volume code to see how the below scenarios are handled

  1. multiple SSVMs. Which agent should handle the upload? 

  2. cross zone template upload
  3. system vm template upload
  4. default timeouts and configurable values
  5. multiple management servers with the one monitoring the download crashed (How is the template status updated? A. when the download hasnt started yet. B. upload in progress C. upload done)
  6. How does the agent return template status for DownloadProgressCommand? (in memory hashmap of in progress items. need to check of it recovers the status during a restart or when done)
  • No labels