...

SSVM agent and SSVM apache interaction

TBD

Upload failures

The below failures can happen while SSVM agent is receiving the template. When management server polls for the status of the template/volume, if any of these below errors happen, the agent returns appropriate information and management server updates the state accordingly.

• pre-validation errors (file upload hasn't started yet)
  • SSL handshake failed
  • params validation with PSK failed
  • max file size from the header crossed the limit
• post validation error 
  • file successfully uploaded
    • file uploaded is not of the type specified (for ex: if the template is said to be of type tar, the actual downloaded file is not if this type)
    • max upload limit crossed
  • partial file upload (http errors can be caught through appropriate event handlers: exception,timeout,closed (http://hc.apache.org/httpcomponents-core-ga/tutorial/html/nio.html))
    • user stopped/canceled the post request before completion
    • uploading the template failed due to a network issue
    • browser request timed out
    • browser crashed
    • SSVM is down/restarted
    • SSVM apache down/restarted
    • secondary storage is not accessible
    • there isn't enough space in secondary storage and hence agent can't write to it

Upload successful

...

If the volume/template install is successful upload is successfully completed (and installed in case of template) within the polling timeout, when the management server asks for the status of it, . In this agent returns a success state and it is updated by the to management server accordinglyduring the status poll.

State machine 

Template/Volume go through below state machine state transitions, when it goes through the browser based upload.

...

NOT_UPLOADED: getUploadParams API was successful and the template is registered. But, upload hasnt hasn't started yet

UPLOAD_IN_PROGRESS: User has initiated the POST request on SSVM Apache and the validation/download upload is in progress

UPLOADED: SSVM Agent agent successfully Downloaded copied the template to secondary storage

UPLOAD_ERROR: Any Failure failure in between user initiating a post the POST request and template being downloaded OR failure to complete within configured timeout

ABANDONED: the post POST request isnt isn't called in the stipulated time (check sequence diagram point 4). In this case the SSVM agent has not record of the upload (TBD: Do we really need to distinguish ERROR and ABANDONED?)

State Transitions:

• Transition1: This state transition happens, when the user initiates the post request
• Transition2: Post request is active SSVM agent is downloading validation or downloading the template but, not done yet Upload is in progress
• Transition3: On successful completion upload/copy (and installation in case of template) Transition3: on successful completion of template installation to secondary storage
• Transition4: This can happen in multiple ways outlined in the failures section above. All these errors will result in the same state for the template/volume as there

...

• isn't any way to recover from them or the handling

...

• isn't different for these. Additional error information will be provided in the form of a message for respective errors in the agent logs.

...

• Transition5: The post POST request hasn't happened been initiated and the timeout has reached

Cleanup

Of the error categories defined in the failures above,

• pre validation errors will not result in any download and hence no cleanup is required.
• post validation errors, template/volume will be cleaned
• http errors shall be caught in the appropriate events and the partially downloaded file will be cleaned.
• application errors and interruptions: a new thread UploadToStorageGarbageCollector will be run periodically(based on a global configuration) on the management server. It will look at any template/volume which are not deleted and in UPLOAD_ERROR/ABANDONED state and send a DeleteCommand to the Agent for the same. It will also update the status of the template/volume to be deleted.

recovery mechanisms

A GC thread will be running at regular intervals (configurable, provide details). It will pick up all volume/template with upload state as "UPLOAD_ERROR" and "ABANDONED" and send agent command to cleanup any partial data from secondary store.

Recovery mechanisms

There isn't There isnt any recovery or retry mechanism as this is a POST request. Once errorederror happens, template/volume will remain in the error state and admin will be able to troubleshoot it based on the appropriate log messages in management server log, agent log, apache access/error log files. This will be cleaned by the garbage collector eventually. The user has to reinitiate the upload by calling getUploadParams and getting a new POST urlagain.

...

One time POST url and parallel uploads

The POST url returned by the management server to upload the file template/volume is expected to be used only once. If there is a download in-progress/error/success state for template/volume in the URL and the SSVM agent gets a new request for the same URL, the agent rejects this request saying there is another download in progress (409 conflict).

Open item: The limitation here is that if the template/volume cleanup happened and the template is cleaned, agent would start accepting the url again. This will be fixed later. 

...

A new API getUploadParams which takes all the params as the registertemplate/uploadvolume except for URL and  which which returns all the required params plus a post URL to be used to uploadVolumeupload volume/registerTemplatetemplate

• Request 
  • type: volume or template : required
  • params while uploading a volume
    • format: the format for the volume. Possible values include QCOW2, OVA, and VHD : required
    • name: the name of the volume : required
    • zoneid: the ID of the zone the volume is to be hosted on : required
    • account: an optional accountName. Must be used with domainId.
    • checksum: the MD5 checksum value of this volume
    • domainid: an optional domainId. If the account parameter is used, domainId must also be used.
    • imagestoreuuid: Image store uuid
    • projectid: Upload volume for the project
  • params while uploading a tempalte
    • displaytext: the display text of the template. This is usually used for display purposes: required
    • format: the format for the template. Possible values include QCOW2, RAW, and VHD : required
    • hypervisor: the target hypervisor for the template : required
    • name: the name of the template : required
    • ostypeid: the ID of the OS Type that best represents the OS of this template : required
    • zoneid: the ID of the zone the template is to be hosted on : required
    • account: an optional accountName. Must be used with domainId.
    • bits: 32 or 64 bits support. 64 by default
    • checksum: the MD5 checksum value of this template
    • details: Template details in key/value pairs.
    • domainid: an optional domainId. If the account parameter is used, domainId must also be used.
    • isdynamicallyscalable: true if template contains XS/VMWare tools inorder to support dynamic scaling of VM cpu/memory
    • isextractable: true if the template or its derivatives are extractable; default is false
    • isfeatured: true if this template is a featured template, false otherwise
    • ispublic: true if the template is available to all accounts; default is true
    • isrouting: true if the template type is routing i.e., if template is used to deploy router
    • passwordenabled: true if the template supports the password reset feature; default is false
    • projectid: Register template for the project
    • requireshvm: true if this template requires HVM
    • sshkeyenabled: true if the template supports the sshkey upload feature; default is false
    • templatetag: the tag for this template.

• Response
  • postURL: post url to upload the file to. Ex: "https://ssvmpublicip/upload/uuid/signature" (signature in the postURL is SHA1 key generated using PSK initially shared between SSVM agent and Management Server and the params in the response.)the params in the response.) The URL also has the uuid to query the status using listTemplate/Volumes API subsequently
  • params: params to be sent in the post request in JSON format Ex: params:{"store":{"com.cloud.agent.api.to.NfsTO":"nfs://10.147.28.7/export/home/rajani/lxc/secondary"}, "hvm":false,"description":"lxc1","maxDownloadSizeInBytes":53687091200,"id":207,"resourceType":"TEMPLATE","installPath":"template/tmpl/2/207", "format":"TAR","accountId":2,"name":"207-2-09c78adc-065f-3dc8-9621-9e7a036d7dc9"}

...

• Request: (this is a POST request)
  • params: json JSON string of params returned from previous getUploadParams api call file: file to upload
  • file: the location of the template/volume to be posted a multipart/form-data
• Response: 
  • '200 OK' on successful upload
  • '401 Authorization authorization error' incase the signature key validation or any other validation failed
  • '500 internal server error' incase the file upload fails
  • '409 ' conflict' incase there is an already another upload with same params in-progress/success/error

At any point, the api user should be able to call the listtemplates/listvolumes api to get the installation status of the template.

If the template failed to upload(401 or 500 above), the user should request for a new upload url and upload it again. He shouldn't be reusing the same upload URL again.

...

• The NFS Store url, path and some other information are exposed to the end user in the response to getUploadParams. This is done so that agent gets all the information required to download the template with the POST request. One option can be to encrypt the entire JSON and then send it across as part of response
• Proxy proxy between apache and agent is internal to the SSVM. Hence , no security threat here.
• SSVM PSK location can only be accessed by admin. Hence , no security threat.

...

Performance

• To increase the number of parallel uploads, multiple SSVMs needs to be created. This will be handled automatically by the management server using the SSVM scale up mechanism (system.vm.auto.reserve.capacity, secstorage.capacity.standby and secstorage.session.max configuration variables).
• TBD: number of parallel uploads a single SSVM can handle

...