...

• feature shall support upload of template through https POST request
• feature shall support upload of volume through https POST request
• uploading same volume or template shall be treated as new upload by management server
• failure to upload of volume or template is irrecoverable
• management server shall ensure cleanup of partially uploaded volume or template in case of failure
• management server shall be able to give status (percentage, bytes etc) of the upload
• a new global param to limit the max upload size
• management server will use the current resource limits for templates/volumes and would error on reaching the limits. 
• This is hypervisor agnostic and should work for all hypervisors 
• SSVM apache will be able to handle parallel upload and will be limited by the number of connections in the apache settings

Assumptions & Out-of-Scope

...

1. DownloadProgressCommand will be used the check the download progress (from Management server to Agent)
2. DownloadCommand will be used to start the download(SSVM Apache to SSVM Agent)

...

fault scenarios & cleanup

This section shall describe how the request to upload a volume is handled by management servermanagement server shall detect a failed volume/template upload.

• how does the management server decide which SSVM to pick?
  • SSVM in the zone will be selected
• how does management server decide which NFS server to store?
  • TBD
• if there is enough load on the SSVM, will management server try launch new SSVM
  • YES. This will be handled automatically by the management server using the system.vm.auto.reserve.capacity, secstorage.capacity.standby and secstorage.session.max configuration variables.
• how does the admin see the outstanding browser based uploads?
  • only through the logs
• how does the user know the state of browser based upload?
  • user should be able to call the listtemplates/listvolumes api to get the upload status of it.
• how does the management server know download complete?
  • by sending DownloadProgressCommand to the SSVM agent
• how does the management server know the % of download
  • it is in the response of DownloadProgressCommand
• how does the management server know the failed upload
  • it is in the response of DownloadProgressCommand
• where does the management server persist the state & percentage of upload
  • in the db(template_store_ref)
• How does the agent return template status for DownloadProgressCommand? 
  • in memory hashmap of in progress items. during a restart or crash, its starts the wget download again.
• multiple management servers with the one monitoring the download crashed (How is the template status updated? A. when the download hasnt started yet. B. upload in progress C. upload done) 
  • Template status wont be updated unless the SSVM is restarted and is connected to new management server
• cross zone template upload - file to be uploaded to one zone and then copied to all the other zones
• system vm template upload - same as the current. will be differentiated using the filename.

• No. of parallel uploads SSVM can handle? - a default value should be provided to tune the apache to handle those many parallel connections. These should be determined based on the CPU RAM of SSVM - a proper error message to the user on reaching this limit

fault scenarios & recovering failed uploads 

This section shall describe how the management server shall detect a failed volume/template upload.

• how does the management server detect failed tasks
• when will it take action
• what it will perform to cleanup the storage.

management server and SSVM agent interaction

• detect failed tasks
• when will it take action
• what it will perform to cleanup the storage.

management server and SSVM agent interaction

When the management server receives getUploadParams api call, it selects SSVM and secondary store in the zone requested by the user and returns a POST url to that SSVM. There can be multiple SSVMs in the same zone. Management servers decides which SSVM to handle this based on the load. SSVM agent will be able to return the status of the upload at any point using the in memory hash map if the download is in progress or by the file status on the secondary storage. If the template/volume is selected to be cross-zone, it will uploaded to one zone and then copied to other zones from it. 

management server management server owns the state machine of the volume/template. It polls the SSVM agent(using DownloadProgressCommand of the agent) to get the status and updates it. Failure at any point during the form upload to ssvm apache would return error to the user and UPLOAD_ERROR when the management server pings for download status.

...

If the management server itself goes down during the upload, if the timeout hasnt crossed, it will poll for the status once its up. Otherwise the state of the template will be transitioned to ERROR. In a load balanced situation where the management server handling the ssvm upload goes down, the status wont be updated unless the SSVM is connected to a new management server.

status update

detecting failures

...