Introduction

Purpose

With the current code for snapshots, cloudStack always creates snapshot on the host where vm is Running (for vms in Running state) or on the host where vm used to run the last time (for vms in Stopped state). As the commands are not synchronized on the agent side, the case when multiple commands are send to the backend at the same time can lead to the performance issues on the hypervisor side.  At the end there is a high possibility that createSnapshot command might time out on the Xen side_._

The solution is to synchronize number of concurrent snapshots per host basis. The threshold should be configurable as the customer usually knows how many snapshots at a time the backend can handle. While the concurrent snapshots are being processed by the backend, all subsequent snapshot commands scheduled for execution on the same host, should wait in the queue. And the wait timeout should also be a configurable value.

References

• Byron Requirement: snapshots improvements

Feature Specifications

• Requirements- introduce synchronization algorithm for createSnapshot calls. Number of synchronized threads as well as the wait timeout for threads waiting for execution, should be configurable vial cloudStack global config parameter
• User permissions - only admin can set global configuration parameters.

Use cases

Admin knows that his backend host can process 10 snapshots at a time with the current vms load. So he sets the global configuration concurrent.snapshots.threshold.perhost to 10 and configures the desired expiration time for the create snapshot job to wait for execution.

Architecture and Design description

• We are going to re-use existing Async Job management logic with a couple of new additions for createSnapshot scenario. Other async jobs won't get affected.
• createSnapshot API now going to be synchronized on Host object. The host object would be determined even before the command is scheduled for execution - using cloud.host table information (host_id field for snapshots created for vms in Running state, and last_host_id for vms in Stopped state)

Configuration parameters changes.

• 1 new global config parameters are added: concurrent.snapshots.threshold.perhost (default is NULL - unlimited)

DB changes

sync_queue table

•  new fields queue_size and queue_size_limit are added  (first determines how many jobs are being processed at the moment, second - the max number of jobs that can be processed at the same time). The default value for queue_size is 0. Field queue_size_limit is 1 for  all jobs other than createSnapshot. For createSnapshot its determined by concurrent.snapshots.threshold.perhost.
• removed field queue_proc_msid. This field was never used in the code, and as now the same queue can have items processed by different management servers, we always refer to sync_queue_item.queue_proc_msid when need to figure out which management server is processing the job.
• removed queue_proc_time field. As before we could only process one queue item at at time, and after the change it can be more than one, this field was moved to sync_queue_item table.

...

• new filed - queue_proc_time (got moved from sync_queue_table).

Code flow

Once the createSnapshot operation is requested either by the user via API, or by cloudStack snapshot scheduler, the code:

...

4) If the createSnapshot job waits for more than job.expire.minutes for execution, then it gets expired and failure is returned to the API caller.

Web services APIs

No new APIs are added. Existing createSnapshot API now going to be synchronized on the Host object, but no new parameters are exposed to the end user.

UI flow

No UI changes for the feature.