• Scenario 1: Connection failures between the agent and the management server

Agent: (for reconcile command only)

1. Save command info as JSON file
2. Update command state in JSON when start/process/complete the command
   1. STARTED
   2. PROCESSING
   3. COMPLETE/FAILED
3. Save Answer in the JSON file
4. Every minute
   1. Load command/answer from JSON files
   2. Send with with PingCommand
   3. Receive the PingAnswer from management server
   4. Remove the JSON file if state is COMPLETE/FAILED

Management server: (for reconcile command only)

1. When create the reconcile command, insert a record with state CREATED into  reconcile_commands table
2. When receive the PingCommand, Update command state and answer in reconcile_commands table
3. When wait for the answer of command (reconcile command only)
   1. Every 10 seconds, check answer of the reconcile_commands table
   2. If answer is found, parse the answer
   3. Returna and continue with the answer
   4. No need to wait until timeout, if there are connection failures. 
4. if timed out, update command state to TIMED_OUT in reconcile_commands table

• Scenario 2: Agent crash (or force killing the agent process)

Management server (TODO)

1. When Host Status is determined as Down (only cloudstack-agent is DOWN?)
2. Update reconcile commands (to the agent (from ) in reconcile_command table)
   1. to INTERRUPTED state
   2. INTERRUPTED
3. Reconcile the command
   1. Migrate VM: via destination host (TODO)
   2. Migrate Volume: via other host (TODO)
   3. Copy Command: via other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host (TODO)see "3.6 Reconcile the command"
4. Update state to RECONCILED, if command is reconciled and no need more reconciliation

• Scenario 3: Agent restart

Agent (when restart)

1. When stop the agent, updates state of processes  (TODO)
   1. PROCESSING to INTERRUPTED
   2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
2. When start the agent, updates state of processes 
   1. PROCESSING to INTERRUPTED
   2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND

1. Agent send CommandInfo to management server via PingCommand every minute
   1. with new state

1. Management server reconcile commands every minute (from reconcile_command table) , for commands in statein state

• Scenario 4: Agent has completed but timed out

Management server

1. Update state_by_management to TIMED_OUT
2. Update state_by_agent to COMPLETED (from PROCESSING)
3. Reconcile the command
   1. see "3.6 Reconcile the command"

• Scenario 4: Agent has completed but timed out

• Scenario 5: Management restart

Agent process the command

• has completed and send Answer to management server, but no action on management server
• has not completed, and process command

Management server

1. Update state_by_management to INTERRUPTED when mgmt server is stopped.
2. When another management server is detected DOWN, update reconcile_command to INTERRUPTED state ? (TODO)