Page History

• Scenario 1: Connection failures between the agent and the management server

Agent: (for reconcile command only)

1. Save command info as JSON file
2. Update command state in JSON when start/process/complete the command
   1. STARTED
   2. PROCESSING
   3. COMPLETE/FAILED
3. Save Answer in the JSON file
4. Every minute
   1. Load command/answer from JSON files
   2. Send with with PingCommand
   3. Receive the PingAnswer from management server
   4. Remove the JSON file if state is COMPLETE/FAILED

Management server: (for reconcile command only)

1. When create the reconcile command, insert a record with state CREATED into  reconcile_commands table
2. When receive the PingCommand, Update command state and answer in reconcile_commands table
3. When wait for the answer of command (reconcile command only)
   1. Every 10 seconds, check answer of the reconcile_commands table
   2. If answer is found, parse the answer
   3. Returna and continue with the answer
   4. No need to wait until timeout, if there are connection failures. 
4. if timed out, update command state to TIMED_OUT in reconcile_commands table

• Scenario 2: Agent crash (or force killing the agent process)

Management server

1. When Host Status is determined as Down
2. Update reconcile commands (to the agent) in reconcile_command table
   1. to INTERRUPTED state
3. Reconcile the command
   1. see "3.6 Reconcile the command"

• Scenario 3: Agent restart

Agent (when restart)

1. When stop the agent, updates state of processes
   1. PROCESSING to INTERRUPTED
   2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
2. When start the agent, updates state of processes 
   1. PROCESSING to INTERRUPTED
   2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
3. Agent send CommandInfo to management server via PingCommand every minute
   1. with new state
4. Management server reconcile commands every minute (from reconcile_commands table) , for commands in state

• Scenario 4: Agent has completed but timed out

Management server

1. Update state_by_management to TIMED_OUT
2. Update state_by_agent to COMPLETED (from PROCESSING)
3. Reconcile the command
   1. see "3.6 Reconcile the command"
4. TODO: what should be the correct resource state ?4

• Scenario 5: Management restart

Agent process the command

• has completed and send Answer to management server, but no action on management server
• has not completed, and is processing command

Management server

1. Update state_by_management to INTERRUPTED when mgmt server is stopped.
2. When another management server is detected DOWN
   1. update reconcile_command to INTERRUPTED state by management server

• iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP
  • iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP
• intermittent failure
  • agent updates answer when connection is back to normal
  • mgmt server gets the answer from DB

• • every 10 seconds
  • mgmt server processes the answer if found

• continuous failure
  • agent

• • • Migration thread of VM [i-2-1198-VM] finished.
  •  management
    • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • Behavior:
    • (new) check via destination host (if dest is Up)
      • if vm is Running on destination, consider the vm migration is successful.
  • • Active migration command so scheduling a restart
    • vm is Stopped on destination and Running on source (actually it is not)

• kill -9  $(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}')
•  agent
  • agent is started again by systemctl

• management server

  • Resource [Host:

• • 1] is unreachable: Host 1: Operation timed out on migrating VM instance

  • VM is Running on source host
• Note:

• • In edge case, if the vm is migrated before agent is restarted (10 seconds) , the vm is Stopped (by ACS on destination host)

• systemctl restart cloudstack-agent
• agent 
  • abort the migration job by disconnect hook
  • written by Marcus
• management server
  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • VM is Running on source host
• Note:
  • In edge case, the vm is migrated before aborting the job, the vm is Stopped (by ACS on destination host)

If VM is Migrating (mgmt server is restarted)

• check via source host (if source is Up)
• check via destination host (if dest is Up)
• determine the state and update