...

            -> RECONCILED (all good) / RECONCILE_FAILED (failed, will retry) / RECONCILED_RETRY (success, but need more information, will retry)

3.7 Summary

...

scenario 1

Connection failures between agent and management server

• iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP
  • iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP

...

scenario 2

Agent crash (or force killing the agent process)

• pid=$(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}') && kill -9 $pid

...

scenario 3

Agent is restarted manually

• systemctl restart cloudstack-agent

...

scenarios 5 (Needed ?)

mgmt server is restarted

• systemctl restart cloudstack-management

...

Migrate VM

• MigrateVMCommand
• MigrateCommand (internal)

intermittent failure

• • agent updates answer when connection is back to normal
  • mgmt server gets the answer from DB every 10 seconds
  • mgmt server processes the answer if found
  • Operation times out

continuous failure

• • agent
    • Migration thread of VM [i-2-1198-VM] finished.
  •  management
    • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • Behavior:
    
    • Active migration command so scheduling a restart
    • Current: vm is Stopped on destination and Running on source (actually it is not if migration is successful)
    • New: vm is Running on destination host if found
    • (new) check via destination host (if dest is Up)
      • if vm is Running on destination, consider the vm migration is successful.

•  agent
  • agent is started again by systemctl

• management server

  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance

  • (new) check via destination host (if dest is Up)
    • if vm is Running on destination, consider the vm migration is successful.
  • Current: VM is Running on source host
  • New: vm is Running on destination if found

• Note:
  • • In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
    • solved

• agent 
  • abort the migration job by disconnect hook
  • written by Marcus

• management server
  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • (new) check via destination host (if dest is Up)
    • if vm is Running on destination, consider the vm migration is successful.
  • Current: VM is Running on source host
  • New: vm is Running on destination if found

• Note:
  • In edge case, the vm is migrated before aborting the job, the vm is Stopped (by ACS on destination host)

If VM is Migrating (TO be verified)

...

• if Running on dest, Running
• If Paused on dest, Migrating
• If Stopped on dest, Running (if found on source) or Stopped (if not found on source)

Migrate VM with volumes

• (between NFS)
• MigrateVirtualMachineWithVolumeCmd
• MigrateCommand (internal)

Note: this action is not supported on PowerFlex

intermittent failure

• • same as above

continuous failure

• • current: vm is Stopped on destination and Running on source  and pool (actually it is not if migration is successful)
  • New: vm is Running on destination host and pool if found on destination host

• management server
  • Failed to migrate VM [VM instance xxx along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) xxx failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Commands 4446460207098233056 to Host 1 timed out after 21600].].
  • Current:
    • vm is Running on source host
    • volume is Ready on source pool
    • new volume is Migrating on destination pool (solved)
  • new
    • check via destination host (if dest is Up)
      • if vm is Running on destination, consider the vm migration is successful. 
    • (after fix) new volume is Destroy on destination pool if fail

• Note:
  • In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)  
  • solved

...

Migrate Volume (on NFS)

• MigrateVolumeCmd (API)

• CopyCommand (from primary1 to sec1)
• CopyCommand (from sec1 to primary2)

ROOT/DATA volume of Stopped VM

intermittent failure

• 1st CopyCommand: wait until connection is recoved
• 2nd CopyCommand: wait until connection is recoved

continuous failure

• 1st CopyCommand: operation timeout
  • Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1159","uuid":"ed322109-8e54-47ca-8183-3c8f2a0f37e6"}] migration failed due to [com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 5302988561228766380 to Host 1 timed out after 1800].
  • new volume is Creating with empty path on destination pool (solved)
• 2nd CopyCommand
  • Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1159","uuid":"ed322109-8e54-47ca-8183-3c8f2a0f37e6"}] migration failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7457960982925541434 to Host 1 timed out after 2400].
  • new volume is Creating with empty path on destination pool  (solved)

New

• new volume is Destroy state with empty path on destination pool

• 1st CopyCommand
  
  • copy object failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 15481123719087037 to Host 1 timed out after 1800
  • new volume is Destroy state
    • expungeVolumeAsync is called in copyVolumeCallBack

• 2nd CopyCommand
  • copy failed com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 398850041998999773 to Host 1 timed out after 2400
  • new volume is Destroy state
    • expungeVolumeAsync is called in copyVolumeCallBack

• ALL looks good

...

Migrate Volume (on Powerflex)

• Running VM
• Stopped VM

...

Current:

TODO

...

Current:

TODO

...

Current:

TODO

Migrate VM with volumes  (If VM is Migrating)

...

Migrate VM

• If VM is Migrating or Running
  • check via source host (if source is Up)
  • check via destination host (if dest is Up)
  • determine the state and update
    • if Running on dest, Running
    • If Paused on dest, Migrating
    • If Stopped on dest, Running (if found on source) or Stopped (if not found on source)
  • Update vm state

Migrate VM with volumes  (If VM is Migrating)

• check VM state and volume states
• Check if there are reconcile commands for the VM
  • If no, check the state on source host (last_host_id) and destination host (host_id)
  • PrepareForMigrationCommand (Update VM state)
  • MigrateCommand
    • check via source host (if source is Up)
    • check via destination host (if dest is Up)
    • determine the state and update

Migrate Volumes  (If volume is Migrating)

• CopyCommand (from primary1 to secondary)
  • skipped.
  • check if there are other Command on same volume ?
• CopyCommand (from secondary to primary2)
  • check if volume exists on primary2 (via the host, other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host)
  • check if volume is changed on primary2
    • if yes, still Copying
    • if no, update state
• CopyCommand (from primary1 to primary2)
  • check if volume exists on primary1
  • check if volume exists on primary2
  • check if volume is changed on primary2
    • if yes, still Copying
    • if no, update state
• MigrateVolumeCommand (from primary1 to primary2)
  • check if volume exists on primary1
  • check if volume exists on primary2
  • check if volume is changed on primary2
    • if yes, still Migrating
    • if no, update state

3.7 Limitations

• Assumption
  • Each request does not have multiple commands with same name
  • for example, 700028267079401857-org.apache.cloudstack.storage.command.CopyCommand
• Only support:
  • 3 commands
    • CopyCommand
    • MigrateCommand
    • MigrateVolumeCommand
  • resources in Migration state (vm, source volume, dest volume)
    • skipped if the resource state is not Migrating
    • skipped if the source or dest host/pool are inconsistent with command
  • Hypervisor: KVM
  • Storage: NFS, Local, Powerflex

3.8 To be Discussed (TODO)

1. How to distribute the reconciiation tasks if there are multiple management servers ?
   1. now: Reconciliation is processed by first management server
2. How to better handle the state DANGLED_IN_BACKEND ?
   1. now: no special action
3. How to better handle the state COMPLETED and FAILED ? Do not reconcile via hosts if state_by_agent to COMPLETED ?
   1. now: no special action

4. Test cases

4.1 Backend commands of some migrations

Please note: 

Migration between NFS and Local requires the fix: https://github.com/apache/cloudstack/pull/10266

4.2 How to test

• agent is restarted
  • systemctl restart cloudstack-agent
• management server is restarted
  • systemctl restart cloudstack-management
• agent and management server communication failure
  • iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP
  • iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP
• Agent crash
  • pid=$(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}')
  • kill -9 $pid
• Agent has completed but timed out

4.3 Summary of test results

...

...

Migrate Volumes  (If volume is Migrating)

• CopyCommand (from primary1 to secondary)
  • skipped.
  • check if there are other Command on same volume ?
• CopyCommand (from secondary to primary2)
  • check if volume exists on primary2 (via the host, other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host)
  • check if volume is changed on primary2
    • if yes, still Copying
    • if no, update state
• CopyCommand (from primary1 to primary2)
  • check if volume exists on primary1
  • check if volume exists on primary2
  • check if volume is changed on primary2
    • if yes, still Copying
    • if no, update state
• MigrateVolumeCommand (from primary1 to primary2)
  • check if volume exists on primary1
  • check if volume exists on primary2
  • check if volume is changed on primary2
    • if yes, still Migrating
    • if no, update state

3.8 Limitations

• Assumption
  • Each request does not have multiple commands with same name
  • for example, 700028267079401857-org.apache.cloudstack.storage.command.CopyCommand
• Only support:
  • 3 commands
    • CopyCommand
    • MigrateCommand
    • MigrateVolumeCommand
  • resources in Migration state (vm, source volume, dest volume)
    • skipped if the resource state is not Migrating
    • skipped if the source or dest host/pool are inconsistent with command
  • Hypervisor: KVM
  • Storage: NFS, Local, Powerflex

3.9 To be Discussed (TODO)

1. How to distribute the reconciiation tasks if there are multiple management servers ?
   1. now: Reconciliation is processed by first management server
2. How to better handle the state DANGLED_IN_BACKEND ?
   1. now: no special action
3. How to better handle the state COMPLETED and FAILED ? Do not reconcile via hosts if state_by_agent to COMPLETED ?
   1. now: no special action

4. Test cases

4.1 Backend commands of some migrations

Please note: 

Migration between NFS and Local requires the fix: https://github.com/apache/cloudstack/pull/10266

...

PrepareForMigrationCommand (dest)

MigrateCommand (source)

...

PrepareForMigrationCommand (dest)

MigrateCommand (source)

...

CopyCommand (template to primary if needed)

CreateObjectCommand (new volume)

ModifyTargetsCommand

PrepareForMigrationCommand (dest)

MigrateCommand (source)

DeleteCommand (source)

...

Migrating a volume online with KVM from managed storage is not currently supported.

...

MigrateVolumeCommand

• Same volume

...

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

...

CopyCommand (primary1 to primary2)

• Different volume IDs

...

MigrateVolumeCommand

• Same volume

...

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

...

CopyCommand (primary1 to primary2)

• Different volume IDs

...

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

...

CopyCommand (primary1 to primary2)

• Different volume IDs

...

4.2 How to test

...

• systemctl restart cloudstack-agent

...

• systemctl restart cloudstack-management

...

• iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP
• iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP

...

• pid=$(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}')
• kill -9 $pid

...