Page History

• iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP
  • iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP

• intermittent failure
  • agent updates answer when connection is back to normal
  • mgmt server gets the answer from DB every 10 seconds
  • mgmt server processes the answer if found
  • Operation times out

• continuous failure
  • agent
    • Migration thread of VM [i-2-1198-VM] finished.
  •  management
    • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • Behavior:
    
    • Active migration command so scheduling a restart
    • Current: vm is Stopped on destination and Running on source (actually it is not if migration is successful)
    • New: vm is Running on destination host if found
    • (new) check via destination host (if dest is Up)
      • if vm is Running on destination, consider the vm migration is successful.

• kill -9  $(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}')

•  agent
  • agent is started again by systemctl

• management server

  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance

  • (new) check via destination host (if dest is Up)
    • if vm is Running on destination, consider the vm migration is successful.
  • Current: VM is Running on source host
  • New: vm is Running on destination if found

• Note:
  • • In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
    • solved

• systemctl restart cloudstack-agent

• agent 
  • abort the migration job by disconnect hook
  • written by Marcus
• management server
  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • (new) check via destination host (if dest is Up)
    • if vm is Running on destination, consider the vm migration is successful.
  • Current: VM is Running on source host
  • New: vm is Running on destination if found

• Note:
  • In edge case, the vm is migrated before aborting the job, the vm is Stopped (by ACS on destination host)

If VM is Migrating

• check via source host (if source is Up)
• check via destination host (if dest is Up)
• determine the state and update
  • if Running on dest, Running
  • If Paused on dest, Migrating
  • If Stopped on dest, Running (if found on source) or Stopped (if not found on source)

Migrate VM with volumes

(between NFS)

Note: this action is not supported on PowerFlex

• intermittent failure
  • same as above
• continuous failure
  • current: vm is Stopped on destination and Running on source  and pool (actually it is not if migration is successful)
  • New: vm is Running on destination host and pool if found on destination host

• management server
  • Failed to migrate VM [VM instance xxx along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) xxx failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Commands 4446460207098233056 to Host 1 timed out after 21600].].
  • Current:
    • vm is Running on source host
    • volume is Ready on source pool
    • new volume is Migrating on destination pool
  • new
    • check via destination host (if dest is Up)
      • if vm is Running on destination, consider the vm migration is successful. 
    • (after fix) new volume is Destroy on destination pool if fail

• Note:
  • In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
  • solved

Migrate Volume (on NFS)

• CopyCommand (from primary1 to sec1)
• CopyCommand (from sec1 to primary2)

ROOT/DATA volume of Stopped VM

ROOT/DATA volume of Stopped VM

Migrate Volume (on Powerflex)

• Running VM
• Stopped VM