Page History

• iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP
  • iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP

• intermittent failure
  • agent updates answer when connection is back to normal
  • mgmt server gets the answer from DB every 10 seconds
  • mgmt server processes the answer if found
  • Operation times out
• continuous failure
  • agent
    • Migration thread of VM [i-2-1198-VM] finished.
  •  management
    • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • Behavior:
    • (new) check via destination host (if dest is Up)
      • if vm is Running on destination, consider the vm migration is successful.
    • Active migration command so scheduling a restart
    • Current: vm is Stopped on destination and Running on source (actually it is not if migration is successful)
    • New: vm is Running on destination host if found

• kill -9  $(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}')

•  agent
  • agent is started again by systemctl

• management server

  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance

  • (new) check via destination host (if dest is Up)
    • if vm is Running on destination, consider the vm migration is successful.
  • Current: VM is Running on source host
  • New: vm is Running on destination if found

• Note:
  • • In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
    • solved

• systemctl restart cloudstack-agent

• agent 
  • abort the migration job by disconnect hook
  • written by Marcus
• management server
  • Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
  • (new) check via destination host (if dest is Up)
    • if vm is Running on destination, consider the vm migration is successful.
  • Current: VM is Running on source host
  • New: vm is Running on destination if found

• Note:
  • In edge case, the vm is migrated before aborting the job, the vm is Stopped (by ACS on destination host)

If VM is Migrating (mgmt server is restarted)check via source host (if source is Up)

• check via destination host (if dest is Up)
• determine the state and update
  • if Running on dest, Running
  • If Paused on dest, Migrating
  • If Stopped on dest, Running (if found on source) or Stopped (if not found on source)

• intermittent failure
  • same as above
• continuous failure
  • current: vm is Stopped on destination and Running on source  and pool (actually it is not if migration is successful)
  • New: vm is Running on destination host and pool if found on destination host

• management server
  • Failed to migrate VM [VM instance xxx along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) xxx failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Commands 4446460207098233056 to Host 1 timed out after 21600].].
  • Current:
    • vm is Running on source host
    • volume is Ready on source pool
    • new volume is Migrating on destination pool
  • new (after reconciliationfix)
    • new volume is Destroy on destination pool

• Note:
  • In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
  • solved