Page History

...

Action

scenario 1: Connection failures between agent and management server

Scenario 4: Agent has completed but timed out

scenario 2

Agent crash (or force killing the agent process)

scenario 3

Agent is restarted manually

scenarios 5 (Needed ?)

mgmt server is restarted

How to test

iptables -I OUTPUT -p tcp -m tcp --dport 8250 -j DROP

iptables -D OUTPUT -p tcp -m tcp --dport 8250 -j DROP

pid=$(ps -ef |grep cloudstack-agent |grep -v grep |awk '{print $2}') && kill -9 $pid

systemctl restart cloudstack-agent

systemctl restart cloudstack-management

1. Migrate VM

MigrateVMCommand
MigrateCommand (internal)

Supported by NFS or Powerflex

intermittent failure

- agent updates answer when connection is back to normal (every reconcile.command.interval in PingRoutingCommand)
  - mgmt server saves the answer into DB
- mgmt server gets the answer from DB every 10 seconds
- mgmt server processes the answer if found
- Operation times out

continuous failure

- agent
  - Migration thread of VM [i-2-1198-VM] finished.
- management
  - Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
- Behavior:
  - Active migration command so scheduling a restart
  - Current: vm is Stopped on destination and Running on source (actually it is not if migration is successful)
  - New: vm is Running on destination host if found (via libvirt)
  - (new) check via destination host (if dest is Up)
    - if vm is Running on destination, consider the vm migration is successful.

agent
- agent is started again by systemctl

management server
- Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
- (new) check via destination host (if dest is Up)
  - if vm is Running on destination, consider the vm migration is successful.
- Current
  - VM is set to Stopped state on destination host and then Running on the source host
- New:
  - vm is Running on destination if found
  - if not found, set to Stopped state on destination host and then Running on the source host (same as current)

Note:
- - In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
  - solved

agent
- when agent is restarted or disconnected
- abort the migration job by a disconnect hook (MigrationCancelHook)
  - written by Marcus
  - tested with multiple mgmt servers

management server
- Resource [Host:1] is unreachable: Host 1: Operation timed out on migrating VM instance
- (new) check via destination host (if dest is Up)
  - if vm is Running on destination, consider the vm migration is successful.
- Current: VM is set to Stopped state on destination host and then Running on the source host
- New:
  - vm is Running on destination if found
  - if not found, set to Stopped state on destination host and then Running on the source host (same as current)

Note:
- In edge case, the vm is migrated before aborting the job, the vm is Stopped (by ACS on destination host)
- solved

Issues

mgmt server
- Cancel left-over job-17783
- Cleaning up Instance with Id: 1226
- CleanUp Async Jobs after mgmt server maintenance (#8394)
- vm is Running on source host (this will be updated when mgmt server gets vm report from agent)
agent (new)
- abort the migration job by disconnect hook
- written by Marcus
Edge case: both scenario 1 and 5 happen
- VM is migrated but state is Running on source
- fixed by Abhishek's PR engine/orchestration: fix missing vm powerstate update vm state #10407
- ACS updates the host_id of vm to destination host

No issues found, ALL looks ok

2. Migrate VM with volumes

(between NFS)
MigrateVirtualMachine

WithVolumeCmd

MigrateCommand (internal)

Note: this action is not supported on PowerFlex

Two volumes in DB

source volume
dest volume (last_id = source_volume_id)

intermittent failure

- same as above

continuous failure

- current: vm is Stopped on destination and Running on source and pool (actually it is not if migration is successful)
- New: vm is Running on destination host and pool if found on destination host

management server
- Failed to migrate VM [VM instance xxx along with its volumes due to [com.cloud.utils.exception.CloudRuntimeException: Copy volume(s) to storage(s) xxx failed in StorageSystemDataMotionStrategy.copyAsync. Error message: [Commands 4446460207098233056 to Host 1 timed out after 21600].].
- Current:
  - vm is Running on source host
  - volume is Ready on source pool
  - Bug: new volume is Migrating on destination pool (solved)
- new
  - check via destination host (if dest is Up)
    - if vm is Running on destination, consider the vm migration is successful.
  - (after fix) new volume is Destroy on destination pool if fail

Note:
- In edge case, if the vm is migrated before agent is restarted (in 10 seconds) , the vm is Stopped (by ACS on destination host)
- solved

same as left

Issues

management server
- vm is Running on source host
- Bug: both volumes are stuck at Migrating state (need to reconcile)

Edge case: both scenario 1 and 5 happen
- VM is migrated but state is Running on source

How to reconcile (Verified):

only if
- VM is Running state
- volume is Migrating state
check via host (if source is Up)
check disk path and update volume state
- - if source volume is found but destination is not found, mark source as Ready and dest as Destroy
  - if source volume is not found but destination is found, mark source as Destroy and dest as Ready

3. Migrate Volume (on NFS)

MigrateVolumeCmd (API)

CopyCommand (from primary1 to sec1)
CopyCommand (from sec1 to primary2)

ROOT/DATA volume of Stopped VM

intermittent failure

1st CopyCommand: wait until connection is recoved
2nd CopyCommand: wait until connection is recoved

continuous failure

1st CopyCommand: operation timeout
- Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1159","uuid":"ed322109-8e54-47ca-8183-3c8f2a0f37e6"}] migration failed due to [com.cloud.utils.exception.CloudRuntimeException: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 5302988561228766380 to Host 1 timed out after 1800].
- Bug: new volume is Creating with empty path on destination pool (solved)
2nd CopyCommand
- Resource [StoragePool:1] is unreachable: Volume [{"name":"ROOT-1159","uuid":"ed322109-8e54-47ca-8183-3c8f2a0f37e6"}] migration failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 7457960982925541434 to Host 1 timed out after 2400].
- Bug: new volume is Creating with empty path on destination pool (solved)

New

new volume is Destroy state with empty path on destination pool

1st CopyCommand
- copy object failed: com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 15481123719087037 to Host 1 timed out after 1800
- new volume is Destroy state
  - expungeVolumeAsync is called in copyVolumeCallBack

2nd CopyCommand
- copy failed com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 398850041998999773 to Host 1 timed out after 2400
- new volume is Destroy state
  - expungeVolumeAsync is called in copyVolumeCallBack

No issues found, ALL looks good

same as left

Issues

1st CopyCommand
- new volume is Creating state
- record in volume_store_ref is Creating state (to be cleaned)
2nd CopyCommand
- new volume is Creating state
- record in volume_store_ref is Copying state (to be cleaned)

How to reconcile (verified):

1st CopyCommand
- new volume is Destroy state and removed=now()
- volume_store_ref with Creating state is removed
2nd CopyCommand
- check the size of new volume. If size is changed, the process is still ongoing.
- if size is not changed, new volume is Destroy state and path is set (but not removed)
- volume_store_ref with Copying state is removed

4. Migrate Volume of Running VM (on Powerflex)

MigrateVolumeCommand

agent: succeed
- answer: {"volumePath":"1fe84f0700000002:vol-610-4d48-306f","result":true,"contextMap":{},"wait":0,"bypassHostMaintenance":false}
management server
- volume path is set to new, then set to old when fail
- Bug: Actually, on KVM host, the VM is running with new volume, but the new volume is removed
  - Input/Output error, read-only system
- New:
  - check volume statistics on destination pool via ScaleIO gateway
  - if allocation size = provisioned size, volume migration is done. remove volume on source pool.
  - if allocation size is not same to provisioned size, volume migration failes, remove volume on destination pool.

management
- Resource [StoragePool:12] is unreachable: Migrate volume failed: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 2520889891420635168 to Host 1 timed out after 2400
- Failed to migrate PowerFlex volume
- volume path is set to new, then set to old when fail (looks good)
  - revertBlockCopyVolumeOperations

No issues found, ALL looks good

changes on agent

it uses "dm.blockCopy" (introduced by Hari)
when agent is disconnected or restarted
- abort the block copy job by a disconnect hook (VolumeMigrationCancelHook)
- similar as MigrationCancelHook

same as left

Issue:

one volume in Migrating in DB
- path, iscsi_name, pool_id are info on destination pool
when mgmt server is restarted, it updates all Migrating volume to Ready
- this is disabled if reconcile is enabled

How to reconcile:

only if vm is Running and volume in Migrating state
- now:
The current pool is destination pool of reconcile command
check vm state and vm disks via agent
- if volume is Ready on source pool,
  - revert path, iscsi_name and pool_id to source pool
  - create a dummy volume on dest pool
  - remove the dummy volume
- if volume is not found on source pool
  - update state to Ready
  - create a dummy volume on source pool
  - remove the dummy volume
- if vm is running on source pool (check by fullpath) = volume is ready on source pool
- if vm is running on dest pool, update volume state to Ready, and remove volume from source pool

5. Migrate Volume of Stopped VM (on Powerflex)

CopyCommand

intermittent failure

CopyCommand: wait until connection is recoved

continuous failure

Resource [StoragePool:11] is unreachable: Volume [{"name":"ROOT-533","uuid":"ff8a53a0-d513-4bfc-8920-352948ea066b"}] migration failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 8886446489732120778 to Host 1 timed out after 2400]
- 2400 = wait (1200) * 2
- new volume is Expunged
  - due to expungeVolumeAsync in copyManagedVolumeCallBack
- Not an issue since the VM is Stopped so no data loss

management server

- Resource [StoragePool:12] is unreachable: Volume [{"name":"ROOT-531","uuid":"4225516d-52fe-465d-a46d-81f97a2e55b2"}] migration failed due to [com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:1, com.cloud.exception.OperationTimedoutException: Commands 3050062847636668430 to Host 1 timed out after 2400].
- new volume is Expunged
  - due to expungeVolumeAsync in copyManagedVolumeCallBack

No issues found, ALL looks good

same as left

Issue: two volumes in DB

source volume: Migrating state
dest volume: Creating state
- dest.last_id = source_volume_id

How to reconcile

check volume information via host
determine state of source and dest volumes
if source volume is Ready
- update source from Migrating to Ready
- update dest volume from Creating to Destroy
if dest volume is Ready
- update dest from Creating to Ready
- update source volume from Migrating to Destroy

TO be fixed

when migrate volume from powerflex to powerflex with same system id, it will not create CopyCommand

Space shortcuts

Child pages

Versions Compared

Old Version 65

New Version Current

Key