Versions Compared

Old Version 14

changes.mady.by.user Unknown User (weizhou)

Saved on

compared with

New Version 15

changes.mady.by.user Unknown User (weizhou)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

3.4 Considerations

  • Analysis
ScenariosCurrent behaviorHow to deal with it
  • Scenario 1: Connection failures between the agent and the management server
Agent cannot send Answer to management server, therefore timed out on management server

Agent: (for reconcile command only)

  1. Save command info as JSON file
  2. Update command state in JSON when start/process/complete the command
    1. STARTED
    2. PROCESSING
    3. COMPLETE/FAILED
  3. Save Answer in the JSON file
  4. Every minute
    1. Load command/answer from JSON files
    2. Send with with PingCommand
    3. Receive the PingAnswer from management server
    4. Remove the JSON file if state is COMPLETE/FAILED


Management server: (for reconcile command only)

  1. When create the reconcile command, insert a record with state CREATED into  reconcile_commands table
  2. When receive the PingCommand, Update command state and answer in reconcile_commands table
  3. When wait for the answer of command (reconcile command only)
    1. Every 10 seconds, check answer of the reconcile_commands table
    2. If answer is found, parse the answer
    3. Returna and continue with the answer
    4. No need to wait until timeout, if there are connection failures. 
  4. if timed out, update command state to TIMED_OUT in reconcile_commands table
  • Scenario 2: Agent crash (or force killing the agent process)
Agent interrupt the process or processing in backend

Management server (TODO)

  1. When Host Status is determined as Down (only cloudstack-agent is DOWN?)
  2. Update reconcile commands (to the agent) in reconcile_command table
    1. to INTERRUPTED state
  3. Reconcile the command
    1. see "3.6 Reconcile the command"
  4. Update state to RECONCILED, if command is reconciled and no need more reconciliation
  • Scenario 3: Agent restart
Agent interrupt the process or processing in backend
)

Agent (when restart)

  1. When stop the agent, updates state of processes  (TODO)
    1. PROCESSING to INTERRUPTED
    2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
  2. When start the agent, updates state of processes 
    1. PROCESSING to INTERRUPTED
    2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
  3. Agent send CommandInfo to management server via PingCommand every minute
    1. with new state
  4. Management server reconcile commands every minute (from reconcile_
command
  1. commands table) , for commands in state
  • Scenario 4: Agent has completed but timed out
timed out on management server

Management server

  1. Update state_by_management to TIMED_OUT
  2. Update state_by_agent to COMPLETED (from PROCESSING)
  3. Reconcile the command
    1. see "3.6 Reconcile the command"
    2. TODO: do not reconcile via hosts, etc
  • Scenario 5: Management restart

Agent process the command

  • has completed and send Answer to management server, but no action on management server
  • has not completed, and
process
  • is processing command

Management server

  1. Update state_by_management to INTERRUPTED when mgmt server is stopped.
  2. When another management server is detected DOWN, update reconcile_command to INTERRUPTED state ? (TODO)



3.5 State transitions


Management server (all good): CREATED -> COMPLETED

...

  • Assumption
    • Each request does not have multiple commands with same name
  • Only support:
    • CopyCommand, MigrateCommand, MigrateVolumeCommand
    • resources in Migration state (vm, source volume, dest volume)
    • Hypervisor: KVM
    • Storage: NFS, Local, Powerflex


  1. How to distribute the reconciiation tasks if there are multiple management servers ? (TODO)
  2. How to handle the state DANGLED_IN_BACKEND ? (TODO)

4. Test cases


Please note: 

Migration between NFS and Local requires the fix: https://github.com/apache/cloudstack/pull/10266


NFS to NFSNFS to LocalLocal to LocalLocal to NFSPowerflex to PowerflexPowerflex
to NFSNFS to Powerflex
<-> NFS
Migrate VM

PrepareForMigrationCommand (dest)

MigrateCommand (source)

---

PrepareForMigrationCommand (dest)

MigrateCommand (source)

-

-
Migrate VM with volumes

CopyCommand (template to primary if needed)

CreateObjectCommand (new volume)

ModifyTargetsCommand

PrepareForMigrationCommand (dest)

MigrateCommand (source)

DeleteCommand (source)

same as "NFS to NFS"same as "NFS to NFS"same as "NFS to NFS"

Migrating a volume online with KVM from managed storage is not currently supported.

Pool [%s] is not compatible with volume [%s], skipping it.
Migrate ROOT Volume (of Running VM)KVM does not support volume live migrationdue to the limited possibility to refresh VM XML domain. Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. Use 'migrateVirtualMachineWithVolumes' instead.samesamesameMigrateVolumeCommandStorage pool pr503-t11980-kvm-ol8-kvm-pri3 is not suitable to migrate volume 
Migrate ROOT Volume (of Stopped VM)

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

samesamesameCopyCommandsame as above







Migrate DATA Volume (of Running VM)KVM does not support volume live migrationdue to the limited possibility to refresh VM XML domain. Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. Use 'migrateVirtualMachineWithVolumes' instead.samesamesameMigrateVolumeCommandsame as above
Migrate DATA Volume (of Stopped VM)

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

samesamesameCopyCommandsame as above
Migrate DATA Volume (unattached)

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

samesamesameCopyCommand
Storage pool pr503-t11980-kvm-ol8-kvm-pri3 is not suitable to migrate volume 
same as above