Versions Compared

Old Version 16

changes.mady.by.user Unknown User (weizhou)

Saved on

compared with

New Version 17

changes.mady.by.user Unknown User (weizhou)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagejava
    public enum State {
        CREATED,        // Command is created by management server
        STARTED,        // Command is started by agent
        PROCESSING,     // Processing by agent
        PROCESSING_IN_BACKEND,  // Processing in backend by agent
        COMPLETED,      // Operation succeeds by agent or management server
        FAILED,         // Operation fails by agent
        RECONCILE_READY,        // Ready for reconciliation
        RECONCILING,    // Being reconciled by management server
        RECONCILED,     // Reconciled by management server
        RECONCILE_FAILED,       // Fail to reconcile by management server
        TIMED_OUT,      // Timed out on management server or agent
        INTERRUPTED,    // Interrupted by management server or agent (for example agent is restarted),
        DANGLED_IN_BACKEND     // Backend process which cannot be processed normally (for example agent is restarted)
    }


new method for each Command

Code Block
languagejava
    public boolean isReconcile() {
        return false;
    }

...

ScenariosCurrent behaviorHow to deal with it
  • Scenario 1: Connection failures between the agent and the management server
Agent cannot send Answer to management server, therefore timed out on management server

Agent: (for reconcile command only)

  1. Save command info as JSON file
  2. Update command state in JSON when start/process/complete the command
    1. STARTED
    2. PROCESSING
    3. COMPLETE/FAILED
  3. Save Answer in the JSON file
  4. Every minute
    1. Load command/answer from JSON files
    2. Send with with PingCommand
    3. Receive the PingAnswer from management server
    4. Remove the JSON file if state is COMPLETE/FAILED


Management server: (for reconcile command only)

  1. When create the reconcile command, insert a record with state CREATED into  reconcile_commands table
  2. When receive the PingCommand, Update command state and answer in reconcile_commands table
  3. When wait for the answer of command (reconcile command only)
    1. Every 10 seconds, check answer of the reconcile_commands table
    2. If answer is found, parse the answer
    3. Returna and continue with the answer
    4. No need to wait until timeout, if there are connection failures. 
  4. if timed out, update command state to TIMED_OUT in reconcile_commands table
  • Scenario 2: Agent crash (or force killing the agent process)
Agent interrupt the process or processing in backend

Management server (TODO)

  1. When Host Status is determined as Down (only cloudstack-agent is DOWN?)
  2. Update reconcile commands (to the agent) in reconcile_command table
    1. to INTERRUPTED state
  3. Reconcile the command
    1. see "3.6 Reconcile the command"
  4. Update state to RECONCILED, if command is reconciled and no need more reconciliation
  • Scenario 3: Agent restart
Agent interrupt the process or processing in backend

Agent (when restart)

  1. When stop the agent, updates state of processes  (TODO)
    1. PROCESSING to INTERRUPTED
    2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
  2. When start the agent, updates state of processes 
    1. PROCESSING to INTERRUPTED
    2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
  3. Agent send CommandInfo to management server via PingCommand every minute
    1. with new state
  4. Management server reconcile commands every minute (from reconcile_commands table) , for commands in state
  • Scenario 4: Agent has completed but timed out
timed out on management server

Management server

  1. Update state_by_management to TIMED_OUT
  2. Update state_by_agent to COMPLETED (from PROCESSING)
  3. Reconcile the command
    1. see "3.6 Reconcile the command"
    2. TODO: do not reconcile via hosts , etcif state_by_agent to COMPLETED
  • Scenario 5: Management restart

Agent process the command

  • has completed and send Answer to management server, but no action on management server
  • has not completed, and is processing command

Management server

  1. Update state_by_management to INTERRUPTED when mgmt server is stopped.
  2. When another management server is detected DOWN, update reconcile_command to INTERRUPTED state ? (TODO)

...

            -> RECONCILED (all good) / RECONCILE_FAILED (failed, will retry) / RECONCILED_READY (success, but need more information, will retry)


Migrate VM w/wo volumeswithout volumes (If VM is Migrating)

  • PrepareForMigrationCommand (Update VM state)
  • MigrateCommand
    • check via source host (if source is Up)
    • check via destination host (if dest is Up)
    • determine the state and update

Migrate Volume

...

VM with volumes  (If

...

VM is

...

Migrating)

  • check

...

  • VM state and volume states
  • Check if there are reconcile commands for the VM
    • If no, check the state on source host (last_host_id) and destination host (host_id)
    • PrepareForMigrationCommand (Update VM state)
    • MigrateCommand
      • check via source host (if source is Up)
      • check via destination host (if dest is Up)
      • determine the state and update

Migrate Volumes  (If volume is Migrating)

  • CopyCommand (from primary1 to secondary)
    • skipped.
    • check if there are other Command on same volume ?
  • CopyCommand (from secondary to primary2)
    • check if volume exists on primary2 (via the host, other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host) TODO
    • check if volume is changed on primary2
      • if yes, still Copying
      • if no, update state
  • CopyCommand (from primary1 to primary2)
    • check if volume exists on primary1
    • check if volume exists on primary2
    • check if volume is changed on primary2
      • if yes, still Copying
      • if no, update state
  • MigrateVolumeCommand (from primary1 to primary2)
    • check if volume exists on primary1
    • check if volume exists on primary2
    • check if volume is changed on primary2
      • if yes, still Migrating
      • if no, update state

...



3.7 Limitations


  • Assumption
    • Each request does not have multiple commands with same name
  • Only support:
    • CopyCommand, MigrateCommand, MigrateVolumeCommand
    • resources in Migration state (vm, source volume, dest volume)
    • Hypervisor: KVM
    • Storage: NFS, Local, Powerflex

...


NFS to NFSNFS to LocalLocal to LocalLocal to NFSPowerflex to PowerflexPowerflex <------> NFS
Migrate VM

PrepareForMigrationCommand (dest)

MigrateCommand (source)

---

PrepareForMigrationCommand (dest)

MigrateCommand (source)

-
Migrate VM with volumes

CopyCommand (template to primary if needed)

CreateObjectCommand (new volume)

ModifyTargetsCommand

PrepareForMigrationCommand (dest)

MigrateCommand (source)

DeleteCommand (source)

same as "NFS to NFS"same as "NFS to NFS"same as "NFS to NFS"

Migrating a volume online with KVM from managed storage is not currently supported.

Pool [%s] is not compatible with volume [%s], skipping it.
Migrate ROOT Volume (of Running VM)KVM does not support volume live migrationdue to the limited possibility to refresh VM XML domain. Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. Use 'migrateVirtualMachineWithVolumes' instead.samesamesame

MigrateVolumeCommand

  • Same volume
Storage pool pr503-t11980-kvm-ol8-kvm-pri3 is not suitable to migrate volume 
Migrate ROOT Volume (of Stopped VM)

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

samesamesame

CopyCommand (primary1 to primary2)

  • Different volume IDs
same as above







Migrate DATA Volume (of Running VM)KVM does not support volume live migrationdue to the limited possibility to refresh VM XML domain. Therefore, to live migrate a volume between storage pools, one must migrate the VM to a different host as well to force the VM XML domain update. Use 'migrateVirtualMachineWithVolumes' instead.samesamesame

MigrateVolumeCommand

  • Same volume
same as above
Migrate DATA Volume (of Stopped VM)

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

samesamesame

CopyCommand (primary1 to primary2)

  • Different volume IDs
same as above
Migrate DATA Volume (unattached)

CopyCommand (primary1 to secondary)

CopyCommand (secondary to primary2)

DeleteCommand (secondary)

DeleteCommand (primary1)

samesamesame

CopyCommand (primary1 to primary2)

  • Different volume IDs
same as above