Versions Compared

Old Version 6

changes.mady.by.user Unknown User (weizhou)

Saved on

compared with

New Version 7

changes.mady.by.user Unknown User (weizhou)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

3.4 Considerations

  • Analysis
ScenariosCurrent behaviorHow to deal with it
  • Scenario 1: Connection failures between the agent and the management server
Agent cannot send Answer to management server, therefore timed out on management server

Agent: (for reconcile command only)

  1. Save command info as JSON file
  2. Update command state in JSON when start/process/complete the command
    1. STARTED
    2. PROCESSING
    3. COMPLETE/FAILED
  3. Save Answer in the JSON file
  4. Every minute
    1. Load command/answer from JSON files
    2. Send with with PingCommand
    3. Receive the PingAnswer from management server
    4. Remove the JSON file if state is COMPLETE/FAILED


Management server: (for reconcile command only)

  1. When create the reconcile command, insert a record with state CREATED into  reconcile_commands table
  2. When receive the PingCommand, Update command state and answer in reconcile_commands table
  3. When wait for the answer of command (reconcile command only)
    1. Every 10 seconds, check answer of the reconcile_commands table
    2. If answer is found, parse the answer
    3. Returna and continue with the answer
    4. No need to wait until timeout, if there are connection failures. 
  4. if timed out, update command state to TIMED_OUT in reconcile_commands table
  • Scenario 2: Agent crash (or force killing the agent process)
Agent interrupt the process or processing in backend)

Management server (TODO)

  1. When Host Status is determined as Down (only cloudstack-agent is DOWN?)
  2. Update reconcile commands to the agent (from reconcile_command table) to state
    RECONCILE_READY
      1. INTERRUPTED
    2. Reconcile the command
      1. Migrate VM: via destination host (TODO)
      2. Migrate Volume: via other host (TODO)
      3. Copy Command: via other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host (TODO)
    remove it from  reconcile_command_jobs
    1. Update state to RECONCILED, if command is
    determined to be COMPLETED or FAILED.
    1. reconciled and no need more reconciliation
    • Scenario 3: Agent restart
    		Agent interrupt the process or processing in backend)

    Agent (when restart)

    updates state of processes
    1. When stop the agent, updates state of processes  (TODO)
      1. PROCESSING to INTERRUPTED
      2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
    2. When start the agent, updates state of processes 
      1. PROCESSING to INTERRUPTED
      2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
    1. Agent send CommandInfo to management server via PingCommand every minute
      1. with new state


    1. Management server reconcile commands every minute (from reconcile_command table) , for commands in state
    • Scenario 4: Agent has completed but timed out
    		timed out on management server
    • Scenario 5: Management restart

    Agent process the command

    • has completed and send Answer to management server, but no action on management server
    • has not completed, and process command

    Management server

    1. Update state_by_management to INTERRUPTED when mgmt server is stopped.
    2. When another management server is detected DOWN, update reconcile_command to INTERRUPTED state ? (TODO)



    3.5 State

    ...

    transitions


    Management server : CREATEDAgent (all good): CREATED -> COMPLETED

    Management server (failed): CREATED -> FAILED

    Management server is restarted: CREATED -> INTERRUPTED 

    Management server time out: CREATED -> TIMED_OUT


    Agent (all good): STARTED → PROCESSING → COMPLETED → Send Answer to management server STARTED → PROCESSING → COMPLETED → Send Answer to management server → Mgmt server removes the reconcile command jobs (new table: reconcile_command _jobs ?) → Agent removes the JSON file if job is not found (TODO).

    Agent (wrong): STARTED → PROCESSING → FAILED → Send Answer to management server → Mgmt server removes the reconcile command jobs (new table: reconcile_command _jobs ?) → Agent removes the JSON file if job is not found.

    ...

    Agent timed out: STARTED → PROCESSING → TIMEDOUT (processed by java)→ Send Send  Answer to Answer management server

    Agent timed out: STARTED → PROCESSING_IN_BACKEND → DANGLED_IN_BACKEND (processed in backend) → Send Answer to management server and Send CommandInfo via PingCommand every minute

    Management server (TIMEDOUT) → Add CommandInfo to reconcile list

    Management server (Timeout) → Add CommandInfo to reconcile list


    • Multiple Multiple management servers

    Agent → First Management server → (If it is not the source mgmt server and not found in the memory, send to) Other management server to reconcile.


    3.

    ...

    6 Reconcile the command

    States: INTERRUPTED/TIMED_OUT/RECONCILE_READY/RECONCILE_FAILED

                -> RECONCILING

                -> RECONCILED (all good) / RECONCILE_FAILED (failed, will retry) / RECONCILED_READY (success, but need more information, will retry)


    • Migrate VM: via destination host (TODO)
    • Migrate Volume: via other host (TODO)
    • Copy Command: via other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host (TODO)


    4. Test cases



    		NFS to NFSNFS to LocalLocal to LocalLocal to NFSPowerflex to PowerflexPowerflex to NFSNFS to Powerflex
    Migrate VM
    		----
    		-
    Migrate VM with volumes






    Migrate Volume (of Running VM)






    Migrate Volume (of Stopped VM)

    ...