Following the sequence diagram for the above scenario:

2.4 Considerations

Analysis

...

Scenario 1: Connection failures between the agent and the management server

...

Agent: (for reconcile command only)

Save command info as JSON file
Update command state in JSON when start/process/complete the command
1. STARTED
2. PROCESSING
3. COMPLETE/FAILED
Save Answer in the JSON file
Every minute
1. Load command/answer from JSON files
2. Send with with PingCommand
3. Receive the PingAnswer from management server
4. Remove the JSON file if state is COMPLETE/FAILED

Management server: (for reconcile command only)

When create the reconcile command, insert a record with state CREATED into reconcile_commands table
When receive the PingCommand, Update command state and answer in reconcile_commands table
When wait for the answer of command (reconcile command only)
1. Every 10 seconds, check answer of the reconcile_commands table
2. If answer is found, parse the answer
3. Returna and continue with the answer
4. No need to wait until timeout, if there are connection failures.
if timed out, update command state to TIMED_OUT in reconcile_commands table

...

Scenario 2: Agent crash (or force killing the agent process)

...

Management server

When Host Status is determined as Down (only cloudstack-agent is DOWN?)
Update reconcile commands to the agent (from reconcile_command table) to state
1. RECONCILE_READY
Reconcile the command (via other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host)
remove it from reconcile_command_jobs, if command is determined to be COMPLETED or FAILED.

...

Scenario 3: Agent restart

...

Agent (when restart)

updates state of processes
1. PROCESSING to INTERRUPTED
2. PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND
Agent send CommandInfo to management server via PingCommand every minute
1. with new state

Management server reconcile commands every minute (from reconcile_command table) , for commands in state

...

Scenario 4: Agent has completed but timed out

...

Scenario 5: Management restart

...

Agent process the command

has completed and send Answer to management server, but no action on management server
has not completed, and process command

...

Management server

Update state_by_management to INTERRUPTED when mgmt server is stopped.
When another management server is detected DOWN, update reconcile_command to INTERRUPTED state ? (TODO)

State transition flows

Management server: CREATED

Agent (all good): STARTED → PROCESSING → COMPLETED → Send Answer to management server → Mgmt server removes the reconcile command jobs (new table: reconcile_command_jobs ?) → Agent removes the JSON file if job is not found.

Agent (wrong): STARTED → PROCESSING → FAILED → Send Answer to management server → Mgmt server removes the reconcile command jobs (new table: reconcile_command_jobs ?) → Agent removes the JSON file if job is not found.

Agent is restarted: STARTED → PROCESSING → INTERRUPTED (processed by java) → Send CommandInfo to management server via PingCommand every minute

Agent is restarted: STARTED → PROCESSING_IN_BACKEND → DANGLED_IN_BACKEND (processed in backend)→ Send CommandInfo to management server via PingCommand every minute

Agent timed out: STARTED → PROCESSING → TIMEDOUT (processed by java)→ Send to Answer management server

Agent timed out: STARTED → PROCESSING_IN_BACKEND → DANGLED_IN_BACKEND (processed in backend) → Send Answer to management server and Send CommandInfo via PingCommand every minute

Management server (TIMEDOUT) → Add CommandInfo to reconcile list

Management server (Timeout) → Add CommandInfo to reconcile list

Multiple management servers

Agent → First Management server → (If it is not the source mgmt server and not found in the memory, send to) Other management server to reconcile.

How to reconcile

3. Implementation

API, database, service layer and agent changes

3.1 API changes

No.

3.2 Database changes

Code Block

language	java

-- Add table for reconcile commands
CREATE TABLE IF NOT EXISTS `cloud`.`reconcile_commands` (
    `id` bigint unsigned NOT NULL UNIQUE AUTO_INCREMENT,
    `management_server_id` bigint unsigned NOT NULL COMMENT 'node id of the management server',
    `host_id` bigint unsigned NOT NULL COMMENT 'id of the host',
    `request_sequence` bigint unsigned NOT NULL COMMENT 'sequence of the request',
    `state_by_management` varchar(255) COMMENT 'state of the command updated by management server',
    `state_by_agent` varchar(255) COMMENT 'state of the command updated by cloudstack agent',
    `command_name` varchar(255) COMMENT 'name of the command',
    `command_info` MEDIUMTEXT COMMENT 'info of the command',
    `answer_name` varchar(255) COMMENT 'name of the answer',
    `answer_info` MEDIUMTEXT COMMENT 'info of the answer',
    `created` datetime COMMENT 'date the reconcile command was created',
    `removed` datetime COMMENT 'date the reconcile command was removed',
    `updated` datetime COMMENT 'date the reconcile command was updated',
    `retry_count` bigint unsigned DEFAULT 0 COMMENT 'The retry count of reconciliation',
    PRIMARY KEY(`id`),
    INDEX `i_reconcile_command__host_id`(`host_id`),
    CONSTRAINT `fk_reconcile_command__host_id` FOREIGN KEY (`host_id`) REFERENCES `host`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

3.3 Core changes

Command.java

Code Block

language	java

    public enum State {
        CREATED,        // Command is created by management server
        STARTED,        // Command is started by agent
        PROCESSING,     // Processing by agent
        PROCESSING_IN_BACKEND,  // Processing in backend by agent
        COMPLETED,      // Operation succeeds by agent or management server
        FAILED,         // Operation fails by agent
        RECONCILE_READY,        // Ready for reconciliation
        RECONCILING,    // Being reconciled by management server
        RECONCILED,     // Reconciled by management server
        RECONCILE_FAILED,       // Fail to reconcile by management server
        TIMED_OUT,      // Timed out on management server or agent
        INTERRUPTED,    // Interrupted by management server or agent (for example agent is restarted),
        DANGLED_IN_BACKEND     // Backend process which cannot be processed normally (for example agent is restarted)
    }

new method

...

language	java

...

3. Implementation

API, database, service layer and agent changes

3.1 API changes

No.

3.2 Database changes

Code Block

language	java

-- Add table for reconcile commands
CREATE TABLE IF NOT EXISTS `cloud`.`reconcile_commands` (
    `id` bigint unsigned NOT NULL UNIQUE AUTO_INCREMENT,
    `management_server_id` bigint unsigned NOT NULL COMMENT 'node id of the management server',
    `host_id` bigint unsigned NOT NULL COMMENT 'id of the host',
    `request_sequence` bigint unsigned NOT NULL COMMENT 'sequence of the request',
    `state_by_management` varchar(255) COMMENT 'state of the command updated by management server',
    `state_by_agent` varchar(255) COMMENT 'state of the command updated by cloudstack agent',
    `command_name` varchar(255) COMMENT 'name of the command',
    `command_info` MEDIUMTEXT COMMENT 'info of the command',
    `answer_name` varchar(255) COMMENT 'name of the answer',
    `answer_info` MEDIUMTEXT COMMENT 'info of the answer',
    `created` datetime COMMENT 'date the reconcile command was created',
    `removed` datetime COMMENT 'date the reconcile command was removed',
    `updated` datetime COMMENT 'date the reconcile command was updated',
    `retry_count` bigint unsigned DEFAULT 0 COMMENT 'The retry count of reconciliation',
    PRIMARY KEY(`id`),
    INDEX `i_reconcile_command__host_id`(`host_id`),
    CONSTRAINT `fk_reconcile_command__host_id` FOREIGN KEY (`host_id`) REFERENCES `host`(`id`) ON DELETE CASCADE
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

3.3 Core changes

Command.java

Code Block

language	java

    public enum State {
        CREATED,        // Command is created by management server
        STARTED,        // Command is started by agent
        PROCESSING,     // Processing by agent
        PROCESSING_IN_BACKEND,  // Processing in backend by agent
        COMPLETED,      // Operation succeeds by agent or management server
        FAILED,         // Operation fails by agent
        RECONCILE_READY,        // Ready for reconciliation
        RECONCILING,    // Being reconciled by management server
        RECONCILED,     // Reconciled by management server
        RECONCILE_FAILED,       // Fail to reconcile by management server
        TIMED_OUT,      // Timed out on management server or agent
        INTERRUPTED,    // Interrupted by management server or agent (for example agent is restarted),
        DANGLED_IN_BACKEND     // Backend process which cannot be processed normally (for example agent is restarted)
    }

new method

Code Block

language	java

    public boolean isReconcile() {
        return false;
    }

3.4 Considerations

Analysis

Scenarios	Current behavior	How to deal with it
Scenario 1: Connection failures between the agent and the management server	Agent cannot send Answer to management server, therefore timed out on management server	Agent: (for reconcile command only) Save command info as JSON file Update command state in JSON when start/process/complete the command STARTED PROCESSING COMPLETE/FAILED Save Answer in the JSON file Every minute Load command/answer from JSON files Send with with PingCommand Receive the PingAnswer from management server Remove the JSON file if state is COMPLETE/FAILED Management server: (for reconcile command only) When create the reconcile command, insert a record with state CREATED into reconcile_commands table When receive the PingCommand, Update command state and answer in reconcile_commands table When wait for the answer of command (reconcile command only) Every 10 seconds, check answer of the reconcile_commands table If answer is found, parse the answer Returna and continue with the answer No need to wait until timeout, if there are connection failures. if timed out, update command state to TIMED_OUT in reconcile_commands table
Scenario 2: Agent crash (or force killing the agent process)	Agent interrupt the process or processing in backend)	Management server When Host Status is determined as Down (only cloudstack-agent is DOWN?) Update reconcile commands to the agent (from reconcile_command table) to state RECONCILE_READY Reconcile the command (via other host on same cluster or pod or zone, depends on the scope of storage, or cluster of host) remove it from reconcile_command_jobs, if command is determined to be COMPLETED or FAILED.
Scenario 3: Agent restart	Agent interrupt the process or processing in backend)	Agent (when restart) updates state of processes PROCESSING to INTERRUPTED PROCESSING_IN_BACKEND to DANGLED_IN_BACKEND Agent send CommandInfo to management server via PingCommand every minute with new state Management server reconcile commands every minute (from reconcile_command table) , for commands in state
Scenario 4: Agent has completed but timed out	timed out on management server
Scenario 5: Management restart	Agent process the command has completed and send Answer to management server, but no action on management server has not completed, and process command	Management server Update state_by_management to INTERRUPTED when mgmt server is stopped. When another management server is detected DOWN, update reconcile_command to INTERRUPTED state ? (TODO)

State transition flows