Overview

All nodes (servers and clients) are able to host services, but the client nodes are excluded from service deployment by default. The only way to deploy service on client nodes is to specify node filter in ServiceConfiguration.

All deployed services are identified internally by “serviceId” (IgniteUuid). This allows us to build a base for such features as hot redeployment and service versioning. It’s important to have an ability to identify and manage services with the same name, but a different version.

Deployment process

Services deployment process - is a process of change deployed services state.

Unified steps of service deployments process:

• Each node makes a decision if it should apply actions (deploy/undeploy) on local services and sends single deployments message to the coordinator through comm-spi (p2p). The message contains deployments errors and count of locally deployed instances of services related to the current deployment process;
• Coordinator aggregates single deployments messages, then builds and sends full deployments message over the to all nodes through disco-spi;
• All nodes update services information and complete initiators futures if needed;

Triggers:

• Users deploy/undeploy requests;
• Affinity topologies change if affinity services exist;
• Topology change events (EVT_NODE_JOINED / LEFT / FAILED);

Example of user’s service deployment request according to unified flow:

1. Initiator sends DynamicServicesChangeRequestBatchMessage request using discovery spi to change service state [deploy, undeploy]. The request will be stored by all nodes in own queue to be processed, event if the coordinator failed;
2. All nodes process tasks from queue, when deployment request received each node calculates a new service’s assignments independently using deterministic assignments function and apply actions (deploy, undeploy) if needed, then node builds single deployments message ServicesSingleMapMessage that contains services id and amount of instances were deployed on this single node and sends the message to coordinator (p2p);
3. Once the coordinator receives all single deployments messages then it builds ServicesFullMapMessage that contains services deployments across the cluster and sends the message through disco-spi to be processed by all nodes;

Messages

class DynamicServicesChangeRequestBatchMessage {

          Collection<DynamicServiceChangeRequest> reqs;

}

class DynamicServiceChangeRequest {

          IgniteUuid srvcId; // Unique service id (generates to deploy, existing used to undeploy)

          ServiceConfiguration cfg; // Empty in case of undeploy

          byte flags; // Change’s types flags [deploy, undeploy, etc.]

}

class ServicesSingleDeploymentsMessage {

          ServicesDeploymentProcessId depId;

          Map<IgniteUuid, ServiceSingleDeploymentsResults> results;

}

class ServiceSingleDeploymentsResults {

          int cnt; // Deployed instances count, 0 in case of undeploy

          Collection<byte[]> errors; // Serialized exceptions to avoid issues at spi-level

}

class ServicesFullDeploymentsMessage  {

          ServicesDeploymentProcessId depId;

          Collection<ServiceFullDeploymentsResults> results;

}

class ServiceFullDeploymentsResults {

          IgniteUuid srvcId;

          Map<UUID, ServiceSingleDeploymentsResults> results; // Per node

}

class ServicesDeploymentProcessId {

          AffinityTopologyVersion topVer;

          IgniteUuid reqId; // Unique id of custom discovery message

}

Coordinator change

All nodes handle requests of service’s state changes and put them into deployment queue, but only the coordinator aggregates single deployments messages received from nodes and builds a full deployments to finish exchange. If coordinator left or fail during deployment exchange another server node will become the coordinator and continue the process.

Topology change

Each topology change event (NODE_JOIN/LEFT/FAILED) causes  deployment task. Assignments will be recalculated and applied for each deployed service if needed.

Services reassignment process takes into account previous assignments to avoid redundant redeployments.

If left/failed node had not deployed any services, then deployment task will be finished without sending messages.

Cluster activation/deactivation

• On deactivation:

• • local services are being undeployed;
  • requests are not handling (including deployment / undeployment);

• On activation:

• • local services are being redeployed;
  • requests are handling as usual;

Deployment errors propagation

All error occurred during service deployment exchange are propagated across the cluster and are available on any node. Current implementation covers the following errors’ causes:

• Errors during assigned nodes definition, e.g. when failed to determine suitable nodes for deploy;
• Deployment errors, e.g. when failed to load service class;
• Service#init errors, e.g. any users failures;