Page History

Overview

All nodes (servers and clients) are able to host services, but the client nodes are excluded from service deployment by default. The only way to deploy service on client nodes is to specify node filter in ServiceConfiguration.

All deployed services are identified internally by “serviceId” (IgniteUuid). This allows us to build a base for such features as hot redeployment and service versioning. It’s important to have an ability to identify and manage services with the same name, but a different version.

Deployment process

Services Service deployment map exchange - is a process of change deployed services state.

Unified steps of service deployments exchange:

...

is managed via DiscoverySpi and CommunicationSpi messages.

Users requests (deploy/undeploy)

...

are represented as discovery custom message ServiceChangeBatchRequest which contains collection of actions intended changing services states. The action has to extend ServiceChangeAbstractRequest.

The requests are sent via DiscoverySpi, it allows us to get following guarantees:

• The request will be received by all the nodes in the topology. This allows continuing processing request if the coordinator failed and to obtain all needed metadata by one message to handle the request (service configuration, id, class etc.);

• A strict order of requests, to be able to validate request in case of duplication or conflicts in services configuration;

• The relation (order) between user requests and exchange of node joining process (which is also managed via DiscoverySpi). This allows sending a complete set of existing services metadata (also waiting for deployment) to joining node and not to lose any metadata (IgniteServiceProcessor#registeredServices);

Once the request is received, it is stored in a deployment queue as ServiceDeploymentTask to be processed in a separate thread because of requiring significant time. That means each request will be processed in order of queue.

The deployment queue is managed by ServiceDeploymentManager. Deployment worker (special thread) takes from the queue a deployment task and calls ServiceDeploymentTask#init to start the deployment process:

• The task performs actions of changing services states. In case of deploy request, each node calculates assignments independently using a deterministic function (IgniteServiceProcessor#reassign);

• Deployment results are represented as communication message ServiceSingleNodeDeploymentResultBatch, which is sent to coordinator via CommunicationSpi (p2p) once actions were performed. The message contains deployments errors and count of locally deployed instances of services related to the current deployment process;

• The coordinator aggregates the deployment results from the cluster. The result of the whole deployment process is represented as discovery custom message ServiceClusterDeploymentResultBatch which is built and sent to all nodes via DiscoverySpi once all single nodes results were received.

• Each node handles ServiceClusterDeploymentResultBatch, updates deployments information and complete initiators futures if needed, then finishes the deployment process;

The following events cause deployment process:

• Users deploy/undeploy requests;

• Affinity topologies change if affinity services exist;

Topology change events (EVT_NODE_JOINED/LEFT/FAILED);

Topology/coordinator change

Each topology change event

Triggers:

• Users deploy/undeploy requests;
• Affinity topologies change if affinity services exist;
• Topology change events (EVT_NODE_JOINED / LEFT / FAILED);

Example of user’s service deployment request according to unified flow:

1. Initiator sends DynamicServicesChangeRequestBatchMessage request using discovery spi to change service state [deploy, undeploy] . The request will be stored by all nodes in own queue to be processed, event if the coordinator failed;
2. All nodes process tasks from queue, when deployment request received each node calculates a new service’s assignments independently using deterministic assignments function and apply actions (deploy, undeploy) if needed, then node builds single map message ServicesSingleMapMessage that contains services id and amount of instances were deployed on this single node and sends the message over comm-spi to coordinator (p2p);
3. Once the coordinator receives all single map messages then it builds ServicesFullMapMessage that contains services deployments across the cluster and sends the message over discovery spi to be processed by all nodes;

Messages

class DynamicServicesChangeRequestBatchMessage {

          Collection<DynamicServiceChangeRequest> reqs;

}

class DynamicServiceChangeRequest {

          IgniteUuid srvcId; // Unique service id (generates to deploy, existing used to undeploy)

          ServiceConfiguration cfg; // Empty in case of undeploy

          byte flags; // Change’s types flags [deploy, undeploy, etc.]

}

class ServicesSingleMapMessage {

          ServicesDeploymentExchangeId exchId;

          Map<IgniteUuid, ServiceSingleDeploymentsResults> results;

}

class ServiceSingleDeploymentsResults {

          int cnt; // Deployed instances count, 0 in case of undeploy

          Collection<byte[]> errors; // Serialized exceptions to avoid issues at spi-level

}

class ServicesFullMapMessage  {

          ServicesDeploymentExchangeId exchId;

          Collection<ServiceFullDeploymentsResults> results;

}

class ServiceFullDeploymentsResults {

          IgniteUuid srvcId;

          Map<UUID, ServiceSingleDeploymentsResults> results; // Per node

}

class ServicesDeploymentExchangeId {

          AffinityTopologyVersion topVer;

          IgniteUuid reqId; // Unique id of custom discovery message

}

Coordinator change

All server nodes handle requests of service’s state changes and put it into deployment queue, but only coordinator process them. If coordinator left or fail they will be processed on a new coordinator.

Topology change

Each topology change (NODE_JOIN/LEFT/FAILED event) causes service's states deployment  deployment task. Assignments will be recalculated and applied for each deployed service if needed.

Services reassignment process takes into account previous assignments to avoid redundant redeployment.

If left/failed node had not deployed any services, then a deployment task will be finished without sending messages.

If a coordinator was changed during the service deployment process then all nodes send of ServiceSingleNodeDeploymentResultBatch to a new coordinator which continues the process as usual.

Cluster activation/deactivation

• On deactivation:

• • local services are being undeployed;
  • requests are not handling (including deployment / undeployment);

...

• • local services are being redeployed;
  • requests are handling as usual;

Service deployment failures policy

ServiceDeploymentFailuresPolicy describes rules of handling deployment errors during deployment exchange process. Policy is configurable per service level using ServiceConfiguration.

There were implemented 2 basic policy:

•  IGNORE - ignores deployment errors and deploy services as is;
• CANCEL - cancels deployed instances in case of any errors;

Deployment errors propagation

All error occurred during service deployment exchange are propagated across the cluster and are available on any node. Current implementation covers the following errors’ causes:

• errors Errors during assigned nodes definition, e.g. when failed to determine suitable nodes for deploy;
• deployment Deployment errors, e.g. when failed to load service class;
• Service#init errors, e.g. any users failures.;