Service Grid redesign. Phase 1. Implementation details.

Overview

All nodes (servers and clients) are able to host services, but the client nodes are excluded from service deployment by default. The only way to deploy service on client nodes is to specify node filter in ServiceConfiguration.

All deployed services are identified internally by “serviceId” (IgniteUuid). This allows us to build a base for such features as hot redeployment and service versioning. It’s important to have an ability to identify and manage services with the same name, but a different version.

Deployment process

Services deployment map exchange - is a process of change deployed services state.

Unified steps of service deployments exchange:

• Each node makes a decision if it should apply actions (deploy/undeploy) on local services and send map of local deployed services to the coordinator;
• Coordinator aggregates single maps and builds and sends full services deployments map to all nodes;
• All nodes update services informations and complete initiators futures if needed;

Triggers:

• Users deploy/undeploy requests;
• Affinity topologies change if affinity services exist;
• Topology change events (EVT_NODE_JOINED / LEFT / FAILED);

Example of user’s service deployment request according to unified flow:

1. Initiator sends DynamicServicesChangeRequestBatchMessage request using discovery spi to change service state [deploy, undeploy]. The request will be stored by all nodes in own queue to be processed, event if the coordinator failed;
2. All nodes process tasks from queue, when deployment request received each node calculates a new service’s assignments independently using deterministic assignments function and apply actions (deploy, undeploy) if needed, then node builds single map message ServicesSingleMapMessage that contains services id and amount of instances were deployed on this single node and sends the message over comm-spi to coordinator (p2p);
3. Once the coordinator receives all single map messages then it builds ServicesFullMapMessage that contains services deployments across the cluster and sends the message over discovery spi to be processed by all nodes;

Messages

class DynamicServicesChangeRequestBatchMessage {

          Collection<DynamicServiceChangeRequest> reqs;

}

class DynamicServiceChangeRequest {

          IgniteUuid srvcId; // Unique service id (generates to deploy, existing used to undeploy)

          ServiceConfiguration cfg; // Empty in case of undeploy

          byte flags; // Change’s types flags [deploy, undeploy, etc.]

}

class ServicesSingleMapMessage {

          ServicesDeploymentExchangeId exchId;

          Map<IgniteUuid, ServiceSingleDeploymentsResults> results;

}

class ServiceSingleDeploymentsResults {

          int cnt; // Deployed instances count, 0 in case of undeploy

          Collection<byte[]> errors; // Serialized exceptions to avoid issues at spi-level

}

class ServicesFullMapMessage  {

          ServicesDeploymentExchangeId exchId;

          Collection<ServiceFullDeploymentsResults> results;

}

class ServiceFullDeploymentsResults {

          IgniteUuid srvcId;

          Map<UUID, ServiceSingleDeploymentsResults> results; // Per node

}

class ServicesDeploymentExchangeId {

          AffinityTopologyVersion topVer;

          IgniteUuid reqId; // Unique id of custom discovery message

}

Coordinator change

All nodes handle requests of service’s state changes and put it into deployment queue, but only the coordinator aggregates single maps messages received from nodes and builds a full map to finish exchange. If coordinator left or fail during deployment exchange another server node will become the coordinator and continue the process.

Topology change

Each topology change (NODE_JOIN/LEFT/FAILED event) causes service's states deployment task. Assignments will be recalculated and applied for each deployed service.

Services reassignment process takes into account previous assignments to avoid redundant redeployments.

Cluster activation/deactivation

• On deactivation:

• • local services are being undeployed;
  • requests are not handling (including deployment / undeployment);

• On activation:

• • local services are being redeployed;
  • requests are handling as usual;

Service deployment failures policy

ServiceDeploymentFailuresPolicy describes rules of handling deployment errors during deployment exchange process. Policy is configurable per service level using ServiceConfiguration.

There were implemented 2 basic policy:

• IGNORE - ignores deployment errors and deploy services as is;
• CANCEL - cancels deployed instances in case of any errors;

Deployment errors propagation

All error occurred during service deployment exchange are propagated across the cluster and are available on any node. Current implementation covers the following errors’ causes:

• Errors during assigned nodes definition, e.g. when failed to determine suitable nodes for deploy;
• Deployment errors, e.g. when failed to load service class;
• Service#init errors, e.g. any users failures;