Motivation

Ignite provide many ways to execute user provided code:

• Compute jobs.
• Services.
• SQL UDF.
• CacheInterceptors
• Query filters
• etc.

Currently, Ignite provides several ways to load user-provided code:

• add user jars to Ignite server node classpath.
• Configure DeploymentSpi implementation via IgniteConfiguration#setDeploymentSpi, IgniteConfiguration#setDeploymentMode.
  Monitored by GridDeploymentInfo, UriDeploymentSpiMBean, LocalDeploymentSpiMBean.
• Add dependencies to client node and enable peer to peer class loading.

However, all this ways lack several crucial features:

• Clear way to know which classes loaded from which nodes.
  It expected that all classes will be the same on all nodes, which is not alway true.
• Tools to view jars available.
• Clear way to update user jars.
• Having several versions of user jars: blue-green, canary deployments.
• Remove classes from Ignite process.
• Restrictions (security) for client nodes to provide classes.
  In real world cases not all client nodes can be treated as trusted
  so in many production cluster p2p class loading disabled and forbidden to enable.

We must reinvent the way Ignite loads user classes to fill the gap. 

Description

It proposed to add new entity to Ignite - IgniteClassPath.

IgniteClassPath properties:

• Name - unique string identifier (app_v1, 01_07_2025, etc).
• Status - lifecycle status: CREATING, READY, REMOVING, etc.
• Set of libs and resources associated with IgniteClassPath.
• Version - version of ICP in the form of x.y.z (1.0.3, for example). 
  max version can be referenced as "latest".
• Each instance available on each Ignite node, maybe on demand.
• Can be created and removed runtime - without any restarts.
• Instances independent from each other.

API changes (list will be widen):

• IgniteCache#withClassPath
• IgniteCompute#withClassPath
• ServiceConfiguration#setClassPath
• JDBC driver params to point to classpath used by UDFs and other things during queries execution.
• Deprecation and removal of DeploymentSpi and related configuration properties.
• Deprecation and removal p2p class loading and related configuration properties.
• New security permission to restrict creation/deletion of IgniteClassPath for specific users.
• API to access IgniteClassPath object inside job, service, compute.
  Note, we need to clearly understand cases when regular java API (accessing current ClassLoader) not enough.

Required tools:

• control.sh command to create/remove/list resources.
• control.sh command to register class path from server node resource.
  In strict environments it often prohibited for process to modify it's code (jars).
  So, regular flow with the "Ignite, please deploy files from the local disk to cluster" not working.
  We must provide a way to register and validate IgniteClassPath from server node local disks. 
• system view to list available instances.
• Metrics that contains class loader version: which version of job invoked how many times, how many errors, etc.
• Log, especially error log, must contain classpath name, if applicable.

Examples:

$ ./control.sh --ignite-classpath create --name superapp_v1 --files ./my-app-v1.jar,./grpc-context-1.19.0.jar,./error_prone_annotations-2.1.3.jar
$ ./control.sh --ignite-classpath remove --name superapp_v0
$ ./control.sh --system-view IGNITE_CLASSPATHES

Ignite ignite = ...

IgniteCompute compute = ignite.compute(ignite.cluster().forServers());

// Running compute with the provided jars.
compute.withClasspath("superapp_v1").() -> {
    MyLibClass libClass = MyLibClass.getInstance();
    // other code.
};


ServiceConfiguration serviceCfg = new ServiceConfiguration();

serviceCfg.setName("mysuperappService_v1");
serviceCfg.setService(new MySuperappServiceImpl());
serviceCfg.setIgniteClasspath("mysuperapp_v1");

ignite.services().deploy(serviceCfg);

Other systems approaches:

Apache Spark

Provides a special tool - spark-submit  to deploy and execute jobs inside cluster.
It supports --jars argument to list jar files required by job. (1)

Apache Tomcat (2)

J2EE spec itself provide a clear way to:

• pack dependencies to single WAR file
• update application versions by deploying new WAR file to servlet container.
• Well known directories /WEB-INF/classes, /WEB-INF/libs contains all applcation dependencies that added to classpath.
• Tools to deploy/undeploy application provided, also.

Apache Flink (3) (4)

Provides an options to specify job classpath during starttime.

Example: 

$ ./bin/flink run -C /path/to/dependency.jar -c com.example.MyJob my-flink-job.jar

Design

There are two perspectives for ICP operations: a user perspective and a cluster perspective.

The following operations with ICP must be implemented:

• create - when jars for ICP stored on control.sh running node.
• register - when jars for ICP exists on all server nodes, already. 
• check - for registered ICP, check that 
• safe remove from the cluster. Will wait until all code running on the top of ICP stops.
• unsafe remove. Will forcefully remove ICP regardless of code currently running. 
• get ICP status in the cluster: list of all deployment units in the cluster with their statuses (will be defined below).
• get ICP status on a specific node. The main scenario for this operation is troubleshooting.

From a cluster perspective a deployment unit could be deployed on the cluster, but it could be not deployed on a particular node. So additional operation is required: create ICP on the target node (on demand).

Directories

All ICP must be placed in the ICP base directory which is a subdirectory under Ignite work directory:

<ignite_work_dir>/icp

For each ICP a directory should be created under the base directory. The name of this directory must be the same as ICP name and a nested directory for a particular version must be created. 

Example:

Operations

create

1. Send all jars to choosen server node:
   1. Select random server node - deploy coordinator.
   2. Initiate sending jar files to deploy coordinator.
   3. Deploy coordinator check metastorage for the ICP (name, version) record.
      1. if not exists then creates: (name, version, self_node_id, CREATING).
      2. if exists and node alive then return "already exists" to the user.
      3. if status not CREATING then return "already exists" to the user.
      4. remove existing record and create new: (name, version, self_node_id, CREATING).
   4. Deploy coordinator stores files into some temporary directory.
2. After uploading complete, start the "Upload new ICP" distributed process.
   1. Sent UploadNewICP message by discovery.
   2. On receive UploadNewICP message each server node request ICP files one by one from the deployment coordinator.
      1. File download protocol used to receive file. Same protocol used during snapshot preload.
   3. Send UploadNewICPDone single message on completion or fail.
   4. Deploy coordinator on receiving all single messages completes "Upload new ICP" process with the corresponding status.
3. If "Upload new ICP" succeed then change metastorage record status to READY, otherwise remove it. 
4. Return status of "Upload new ICP" process to the user.  

register

1. Send command to choosen server node
   1. Select random server node - register coordinator.
2. Register coordinator check metastorage for the ICP (name, version) record.
   1. if not exists then creates: (name, version, self_node_id, CREATING).
   2. if exists and node alive then return "already exists" to the user.
   3. if status not CREATING then return "already exists" to the user.
   4. remove existing record and create new: (name, version, self_node_id, CREATING).
3. Register coordinator starts "Register new ICP" distributed process.
   1. Sent RegisterNewICP message by discovery.
   2. On receive RegisterNewICP message each node checks local files.
      1. Get file list. 
      2.  Calculate consistent control sum for each file.
      3. Send RegisterNewICPDone single message on completion or fail.
   3. Register coordinator on receive all single messages checks that all lists and control sums are the same.
      1. OK if true, FAIL otherwise.
4. If "Register new ICP" succeed then change metastorage record status to READY, otherwise remove it. 
5. Return status of "Register new ICP" process to the user. 

check

Invoke same procedure as register but without metastorage registration.

ICP must exists in cluster, already.

safe remove

1. Send command to random server node.
2. Update ICP status in metastorage.
3. Return result to the user.
4. When some code tries to start on top of ICP Ignite must check ICP status and throw exception if status not READY.
5. Each server node must track usage count for ICP:
   1. When some user code starts - increase usageCount.
   2. decrease usageCount on finish.
6. Eventually usageCount will become zero. Node must clean local resources then.

unsafe remove

1. Send command to all server node.
2. Remove ICP status in metastorage.
3. Remove all files from local disk.
4. Return result to the user.

Risks and Assumptions

We assume that p2p and deployment SPI will be removed from Ignite.

Users may want to recompile and refresh classes without changing (remove, create) existing IgniteClassPath during development or hotfix.
Currently, we don't have plans to support this kind of scenario.

We assume the following user scenarios:

Deploy and run first version of application:

1. Pack jar files by regular procedures.
2. Create IgniteClassPath in running cluster.
3. Set name of classpath in app config.
4. Start Ignite client nodes or connect via thin clients.
5. Use classpath when starting jobs, services, etc.

Run second version of application in the same cluster:

1. Pack jar files by regular procedures.
2. Create second IgniteClassPath in running cluster.
3. Restart some instances of user application with the updated version in config.
4. Use updated classpath for new instances of application.
5. When tested update all application instances.

With this approach, we can update applications without any outages.

Discussion Links

// Links to discussions on the devlist, if applicable.

Reference Links

1. https://spark.apache.org/docs/latest/submitting-applications.html
2. https://tomcat.apache.org/tomcat-11.0-doc/deployer-howto.html
3. https://nightlies.apache.org/flink/flink-docs-stable/docs/ops/debugging/debugging_classloading/
4. https://nightlies.apache.org/flink/flink-docs-release-2.2/docs/ops/rest_api/
5. IEP-103: Code Deployment

Tickets