Ignite-3: .NET Native Compute and future support for other platforms.

Motivation

...

public interface IComputeJob<TArg, TResult>
{
   IMarshaller<TArg>? InputMarshaller => null;
   IMarshaller<TResult>? ResultMarshaller => null;

   ValueTask<TResult> ExecuteAsync(IJobExecutionContext context, TArg arg, CancellationToken cancellationToken);
}

public interface IJobExecutionContext
{
   IIgnite Ignite { get; }
   bool Cancelled { get; }
}

Example job implementation:

public class ToStringJob : IComputeJob<object?, string?>
{
   public async ValueTask<string?> ExecuteAsync(IJobExecutionContext context, object? arg, CancellationToken cancellationToken)
   {
       await context.Ignite.Tables.GetTablesAsync();

       return arg?.ToString();
   }
}

...

• New handshake extension COMPUTE_EXECUTOR_ID
• New feature flag PLATFORM_COMPUTE_JOB (send non-Java jobs to server; all clients are supposed to support this)
• New feature flag PLATFORM_COMPUTE_EXECUTOREXECUTOR (execute jobs requested by server; only .NET client will support this for now)
• New response header flag SERVER_OP_FLAG (indicates a server->client request)
  • Builds on existing notification mechanism
  • Generic, allows different server ops (e.g. start job, cancel job, etc)
• New operation ClientOp.SERVER_OP_RESPONSE (client responds to a request with SERVER_OP_FLAG)
• ServerOp enum

  • COMPUTE_JOB_EXEC

  • COMPUTE_JOB_CANCEL

  • DEPLOYMENT_UNIT_UNDEPLOY

Limitations

No limitations. We expect to be able to run any user code as part of a Compute Job with this approach.

...

None. Client protocol change will be handled via feature flag, preserving backwards and forwards compatibility.

Discussion Links

...

• Dev list: https://lists.apache.org/thread/65xzmcjym9y81rd7tq1j227hbxdbn1h1

Reference Links

• Working POC: https://github.com/apache/ignite-3/pull/5556

Tickets