Communication protocol

Goal is a communication protocol that able to work between nodes with different Ignite versions.

Prerequisites:

1. It is proposed to provide compatibility between versions that differ by 1 minor version, for example between 2.20.X and 2.19.X (but not between 2.20.X and 2.18.X).
2. Peers are aware of messages schemas (Message class). Older versions aren't aware of new fields only.
3. Schema changes are rare.
4. Code changes should be minimal.

Communication protocol

Communication protocol consist of 2 parts:

1. Data - set of declared Message classes, including ser/des algorithm.
2. Transport - algorithm of transport the Messages to remote node.

Data compatibility

Message - is base class for all messages transported between nodes. Proposed changes:

1. Message#writeTo consumes IgniteProductVersion of destination node and use it for serializing data for this version (mostly, for ignoring some fields).

2. Message#readFrom consumes IgniteProductVersion of source node and use it for deserializing data (mostly, for setting default values of new fields).

   Message

   public interface Message {    
       public boolean writeTo(ByteBuffer buf, MessageWriter writer, IgniteProductVersion destVer);
   
       public boolean readFrom(ByteBuffer buf, MessageReader reader, IgniteProductVersion srcVer);
   
       public short directType();
   }

3. Introduce annotations @Since and @Until for Message classes and Message fields, to use it for generating code for Message#writeTo and Message#readFrom:
   
   

@Since(version = "2.19.0")
public class MyMessage {
    private int id;
    
    @Until(version = "2.20.0")
    private String rmFld;

    @Since(version = "2.20.0")
    private String newFld;

    // Generated code from the schema ^.

    @Override public boolean writeTo(ByteBuffer buf, MessageWriter writer, IgniteProductVersion destVer) {
        if (destVer.lessThan(2, 19, 0))
            throw new IgniteException("Must not send the message to destination node");

        if (!writer.writeString(id))
            return false;

        if (destVer.lessThan(2, 20, 0)) {
            if (!writer.writeString(rmFld))
                return false;
        }

        if (destVer.greaterThanEqual(2, 20, 0)) {
            if (!writer.writeString(newFld))
                return false;
        }

        return true;
    }

    @Override public boolean readFrom(ByteBuffer buf, MessageReader reader, IgniteProductVersion srcVer) {
        id = reader.readString();

        if (srcVer.greaterThanEqual(2, 20, 0))
            newFld = reader.readString();
        else
            newFld = null;

        return true;
    }
}

Rules to describe Message (must be automated and validated):

1. Do not remove Message class or Message fields, but annotate it with @Until. 
2. Do not change types or order of fields.
3. New fields must be annotated with @Since. All such fields must be optional, default value is null. Handling the nulls is care of Message consumer on the reader side.

Removing annotated entities is allowed after current version is greater than (@Until + 1) or (@Since + 1).

Transport compatibility 

FeatureTable

In case of transport protocol should be changed, Ignite must support -1 Ignite version. 

1. It's proposed to add FeatureTable class that used to track changes in transport protocol. Each entry must be annotated with @Since version.
2. Ignite release must disable a communication feature with Ignite node with (@Since - 1) version. 
3. On release Ignite should check the @Since version and notify release manager to drop support of old versions.

Communication handshake

Handshake algorithm is extended on new step - validating TcpCommunicationConfiguration consistency. Settings that affects both communicating nodes must be same:

1. usePairedConnections
2. connectionsPerNode

Marshaller compatibility

1. JdkMarshaller - current version RU does not support work with different Java versions. TODO: hide jdkMarshaller?
2. BinaryMarshaller - backward compatibility is guaranteed. Require API for getting marshaller for specific version.

Links

Protobuf

https://protobuf.dev/programming-guides/encoding/

1. Field numbers are serialized with data:
   1. Do not send null fields, ignore unknown fields.
   2. Order of fields isn't guaranteed: message can be concatenations of same fields in different order. For optimizations (compression)?
   3. Make possible easily change schema (but user must preserve field ids).
   4. Serializer must know len of varlen fields (Messages). 
2. It can be worked in streaming way using CodedOutputStream. To customize serialization, but it still requires len for Messages be written before.
3. It allows deserialize only required fields using CodedInputStream. It requires writing a code for iterating over tags and skipping fields. 

FlatBuffers

https://flatbuffers.dev/internals/

1. Offsets are serialized with data:
   1. Order of fields is not guaranteed - for optimization like compaction.
   2. First 4bytes - offset to the root of vtable, that stores offsets to other fields. Vtable can be anywhere relative to fields.
   3. Size of data must be know before serializing to prepare the vtable.
   4. It starts serializing from nested objects, calculate it sizes and fill tables, and then write root object.
2. No streaming is possible.

Avro

Bson