Goal is a communication protocol that able to work between nodes with different Ignite versions.

It is proposed to work between versions that differ by 1 minor version, for example 2.20.X between 2.19.X (but not between 2.20.X and 2.18.X).

Communication protocol

Communication protocol consist of 2 parts:

1. Data - set of declared Message classes, including ser/des algorithm.
2. Transport - algorithm of transport the Messages to remote node.

Data compatibility

Message - is base class for all messages transported between nodes. Proposed changes:

1. Message#writeTo consumes IgniteProductVersion of destination node and use it for serializing data for this version (mostly, for ignoring some fields).

2. Message#readFrom consumes IgniteProductVersion of source node and use it for deserializing data (mostly, for setting default values of new fields).

   public interface Message {    public boolean writeTo(ByteBuffer buf, MessageWriter writer, IgniteProductVersion destVer);     public boolean readFrom(ByteBuffer buf, MessageReader reader, IgniteProductVersion srcVer);     public short directType(); }

   
   

3. Introduce annotations @Since and @Until for Message classes and Message fields, to use it for generating code for Message#writeTo and Message#readFrom:
   
   

@Since(version = "2.19.0")
public class MyMessage {
    private int id;
    
    @Until(version = "2.20.0")
    private String rmFld;

    @Since(version = "2.20.0")
    private String newFld;

    @Override public boolean writeTo(ByteBuffer buf, MessageWriter writer, IgniteProductVersion destVer) {
        if (destVer.lessThan(2, 19, 0))
            throw new IgniteException("Must not send the message to destination node");

        if (!writer.writeString(id))
            return false;

        if (destVer.lessThan(2, 20, 0)) {
            if (!writer.writeString(rmFld))
                return false;
        }

        if (destVer.greaterThanEqual(2, 20, 0)) {
            if (!writer.writeString(newFld))
                return false;
        }

        return true;
    }

    @Override public boolean readFrom(ByteBuffer buf, MessageReader reader, IgniteProductVersion srcVer) {
        id = reader.readString();

        if (srcVer.greaterThanEqual(2, 20, 0))
            newFld = reader.readString();
        else
            newFld = null;

        return true;
    }
}

Introduce Communication Protocol Version (ProtoVer) that depends on changes in the parts:

• Version is Major.Minor, initial 1.0.
• Minor version is upgraded by adding new fields in any Message. Version is updated along with a commit that introduce a change.
• Major version is upgraded by breaking changes in protocol (changes in handshake or ser/des algorithm, adding new Message).

Rules to support compatibility:

1. It's prohibited to remove fields, change types of fields in Messages. Only adding new fields is allowed.
2. Breaking changes are introduced as communication features (FeatureTable).
   1. Every communication feature must be declared with @since Ignite version.
   2. Ignite release must disable a communication feature with Ignite node with (@since - 1) version. 
   3. On release Ignite should check the @since version and notify release manager to drop support of old versions.

ProtoVer is fixed for Ignite release version. Nodes exchange theirs ProtoVer, FeatureMask on joining node on discovery:

1. If ProtoVer.Major is different then FeatureMask is validated. Let the cluster version V, the joining node version V+2. If the joining node mask contains features since V + 1, then this node must not be join the cluster.
2. If ProtoVer.Minor is differrent then enable Rolling upgrade mode for communication protocol between nodes with different versions.
3. Otherwise communication protocol runs in Stable mode.

Communication handshake

Handshake algorithm is extended on new step - validating TcpCommunicationConfiguration consistency. Settings that affects both communicating nodes must be same:

1. usePairedConnections
2. connectionsPerNode

Stable mode

In Stable mode logic of exchanging messages is not changed. Between nodes open channel, messages are written to the channel one by one as byte stream. There is no delimeters between messages, each message starts with direct type. Reader knows reads full message with Message#readFrom logic.

Rolling upgrade mode 

In Rolling upgrade mode logic is different, because node can't know whether it reads full message - Message#readFrom can't guarantee that.

• Sender splits every message on frames. One frame cannot contain more than one message.
• Frame header contains: 
  • continuation bit - flag that shows whether this frame is a part of previous message, or start of new message.
  • frame size - can be default for varlen messages, or defined by Message class for fixed sized messages.
• Message header contains direct type and fields count:
  
  • Let receiver expects N fields, but sender sends (N-1) fields. This field protect of reading random bytes to the Nth field.
• After message is fully written, writer fills remaining frame buffer with zeros. 
• Reader reads message from frames util Message#readFrom returns true or fields count is achieved. Then the message is processed.
  • After that, reader reads to /dev/null until continuation bit is unset.