Motivation

Currently Communication protocol doesn't support message exchange with node of another version. For making Rolling upgrade feature possible we must make the protocol compatible between Ignite versions. 

Prerequisites:

1. Communication protocol is peer-to-peer protocol: every message is sent between one node-sender and one node-receiver.
2. Communication protocol is an internal protocol used for system messages exchange

Goal is to design a communication protocol that supports messages exchange between nodes with different Ignite versions.

Prerequisites:

1. It's an internal communication protocol between Ignite nodes:
   1. There is a limited amount of peers, and all peers are aware of versions of each other.
   2. All peers are aware of all possible messages and their schemas.
   3. At most time peers use schemas are of same version. In short period of time (during RU) messags schemas might differ, but not much (few messages might differ with few fields only).
2. It is proposed to provide compatibility between versions that differ by 1 minor version, for example between 2.20.X and 2.19.X (but not between 2.20.X and 2.18.X).
3. Messages can be pretty big, contains cache entries (putAll, historic rebalance).
4. Code changes should be minimal.

Current implementation of Communication protocol is inspired by Avro::

• All peers hold predefined messages schemas.
• Messages follow schema in strict way - no fields are skipped and order Order of fields is guaranteed.
• Peers send → it sends messages as byte stream serializing fields one by one, no delimiters are used between fields and messages.There is API for sending field names for schema resolving

Proposed changes

Message DTO changes

1. Ser/des logic is depends on a remote peer version.
2. Code of ser/des is auto-generated and stored separately from message DTO classes.
3. Add @Since, @Until  annotations for Message classes and fields.

Version check

1. Ignite CI should notify for IF-clauses with condition based on IgniteVersion older than (curVer - 1).

Nice to have (for later research)

1. Optional tagged fields. Such fields is not part of Message schema. The fields can be attached to any message. Cases: securityId, traceId, incremenalIndex, sessionAttributes, etc.
   Current approach - is creating a new message-wrapper (IncrementalSnapshotAwareMessage, TransactionAttributesAwareRequest) that wraps original Message with extra data.
   
   
2. Lazy deserialization/unmarshalling of specific fields. Cases: skip deserializing optional fields, transfer cache entries as byte arrays. It can be achieved by storing these fields as byte array.
3. Compact length of varlen/collections - currently we use 4 bytes (int) to write length of collection or varlen type. In most cases this too much and the length can be encoded with less data (using 1-2 bytes instead).

Communication protocol

Communication protocol consist of 2 parts:

...

Removing annotated entities is allowed after current version is greater than (@Until + 1) or (@Since + 1).

Transport compatibility 

FeatureTable

In case of transport protocol should be changed, Ignite must support -1 Ignite version. 

...

Communication handshake

Handshake algorithm is extended on new step - validating TcpCommunicationConfiguration consistency. Settings that affects both communicating nodes must be same:

...

1. All messages are size delimited
2. Fields order is preserved in default serialization, but there is an optional flexible schema - for adding extra fields or skip sending some fields.
3. Clients and brokers are aware of versions of each other and send messages in the form for specific versions known by each others.
4. There are optional tagged fields beyond a message schema, that can be attached to messages.