Motivation

Ignite use a few protocols of inter-node message exchange:

1. Communication protocol
2. Discovery protocol

All protocols have own serialization mechanisms and Currently Communication protocol doesn't support message exchange with node of another version. For making Rolling upgrade feature possible we must make the protocol these protocols compatible between Ignite versions. 

Prerequisites

...

1. Communication protocol is peer-to-peer protocol: every message is sent between one node-sender and one node-receiver.
2. Communication protocol is an internal protocol used for system messages exchange between Ignite nodes:
   1. There is a limited amount of peers, and all peers are aware of versions of each other.
   2. All peers are aware of all possible messages and their schemas.
   3. At most time peers are of same version. In short period of time (during RU) messags schemas might differ, but not much (few messages might differ with few fields only).
3. It is proposed to provide compatibility between versions that differ by 1 minor version, for example between 2.20.X and 2.19.X (but not between 2.20.X and 2.18.X).
4. Messages can be pretty big, contains cache entries (putAll, historic rebalance).
5. Users applications deployed on cluster (services, etc) must support compatibility by itself.

Current implementation

...

Communication protocol

...

• All peers hold predefined messages schemas (described with java classes).
• Messages follow schema in strict way - no fields are skipped and order of fields is guaranteed.
• Peers send messages as byte stream serializing fields one by one, no delimiters are used between fields and messages.
• For serializing POJO fields different techniques are used:
  • Message consumes already serialized byte array (e.g. JdkMarshaller for IgniteDiagnosticMessage, SchemaOperationStatusMessage... BinaryMarshaller for GridJobExecuteRequest#jobBytes).
  • User's BinaryObjects, Binarylizable are marshalled to byte array prior to serializing a message (the array is stored in cache structures).
  • Some messages contain POJO field annotated with GridDirectTransient, and additional byte array field fill with GridCacheMessage#prepareMarshal (e.g. BinaryMarshaller for GridChangeGlobalStateMessageResponse).
  • Some messages contains POJO that implements Message interface. 
  GridCacheMessage holds additional field depInfo
  • .
   

Discovery protocol

...

• All peers hold predefined messages schemas (described with java classes).
• Peers use JdkMarshaller (java serialization) to serialize and deserialize full message. Borders between messages are controlled by java serialization.
• Responses with status code or ResponseMessage.

Proposed changes

Message DTO

...

1. Optional tagged fields. Such fields is not part of Message schema. The fields can be attached to any message. Cases: securityId, traceId, incremenalIndex, sessionAttributes, depInfo, etc.
   Current approach - is creating a new message-wrapper (IncrementalSnapshotAwareMessage, TransactionAttributesAwareRequest) that wraps original Message with extra data.
2. Lazy deserialization/unmarshalling of specific fields. Cases: skip deserializing optional fields, transfer cache entries as byte arrays. It can be achieved by storing these fields as byte array.
3. Compact length of varlen/collections - currently we use 4 bytes (int) to write length of collection or varlen type. In most cases this too much and the length can be encoded with less data (using 1-2 bytes instead).

Communication protocol

Communication protocol consist of 2 parts:

...