Compatibility for Communication protocol

Communication protocol consist of 2 parts:

  1. Data - set of declared Message classes, including ser/des algorithm.
  2. Transport - algorithm of transport the Messages to remote node.

Introduce Communication Protocol Version (ProtoVer) that depends on changes in the parts:

Rules to support compatibility:

  1. It's prohibited to remove fields, change types of fields in Messages. Only adding new fields is allowed.
  2. Breaking changes are introduced as communication features (FeatureMask).
    1. Every communication feature must be declared with @since Ignite version.
    2. Ignite release must disable a communication feature with Ignite node with (@since - 1) version. 
    3. On release Ignite should check the @since version and notify release manager to drop support of old versions.

ProtoVer is fixed for Ignite release version. Nodes exchange theirs ProtoVer, FeatureMask on joining node on discovery:

  1. If ProtoVer.Major is different then FeatureMask is validated. Let the cluster version V, the joining node version V+2. If the joining node mask contains features since V + 1, then this node must not be join the cluster.
  2. If ProtoVer.Minor is differrent then enable Rolling upgrade mode for communication protocol between nodes with different versions.
  3. Otherwise communication protocol runs in Stable mode.

Communication handshake

Handshake algorithm is extended on new step - validating TcpCommunicationConfiguration consistency. Settings that affects both communicating nodes must be same:

  1. usePairedConnections
  2. connectionsPerNode

Stable mode

In Stable mode logic of exchanging messages is not changed. Between nodes open channel, messages are written to the channel one by one as byte stream. There is no delimeters between messages, each message starts with direct type. Reader knows reads full message with Message#readFrom logic.

Message headerMessage

2 bytes: direct type

fields

Rolling upgrade mode 

In Rolling upgrade mode logic is different, because node can't know whether it reads full message - Message#readFrom can't guarantee that.

Frame headerMessage headerMessage

1 byte:

1bit - continuation bit

2-7bit - reserved

2 bytes:

unsigned frame size

(frame size limit 64Kb)

2 bytes:

direct type

1 byte:

fields count

(max 255 fields)

fieldstrailing zeros