...

• communication: Communication messages consists of two parts:
  
  • Message format: message format itself. Communication API should be reworked to force backward compatible messages.
  • User data: Message can store user data. User data format must be backward compatible.
• discovery:
  • Message format: message format itself. Communication API must be reworked to force backward compatible messages.
• binary marshaller:
  • Currently, binary marshaller code highly coupled with the other Ignite code. We must modularize Binary infrastructure (IEP-119 Binary infrastructure modularization) and provide compatibility guarantees for each part of it.
• features:
  • Framework to enable/disable features for mixed version clusters must be developed.
• affinity: 
  • Affinity function must the same for each online node version.

Current Ignite codebase

• Is there any primitives, building blocks to provide compatibility?
• Difficulties for day by day coding.
• Compatibility testing.
• New feature implementation, enabling.

Additional materials

Many distributed open source systems has rolling upgrade feature, already.
We must study implemented approaches to gain some insights from them.

...

1. Is there rolling upgrade feature?
2. How it implemented?
3. How it tested?
4. When it tested: Each PR? Weekly? Only on release?
5. Network message format.
6. Serdes implementation implementation.
7. How many earlier releases can be upgraded.

Apache Cassandra

1. Rolling upgrade implemented. Guide -
   1. https://www.datastax.com/learn/whats-new-for-cassandra-4/migrating-cassandra-4x
   2. https://docs.datastax.com/en/luna-cassandra/guides/upgrade/overview.html
2. Server-client compatibility works similar to Ignite Thin Client protocol.
3. Compatibility checked with the special test framework - https://github.com/apache/cassandra-dtest/blob/trunk/upgrade_tests/README.md
   At a first glance, there are no special source code checks to ensure compatibility in day by day coding.
4. On release or by request.
5. POJO
6. Internal serdes.
7. ?

Apache Kafka

1. Rolling upgrade implemented. Guide - https://kafka.apache.org/documentation/#upgrade
2. Message formats checked on PR reivew. 
   At a first glance, there are no special source code checks to ensure compatibility in day by day coding.
   But, all machinery to code compatible implemented in code generation framework - https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/AlterPartitionResponse.json
3. Compatibility checked with the special test framework ducktest - https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/kraft_upgrade_test.py
4. On release or by request.
   Kafka doesn't provide public resources to run ducktests. Run done by contributors or by confluent employers on private hosts. 
5. POJO
6. Internal autogenerated serdes.
7. ?

Yugabyte

1. Rolling upgrade implemented. Doc - https://docs.yugabyte.com/preview/manage/upgrade-deployment/
2. It seems compatibility checked on review (see commit message section "Upgrade/Rollback safety") - https://github.com/yugabyte/yugabyte-db/commit/e9ab17dea0d3b4f1673531c07404a290f9fbd8f2
3. ???
4. ???
5. simple data structures.
6. protobuf.
7. ?

YDB

1. Rolling upgrade implemented.
2. Message formats checked on PR rivew - grpc+protobuf helps to maintain compatibility.
3. Compatibility checked with the special test framework - https://github.com/ydb-platform/ydb/blob/main/ydb/tests/functional/restarts/
4. On request. 
5. simple data structures
6. protobuf
7. ?

Cockroach DB

1. Rolling upgrade implemented.
2. g?
3. ?
4. ?
5. simple data structures.
6. protobuf.
7. ?

Alternative designs

• subsystem API versions (like in REST API)
• runtime component(IgniteProcessor, IgniteManager) upgrade with the dynamic class loading.
• ability to run tests with random node versions
• patterns for implementing commons cases: new version of algorithmes, testing against new versions.