Motivation

For now, Ignite cluster can be updated only by full cluster restart.

Procedure must be the following:

1. stop all cluster nodes.
2. update files on each node.
3. start nodes one by one.

So, each version update is a cluster temporary unavailability.

This is extremely inconvenient for the user, especially one who use Ignite as a primary data storage.

We must add rolling upgrade feature to the Ignite.

Description

Rolling Upgrade assume that cluster can consist of Ignite versions of different versions.
So Ignite must provide backward compatibility on network level and ability to work in mixed topology.
Development time checks, tests must be added to provide protection of incorrect patches.
Ignite codebase must be reorganized in a way to clearly distinguish those parts that require compatibility and those who don't.

Let's define clearly, what "backward compatibility" for the network messages means:

• New Ignite server MUST be able to read previous version of message.
• Old Ignite server MUST be able to read new version of message
  • Newly added fields MUST be ignored.
  • Removed fields MUST have default values. 

Let's list subsystems that must be reworked to provide compatibility:

• communication: Communication messages consists of two parts:
  
  • Message format: message format itself. Communication API should be reworked to force backward compatible messages.
  • User data: Message can store user data. User data format must be backward compatible.
• discovery:
  • Message format: message format itself. Communication API must be reworked to force backward compatible messages.
• binary marshaller:
  • Currently, binary marshaller code highly coupled with the other Ignite code. We must modularize Binary infrastructure (IEP-119 Binary infrastructure modularization) and provide compatibility guarantees for each part of it.
• features:
  • Framework to enable/disable features for mixed version clusters must be developed.
• affinity: 
  • Affinity function must the same for each online node version.

Additional materials

Many distributed open source systems has rolling upgrade feature, already.
We must study implemented approaches to gain some insights from them.

1. Is there rolling upgrade feature?
2. How it implemented?
3. How it tested?
4. When it tested: Each PR? Weekly? Only on release? 

Apache Cassandra

1. Rolling upgrade implemented. Guide -
   1. https://www.datastax.com/learn/whats-new-for-cassandra-4/migrating-cassandra-4x
   2. https://docs.datastax.com/en/luna-cassandra/guides/upgrade/overview.html
2. Server-client compatibility works similar to Ignite Thin Client protocol.
3. Compatibility checked with the special test framework - https://github.com/apache/cassandra-dtest/blob/trunk/upgrade_tests/README.md
   At a first glance, there are no special source code checks to ensure compatibility in day by day coding.
4. ??? 

Apache Kafka

1. Rolling upgrade implemented. Guide - https://kafka.apache.org/documentation/#upgrade
2. Message formats checked on PR reivew. 
   At a first glance, there are no special source code checks to ensure compatibility in day by day coding.
   But, all machinery to code compatible implemented in code generation framework - https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/AlterPartitionResponse.json
3. Compatibility checked with the special test framework ducktest - https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/kraft_upgrade_test.py
4. On release, or by request. Kafka doesn't provide public resources to run ducktests. Run done by contributors or by confluent employers on private hosts. 

Yugabyte

Cockroach DB

Alternative designs

• subsystem API versions (like in REST API)
• runtime component(IgniteProcessor, IgniteManager) upgrade with the dynamic class loading.
• ability to run tests with random node versions
• patterns for implementing commons cases: new version of algorithmes, testing against new versions.