Versions Compared

Old Version 24

changes.mady.by.user Unknown User (nizhikov)

Saved on

compared with

New Version 25

changes.mady.by.user Unknown User (nizhikov)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Let's see how rolling upgrade implemented in other systems:

Other systems

Many distributed open source systems has rolling upgrade feature, already.
We must study implemented approaches to gain some insights from them.

  1. Is there rolling upgrade feature?
  2. How it implemented?
  3. How it tested?
  4. When it tested: Each PR? Weekly? Only on release?
  5. Network message format.
  6. Serdes implementation.
  7. How many earlier releases can be upgraded.

Apache Cassandra

...

Name

Is there rolling upgrade feature?

How it implemented?

How it tested?

When it tested?

Network message format

Serdes implementation

How many earlier releases can be upgraded?

Apache CassandraYes (1), (2)

...

Server-client compatibility works similar to Ignite Thin Client protocol.

Compatibility checked with the special test framework

...

(3)
At a first glance, there are no special source code checks to ensure compatibility in day by day coding

...

On release or by request

...

POJO

...

Custom serdes implementation.

...


Apache Kafka

...

Yes (4)

Message formats checked on PR reivew. 
At a first glance, there are no special source code checks to ensure compatibility in day by day coding.
But, all machinery to code compatible implemented in code generation framework

...

(5)

Compatibility checked with the special test framework ducktest

...

(6)

On release or by request.
Kafka doesn't provide public resources to run ducktests. Run done by contributors or by confluent employers on private hosts

...

POJO

...

Custom serdes implementation.

...

All
Yugabyte

...

Yes (7)

checked on review (see commit message section "Upgrade/Rollback safety"

...

) (8)



plain objectsProtobuf
YDBYes

...

YDB

...

Message formats checked on PR rivew - grpc+protobuf helps to maintain compatibility

...

Compatibility checked with the special test framework

...

(9)

On request.

...

...

plain objectsProtobuf

...


Cockroach DB
  1. Rolling upgrade implemented.
  2. ?
  3. ?
  4. ?
  5. simple data structures.
  6. protobuf.
  7. ?

Hazelcast

  1. Rolling upgrade implemented
  2. ?
  3. ?
  4. ?
  5. ?
  6. ?
  7. ?
Yes




plain objectsProtobuf
HazelcastYes








Description

Rolling Upgrade assumes that cluster can consist of Ignite nodes of different versions.
So Ignite must provide backward compatibility on network level and ability to work in mixed topology.
Development time checks, tests must be added to provide protection of incorrect patches.
Ignite codebase must be reorganized in a way to clearly distinguish those parts that require compatibility and those who don't.

Let's define clearly, what "backward compatibility" for the network messages means:

  • New Ignite server MUST be able to read previous version of message.
  • Old Ignite server MUST be able to read new version of message
    • Newly added fields MUST be ignored.
    • Removed fields MUST have default values. 

Note, there are guide for PDS compatibility, already (11).

Let's list subsystems that must be reworked to provide compatibility:

  • communication: Communication messages consists of two parts:
    • Message format: message format itself. Communication API should be reworked to force backward compatible messages.
    • User data: Message can store user data. User data format must be backward compatible.
  • discovery:
    • Message format: message format itself. Communication
      Discovery API must be reworked to force backward compatible messages.
  • binary marshaller:
    • Currently, binary marshaller code highly coupled with the other Ignite code.
      We must modularize Binary infrastructure (IEP-119 Binary infrastructure modularization10) and provide compatibility guarantees for each part of it.
  • features:
    • Framework to enable/disable features for mixed version clusters must be developed.
  • affinity: 
    • Affinity function must the same for each online node version.
  • management commands
    • arguments
    • results
    • tasks

...

  1. PDS
    1. partition data
    2. WAL records
  2. Metadatametadata
    1. binary meta
    2. marshaller
  3. communication messages
  4. discovery messages
  5. binary marshaller format
  6. affinity results.
  7. Communication SPI
  8. Discovery SPI
  9. Binary Marshaller
  10. Affinity.
  11. Management APImanagement commands
    1. argument 
    2. results
    3. tasks (class names).
  12. ThinClient Protocol

Current Ignite codebase

  • Is there any primitives, building blocks to provide compatibility?
  • Difficulties for day by day coding.
  • Compatibility testing.
    • explicit tests.
    • ability to run tests with random node versions
  • New feature implementation, enabling.
  • patterns for implementing commons cases: new version of algorithmes, testing against new versions.
  • Scope of compatibility: 
    • Currently any third-party module can register own ports (messages?) and must be able to track compatibility. 

Code cleanup

  • remove DirectByteBufferStreamImpl  V1-V3, keep V4, only.
  • remove all items from IgniteFeatures  and corresponding checks.
  • remove all code and checks for GridContinuousProcessor#discoProtoVer
  • TcpDiscoverySpi#setForceServerMode
  • All old version of classes - keep only max from V2, V3, etc. versions. StartRequestV2 that belongs to internal communication.
    Classes releated to thin client, jdbc, odbc interaction must stay.
    • StartRequest must be deleted. Rename StartRequestV2 → StartRequest.
    • CacheMetricsSnapshot must be deleted. Rename CacheMetricsSnapshotV2 → CacheMetricsSnapshot.
  • MessageFactory

...

  • subsystem API versions (like in REST API)
  • runtime component(IgniteProcessor, IgniteManager) upgrade with the dynamic class loading. 

Reference Links

  1. https://www.datastax.com/learn/whats-new-for-cassandra-4/migrating-cassandra-4x
  2. https://docs.datastax.com/en/luna-cassandra/guides/upgrade/overview.html
  3. https://github.com/apache/cassandra-dtest/blob/trunk/upgrade_tests/README.md
  4. https://kafka.apache.org/documentation/#upgrade
  5. https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/AlterPartitionResponse.json
  6. https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/core/kraft_upgrade_test.py
  7. https://docs.yugabyte.com/preview/manage/upgrade-deployment/
  8. https://github.com/yugabyte/yugabyte-db/commit/e9ab17dea0d3b4f1673531c07404a290f9fbd8f2
  9. https://github.com/ydb-platform/ydb/blob/main/ydb/tests/functional/restarts/
  10. IEP-119 Binary infrastructure modularization
  11. PDS Compatibility Guide (WIP)