Motivation

Apache Ignite provides a good horizontal scalability enabling applications built on top of it to store more data and handle more user requests. But at some scale new requirements of disaster resilence and recovery emerge.

Disaster resilence usually means readiness of a system for a catastrophic scenarios like a complete outage of a datacenter where the system is hosted. And one approach to provide such guarantees is to enable a cluster deployment to span more than one data center.

Though it is possible to stretch a single Ignite cluster into several datacenters even now, such cluster would work inefficiently. The reason is that internal components cannot optimize their functions for local DC usage as they are not aware of different data centers. And this would lead to lower throughput and higher latency of user operations as well as decreased stability of the cluster (at least for ring-shaped cluster topology used by TcpDiscoverySpi).

Right now Ignite 2.x doesn't support MDC deployments in an efficient way and this IEP describes improvements needed to close this gap in functionality.

Description

Support for MDC isn't a separate big feature but rather a set of improvements of existing components allowing Ignite cluster to reach the following set of goals:

1. Minimizing cluster stability impact in MDC environment for ring-shaped topology.
2. Maximazing efficiency of user operations by making internal components aware of MDC configuration.
3. Addressing increased risks of situations like split-brain.

The first improvement is related to Discovery SPI implementation and aims at minimizing cross-DC edges in ring topology thus increasing cluster stability. It enables the subsequent set of improvements.

Improvements of all other components like KV API, SQL, rebalancing and so on could be done in parallel in one or more phases after Discovery SPI improvement is implemented. All necessary changes for each component will be covered in subsequent sections of this document.

On top of modified Discovery SPI could be built more effective mechanisms for detecting and mitigating split-brain situations.

High-level plan

Dicsovery SPI improvements

Modification of TcpDiscoverySpi is needed to fulfill the following requirements:

1. No more than 2 cross-DC connections between two datacenters are allowed in the ring.
2. In order to achive the first requirement a join protocol should be modified to support adding a node at an arbitrary point in the ring in contrast with current implementation (new server node is always added to the very end of the ring).
3. To enable the second requirement new node attribute is introduced labeling the node as belonging to a particular DC (e.g. NODE_DC with String values).
4. TcpDiscoveryNode properties order and internalOrder are reworked to protect higher-level code from changed behavior and to preserve as much internal discovery invariants as possible (to avoid throwing away battle-tested code).

At the same time any improvements to TcpDiscoverySpi should not affect single-DC installations (no additional complexity or overhead is added for an existing use-cases).

Risks and Assumptions

Current approach to this IEP introduces components' mostly internal logic modifications, no public API changes or breaking binary compatibility are needed. Protocols stay the same as well with some internal tweaks and possibly some refactoring.

Discussion Links

[Will be added later]

Tickets

// Links or report with relevant JIRA tickets.