...

Apache Ignite provides a good horizontal scalability enabling applications built on top of it to store more data and handle more user requests as applications grow. But at some scale new requirements of disaster resilence and recovery emerge.

Disaster resilence usually means readiness of a system for a catastrophic scenarios like a complete outage of a datacenter where the system is hosted. And one approach to provide such guarantees is to enable a cluster deployment to span more than one data center.

And at some point support for deployment of an application to more than one datacenter is required to ensure disaster resilence and recovery.

Right now Ignite doesn't support 2+ DC deployments effectively though it is not prohibited even in current versions. The reason for that are internal components that are not optimized to perform their functions in MultiDC environment. For instance SQL engine could requiest pieces of data from remote DC even though original request could be fulfilled from a local DC. This leads Though it is possible to stretch a single Ignite cluster into several datacenters even now, such cluster would work inefficiently. The reason is that internal components cannot optimize their functions for local DC usage as they are not aware of different data centers. And this would lead to lower throughput and higher latency of user operations as well as decreased stability of the cluster (at least for ring-shaped cluster topology used by TcpDiscoverySpi).

...

High-level work decomposition

Preparation phase

Introduce new attributes to label Ignite nodes related to different DCs, develop components necessary for operations in MultiDC environment (TopologyValidator, SegmentationResolver with suitable segmentation policy, tooling for managing stretched cluster).

Dicsovery SPI improvements

...

1. No more than 2 cross-DC connections between two datacenters are allowed in the ring.
2. In order to achive the first requirement a join protocol should be modified to support adding a node at an arbitrary point in the ring in contrast with current implementation (new server node is always added to the very end of the ring).
3. To enable the second requirement new node attribute is introduced labeling the node as belonging to a particular DC (e.g. NODE_DC with String values).
4. TcpDiscoveryNode properties order and internalOrder are reworked to protect higher-level code from changed behavior and to preserve as much internal discovery invariants as possible (to avoid throwing away battle-tested code).

At the same time any improvements to TcpDiscoverySpi should not affect single-DC installations (no additional complexity of configuration or overhead is added for an existing use-casesperformance penalty is introduced).

Other components' improvements

...

However proposed improvements may enable more efficient algorithms for detecting and handling split-brain, e.g. fast detecting of a split-brain on the edges of DCs or simpler and more efficient TopologyValidator implementations based on the notion of "Leader/Follower DC". However all these ideas are a matter of discussion and clarification.

Thin client improvements

Thin client has partition-awareness feature. With this feature client connects to all cluster server nodes and tries to find correct node for key and send request to this node (see IEP-23: Partition Awareness for Thin Clients). To achive this client requests primary node map for cache from server. Non partition-awareness requests (not related to cache requests, or without specified key, which can be mapped to node) distributed randomly to all nodes.

In case of multi DC we can make following improvements to PA mechanism:

• PA "write" requests still should be sent to primary nodes (nothing changed), but PA "read" requests (if readFromBackup flag is set and write synchrinization mode is not PRIMARY_SYNC) can be sent to backup node, if node is located in the same DC as client (if there is no partitions in the same DC, request should be sent to primary node).
• Non-PA requests should sent to random node in the same DC as client. 

It's proposed to change "cache partitions" messages (add information about partition in current DC) and introduce new "data center nodes" messages to achive these improvements. Alternatevely, instead of introducing new message we can reuse CLUSTER_GROUP_GET_NODE_IDS message, but in this case client should know about server attribute name to store DC ID (which is server internal information), and everything can be broken if server will change DC ID storage place. 

Protocol changes

Operation codes

The new operation for "data center nodes" request is required:

OP_CLUSTER_GET_DC_NODES message format

OP_CACHE_PARTITIONS message format changes

Risks and Assumptions

Current approach to this IEP introduces components' mostly internal logic modifications, no public API changes or breaking binary compatibility are needed. Protocols stay the same as well with some internal tweaks and possibly some refactoring.

...

[Will be added later]

Tickets