Motivation

Replicated zones are zones with partitions that have replicas on every node of the cluster that matches the zone filter and storage profile filter. These zones can provide better performance on SQL joins. 

Currently in Apache Ignite 3, we have an ability to set any number of replicas for any zone regardless of its initial replica number. The problem is that in large clusters, if we create a zone with a number of replicas that would be equal to the cluster size, there will be too many members in each replication group to maintain consensus. This will have a serious impact on performance. Moreover, we don’t have automatic scale up for this case: if the user decides to scale up their cluster, they would need to change the replica number manually (unless it is big enough, exceeding the nodes number).

Therefore, we need special options for zones to create members of replication groups that would not be a part of consensus, and to be able to scale the zones along with the cluster.

Description

Requirements

• user should be able to create a zone with partitions that have replicas on every node of the cluster that matches the zone filter and storage profiles;
• replicas should be operational in a case when they are not included into the consensus group, let’s call them learners. Learners are created on every node that matches the zone filters where regular replica is not created;
• read-write transactions work as on regular zones (i.e. through primary replicas);
• read-only transactions and SQL within read-only transactions use learners as regular non-primary replicas;
• filters and storage profiles for the zones having learners should work in the same way as for regular zones;
• auto scaling: if new nodes are included into the cluster, learners are created there automatically (if the nodes match the zone filters).

Proposal

Definitions

Replication group - full set of replicas of a partition.

Consensus group - subset of replicas of a partition that maintains the data consistency in the replication group. For example, in the case of Raft it is the set of voting members.

Consensus replicas - replicas that form the consensus group.

Learners - replicas of a replication group that are not included into the consensus group.

Quorum - the minimal subset of replicas in the consensus group that is required for it to be fully operational and maintain the data consistency, in the case of Raft it is the majority of voting members.

Data node - a node of the cluster that is included into the distribution zone, matches its filters and storage profiles, and therefore, is used for storing the data and replicas are created there.

Changes of distribution zone parameters

REPLICAS

This is an existing parameter of integer type that specifies the constant size of a replication group. The proposal is to add one more acceptable value for this parameter: ALL, meaning that replicas should be created on every data node of the zone.

QUORUM_SIZE

This is the new parameter that is proposed here. It specifies the size of the majority quorum. In fact, the size of consensus group can be derived from it, depending on the replication algorithm, in the case of Raft it is calculated as QUORUM_SIZE * 2 - 1, so the quorum size is the size of the majority of nodes in the consensus of a replication group. This implies that the size of the consensus group will be an odd number, if there are enough replicas.

The QUORUM_SIZE parameter may be set with any sufficient number of replicas, either ALL or not. This means that we can have, for example, 10 data nodes, 7 replicas and quorum size 3, meaning that 5 replicas will form the consensus group and 2 will be learners.

The default value should be

• If there are 4 or less data nodes: min(2, data_nodes_count);
• If there are at least 5 data nodes: 3.

Lower and upper boundaries:

• Lower: 1 if there is only one replica and 2 if there is more than 1 node. Having the quorum of 1 node where there are more replicas makes no sense and decreases reliability;
• Upper: no less than lower bound, but making the consensus group fit into the configured replicas count.

Quorum size parameter can conflict with incorrect replicas count and insufficient data nodes count. We should throw an exception in this case to not create a group without a majority. Also, there should be special error codes for each case.

Quorum is chosen because it is quite straightforward for the user: if they are able to keep QUORUM_SIZE nodes in the zone, there should be no data loss, unless they lose multiple nodes simultaneously so that the quorum would be not able to transfer to nodes that remain online.

Also, many widely-used replication algorithms with strict consistency are based on consensus algorithms that require majority quorum, which has size floor(nodes / 2) + 1. For example, Raft [1] and Zookeeper [2] use majority quorum to replicate the data. Membership-based protocols, such as Hermes [3], may not use majority for replication but need majority-based protocol to maintain the consistent view of the cluster topology on every node. So, the size of the consensus group can be calculated using the quorum size.

SQL API changes

The aforementioned parameters of distribution zones should be accessible via SQL commands CREATE ZONE and ALTER ZONE.

QUORUM_SIZE should be added as an optional parameter of these commands. It should be of type int.

For example:

CREATE ZONE ‘MY_ZONE’ WITH REPLICAS = ALL, QUORUM_SIZE = 2

This means that replicas will be created on all nodes of the cluster, and the size of consensus group will be 3. So, if there are 7 nodes, there will be 7 replicas for each partition (3 consensus replicas and 4 learners).

Risks and assumptions

Zone scale up and scale down

We assume that both consensus replicas and learners should be located with partition distribution algorithm, which should became aware of the replica type (learner or consensus). So, the new replicas will be created on the new nodes on zone scale up, it can be either learners or not, thus the replicas on old nodes can change their type in order to preserve the size of consensus group. On the zone scale down, we may upgrade some learners to consensus replicas to save the number them and reduce the risk of data loss in a case of further node failures. The upgrade is not possible in a case of the quorum loss: this should be handled by disaster recovery due to the risks of split-brain.

This implies that the rebalancing algorithm should be able to create both learners and consensus replicas, and to switch the replica type.

Disaster recovery

In a case of quorum loss, partitions may be reset in the same way as it was designed for regular zones without learners. We will have to ensure this by extending the test coverage for disaster recovery scenarios:

• if the quorum is lost, and there is no ability to turn a learner into consensus replica (they may be offline, outdated, etc.), then disaster recovery should work as for regular zones without learners;
• the case when there are less nodes than configured quorum size, meaning that the configured value of quorum size will be overridden.

Reference links

[1] https://raft.github.io/raft.pdf

[2] https://zookeeper.apache.org/doc/current/zookeeperInternals.html#sc_quorum

[3] Hermes protocol https://arxiv.org/pdf/2001.09804 (2.4, 3.4)

Tickets

IGNITE-23790 - Getting issue details... STATUS