...

This means that replicas will be created on all nodes of the cluster, and the size of consensus group will be 3. So, if there are 7 nodes, there will be 7 replicas for each partition (3 consensus replicas and 4 learners).

Risks and assumptions

Zone scale up and scale down

Assignments

Now the assignments should be calculated with taking the size of the quorum (and consensus group) into account, using the flags Assignment.forPeer and Assignment.forLearner.

The additional requirement here is that consensus group replicas should be distributed by cluster as evenly as the algorithm allows for all replicas. This is required because the consensus group bears additional load related to providing the data consistency; also, there are group leaders which are chosen from the members of the consensus group.

The assignments calculation for learners via Rendezvous should be simple: Rendezvous algorithm builds the list of nodes for each partition, then takes first N of them (where N is replica factor) as assignments. We should modify it so that it would take the first N nodes as a consensus group (where N is the size of this group), and the next M nodes as learners (where M is learners count).

Distribution zone scale up and scale down

Scale up

On node join, if the node matches the conditions for zone filter and has appropriate storage profile, it may be included into the zone according to the zone scale-up timer. In this case, assignments should be recalculated as if it was the scale-up for a regular zone, so We assume that both consensus replicas and learners should be located with partition distribution algorithm, which should became aware of the replica type (learner or consensus). So, the new replicas will be created on the new nodes on zone scale up, it can be either learners or not, thus the replicas on old nodes can change their type in order to preserve the size of consensus group. On the zone scale down, we may upgrade some learners to consensus replicas to save the number them and reduce the risk of data loss in a case of further node failures. The upgrade is not possible in a case of the quorum loss: this should be handled by disaster recovery due to the risks of split-brain.node. The replicas will be either learners or consensus ones, according to the new assignments: the moving of some consensus replicas to the new node will allow it to take some consensus-related load. This will also launch the rebalancing process to move the data to the new node. It can be also related to upgrading new nodes or learners to consensus replicas and downgrading of some consensus replicas on older nodes to learners, in a case if the new consensus replicas are created on new nodes, and we want to preserve the consensus group size.

Scale down

On node shutdown or failure, the zone scale-down may happen according to the scale-down timer. This will also trigger the assignments recalculation. There are several possibilities (for each partition group):

• Learner(s) left the cluster: no actions required, it is not expected that the location of other consensus or learner replicas will change;
• Consensus replica left the cluster, there are learners, majority quorum is preserved: to maintain the number of consensus replicas in the zone, some learner should be promoted to the consensus replica and included into consensus group;
• Consensus replica left the cluster, there are no learners, but majority quorum is preserved: no actions required;
• Consensus replica left the cluster, there are no learners and majority quorum is lost: the zone should be able to be recovered using the regular disaster recovery procedure;
• Multiple nodes failed at once, consensus replicas left the cluster, there are learners but majority quorum is lost: also requires disaster recovery procedure. Learners cannot be promoted to consensus replicas automatically, because this may lead to split-brain (for example, in cases when “failed” nodes still work but are in the unavailable network segment).

Zone auto-adjust timers

Default timers are: IMMEDIATE for scale-up and INFINITE for scale-down. They should remain the same for the zones with REPLICAS = ALL.

Risks and assumptions

Read-write transactions

No changes of read-write transaction protocol are planned. It continues working as is - through primary replicas only.

Read-only transactions

Read-only transactions should be able to work with learners in the same way as with regular non-primary replicas. Safe time should be propagated to the learners along with any data changes and with idle propagation requests, as long as there is no difference from the point of view of the transaction protocol. So, safe time based reading from learners will work in the same manner as it works with non-primary replicasThis implies that the rebalancing algorithm should be able to create both learners and consensus replicas, and to switch the replica type.

Disaster recovery

In a case of quorum loss, partitions may be reset in the same way as it was designed for regular zones without learners. We will have to ensure this by extending the test coverage for disaster recovery scenarios:

...