Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The system also have to be horizontally scalable. To achieve scalability, the data will be partitioned using hash or range partitioning method. Each partition is a chunk of parallelism. The exact partitioning method is not important for the purpose of this document. We treat a data partition here as a synonym for a data shard. Each partition is assigned to a cluster node and replicated to a predefined number of replicas to achieve high availability. Adding more nodes increases a number of partitions in the cluster, thus increasing the scalability. A distributed transaction protocol makes cross-partition transactions preserve ACID guaranties 

...

Turns out we want a Google Spanner clone. It seems it was designed keeping the similar goals in mind. Other popular known clones are cockroachdb, yugabyte.

We aim to reuse common replication infrastructure. This means the records will be durably replicated using a consensus based protocol, like RAFT. This approach tolerates f failed nodes from n total nodes, where n >= 2f + 1. Other products can do better, for example FoundationDB tolerates f failed nodes from n, where n >= f + 1. A CC protocol is not tied to the underliying replication protocol.

...