...
...
A simplified diagram demonstrating the flow of information on a schema update (low-level details (timestamp arithmetics for optimization) are not shown):
In real life, the clocks are skewed, and the network is asynchronous (so message delivery might take an arbitrarily long time). This means that we cannot just use the warmup approach of installing the update to the future and processing transactions on the nodes right away.
Briefly, the idea is to do the following:
We must prevent a node processing a transaction at moment Now before the node has received all the schema updates with activation moment not later than Now. We are going to use Meta-storage for storing the schema updates (note that this implies that we use HLC from now on, not the physical clock that we used in the warm-up section). We are going to read schema updates from the Meta-storage on other nodes. To make sure we never read stale data (breaking AP), we could use the same approach that we use to make reads (in RO) from non-primary replicas: Safe Time.
Message M saying ‘Starting at moment Tu, schema is S’ is sent with timestamp Tm (that moves safeTime forward). A schema storage client that wants to obtain a schema active at moment Top waits for Meta-storage’s safeTime>=Top. Tu must be ≥ Tm (because we’ll construct messages in this way), so, if Tm<Top, after waiting, the client sees M (as Tm<Top<=safeTime, so Tm<safeTime, hence M is visible to the client’s node) and knows that S has been activated at Tu before it obtains schema for moment Top.
This means that AP is maintained. Let’s suppose the opposite. This means that a client obtained a schema for some moment Top>=Tu>=Tm while it was not knowing about M. But as the client had to wait for safeTime>=Top+CSmax, Tmmust have become ≤ than safeTime (even in the presence of clock skew), so the client must have seen message M. We have a contradiction.
To maintain the ‘client sees his own schema changes’ property, we must make the client wait for CSmax after installing an update to make sure that Tu has become non-future on any node, so if the client issues a request to any node, they will see their update.
This approach guarantees correctness (AP is always maintained), but every node (that is not the Meta-storage leader), while trying to obtain a schema for the moment Top=Now, will have to wait for safeTime visible on this node to reach Top, which will take PD (propagation duration). So each transaction will be delayed for PD (on average). This creates a performance problem: latency increases.
Let’s make a couple of modifications.
The downside is that schema change becomes visible only after some time. Even if a message is delivered faster than DD (or we are on the leader), we will still not see the schema until DD passes. We can alleviate this by returning the control to the user after additional time (DD) after the schema change is applied by the user, so the user that created the schema update will not be able to see the system in an inconsistent state (that is, not being able to see the change they have just introduced).
Here is a diagram that exemplifies the mechanism with the cases of 2 nodes: one healthy (that does not need to wait), another lagging (that has to wait).
...