...
A schema update U is visible to a node N at moment Top if the node N got a message M containing this update at a moment T<=Top. (Schema update messages are distributed using Meta - Storage RAFT group (so a message consumption is an application of it to a state machine), so the update messages are delivered in order (hence, if a message is not delivered, none of the messages that follow it will be delivered) and not duplicated)). This is due to the fact that there is exactly one Meta - Storage RAFT state machine at each node.
...
A simplified diagram demonstrating the flow of information on a schema update (low-level details (timestamp arithmetics for optimization) are not shown):
The clocks may be skewed In real life, the clocks are skewed, and the network is asynchronous (so message delivery might take an arbitrarily long time). This means that we cannot just use the warmup naive approach of installing the update to the future and processing transactions on the nodes right away.
Briefly, the idea is to do the following:
...
We must prevent a node processing a transaction at moment Now before the node has received all the schema updates with activation moment not later than Now. We are going to use Meta -storage Storage for storing the schema updates (note that this implies that we use HLC from now on, not the physical clock that we used in the warm-up section). We are going to read schema updates from the Meta -storage Storage on other nodes. To make sure we never read stale data (breaking AP), we could use the same approach that we use to make reads (in RO) from non-primary replicas: Safe Time.
Message M saying ‘Starting at moment Tu, schema is S’ is sent with timestamp Tm (that moves safeTime forward). A schema storage client that wants to obtain a schema active at moment Top waits for Meta -storage’s Storage’s safeTime>=Top. Tu must be ≥ Tm (because we’ll construct messages in this way), so, if Tm<Top, after waiting, the client sees M (as Tm<Top<=safeTime, so Tm<safeTime, hence M is visible to the client’s node) and knows that S has been activated at Tu before it obtains schema for moment Top.
...
This approach guarantees correctness (AP is always maintained), but every node (that is not the Meta -storage Storage leader), while trying to obtain a schema for the moment Top=Now, will have to wait for safeTime visible on this node to reach Top, which will take PD (propagation duration). So each transaction will be delayed for PD (on average). This creates a performance problem: latency increases.
...
Even on the Meta -storage Storage leader, a just-installed schema update is not visible right away, because it activates after DD in future. This might be confusing for a user that just issues an ALTER TABLE query because ‘read own writes’ is violated. This can be alleviated by making sure that we complete the future used to install the schema update not later than DD passes, but if a client fails before the wait elapses and reconnects, they might expect to see the update.
...
A primary replica needs to wait for safeTime to reach Top-DD before processing a transaction at a reference timestamp Top. If the node does not get any Meta - Storage updates (i.e. it lags behind the Meta - Storage leader), it cannot proceed.
...
...
An unpleasant thing is that a node lagging too much behind the Meta - Storage freezes transaction processing even in the absence of any schema updates.
...
// N/A
Jira | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|