THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Workflow
The overall process is fairly simple and is outlined below.
Gliffy Diagram name JohnWorkflow pagePin 11
...
| Code Block |
|---|
1) LHS sends (FK-Key, null) to RHS 2) RHS deletes Key from state store 3) RHS sends (Key, null) back to LHS 4) LHS validates Key is still null in LHS-state store, propagates tombstone downstream |
In the event of a foreign key change, a specialized tombstone record is sent Each event from the LHS has a set of instructions sent with it to the RHS. This record tells the RHS to delete the entry from its state store, but to NOT propagate a null back to the LHS. This saves over-the-wire transmission of data which would be discarded anyways.These instructions are used to help manage the resolution of state.
Changing LHS (Key, FK-1) to (Key, FK-2).
| Code Block |
|---|
1) LHS sends (CombinedKey(FK-1,Key), SubscriptionWrapper( |
| Code Block |
1) LHS sends (FK-Key, (value=null, noPropagate=trueinstruction=DELETE_KEY_NO_PROPAGATE)) to RHS-1 2) LHS sends (CombinedKey(FK-2,Key), NewValueSubscriptionWrapper(value=NewValueHash, instruction=PROPAGATE_NULL_IF_NO_FK_VAL_AVAILABLE)) to RHS-2 3) RHS-1 deletes CombinedKey(FK-1,Key) from state store, but does not propagate any other event. 4) RHS-2 updates the local Key-Value store. with (CombinedKey(FK-2,Key), NewValueHash). 5) RHS-2 sends (Key, Data) back to LHS 6) LHS validates Key and Data, performs join logic accordingly. looks up FK-2 in the RHS-2 materialized state store: a) If a non-null result is obtained, RHS-2 propagates to LHS: (Key, SubscriptionResponseWrapper(NewValueHash, RHS-Result, propagateIfNull=false)) b) If a null result is obtained, RHS-2 propagates to LHS: (Key, SubscriptionResponseWrapper(NewValueHash, null, propagateIfNull=true)) - This is done to ensure that the old join results are wiped out, since that join result is now stale/incorrect. 6) LHS validates that the NewValueHash from the SubscriptionResponseWrapper matches the hash of the current key in the LHS table. a) If the hash doesn't match, discard the event and return. b) If the hash does match, proceed to next step. 7) LHS checks if RHS-result == null and propagateIfNull is true. a) If yes, then propagate out a (Key, null) and return b) If no, proceed to next step. 8) Reminder: RHS-result is not null, and NewValueHash is valid and current. LHS performs the join logic on (LHS-Value and RHS-Result), and outputs the resultant event (Key, joinResult(LHS-Value, RHS-Result)) |
| LHS Event (key, extracted fk) | To which RHS-partition? | RHS-0 State | RHS-1 State | Inner Join output | Notes | Inner-Join SubscriptionWrapper Instruction | |
|---|---|---|---|---|---|---|---|
| Publish new event | (k, 1) | RHS-0 | (1,foo) | (k,1,foo) | Normal fk-join induced by LHS event | to RHS-0: | |
| Publish update to event by changing fk | (k, 1) → (k, 2) | RHS-1 | (1,foo) | (k,null) | Must indicate a delete because there is currently no (fk,value) in RHS with key=2, and (k,1,foo) is no longer valid output. | to RHS-0: DELETE_KEY_NO_PROPAGATE to RHS-1: | |
Publish update to event | (k,2) → (k,3) | RHS-0 | (1,foo) | (k,null) | Ideally would not publish a delete, but we do not maintain sufficient state to know that the (k,2) update resulted in a null output and we don't need to do it again. | to RHS-0: DELETE_KEY_NO_PROPAGATE to RHS-1: | |
| Publish a value to RHS Partition 0 | - | - | (1,foo) | - | (k,3,bar) | Performs prefix scan join | - |
| Delete k | (k,3) → (k,null) | RHS-0 | (1,foo) | (k,null) | Propagate null/delete through the sub-topology | to RHS-0: DELETE_KEY_AND_PROPAGATE | |
Compatibility, Deprecation, and Migration Plan
...