THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Figure 2: The format of the state-based storage. The state header contains the version number and is written once per
logical input / output.
Each buffer is prefixed with the channel id for restoring the original assignment to channels.
...
Depending on the implementation of the persistent storage, operators need to store the in-flight data or pointers to it into the operator state. In an operator chain, the head operator will store the input data and the tail operator store the output data.
The actual logic for persisting data is encapsulated in the PersistentInFlightDataStorer and PersistentInFlightDataRetriever that should receive a state handle during initialization. Future versions could store the data in external systems and simply add pointers to the external systems in the state. In the following, we assume some kind of streaming storage with offsets.
For an input channel, the relevant offset is right after the checkpoint barrier. For multiple inputs, multiple offsets need to be saved. If we decide to store keygroups individually for faster rescaling, the same offset will be stored for each key group.
...
For non-keyed data, rescaling semantics is unfortunately a bit fuzzy. For this FLIP, we assume that no data of a given input split can overtake prior data in processing on forward channels. Any fan out or reshuffling will already destroy that ordering guarantee, so we can disregard these cases in this FLIP. If we focus on forward channels, however, we quickly run into situations where the ordering is violated (see Fig. 3).
draw.io Diagram border true viewerToolbar true fitWindow false diagramName rescale.drawio simpleViewer false width 400 diagramWidth 521 revision 3
Figure 3: Split
...
S1 is moved from source instance I to instance II. If the original operators are
back-pressured, data from the new source instance can overtake the original data.
To solve these issues, we have three solutions:
...