Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

For non-keyed data, rescaling semantics is unfortunately a bit fuzzy. For this FLIP, we assume that no data of a given input split can overtake prior data in processing on forward channels. Any fan out or reshuffling will already destroy that ordering guarantee, so we can disregard these cases in this FLIP. If we focus on forward channels, however, we quickly run into situations where the ordering is violated (see Fig. 3).

draw.io Diagram
bordertrue
viewerToolbartrue
fitWindowfalse

...

diagramNamerescale.drawio
simpleViewerfalse
width

...

diagramWidth

...

554

...

revision3

...

Figure 3: Violating ordering guarantees while rescaling of non-keyed data. Split S1 is moved from source instance I to instance II.
If the original operators are back-pressured, data from the new source instance can overtake the original data.



To solve these issues, we have three solutions:

...