Status

Current state: Under Discussion

...

The existing config option table.exec.shuffle-mode will be extended with new values and the old values will still be respected. More details can be found in section Proposed Changes → Global Data Exchange Mode .

Proposed Changes

Pipelined Region Scheduling

...

Note that the above algorithm works for streaming jobs because all tasks in a streaming job are connected with each other by pipelined data exchanges. If the streaming job employs a shuffle, all tasks land in the same pipelined region and the Pipelined Region Scheduler will trivially schedule all tasks at the same time. For streaming jobs that do not employ a shuffle, one may or may not have to apply special considerations (see Embarrassingly parallel Streaming Jobs ).

Pipelined Region Scheduling Strategy

...

The SlotProvider interface should be extended with an bulk slot allocation method which accepts a bulk of slot requests as one of the parameters. See Extended SlotProvider Interface section below for more details.

...