Page History

...

• Lifecycle of TaskExecutor (TM)/Task/ResultPartition: TM starts an internal shuffle service for transporting partition data to consumer side. When task enters FINISHED state, its produced partition might not be fully consumed. Therefore TM container should not be freed until all the internal partitions consumed. It is obvious that there exists coupled implicit constraints among them, but has no specific mechanism for coordinating them work well (more details in design proposal to extend this FLIP).

• Extension of writer/reader: ResultPartition can only be written into local memory for streaming job and single persistent file per subpartition for batch job. It is difficult to extend partition writer and reader sides together based on current architecture. E.g. ResultPartition might be written in sort&merge way or to external storage. And partition might also be transported via external shuffle service on YARN, Kubernetes etc in order to release TM early.

...

Set shuffle implementation config parameter to default netty-based implementation from FLINK-11391 and FLINK-11392, instead of <none> which meant feature flag to use previous non-pluggable legacy implementation. The legacy and feature flag code should be removed.

Next steps

• Shuffle resources lifecycle management (design proposal to extend this FLIP):

  • Implement partition deregister and cleanup logic via ShuffleMaster.

  • Improve TM release by checking partition consumed via ShuffleMaster.

  • Improve task failover by checking producer’s partition available via ShuffleMaster.

• Support job/edge level config for ShuffleType.

• Abstract RecordWriter/Reader interface for handing raw records.

• Refactor ShuffleService interface for returning RecordWriter/Reader.

• Adjust the processes in StreamInputProcessor and StreamRecordWriter based on Writer/Reader interfaces.

• Extend to Yarn/KubernetesShuffleManager implementations based on new interfaces.

...