...

Here we propose the speculative execution strategy [FLINK-10644] to handle the problem. The basic idea is to run a copy of task on another node when the original task is identified to be long tail. In more details, the speculative task will be triggered when the speculative scheduler detects that the running time of a task is greater than a configurable multiple of the median of the running time of other finished executions and the data processing throughput of the task is much slower than others. The speculative task is executed in parallel with the original one and share the same failure retry mechanism. Once either task complete, the scheduler admits its output as the final result and cancel the other running one. The preliminary experiments have demonstrated the effectiveness.  

Proposed Changes

General design

Detection of Long Tail Tasks

Finished Tasks Percentage

Long Running Time

Slow Processing Throughput

Scheduling of Speculative Executions

Limitations

Configuration

Compatibility, Deprecation, and Migration Plan

...