Page History

...

Scheduling of Speculative Executions

In each ExecutionJobVertex there is a SpeculativeScheduler thread used for detect the long tail execution in this ExecutionJobVertex periodically based on above three rules.

Image Added

Because of the introduction of speculative execution, the ExecutionVertex can't default to only one execution is running at the same time.

Member fields of ExecutionVertex need bigger reconstruction.So change the currentExecution in ExecutionVertex to an arrayList named executionList(The purpose of using list is to increase the code extensibility, in case that multiple speculative executions will run at the same time in the future), which means that there can be multiple executions in an ExecutionVertex at the same time and for each execution in executionList there is no difference of the behavior such as failover, request slot etc between them.

private List<Execution> executionList = new ArrayList<>();

The scheduler should scheduling an execution(original or speculative execution) according to the index of this execution in executionList instead of default to scheduling the currentExecution in current code.In each ExecutionJobVertex there is a speculative execution thread used for detect the long tail execution in this ExecutionJobVertex periodically based on above three rules.

todo..

Handle of the situation of failover 

...