Page History

...

这个就是对于batch的sink file，到每个文件后面加全局唯一的后缀，然后最后在作业全部结束的时候，在主节点对于各种情况，去rename或者delete.

FileOutputFormat

How does the speculative execution play together with other sinks? Does it only work for the file based sinks?

The speculative execution could also support sink to Key-value databases, such as Hologres, HBase etc.

Metrics

We can use six metrics to measure and evaluate the efficiency of speculative execution that can be summarized in the job status and Web page.

...

(1)
// 必须要多少execution完成了，才开始预测执行的metrics
How much execution must be completed before you start to predict the metrics to be executed
private AtomicInteger minFinishedForSpeculationThresholdMetrics;

(2)
// 已经完成的execution数的metrics
private Metrics of the number of execution completed
private AtomicInteger finishedExecutionCountMetrics;

(3)
// 有多少execution发生了预测执行的metrics
How many execution metrics predict execution
private Counter speculativeExecutionCountMetrics;

(4)
// execution运行时间维度，发生预测执行的阈值
Execution run time dimension, the threshold of execution prediction
private AtomicDouble speculativeThresholdOfTime;

(5)
// 每个ExecutionVertex中原execution的运行时间的metrics
Metrics of the running time of the original execution in each executionvertex
private Map<String, AtomicDouble> executionIndex2RunningTimespan;

(6)

// 如果最快结束的execution是预测执行的execution，那么对相应的metrics进行汇报
If the fastest ending execution is the predicted execution, report the corresponding metrics

private Counter speculativeExecutionFastestFinishedCountMetrics;

...