Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

这个就是对于batch的sink file,到每个文件后面加全局唯一的后缀,然后最后在作业全部结束的时候,在主节点对于各种情况,去rename或者delete.


Code Block
languagejava
titleDefaultExecutionGraph
public class JobVertex implements java.io.Serializable {
	// 存这个JobVertex对应的ExecutionJobVertex中发生了预测执行的ExecutionVertexs中最快跑完的ExecutionAttemptID
	private Set<ExecutionAttemptID> fastAttemptIdsWithSpeculative = new HashSet<>();

	// 存这个JobVertex对应的ExecutionJobVertex中发生了预测执行的ExecutionVertex中不是最快跑完的ExecutionAttemptID
	private Set<ExecutionAttemptID> slowAttemptIdsWithSpeculative = new HashSet<>();

	// 存这个JobVertex对应的ExecutionJobVertex中没有发生预测执行,且顺利结束的Execution的ExecutionAttemptID
	private Set<ExecutionAttemptID> finishedAttemptIdsWithoutSpeculative = new HashSet<>();

	// 存这个JobVertex的所有Execution的ExecutionAttemptID,包括发生过异常的
	private Set<ExecutionAttemptID> allAttemptIds = new HashSet<>();
}


FileOutputFormat

How does the speculative execution play together with other sinks? Does it only work for the file based sinks?

The speculative execution could also support sink to Key-value databases, such as Hologres, HBase etc.

...

private Counter speculativeExecutionFastestFinishedCountMetrics;


Web UI

After I have implemented this feature. When speculative execution runs faster then original execution, the flink ui will show that this task has been cancelled. But the result of the job is correct, which is in full compliance with our expectations.

再加一张图

todo...

Limitations

(1)JobType is Batch.

...