THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
public class ExecutionVertex
implements AccessExecutionVertex, Archiveable<ArchivedExecutionVertex> {
private Execution fastestFinishedExecution = null;
private volatile boolean vertexFinished = false;
} |
Manage sink files
When batch job write record into file or Key-value databases, this feature could enabled.
- Sink to Key-value databases, nothing should do with speculative execution feature.
- Sink to file, we should add a global unical ExecutionAttemptID suffix after the normal task names. Finally, I will delete or rename these files in finalizeOnMaster().
这个就是对于batch的sink file，到每个文件后面加全局唯一的后缀，然后最后在作业全部结束的时候，在主节点对于各种情况，去rename或者delete.
| Code Block | ||||
|---|---|---|---|---|
| ||||
public class JobVertex implements java.io.Serializable {
// 存这个JobVertex对应的ExecutionJobVertex中发生了预测执行的ExecutionVertexs中最快跑完的ExecutionAttemptID ExecutionAttemptIDs of the fastest finished executions in all speculatived ExecutionVertex in this jobvertex.
private Set<ExecutionAttemptID> fastAttemptIdsWithSpeculative = new HashSet<>();
// 存这个JobVertex对应的ExecutionJobVertex中发生了预测执行的ExecutionVertex中不是最快跑完的ExecutionAttemptID
private Set<ExecutionAttemptID> slowAttemptIdsWithSpeculative = new HashSet<>();
// 存这个JobVertex对应的ExecutionJobVertex中没有发生预测执行，且顺利结束的Execution的ExecutionAttemptIDExecutionAttemptIDs of the non-fastest finished executions in all speculatived ExecutionVertex in this jobvertex.
private Set<ExecutionAttemptID> slowAttemptIdsWithSpeculative = new HashSet<>();
// ExecutionAttemptIDs of the success finished executions in all non-speculatived ExecutionVertex in this jobvertex.
private Set<ExecutionAttemptID> finishedAttemptIdsWithoutSpeculative = new HashSet<>();
// 存这个JobVertex的所有Execution的ExecutionAttemptID，包括发生过异常的 All executionAttemptIDs in this jobvertex.
private Set<ExecutionAttemptID> allAttemptIds = new HashSet<>();
} |
FileOutputFormat
...
} |
Metrics
We can use six metrics to measure and evaluate the efficiency of speculative execution that can be summarized in the job status and Web page.
...