Page History

...

As shown below for batch job with blocking shuffle , because as shown below. Because of introduce speculative execution the down-stream tasks will consume all reduce executions in an ExecutionVertex will consume the resultPartition of upstream of map ExecutionVertex's fastest finished execution. 

...

• When all upstream ExecutionVertexs run finished,

...

• all executions in down stream

...

• ExecutionVertex should be

...

• notifyed to update  inputChannels from unknow to local/remote

...

• so there are some modifys in Execution.updatePartitionConsumers().

• When a task read from a blocking result partition, if its input is not available, the produce ExecutionVertex and all consumer ExecutionVertexs will be revoked. For upstream  produce ExecutionVertex, I think that SpeculativeScheduler thread should still work. For all consumer ExecutionVertexs I think we should kill all speculative executions and only restart original execution.

• Multiple executions of a upstream ExecutionVertex will produce Multiple ResultPartitions. When all upstream ExecutionVertexs run finished, the inputChannel of down stream executions will be updated to consume the fastest finished execution of upstream. So add a member-variable named fastestFinishedExecution in ExecutionVertex used for create PartitionInfo which used for update down stream executions' inputChannels from unknow to local/remote. 

• In order to avoid cause problems when multiple executions in one ExecutionVertex finished at the same time. A member-variable named vertexFinished was added in ExecutionVertex used for indicates whether this ExecutionVertex run finished and double checked locking pattern in ExecutionVertex.executionFinished().In ExecutionVertex.executionFinished(), after double checked locking pattern finishPartitionsAndUpdateConsumers() should be called instead of Execution.markFinished(),then non-fastest executions in this ExecutionVertex should be canceled.

public class ExecutionVertex
        implements AccessExecutionVertex, Archiveable<ArchivedExecutionVertex> {
	private Execution fastestFinishedExecution = null;
	private volatile boolean vertexFinished = false;
}

• In order to avoid cause problems when multiple executions in one ExecutionVertex finished at the same time. A member-variable named vertexFinished was added in ExecutionVertex used for indicates whether this ExecutionVertex run finished and double checked locking pattern in ExecutionVertex.executionFinished().In ExecutionVertex.executionFinished(), after double checked locking pattern finishPartitionsAndUpdateConsumers() should be called instead of Execution.markFinished(),then non-fastest executions in this ExecutionVertex should be canceled.

public class ExecutionVertex
        implements AccessExecutionVertex, Archiveable<ArchivedExecutionVertex> {
	private Execution fastestFinishedExecution = null;
	private volatile boolean vertexFinished = false;
}

• When all upstream ExecutionVertexs run finished, all executions in down stream ExecutionVertex should be notifyed to update  inputChannels from unknow to local/remote so there are some modifys in Execution.updatePartitionConsumers().

...

Manage sink files

这个就是对于batch的sink file，到每个文件后面加全局唯一的后缀，然后最后在作业全部结束的时候，在主节点对于各种情况，去rename或者delete.

...