Page History

...

Make CheckpointBarrierHandler Support EndOfPartition on TM Side

As a whole, for the task side, there are two cases involving finished precedent tasks: 

1. The checkpoint is triggered after the task has received all the EndOfPartition, but not finished yet (due to waiting for the 
2. The checkpoint is 

Currently tasks could be classified into two types according to how checkpoint is notified: the source tasks are trigger via the RPC from the JobManager, while the non-source tasks are triggered via barriers received from precedent tasks. If checkpoints after some tasks finished are considered, the non-source tasks might also get triggered via RPC if they become the new "root" tasks. The implementation for non-source tasks would be different from the source tasks: it would notify the CheckpointBarrierHandler instead of directly performing the snapshot so that the CheckpointBarrierHandler could deal with checkpoint subsuming correctly. Since currently triggerCheckpoint is implemented in the base StreamTask class uniformly and only be called for source StreamTask classes, we would refactor the class hierarchy to support different implementation of  triggerCheckpoint method for the two types of tasks.

If for one task not all its precedent tasks are finished, the task would still get notified via checkpoint barrier from the running precedent tasks. However, some of the precedent tasks might be already finished and the CheckpointBarrierHandler must consider this case. As a whole, EndOfPartition from one input channel would mark this channel as aligned for all the pending and following checkpoints.  Therefore, when received endOfPartition, we will notify the 

Failover and Recovery

Logically we could not restart the finished tasks on failover. We could only restart those tasks who have operators not fully finished. Specially, if all the source operators have been finished, the remaining operator would directly start the process of finishing. 

...