Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

However, except for the normal operator list state and keyed state, Flink also support supports other types of state, namely broadcast state and union list state. 

...

The UnioinListState would be more complex. Unlike the normal operator list state, user might have special usage with union list state. For example, users may only have one subtask to store the state, and initialize the subtask with the shared value after restart to implement a more flexible "broadcast" state. Besides, users may have complex initialization logic based on the whole state, like minus the state from a global set of values, which is impossible for normal operator list state. If a part of subtasks are finished and we only keep the left state in the checkpoint, the subtask would have different initialized state after restarted. However, since the UnionListState is on the way to be deprecated and replaced by the OperatorCoordinator, we could first have a temporary fix by asking the tasks of operators using UnionListState to wait for each other to make sure the snapshot of the operator must be complete. 

TODO ListState (operator state)

Based on the above discussion discarding the final states of the finish tasks would only change behavior for a very little fraction of the possible existing jobs whose non-source operators have special logic on initialization or endOfInput, and these jobs could also be modified to keep the current behaviors.  Therefore, we lean towards option 3, which simplify the implementation and leave more room for optimization in the future. 

...