THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
As part of this FLIP we would like to finally explicitly document what user can do with checkpoints and savepoints. Currently there are a number of things that are not officially supported, but we know users are doing that. So let's start with the status quo:
Pre-existing
Yes | Yes (unofficially)/Maybe (untested) | No (but could be done) | Very difficult/impossible |
Canonical Savepoint | Aligned Checkpoint | Unaligned Checkpoint | |
Statebackend change | |||
State Processor API | (???) | ||
Schema evolution | untested? | untested? | |
Flink minor (1.x → 1.y) version upgrade | (???) | ||
Flink bug/patch (1.14.x → 1.14.y) version upgrade | (???) | (???) | |
Job full upgrade | |||
Job upgrade w/o changing graph shape and record types | |||
Rescaling |
Questions:
What about RocksDB upgrades? If we bump RocksDB version between Flink versions, do we support recovering from a native format snapshot (incremental checkpoint)?
Proposal 1
Canonical Savepoint | Native Savepoint | Aligned Checkpoint | Unaligned Checkpoint | |
Statebackend change | ||||
State Processor API | (???) | (???) | ||
Schema evolution | (???) | (???) | (???) | |
Flink minor (1.x → 1.y) version upgrade | ||||
Flink bug/patch (1.14.x → 1.14.y) version upgrade | (change) | (change) |
Arbitrary job upgrade (changed graph shape/record types) | ||||
Job upgrade w/o changing graph shape and record types | ||||
Rescaling |
Main aim of the first proposal is to unify guarantees between two types of savepoints and two types of checkpoints. The only difference between native and canonical savepoint should be the ability to change statebackend, and officially there would be no difference between aligned and unaligned checkpoints. Hence we would simplify the documentation, as we could avoid documenting the distinction between unaligned and aligned checkpoints.
Proposal 2
Canonical Savepoint | Native Savepoint | Aligned Checkpoint | Unaligned Checkpoint | |
Statebackend change | ||||
State Processor API | (???) | (???) | ||
Schema evolution | (???) | (???) | (???) | |
Flink minor (1.x → 1.y) version upgrade | (change) | |||
Flink bug/patch (1.14.x → 1.14.y) version upgrade | (change) | (change) |
Arbitrary job upgrade (changed graph shape/record types) | (change) | |||
Job upgrade w/o changing graph shape and record types | (change) | (change) | ||
Rescaling |
The main aim of this proposal is to actually document what we can easily provide, based on the fact that native savepoint and aligned checkpoints would be virtually the same thing. The disadvantage of this proposal is that we would need to document the distinction between aligned and unaligned checkpoints.
...