THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Discussion thread | https://lists.apache.org/thread/0oc10cr2q2ms855dbo29s7v08xs3bvqg | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Vote thread | https://lists.apache.org/thread/lrqjg44v0s82shbpvbqp6ojqv873q1wr | ||||||||
| JIRA |
| ||||||||
| Release | 1.1920, 2.0 |
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Additional key naming rule for boolean type options (In discussion): ONLY add 'enabled' when there are more detailed configurations under the same prefix, to prevent one name from serving as a prefix to another. Otherwise, do not add 'enabled' as suffix. Alternatives:
...
Please note that this FLIP does not change any state accessing, checkpointing and recovery logic. To be more specific, the detailed migration plan for the individual options is as follows:
Checkpointing
Original (Current) option | Proposed New option | Type | Note |
state.checkpoints.num-retained | execution.checkpointing.num-retained | Integer | |
state.checkpoint.cleaner.parallel-mode | execution.checkpointing.cleaner.parallel-mode | Boolean | |
state.backend.incremental | execution.checkpointing.incremental | Boolean | |
state.backend.local-recovery | execution.checkpointing.local-backup.enabled | Boolean | Whether to do checkpoint on local disk, another option |
taskmanager.state.local.root-dirs | execution.checkpointing.local-backup.dir | String | |
state.savepoints.dir | execution.checkpointing.savepoint-dir | String | |
state.checkpoints.dir state.checkpoint-storage | execution.checkpointing.dir | String | One URI is enough for specifying the checkpoint directory and storage. 'jobmanager' and 'jm://' are special URI for job manager based storage. |
state.storage.fs.memory-threshold | execution.checkpointing.data-inline-threshold | MemorySize | |
state.storage.fs.write-buffer-size | execution.checkpointing.write-buffer-size | MemorySize | |
execution.checkpointing.checkpoints-after-tasks-finish.enabled | execution.checkpointing.checkpoints-after-tasks-finish | Boolean | Remove the 'enabled' if the naming rule for boolean type option 3 is chosen |
All those options will be moved into `CheckpointingOptions` if they are not currently in.
Recovery
Original (Current) option | Proposed New option | Type | Note |
execution.savepoint.path | execution.state-recovery.path | String | |
execution.savepoint.ignore-unclaimed-state | execution.state-recovery.ignore-unclaimed-state | Boolean | |
execution.savepoint-restore-mode | execution.state-recovery.claim-mode | Enum | Enum of org.apache.flink.runtime.jobgraph.RestoreMode |
state.backend.local-recovery | execution.state-recovery.from-local | Boolean | Specifies whether to recover from local, the local files are created while the |
execution.checkpointing.approximate-local-recovery | execution.state-recovery.approximate-local-recovery | Boolean | |
execution.checkpointing.recover-without-channel-state.checkpoint-id | execution.state-recovery.without-channel-state.checkpoint-id | Long |
All those options will be moved into a newly introduced class `RecoveryOptions` in flink-core.
...
As the most special one, the state.backend.local-recovery defines whether the Flink do checkpoint in local disk, as well as whether the Flink would recover from the local. This FLIP proposes to spilt this option into two, named execution.checkpointing.local-backup and execution.state-recovery.from-local, representing checkpointing and recovery behavior respectively.
Additionally, some existing option classes lack annotations,
...
which are:
org.apache.flink.configuration.CheckpointingOptions
- org.apache.flink.configuration.StateBackendOptions
org.apache.flink.contrib.streaming.state.RocksDBOptions
This FLIP will annotate all such existing classes, alongside any new classes that are introduced, with the @PublicEvolving in 2.0. Note that the migration period for deprecated options in those classes will meet the requirement for @PublicEvolving (Will last for at least one minor release). With above changes, the options for state, checkpointing and recovery are organized in a more straightforward structure. It is simple to categorize new options under appropriate prefixes.
Compatibility, Deprecation, and Migration Plan
In Flink 1.1920, the new options will be introduced and the current ones will be annotated as deprecated. Flink will read new options first, and if the user does not configure them, fallback to the old(current) options. The deprecation will last for two one minor versions(1.19 and 1.20), and in Flink 2.0, the old options will be removed entirely.
It there is a migration tool for current YAML file transition (current flink-conf.yaml to new config.yaml) that FLIP-366 introduced, the migration of above changes will also integrated in the tool.
Test Plan
Existing UT/IT can ensure the compatibility for old options. New tests will cover the new options.
...