THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Release Managers
Jiangjie Qin Xintong Song Martijn Visser Jark Wu
Timeline & Branch Management
- August 1, 2023: 2.0 Status Update
- We are expecting a 1.19 → 1.20 → 2.0 release sequence, with a normal 4-5 months release cycle. We expect to deliver the 2.0 release by the end of 2024, as long as not compromising the quality.
- Public APIs need to be deprecated in 1.19, and PublicEvolving APIs in 1.20, in order to be removed in 2.0. Please try to reach out to the release managers early if you see a risk of planned work items not completed in time.
- The 2.0 branch will be cut around the time of releasing 1.20, from the master.
Work Items
Priority
- Must-have Items
- Things that must be done in release 2.0.
- Typically, this should include only API breaking changes.
- If not finished, the release will be blocked.
- Nice-to-have Items
- Everything else that is planned for the 2.0 release.
- If not finished, the release will not be blocked.
If there's a resource conflict (e.g., requiring efforts from the same group of people, interference due to actively working on the same pieces of codes, etc.), must-have items should be prioritized over nice-to-have items
Legend
| Symbol | Meaning | |
|---|---|---|
Priority |
| Must-have Item |
| Nice-to-have Item | |
| TBD | |
State |  | Done |
 | Will make it | |
 | In danger | |
 | Very unlikely | |
 | Won't make it | |
 | State unclear |
List
Modification of the list would require community decision. Please reach out to the release managers and do not modify the list directly.
| Remove deprecated APIs | ||||||||
|---|---|---|---|---|---|---|---|---|
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| DataSet |  |
| | 2023-08-01 | ||||
Deprecated methods/fields/classes in DataStream | |
| | 2023-08-01 | ||||
| |
| 2023-12-01 | |||||
| |
| | 2023-08-01 | ||||
| |
| ||||||
| Deprecated methods in Table API | |
| ||||||
| Legacy SQL function stack and operators | | Jark Wu |
| |||||
| Queryable state |  | Martijn Visser | Queryable State will not be removed until we have a new substitution. Its timeline depends on the progress of the disaggrated state management | |||||
| | Junrui Lee |
| | 2023-08-22 | Note: This includes removing config options marked @Deprecated and removing config keys placed withDeprecatedKeys(). | ||
| Imperative APIs (DataStream / DataSet / ProcessFunction) | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| Introduce a MVP for a new ProcessFunction API | |
|
| 2023-08-01 | This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.
| |||
| | Rui Fan |
|
| 2024-01-08 | Subsume Flink Time class with Java Duration. | ||
Refactor context-like APIs | |
| Refactor classes like RuntimeContext to return a concrete class containing various bits of metadata, that can be shared in various context implementations. We currently extend various context-like APIs to often, exposing one additional item at a time. | |||||
| Table API / SQL | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
State compatibility for SQL jobs | | Jark Wu |
| State compatibility of upgrading Flink version for SQL jobs. This can be a good API story and milestone for Flink SQL since 2.0.0. | ||||
Move Calcite rules from Scala to Java | |
| Move Flink Calcite rules from Scala to Java. | |||||
| States | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| Disaggregated State Management | |
| Disaggregated State Backend/Management in the cloud-native era.
Note: Only the public API related part is must-have for release 2.0. | |||||
| Eager state declaration. | | Part of the new queryable state API? | This is basically a restart of discussion around FLIP-22: Eager State Declaration. | |||||
|
| Update RocksDB statebackend package | ||||||
| Serialization | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
Plugin serializers | | Fang Yong |
| 2023-12-18 | It is currently too difficult to introduce custom serializers, requiring modifications to the actual program sources, which in some cases isn't feasible. The general idea is that instead of defaulting to Kryo we first ask all found serializers whether they support the given type. This would allow us to get rid of the Kryo-related APIs in DataStream. | |||
Make Kryo usage opt-in | | 2023-12-18 | It is currently too easy to accidentally use Kryo, causing problems down the line with schema evolution. | |||||
Do not eagerly register kryo avro serializer | | 2023-12-18 | Currently the avro serializers is registered with Kryo if flink-avro is on the classpath. This also happens if avro isn't even used by the job, be it due to a mistake in the dependencies setup, branching or flink-avro being in lib. This forces users to always provide flink-avro going forward for affected jobs, because on recovery Flink complains if any of the Kryo serializers cant be loaded. | |||||
Enable built-in Map/List/Set serializers by default | | Fang Yong | 2023-12-18 | These are basic data-structures that are commonly used but currently required Kryo to be used. | ||||
Configuration | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| Introduce a clean configuration layer FLIP-381: Deprecate configuration getters/setters that return/set complex Java objects FLIP-391: Deprecate RuntimeContext#getExecutionConfig | |
|
| 2023-08-22 | Goals: | |||
| Clarify the scopes of configuration options | |
|
| 2023-08-22 | Clarify the scopes of configuration options, i.e. job config, cluster config or both, to avoid misuses of configuration. The scopes should be displayed in Flink configuration documentation. Add a mechanism to prevent setting configuration from wrong scopes, e.g. setting cluster configuration when submitting a job to an existing Flink session, or setting job configuration in cluster components. | |||
| |
|
| 2023-08-22 | Use a YAML compliant configuration parser. Note: While this can be introduced as an optional feature in 1.x (i.e., first parse with standard YAML parser, fall back to custom parsing as a backup), the goal for 2.0 would be to remove the custom parsing logic. | |||
Revisit all config option types and default values | |
| 2023-10-17 | Revisit all config option types and default values to determine whether any modifications are required for Flink 2.0. If any changes are necessary for configuration options marked as 'Public', they should be replaced with a new config option and deprecated in version 1.19; for other options, it should take place in version 1.20. | ||||
| |
|
| 2023-08-22 | It's a plain renaming of the | |||
| | Zhu Zhu | Decide to not do it in Flink 2.0. Go to FLINK-21126 for more details or further discussions. |
| 2023-08-22 | Drop FLINK_PROPERTIES from Flink docker image | ||
REST API | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| Review and refactor the REST API | |
| ||||||
| FLIP-336: Remove "now" timestamp field from REST responses | |
| Remove "now" field from various REST responses. This field contains the current timestamp, which is problematic w.r.t. caching and testing, while providing no value. | |||||
| FLIP-337: Remove JarRequestBody#programArgs | |
| Remove JarRequestBody#programArgs in favor of #programArgsList. | |||||
| FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint | |
| Remove terminationMode parameter from job cancellation endpoint. Effectively it only supports "cancel" as "stop" was moved into a dedicated endpoint. | |||||
|
| Use a consistent naming convention for all REST API fields (camelCase) | ||||||
|
| Remove deprecated query parameters that were subsumed by request body parameters, like the jar handlers. | ||||||
|
| remove rescale REST endpoint. This endpoint hasn't been working for years. We can add it back later if necessary. | ||||||
| REST API returns only numeric values that conform to the JSON spec. Essentially, never return NaN/Inf as these are serious headaches for generated clients. | |||||||
|
| Refactor certain REST return values to be less special, like ConfigurationInfo which is returned as a list of key-value pairs, when it should just be a map. | ||||||
| |
| | 2023-08-22 | Remove deprecated fields from ExceptionHistory API (all-exceptions, timestamp, root-exception, truncated) | |||
| | Enforce single maxExceptions query parameter Note: Could be generalized to change all applicable query parameters. | ||||||
| | Drop YARN-specific mutating GET REST endpoints (yarn-cancel, yarn-stop) We shouldn't continue having such hacks in our APIs to work around YARN deficiencies. | ||||||
Metrics | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| Review and refactor the metrics implementation (revisit and improve/remove incorrect metrics) | |||||||
|
| Drop deprecated Availability/Network metrics: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/metrics/#network | ||||||
| |
| Split MetricGroup interface to target different audiences (reporters that retrieve metadata, users that register metrics/groups) | |||||
| 1.19:
2.0:
| Rework MetricGroup scope APIs.
| ||||||
|
| Drop MetricGroup methods accepting an int as a metric/group name. These turned out to not be useful, and typically encourage bad design since a key-value pair should be used instead to attach some semantic to then number. | ||||||
| 1.19:
2.0:
| Remove brackets around keys returned by MetricGroup#getAllVariables. This has turned out to just be a bother to anyone using it. | ||||||
| 1.19: deprecate method 2.0: remove method | Drop MetricReporter#open Enforce a single configuration path via the constructor. The factory continues to have access to the configuration. | ||||||
|
| Pass configuration instead of Properties into ReporterFactory. Flinks configuration classes are really useful but not available to reporters. Blocked on splitting flink-core. | ||||||
| Attach semantics to Gauges (ratio/count/rate/something else); refactor Counter/Meter to be Gauges with syntactic sugar on top. The goal is to generalize metrics from the perspective of reporters, and allow them to better map gauges to the external metrics backends supported types. This implies subsuming Metric#getMetricType. | |||||||
| | 1.19: introduce warning if non-numeric gauge is used 2.0: force type to Number | Enforce that gauges are numeric. String gauges are exceedingly rare and push a lot of complexity into reporter implementations. | |||||
| |
| Properly expose the logical scope of a metric group. This is currently exposed via a crude casting code path, and is primarily blocked on rework of the scope APIs as the current deficiencies push down to many requirements into the APIs (support for per-call delimiters/filters). | |||||
| Misc | ||||||||
| Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note |
| |
| Add support for Java 17 and make this the default. Nice-to-have: also add support for Java 21 | |||||
Drop support for Java 8 | |
| ||||||
Drop support for Java 11 | |
| ||||||
Restructure modules | | Restructure modules (maybe overlapping with All user APIs should be in a separate module)
Note: These can be done in 1.x or 2.1+, it's just easier to do in 2.0 if you don't need to worry about backwards-compatibility. | ||||||
Upgrade checkstyle | | Martijn Visser | ||||||