...
Remove deprecated APIs | ||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
DataSet |
| 2023-08-01 | ||||||||||||||||||||||
Deprecated methods/fields/classes in DataStream |
| 2023-08-01 | ||||||||||||||||||||||
| ||||||||||||||||||||||||
SinkV1 |
| 2023-08-01 | ||||||||||||||||||||||
TableSource / TableSink |
| |||||||||||||||||||||||
Deprecated methods in Table API | ||||||||||||||||||||||||
Legacy SQL function stack and operators | Jark Wu | |||||||||||||||||||||||
Queryable state | Martijn Visser | |||||||||||||||||||||||
Deprecated Config Options | Junrui Lee | Note: This includes removing config options marked @Deprecated and removing config keys placed withDeprecatedKeys(). | ||||||||||||||||||||||
Imperative APIs (DataStream / DataSet / ProcessFunction) | ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
Introduce a MVP for a new ProcessFunction API |
| 2023-08-01 | This is an either in-place or smooth replacement of DataStream API. It differs from the current DataStream API mainly in the following aspects.
| |||||||||||||||||||||
Refactor the API modules | 2023-08-01 | All user APIs should be in a separate module, to avoid and prevent the need to rely on internals. StreamOperator should not be a PublicEvolving API, but an Internal only. Note:
| ||||||||||||||||||||||
| Subsume Flink Time class with Java Duration. | |||||||||||||||||||||||
Refactor context-like APIs | Refactor classes like RuntimeContext to return a concrete class containing various bits of metadata, that can be shared in various context implementations. We currently extend various context-like APIs to often, exposing one additional item at a time. | |||||||||||||||||||||||
Convert user-facing concrete classes into interfaces. | Xintong Song | Convert user-facing concrete classes into interfaces. Examples:
Notes by Xintong Song: Depending on how we want to gradually retire DataStream API, this may not be necessary, as the new ProcessFunction API should be designed to avoid user-facing concrete classes. | ||||||||||||||||||||||
Table API / SQL | ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
State compatibility for SQL jobs | Jark Wu | State compatibility of upgrading Flink version for SQL jobs. This can be a good API story and milestone for Flink SQL since 2.0.0. | ||||||||||||||||||||||
Move Calcite rules from Scala to Java | Move Flink Calcite rules from Scala to Java.
| |||||||||||||||||||||||
States | ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
Disaggregated State Management | Disaggregated State Backend/Management in the cloud-native era.
Note: Only the public API related part is must-have for release 2.0. | |||||||||||||||||||||||
Eager state declaration. | This is basically a restart of discussion around FLIP-22: Eager State Declaration. | |||||||||||||||||||||||
FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package
| Update RocksDB statebackend package | |||||||||||||||||||||||
Serialization | ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
Plugin serializers | It is currently too difficult to introduce custom serializers, requiring modifications to the actual program sources, which in some cases isn't feasible. The general idea is that instead of defaulting to Kryo we first ask all found serializers whether they support the given type. This would allow us to get rid of the Kryo-related APIs in DataStream. | |||||||||||||||||||||||
Make Kryo usage opt-in | It is currently too easy to accidentally use Kryo, causing problems down the line with schema evolution. | |||||||||||||||||||||||
Do not eagerly register kryo avro serializer | Currently the avro serializers is registered with Kryo if flink-avro is on the classpath. This also happens if avro isn't even used by the job, be it due to a mistake in the dependencies setup, branching or flink-avro being in lib. This forces users to always provide flink-avro going forward for affected jobs, because on recovery Flink complains if any of the Kryo serializers cant be loaded. | |||||||||||||||||||||||
Enable built-in Map/List/Set serializers by default | These are basic data-structures that are commonly used but currently required Kryo to be used. | |||||||||||||||||||||||
Configuration
| ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
Introduce a clean configuration layer | Some things to consider are that all configuration should be expressible via string key-value pairs and use the ConfigOption stack plus removing ExecutionConfig, CheckpointConfig and all member variables of StreamExecutionEnvironment. Note by Chesnay Schepler : This implies reworking the Kryo integration! | |||||||||||||||||||||||
Clarify the scopes of configuration options | Clarify the scopes of configuration options, i.e. job config, cluster config or both, to avoid misuses of configuration. The scopes should be displayed in Flink configuration documentation. Add a mechanism to prevent setting configuration from wrong scopes, e.g. setting cluster configuration when submitting a job to an existing Flink session, or setting job configuration in cluster components. | |||||||||||||||||||||||
| Use a YAML compliant configuration parser. Note: While this can be introduced as an optional feature in 1.x (i.e., first parse with standard YAML parser, fall back to custom parsing as a backup), the goal for 2.0 would be to remove the custom parsing logic. | |||||||||||||||||||||||
Revisit all config option types and default values | ||||||||||||||||||||||||
| It's a plain renaming of the | |||||||||||||||||||||||
REST API
| ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
Review and refactor the REST API |
| |||||||||||||||||||||||
FLIP-336: Remove "now" timestamp field from REST responses | Remove "now" field from various REST responses. This field contains the current timestamp, which is problematic w.r.t. caching and testing, while providing no value. | |||||||||||||||||||||||
FLIP-337: Remove JarRequestBody#programArgs | Remove JarRequestBody#programArgs in favor of #programArgsList. | |||||||||||||||||||||||
FLIP-338: Remove terminationMode query parameter from job cancellation REST endpoint | Remove terminationMode parameter from job cancellation endpoint. Effectively it only supports "cancel" as "stop" was moved into a dedicated endpoint. | |||||||||||||||||||||||
FLIP-352: Use camelCast for all REST API fields/parameters
| Use a consistent naming convention for all REST API fields (camelCase) | |||||||||||||||||||||||
Remove deprecated query parameters that were subsumed by request body parameters, like the jar handlers. | ||||||||||||||||||||||||
emove rescale REST endpoint. This endpoint hasn't been working for years. We can add it back later if necessary. | ||||||||||||||||||||||||
FLIP-351: REST API normalizes +/-Inf / NaN to 0
| REST API returns only numeric values that conform to the JSON spec. Essentially, never return NaN/Inf as these are serious headaches for generated clients. | |||||||||||||||||||||||
Refactor certain REST return values to be less special, like ConfigurationInfo which is returned as a list of key-value pairs, when it should just be a map. | ||||||||||||||||||||||||
| Remove deprecated fields from ExceptionHistory API (all-exceptions , timestamp , root-exception , truncated ) | |||||||||||||||||||||||
| Enforce single maxExceptions query parameter Note: Could be generalized to change all applicable query parameters. | |||||||||||||||||||||||
| Drop YARN-specific mutating GET REST endpoints (yarn-cancel, yarn-stop) We shouldn't continue having such hacks in our APIs to work around YARN deficiencies. | |||||||||||||||||||||||
Metrics
| ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
Review and refactor the metrics implementation (revisit and improve/remove incorrect metrics) | ||||||||||||||||||||||||
Drop deprecated Availability/Network metrics: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/metrics/#network | ||||||||||||||||||||||||
| Split MetricGroup interface to target different audiences (reporters that retrieve metadata, users that register metrics/groups) | |||||||||||||||||||||||
Rework MetricGroup scope APIs.
| ||||||||||||||||||||||||
Drop MetricGroup methods accepting an int as a metric/group name. These turned out to not be useful, and typically encourage bad design since a key-value pair should be used instead to attach some semantic to then number. | ||||||||||||||||||||||||
FLIP-342: Remove brackets around keys returned by MetricGroup#getAllVariables
| Remove brackets around keys returned by MetricGroup#getAllVariables. This has turned out to just be a bother to anyone using it. | |||||||||||||||||||||||
Drop MetricReporter#open Enforce a single configuration path via the constructor. The factory continues to have access to the configuration. | ||||||||||||||||||||||||
Pass configuration instead of Properties into ReporterFactory. Flinks configuration classes are really useful but not available to reporters. Blocked on splitting flink-core. | ||||||||||||||||||||||||
Attach semantics to Gauges (ratio/count/rate/something else); refactor Counter/Meter to be Gauges with syntactic sugar on top. The goal is to generalize metrics from the perspective of reporters, and allow them to better map gauges to the external metrics backends supported types. This implies subsuming Metric#getMetricType. | ||||||||||||||||||||||||
| Enforce that gauges are numeric. String gauges are exceedingly rare and push a lot of complexity into reporter implementations. | |||||||||||||||||||||||
| Properly expose the logical scope of a metric group. This is currently exposed via a crude casting code path, and is primarily blocked on rework of the scope APIs as the current deficiencies push down to many requirements into the APIs (support for per-call delimiters/filters). | |||||||||||||||||||||||
Misc | ||||||||||||||||||||||||
Name / JIRA / FLIP | Priority | Contributor | Reviewer | Milestones | State | % | Last Update | Description / Note | ||||||||||||||||
| Add support for Java 17 and make this the default. Nice-to-have: also add support for Java 21 | |||||||||||||||||||||||
Drop support for Java 8 | ||||||||||||||||||||||||
Drop support for Java 11 | ||||||||||||||||||||||||
Restructure modules | Restructure modules (maybe overlapping with All user APIs should be in a separate module)
Note: These can be done in 1.x or 2.1+, it's just easier to do in 2.0 if you don't need to worry about backwards-compatibility. | |||||||||||||||||||||||
| Drop FLINK_PROPERTIES from Flink docker image |
...