Status
...
Page properties | |
---|---|
|
...
...
...
JIRA: here (<- link to https://issues.apache.org/jira/browse/FLINK-XXXX)
Released: <Flink Version>
...
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
The biggest change includes StreamStatus
albeit it's an internal class. We propose to adjust the definition (JavaDoc) to exclude records and rename the class to WatermarkStatus
and move it to the watermarkstatus package. Finally, we need to rename emitStreamStatus
and processStreamStatus
in a few public evolving interfaces (Input, Output). Here, we can add deprecated shortcuts to the top-level interfaces with the old name.
We should also update the definition of StreamStatus#IDLE to make sure that this a) may introduce late records and b) should be user-controlled. With that change, we also remove all code that automatically let's sources go idle without user-specified idleness.
Since sources may offer users the ability to go IDLE
beyond the WatermarkStrategy#withIdleness
, we propose to also add WatermarkOutput#markActive
to quickly go active again without the need to explicitly emit a watermark first. This addition allows to reduce the number of late records when a source knows that records are available until it processed a batch and actually emit a watermark.
Compatibility, Deprecation, and Migration Plan
- User code using
Input
andOutput
would work without recompilation. Compiling would yield deprecation warningsAll changes are on internal classes or on public methods that been added only in master. So we can assume that this FLIP causes no visible change.
Test Plan
The change of StreamStatus
reflects the current implementation and is thus already covered by tests. We might need to add additional unit/integration tests if we discover paths where we automatically switch to IDLE (a recent change on 1.14-SNAPSHOT does it).
...