THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Jira Boards
Flink 1.15 Burndown: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=505
Sync meeting
The sync meeting is happening every second Tuesday starting on the 16th of November at 9am CET / 4pm CST.
As we are getting closer to the feature freeze we will do the meeting on a weekly base starting on the 25th of January.
Feel free to join on Dingtalk
Timeline
Feature Freeze
Monday, 14th 16th of February 2022, end of business CEST. (pushed by a week on the 27th of January and due again due to CI issues on the 14th of February).
Release
Mid February
Highlight features
please feel free to add/suggest.
Features
List of features announced by contributors and committers that are likely to be ready for the feature freeze:n
NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.
Legend
State
| symbol | meaning | comment |
|---|---|---|
 | validated | through cross team testing |
 | done | well documented with a complete test coverage |
 | will make it | there is no reason this effort should not go into 1.15 |
 | in danger | there are some concerns the effort could be ready for the feature freeze of 1.15 |
 | very unlikely | there are severe concerns the effort could make it to 1.15 |
 | won't make it | it was decided against adding this for the 1.15 release. working on the effort has been stopped. |
 | state unclear | |
 | independent | as the artefact could be released independent of Apache Flink |
Feature Stage
Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).
- MVP: Have a look, consider whether this can help you in the future.
- Beta: You can benefit from this, but you should carefully evaluate the feature.
- Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
- Stable: Unrestricted use in production
- Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
- Deprecated: Start looking for alternatives now
Summary
Numbers are based on the items in the list below, not on the tickets
| | | | | | | | ∑ | % | Remaining weeks | |
| 2021-11-16 | 2 | 29 | 4 | 1 | 0 | 2 | 1 | 39 | 12 | |
| 2021-11-29 | 3 | 31 | 4 | 1 | 0 | 3 | 1 | 43 | ~20% | 10 |
| 2021-12-14 | 5 | 28 | 5 | 2 | 0 | 1 | 1 | 43 | ~36% | 8 |
| 2022-01-11 | 5 | 25 | 7 | 3 | 0 | 1 | 1 | 45 | ~48% | 4 |
| 2022-01-25 | 11 | 20 | 9 | 3 | 3 | 0 | 1 | 47 | ~70% | 2 |
| 2022-02-01 | 14 | 20 | 6 | 4 | 2 | 0 | 1 | 47 | ~76% | 1 (+1) |
| 2022-02-08 | 14 | 20 | 5 | 2 | 5 | 0 | 1 | 47 | ~80% | 0 (+1) |
| 2022-02-15 | 20 | 15 | 3 | 0 | 8 | 0 | 1 | 47 | ~95% | 0.15 |
| 2022-03-01 |
List
Feel free to add categories.
| Runtime | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Beta | | 100% | 24.01.2021 | Yes | Yes | |||
| FLIP-193: Snapshots ownership | Ready and Evolving | | 100% | 31.01.2022 | Yes | Yes | |||
| Dawid Wysakowicz Piotr Nowojski | | 100% / 75% | 21.02.2022 | Yes (with one limitation) | Yes | |||
| Stable | | 100% | 21.02.2022 | Yes | Yes (Chinese doc reviewing) | |||
| Improve out of the box recovery time experience by activating local recovery by default | TBD | this is a stretch goal and therefor red | | 0% | |||||
| Ready and Evolving | | 100% | 15.02.2022 | Yes | Yes | |||
Following up issues on unified shuffle services and batch shuffle
| Stable | | 100% | 15.02.2022 | Yes | Yes | |||
| FLINK-21352 - FLIP-158: Generalized incremental checkpoints | MVP | | 100% | 07.03.2022 | Yes | Yes (not translated) | No | ||
FLINK-21648 - FLIP-151: Incremental snapshots for heap-based state backend | MVP | | 60% | 25.01.2022 | |||||
| Deployment & Cluster Coordination | |||||||||
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | Updated | Implemented | Documented | X-team verified | |
| REST API | |||||||||
| Stable | | 100% | Yes | Only REST API Documentation. | ||||
| Stable | | 100% | 15.03.2022 | Yes | Nothing to document. |
| ||
| Beta | Flink Shaded released now. Can be finished. |
| 100% | |||||
| Reactive Mode | |||||||||
| Reactive Mode: Ready and Evolving or Beta |
| 100% | Yes | Yes | ||||
| Niklas Semmler | Reactive Mode: Ready and Evolving or Beta |
| 100% | 15.2.2022 | Yes | |||
| Niklas Semmler | Reactive Mode: Ready and Evolving or Beta | Almost ready, doing final touches on the PR. |
| 100% | 01.03.2022 | Nothing to document. | ||
| Misc | |||||||||
| Stable |
| 100% | Yes | Only REST API Documentation. | ||||
| FLIP-194: Introduce of the JobResultStore | Stable | Implementation done. |
| 100% | 15.03.2022 | Yes | Configuration options are documented HA documentation & glossary are updated |
| |
| Stable | Implementation done. Blog post in-progress. |
| 100% | 01.03.2022 | Yes | |||
| Stable | FLINK-23976 adds new metrics that subsume the existing ones (that are now deprecated). Improved/reworked up-/downtime metrics moved to later releases. | | 40% | |||||
| @Zhuo Liu Yang Wang | Stable |
| 100% | 25.01.2022 | ||||
| FLIP-???: Strong read-after-write consistency of Flink FileSystems | Stable | No implementation. Only documentation. We'll be done in 1.16 as doing this later won't block the future efforts. | | 30% | 15.02.2022 | ||||
| Stable |
| 100% | 26.01.2022 | Yes | Configuration options are documented | |||
| FLIP-198: Working directory for Flink processes | Stable |
| 100% | 24.01.2022 | Yes | Configuration options are documented | |||
| FLIP-201: Persist local state in working directory | Stable |
| 100% | 21.02.2022 | Yes | Yes | |||
| OLAP | |||||||||
| @Shammon Yangze Guo Xintong Song | Stable | Would be postponed to 1.16 | | 25% | 08.02.2022 | |||
| Connectors, Formats & FileSystems | |||||||||
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Qingsheng Ren | Beta |
| 80% | 08-02-2022 | ||||
| Watermark Alignment across Sources | MVP |
| 100% | 22-02-2022 | Not yet | ||||
| External connector repository | Arvid Heise | | 25% | 24-01-2022 | |||||
| FLIP-191: Extend unified Sink interface to support small file compaction | Beta | | 100% | 21-02-2022 | |||||
| Alexander Preuss, Fabian Paul Arvid Heise | Ready and Evolving | | 100% | 14-12-2021 | Yes | Yes | ||
| Beta | | 90% | 25.1.2022 | |||||
| Beta | | 100% | 14-02-2022 | Yes | Yes | ||||
| Beta | | 100% | 14-02-2022 | Yes | Yes | ||||
| Yuri Gusev | Beta | | 60% | 26-01-2022 | ||||
| Ready and Evolving | | 90% | 24-01-2022 | |||||
| Galen Warren, Xintong Song | Main part finished and doc & bugfix in-review | | 100% | 01-28-2022 | Yes | Yes | ||
| API | |||||||||
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| 0% | |||||||||
| SQL | |||||||||
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| FLIP-188: Introduce Built-in Dynamic Table Storage | Jingsong Lee | MVP | Flink Side remain: FLINK-25177 (Not blocker, can be done in the next version) Table Store Side: Separate release after Flink 1.15 |
| 95% | 15-02-2022 | |||
| FLIP-190: Support Version Upgrades for Table API & SQL Programs | MVP |
| 90% | 24-01-2022 | |||||
| FLIP-195: Improve the name and structure of vertex and operator name for sql job | Longsan, Godfrey He | Stable |
| 100% | 15-02-2022 | ||||
| Stable |
| 80% | 24-01-2022 | |||||
| Nicholas Jiang | Stable | Unlikely because of substantial feedback on the PR |
| 60% | 15-02-2022 | |||
| Stable |
| 95% | 24-01-2022 | |||||
| Stable |
| 100% | 14-12-2021 | |||||
| Jing Zhang | Stable |
| 100% | 29-11-2021 | Yes | Yes | ||
| Jing Zhang | Stable |
| 100% | 29-11-2021 | Yes | Yes | ||
| Jing Zhang | Stable |
| 100% | 29-11-2021 | Yes | No, | |||
| Sergey Nuyanzin | | 20% | ||||||
| API/Python | |||||||||
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| FLIP-206 Future Improvement on the Performance and Usability of the PyFlink API | Xingbo Huang | Beta |
Others would be postpone to 1.16 |
| 30% | 08-02-2022 | |||
| Machine Learning | |||||||||
| Name/JIRA Issue | Responsible Contributor | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| 0% | |||||||||
Status / Follow-ups
2022-03-01
- Cut branch tomorrow
- All testing issues created
- Blocker & Critical issues on board https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=505&view=detail&selectedIssue=FLINK-23740&quickFilter=2263
- Will ask for cross team testers after meeting
- > Fabian
- > Fabian
2022-02-22
State of the features
- Cut branch on Wednesday?
- Still issues with the CI
- Cutting branch might put more pressure on the CI
- Push it by a week
- Create release testing and documentation issues?
- Add documententation
- Blocker & Critical issues on board: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=505&view=detail&selectedIssue=FLINK-23740&quickFilter=2263 (47 remaining)
- FLINK-25968 Joie pings Tomo
- FLINK-25977 wait till CI is fixed
- FLINK-26191 Fabian & team will have a look at it
- FLINK-26274 Konstantin will do that
- FLINK-26281 Konstantin will assign himself and ask someone who might be able to do it
- FLINK-18356 Joe will follow up
- Teams responsible for components should update wether we want to really do stuff about 1.15.
- Are the right issues shown up on this board?
- Ask if all items are tagged with the right fixed version.
2022-02-15
State of the features
- Code freeze planned on Wednesday
- Status of unfinished features
Blockers
-
- New sink interface has wrong numRecordsOut metric
- Fabian would take a look
- Async sync writer issues all blockers?
- Danny has started working on it.
- How to deal with class loader leaks in the TableAPI?
- Not much progress due to occupied in version upgrade story.
-
- PR is ready, can we close it?
- Could not reproduced, need a double look
-
- Problem already exists in 1.14 but it means that on every leader change the JM process needs to restart
- Deco team would assess it.
-
- Would be merged soon.
- Kinesis related issues
Build Stabilities
- Every team, please check the build instabilities in JIRA
2022-02-08
State of the features
- We want to decide what makes it and what not
- FLINK-24474 Mika Naylor
Still chasing some test failures.
Should be done - FLINK-24229 Yuri Gusev
Follow up
Ready to go, should be reviewed this week. - FLIP-190: Support Version Upgrades for Table API & SQL Programs Timo Walther
Merged a lot of stuff already
Looks good to have the MVP experimental in the release - FLINK-24865 Nicholas Jiang
Looking for a reviewer
Johannes Moser will check
- FLINK-24474 Mika Naylor
- FLINK-23883 Dawid Wysakowicz Yun Gao
Some stuff has been done. - FLINK-21513 David, Chesnay Schepler
Mostly done > Could be moved to done
- FLINK-23883 Dawid Wysakowicz Yun Gao
Blockers
-
The reason should be found.
Till Rohrmann will follow up
Timo Walther was assigned. -
Waiting for the PR
PR there, will be reviewed by Timo Walther s team
Build stability
-
Ververica SDK might have reviewed it, Martijn Visser will check with them -
Assigned -
Fix seems to merge soon.
There's a PR, that is reviewed. CI hasn't run yet.
Once it is run it will be merged by Danny Cranmer and team -
Reproduced
Piotr Nowojski will check it. - and Xingbo would have a look
Please put tickets in progress when you work on them.
2022-01-31
State of the features
Blockers
-
For this case, might mainly related to NetworkBuffer & akka submodule class loader, Yun Gao would update later today.
Yun Gao working on it - Fixed. The hostname resoluation issue would be continue investigating under
-
Need owner for this issue
Wait till contributor is back.
What do we do with Java 11.
Build stability
- 148 test instabilities that affect 1.14 or 1.15 (JIRA filter)
- Many test instabilities are caused by connector tests → Plan forward?
-
- Can somebody help with the review?
Review is done.
PR open, still failing tests.
Old tests spawn manual Kafka processes.
- Can somebody help with the review?
-
- Needs somebody looking into it
Looks related
Fabian Paul will take a look at it.
- Needs somebody looking into it
-
- Needs somebody looking into it
Martijn Visser will ping someone.
- Needs somebody looking into it
2022-01-25
Some stats
Run through critical features
- Users can take & restore from savepoints comparably fast as from checkpoints > Native format savepoints can be done
- FLINK-23883 > Probably will make it
- Improve out of the box recovery time experience by activating local recovery by default > Good scenario, could be a tiny PR.
- FLIP-151 > Won't make it, updating status
- FLINK-24474 > Pretty small change
- JobResultStore > PRs ready, besides retry mechanism, relies on some other work.
- FLINK-21513 > Not sure if that much is missing, most of the PRs are merged
- Strong read-after-write consistency of Flink FileSystems > moved to 1.16.
- FLIP-201 > might or might not make it
- FLINK-25286 > Can make PR in this two weeks
- Watermark Alignment across Sources > Piotr is working on this as own project. Priorities might change.
- FLINK-24228 > Some issues, that might be fixed or not
- FLIP-190 > Might be ready as beta, might be tight (extension would help)
- FLINK-24865 > PR is there, but needs to be reviewed. unsure if it can be done before. (extension would help)
Feature freeze date
- Can we leave it on the 6th of February
- Ververica SQL team would benefit (1 week would help)
- Ververica deco team JobResultStore would benefit (1 week would help)
- Joe brings this to the mailing list
Blockers
-
Yun Gao sets up a Slack channel to improve this issue. -
Related to host name resolution in azure, would change to 127.0.0.1 first and the root cause would be further debugged. PR opened.
Commit by Fabian Paul should be in end of this week.
-
Build stability
- 145 test instabilities that affect 1.14 or 1.15 (JIRA filter) (number slightly decreased by 4
) - Kafka related issues, waiting on
-
- Needs somebody
Fabian pings Danny again
- Needs somebody
-
Yun Gao pings Cloudera -
Fabian will have a look -
David will take it -
- What's the state?
Solved by infra overhaul. Fabian will update tickets.
- What's the state?
- Are pulsar related cases all fixed?
Martijn will double check
- 145 test instabilities that affect 1.14 or 1.15 (JIRA filter) (number slightly decreased by 4
Bi-weekly meetings now weekly
Start working on the release blogpost
- Who wants to contribute?
2022-01-11
- Run through features
- Buffer debloating at a ok state
- Flip-187
- Incremental savepoints > Not 100 % sure yet
- Unified batch shuffle > PR open, bigger stretch
- FLIP-158 > Process of testing, couple of issues with testing. Put back progress a bit.
- FLIP-151 > It is in danger, probably won't make it. Not enough people working on it
- FLINK-24474 > Stretch goal didn't address this
- FLINK-14954 > Waiting for Flink shaded release and then it will be done
- FLINK-25277 > Already addressed, yarn issues but fixed this week
- FLIP-194 > Looks good for the release. Work on file base approach. Waiting for retry mechanism.
- FLINK-21513 > Defining what it really means
- FLINK-24947 > Already opened a PR
- Strong read after write consistency > FLIP this week. Not much of a follow up. Should be doable
- FLINK-25318 > Some progress, some PRs. We'll ask for a further update. Also update the ticket.
- FLINK-25286 > Some meta PR, code is almost ready. Review PR with high priority
- Watermark alignment > Will be red at the end of the release as no one took it over.
- External connector repository > Will be done after the release
- FLIP-191 > Opened first PR, rough prototype. Good estimate at the end of the week. Still positive.
- FLINK-20732 > Initial pull request almost month ago. No update. Expectation to not have it in 1.15
- FLINK-24228 > PR open, no review yet.
- FLINK-24703 > Only waiting for review.
- FLIP-188 > ?
- FLIP-190 > Highest priority, but a lot of remaining work. Not confident yet.
- FLIP-195 > Might not get done (about 60 %, some not very large PRs)
- FLINK-24403 > Will be in
- FLINK-24865 > ?
- FLINK-24427 > will be in
- FLINK-21327 > Cross testing
- FLINK-24890 > ? (No subtask done, unlikely it will be done)
- PyFlink > ?
- Blockers
-
- Might need some help from the deco team
-
- Yun would investigate this issue.
-
- Needs somebody to drive forward
- Piotr will take care of it
-
- Build stability
- We had some issues around build stability in the past weeks. We believe it is cause by exceeding the resources of the available workers.
- 149 test instabilities that affect 1.14 or 1.15 (JIRA filter) (this number is increasing
) - Please try to monitor your components and make sure that instabilities get addressed
- Most common failing tests
-
- Needs somebody
-
- Maybe somebody can take over Yufan Sheng
- Danny would open the PR, reviewer
- Some kafka tests failed due to "create test topic xxx failed, the metadata failed to propagate to kafka broker.
-
Issues waiting for kafka container:
QingSheng opened the PR, reviewer
-
2021-12-14
Agenda / doc
- Overview over the effort
- Number of features remained the same
- 2 additional features of done
- 2 shifted towards won't make it
- Overall progress around 36 %
- Numbers are based on items on this list.
- 8 weeks remaining till feature freeze
- We will skip the next sync on the 28th due to holiday season
- Blockers
- Yufei might fix it, who can keep an eye on this ticket? VV SDK team will track it according to Martijn Visser
- Jingsong will track it.
- Has been disabled for years. Now it has been reenabled. It is downgraded from the blockers. Fabian Paul is gonna check if it is still valid.
- Under development, who can keep an eye on this ticket? Martijn Visser will take an eye on it.
- We should make a plan to remove the old Kafka connector. who could take care of this?
- Build stability
- Thanks a lot to everyone helping with improving our build stability!
- 128 test instabilities that affect 1.14 or 1.15 (JIRA filter)
- 50% of build failures over the past 30 days were caused by connectors, python, kafka/gelly build profiles
- Most common failing tests
-
- Etienne is working on it to migrate the tests to test containers.
-
- Would refactor the test infrastructure with Kafka → Needs the removal of the old Kafka connector
Blocked on Python API. It will be reworked by Qingsheng Ren
- Would refactor the test infrastructure with Kafka → Needs the removal of the old Kafka connector
-
- Would refactor the test infrastructure with Kafka → Needs the removal of the old Kafka connector
Blocked on Python API. It will be reworked by Qingsheng Ren
- Would refactor the test infrastructure with Kafka → Needs the removal of the old Kafka connector
-
- Update from Fabian Paul ?
Pull request open. Will fix most of the issues. Will be reviewed and merged this week.
- Update from Fabian Paul ?
-
- Little progress from Yufan's side. Can someone help?
We are in contact with them.
- Little progress from Yufan's side. Can someone help?
-
- Caused by maven failed to download the avro package on startup, should be occasional network issues.
- Caused by maven failed to download the avro package on startup, should be occasional network issues.
-
- Not reproduced yet.
Taken care of.
- Not reproduced yet.
-
- We have fixed ElasticSearch related OOM, would pick it to 1.14/1.13 soon, and we still have some cases in other modules.
- We have fixed ElasticSearch related OOM, would pick it to 1.14/1.13 soon, and we still have some cases in other modules.
-
Agenda
- Overview on the effort.
- 43 features collected (3 done, 31 green)
- It is ok to label features as green or red right away if they are stretch goals
- around 20 % done (rather random number)
- 10 weeks left till feature freeze
- How to report progress on features?
- Please update the feature page accordingly!
- Please set JIRA issues you are working on to "in Progress"
- Blockers
- Build stability
- 125 test instabilities that affect 1.14 or 1.15 (JIRA filter)
- Most common failing tests
-
Update later this week -
Might be done today -
PR open, probably in today -
Fix was opened that should stabilise all those issues. Hopefully the Kafka IT cases are stabilised -
The Ververica SDK team will pick it up -
No update yet, will have a look today -
Looking into it, could be fixed with the other issue -
Closed -
Will be communicated asyncly -
already mentioned -
There is already a PR. They are working on this.
-
- We could fix most of our build system related instabilities
? - Github Action, glue schema was deactivated
Matthias and team will follow up on this one. (Create a new ticket)
2021-11-16
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- 123 test instabilities that affect 1.14 or 1.15 (JIRA filter)
- Total test instabilities 222
- Most common failing tests
- Problems with build infrastructure