THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Table of contents
Jira Boards
Flink 1.17 Burndown: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=572
Sync meeting
The sync meeting is happening every second Tuesday starting on the 1st of November 2022 at 9am CEST / 4pm China Standard Time / 8am UTC.
As we are getting closer to the feature freeze we will do the meeting on a weekly base.
Feel free to join on Google Meet. Local dial-in numbers can be found at https://tel.meet/wcx-fjbt-hhz?pin=1940846765126
Timeline
- Feature Freeze
January, 17th of 2023, end of business CEST.- January 31st, 2023, end of business CEST (ML discussion on change)
- Release
Mid of March 2023- End of March 2023
Chinese new year is 22 January: 1+ week holiday. Ppl will be back in early February
Highlight features
please feel free to add/suggest.
Features
List of features announced by contributors and committers that are likely to be ready for the feature freeze:n
NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.
Legend
State
| symbol | meaning | comment |
|---|---|---|
 | validated | through cross team testing |
 | done | well documented with a complete test coverage |
 | will make it | there is no reason this effort should not go into 1.17 |
 | in danger | there are some concerns the effort could be ready for the feature freeze of 1.17 |
 | very unlikely | there are severe concerns the effort could make it to 1.17 |
 | won't make it | it was decided against adding this for the 1.17 release. working on the effort has been stopped. |
 | state unclear | |
 | independent | as the artifact could be released independent of Apache Flink |
X-Team verification
| symbol | meaning |
|---|---|
| done |
| not required |
Feature Stage
Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).
- MVP: Have a look, consider whether this can help you in the future.
- Beta: You can benefit from this, but you should carefully evaluate the feature.
- Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
- Stable: Unrestricted use in production
- Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
- Deprecated: Start looking for alternatives now
Summary
Numbers are based on the items in the list below, not on the tickets
| | | | | | | | ∑ | Remaining weeks | |
| 2022-11-01 | 13 | |||||||||
| 2022-11-15 | 11 | |||||||||
| 2022-11-29 | 4 | 10 | 0 | 0 | 0 | 23 | 0 | 37 | 9 | |
| 2022-12-13 | 7 | 24 | 1 | 0 | 1 | 9 | 0 | 42 | 7 | |
| 2022-12-27 | 7 | 25 | 1 | 0 | 1 | 8 | 0 | 42 | 5 | |
| 2023-01-10 | 9 | 27 | 0 | 0 | 3 | 5 | 0 | 44 | 3 | |
| 2023-01-17 | 13 | 23 | 0 | 0 | 3 | 3 | 2 | 44 | 2 | |
| 2023-01-24 | 16 | 21 | 0 | 0 | 3 | 3 | 2 | 45 | 1 | |
| 2023-01-31 | 26 | 5 | 0 | 0 | 16 | 0 | 0 | 47 | 0 | |
| 2023-02-14 | 5 | 23 | 3 |
List
Feel free to add categories.
| Runtime | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| FLIP in voting | | ||||||||
| Shuffle | ||||||||||
| Xintong Song | | 100% | 10-01-2023 | 10-01-2023 | 14-02-2023 |
| |||
| AdaptiveBatchScheduler should supports early consumption for dynamic graph. | Weijie Guo | Xintong Song | | |||||||
| Yuxin Tan | Xintong Song | | 100% | 17-01-2023 | 17-01-2023 | 10-02-2023 | self-test | ||
| Further improvement of production availability of hybrid shuffle | | 100% | 17-01-2023 | 17-01-2023 | self-test | |||||
| Deployment & Cluster Coordination | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| REST API | ||||||||||
| | 15-11-2022 | 15-11-2022 | 15-11-2022 | ||||||
| The first stage is finished, and the second stage is improvement, it will be finished in the next version. | | 100% | 09-01-2023 | 05-01-2023 | 09-01-2023 | ||||
| Reactive Mode | ||||||||||
| n/a | ||||||||||
| Scheduler | ||||||||||
| | 100% | 17-01-2022 | |||||||
| Biao Liu | | 100% | 31-01-2022 |
| |||||
| | 100% | 31-01-2023 | |||||||
| Misc | ||||||||||
| n/a | ||||||||||
| OLAP | ||||||||||
| n/a | ||||||||||
| State backend | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
FLIP-263: Improve resolving schema compatibility
| Hangxiang Yu | | 80% | 31-1-2023 | ||||||
Improve File Management in State Backend | | 5% | 31-1-2023 | |||||||
Improve the serializer performace of state change of changelog
| | 100% | 31-1-2023 | self-test | ||||||
Allow to configure Changelog Storage per program
| | 70% | 31-1-2023 | |||||||
Add a metric for back-pressure from the ChangelogStateBackend
| | 40% | 31-1-2023 | |||||||
Frocksdb cannot run on Apple M1
| | 100% | 30-1-2023 | Tested by Martijn Visser | ||||||
Release FRocksDB 6.20.3-ververica-2.0
| | 100% | 31-1-2023 | Tested by Martijn Visser | ||||||
| Checkpoint | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Benchmark Generic Incremental CP + UC + BD | Yuan Mei | | ||||||||
| | 100% |
| |||||||
| Benchmark | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Improve benchmark stability
| Yuan Mei | | 50% | 31-1-2023 | ||||||
| API | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Deprecated | | 100% | 08-11-2022 | 08-11-2022 | 08-11-2022 | Self-tested | |||
| Removed | | 40% | 31-01-2023 | ||||||
| Dawid Wysakowicz | | 90% | 31-01-2022 | ||||||
| FLIP Discussion | Need to be postponed to the next release | | 10-01-2023 | |||||||
| Martijn Visser | Removed | | 100% | 14-11-2022 | 14-11-2022 | 14-11-2022 | Self-tested | ||
| | 90% | 31-01-2022 | Self-tested | ||||||
| Hang Ruan | Coding | ||||||||
| SQL | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| General | ||||||||||
| Godfrey He ,@Yunhong Zheng | | 100% | 2023-01-31 | ||||||
| | 31-01-2023 | ||||||||
| Chesnay Schepler | | 31-01-2023 | |||||||
| Table API | ||||||||||
| | 2023-01-20 | ||||||||
| Calcite Update | ||||||||||
| | |||||||||
| | |||||||||
| Sergey Nuyanzin | | 31-01-2023 | |||||||
| Flink Dialect | ||||||||||
ALTER TABLE API
| | 100% | 2023-02-14 | self-tested | ||||||
| | 100% | 2023-02-14 | self-tested | ||||||
| Hive Dialect | ||||||||||
| |
| ||||||||
| | 100% | 2023-01-31 | |||||||
| | 31-01-2022 | ||||||||
| SQL Gateway | ||||||||||
| Shengkai Fang , Zelin Yu | writing doc | | 100% | 2023-01-10 |
| ||||
| QE | ||||||||||
| doc under review | | 100% | 2023-01-10 |
| |||||
| QO | ||||||||||
| Godfrey He ,@Yunhong Zheng | Need to be postponed to the next release | | 40% | 2023-1-31 | |||||
| API/Python | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| HuangXingbo | | 100% | 10-01-2023 | self-test | |||||
| HuangXingbo | | 0% | 10-01-2023 | ||||||
| HuangXingbo | | 100% | 19-01-2023 | self-test | |||||
| HuangXingbo | | 100% | 19-01-2023 | self-test | |||||
| HuangXingbo | | 100% | 10-01-2023 | self-test | |||||
| Machine Learning | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| n/a | ||||||||||
| CEP | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| n/a | ||||||||||
| Web | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| | 100% | ||||||||
Status / Follow-ups
2022-11-01
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-11-15
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Number of growing test stability issues with "Exit code 137 errors"
- Number of growing test stability issues with "Exit code 137 errors"
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-11-29
- Build instabilities
- : Anton has some new findings on that issue and will get back on it
- Pulsar-related issues:
- Python-related issues:
- Matthias Pohl pings Xingbo Huang / Dian Fu
-
: OOM errors are most likely being caused by
flink-table-planner(Godfrey He might be a person to reach out to about it) - : Qingsheng Ren will look into the PR
- Matthias Pohl will ping contributors on the issue - but it's not that urgent since it's not failing that frequently
- Other topics:
- Externalizing Pulsar connector (test instabilities): Martijn Visser is working on externalizing the code base
- Performance test monitoring: Discussion is happening on the mailing list
- Externalizing connectors in general is work-in-progress but looks good
- Public CI documentation can be improved
- Matthias Pohl will work on a first approval
2022-12-13
- Build instabilities
- → Qingsheng to have a look at the PR
- Pulsar connector has been synced to external connector repository. Martijn Visser to open a PR to remove connector from master
- → Qingsheng to ping Godfrey
- → The PR is still failing for the same issue as this PR should fix. We should ping the author to have a look first
- → Martijn to ping Thomas once more
- → Xingbo is working on this
- How to have monitoring and quality control for the externalized connectors → Need to have a discussion on the Dev mailing list. Martijn Visser to make a proposal and open a discussion thread on this topic.
2022-12-27
- Meeting skipped due to Christmas holiday/sick leaves
- Discussion started on moving the feature freeze from Jan 17 to Jan 31 due to pandemic situation in China (see dev ML discussion thread)
- Pulsar connector has been externalized. Pulsar-related test instabilities were disabled in
release-1.16andrelease-1.15(see and parent task)
2023-01-10
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
-
- Godfrey He and Yunhong Zheng are working on it
- run tests after each other/not in parallel
- don't reuse JVMs
- JUnit has feature to log memory consumption
-
- Priorities of test instabilities (docs about it)
- Test instabilities are prioritized as Critical and become blocker as soon as we notice that they are newly introduced
- Feature freeze extended until Jan 31, 2023
- Switching to weekly calls as we're getting closer to the feature freeze?
- Yes, switch to weekly will happen. Next meeting will be Jan 17 due to Chinese New Year coming up and the feature freeze happening soon
- Votes on FLIPs are stalled due to outstanding votes
- Qingsheng Ren will reach out to Martijn Visser about it
- Share in the Slack dev channel to get more people to look into it
- leader elections lacks test coverage (FLIP-285,
)
- No issues popped up on the mailing list since 1.16
2023-01-17
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
- Blockers
- Performance regressions
- → In progress
- → In progress
- → In progress
- → Qingsheng will look into this one after FLINK-29427 (see below)
- → Qingsheng currently is looking into this
- Performance regressions
- Martijn Visser to check open critical test stabilities to determine if some need to be assigned / if we're OK in the overall direction
- FLIP-272: Generalized delegation token support has been merged in and a blog post will be written and published about it, prior to the 1.17 release
- https://github.com/apache/flink/pull/21606 has been merged into Flink, but should also be taken into account for externalized connectors - This is tracked under
- Blockers
2023-01-24
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
- Performance regressions
- → In progress
- → In progress
- → In progress
- Blockers
- → In progress, needs a status update
- → In progress, needs a status update
- → Resolved
- → In progress, needs a status update
- → Martijn Visser is looking into this one
- → Martijn Visser is looking into this one
- Performance regressions
2023-01-31
- Today is feature freeze day
- 26 features / improvements are in for Flink 1.17 (47 in at Flink 1.16, 20 for Flink 1.15, 27 for Flink 1.14)
- 5 features are still listed as expected to be completed, but are not yet in, 4 of them have been merged and are writing documentation, Martijn Visser to check/update for the status of these items.
- Blockers:
- → Matthias will look into this
- → Leonard review PASS, waiting CI green
- → Fixed by Qingsheng
- → Should be resolved, pending validation by the benchmarks (related to FLINK-30624). Benchmarks look to be improved, downgraded to Critical.
- → Should be resolved, pending validation by the benchmarks (related to FLINK-30625). Benchmarks look to be improved, downgraded to Critical.
- → Martijn to reach out to Dong, Rui Fan, Piotr. We're planning to give them until Friday the 3rd of February to come to a conclusion on this ticket; if no consensus is achieved, then the original commit that introduced the regression should be reverted.
- → Matthias to check if this has already been resolved via another ticket (multiple related tickets)
- → [Critical] PR to update buffers for the test has been merged, will be continued to monitor. Test downgraded to Critical
- → [Critical] Downgraded to Critical as it's only a test-related issue and doesn't indicate a bug in production.
- → [Major] Test downgraded to Major, If the test fail again contributor will increase the waiting interval for this test as a solution.
- → [Major] Leonard downgrade the issue priority to Major as it's a known slack plugin issue
- Martijn Visser to communicate to the Flink community this evening that the feature freeze has started and we plan to cut the release branch at the end of this week (Friday 3rd of February).
2023-02-07
masteris stabilized enough to cut therelease-1.17branch- FLINK-30921 - The Azure apt mirror instabilities seem to have been resolved for now.
- FLINK-30908 - The issue turned out to be a problem that existed in previous releases
- FLINK-30921 - The Azure apt mirror instabilities seem to have been resolved for now.
- Release branch is going to be cut today by Leonard Xu
- Release testing will be announced: 2 weeks will be planned for this
2023-02-14
- Status update on release testing efforts
- End of cross-team testing date is 21st of February 2023. We will monitor the status throughout the week and hopefully conclude everything next week.
- When a feature is cross-team tested, the icon needs to be changed from to to indicate that the testing has been completed
- We always look for volunteers: picking up a cross-team testing task is much appreciated.
- Proposal: Create Jira issues for release management tasks to document what was done to improve review-ability (alternatively, add expected output to release documentation)
- Do this from now on for the next steps in release management (create release candidate etc.)
- Test instabilities:
- : Blocker
- : Starts failing more regularly again. Qingsheng Ren to ping Godfrey He and Yunhong Zheng
- FLINK-30972 (openssl version update necessary) continues to fail because of FLINK-30965 (repo-sync doesn't pick up 1.15 changes anymore)
2023-02-21
- Blocker issues: