Table of contents
Jira Boards
Flink 1.17 Burndown: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=572
Sync meeting
The sync meeting is happening every second Tuesday starting on the 1st of November 2022 at 9am CEST / 4pm China Standard Time / 8am UTC.
As we are getting closer to the feature freeze we will do the meeting on a weekly base.
Feel free to join on Google Meet. Local dial-in numbers can be found at https://tel.meet/wcx-fjbt-hhz?pin=1940846765126
Timeline
- Feature Freeze
January, 17th of 2023, end of business CEST.- January 31st, 2023, end of business CEST (ML discussion on change)
- Release
Mid of March 2023- End of March 2023
Chinese new year is 22 January: 1+ week holiday. Ppl will be back in early February
Highlight features
please feel free to add/suggest.
Features
List of features announced by contributors and committers that are likely to be ready for the feature freeze:n
NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.
Legend
State
symbol | meaning | comment |
---|---|---|
validated | through cross team testing | |
done | well documented with a complete test coverage | |
will make it | there is no reason this effort should not go into 1.17 | |
in danger | there are some concerns the effort could be ready for the feature freeze of 1.17 | |
very unlikely | there are severe concerns the effort could make it to 1.17 | |
won't make it | it was decided against adding this for the 1.17 release. working on the effort has been stopped. | |
state unclear | ||
independent | as the artifact could be released independent of Apache Flink |
X-Team verification
symbol | meaning |
---|---|
done | |
not required |
Feature Stage
Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).
- MVP: Have a look, consider whether this can help you in the future.
- Beta: You can benefit from this, but you should carefully evaluate the feature.
- Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
- Stable: Unrestricted use in production
- Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
- Deprecated: Start looking for alternatives now
Summary
Numbers are based on the items in the list below, not on the tickets
∑ | Remaining weeks | ||||||||
2022-11-01 | 13 | ||||||||
2022-11-15 | 11 | ||||||||
2022-11-29 | 4 | 10 | 0 | 0 | 0 | 23 | 0 | 37 | 9 |
2022-12-13 | 7 | 24 | 1 | 0 | 1 | 9 | 0 | 42 | 7 |
2022-12-27 | 7 | 25 | 1 | 0 | 1 | 8 | 0 | 42 | 5 |
2023-01-10 | 9 | 27 | 0 | 0 | 3 | 5 | 0 | 44 | 3 |
2023-01-17 | 13 | 23 | 0 | 0 | 3 | 3 | 2 | 44 | 2 |
2023-01-24 | 16 | 21 | 0 | 0 | 3 | 3 | 2 | 45 | 1 |
2023-01-31 | 25 | 10 | 0 | 0 | 12 | 0 | 0 | 47 | 0 |
List
Feel free to add categories.
Runtime | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| FLIP in voting | |||||||||
Shuffle | ||||||||||
| Xintong Song | 100% | 10-01-2023 | |||||||
Adaptive Batch Scheduler supports hybrid shuffle mode | Weijie Guo | Xintong Song | ||||||||
| Yuxin Tan | Xintong Song | 100% | 17-01-2023 | ||||||
Further improvement of production availability of hybrid shuffle | 100% | 17-01-2023 | ||||||||
Deployment & Cluster Coordination | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
REST API | ||||||||||
| 15-11-2022 | 15-11-2022 | 15-11-2022 | |||||||
| The first stage is finished, and the second stage is improvement, it will be finished in the next version. | 100% | 09-01-2023 | 05-01-2023 | 09-01-2023 | |||||
Reactive Mode | ||||||||||
Scheduler | ||||||||||
| 100% | 17-01-2022 | ||||||||
| Biao Liu | 95% | 31-01-2022 | |||||||
| 95% | 31-01-2023 | ||||||||
Misc | ||||||||||
OLAP | ||||||||||
State backend | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
FLIP-263: Improve resolving schema compatibility
| Hangxiang Yu | 80% | 31-1-2023 | |||||||
Improve File Management in State Backend | 5% | 31-1-2023 | ||||||||
Improve the serializer performace of state change of changelog
| 100% | 31-1-2023 | ||||||||
Allow to configure Changelog Storage per program
| 70% | 31-1-2023 | ||||||||
Add a metric for back-pressure from the ChangelogStateBackend
| 40% | 31-1-2023 | ||||||||
Frocksdb cannot run on Apple M1
| 100% | 30-1-2023 | ||||||||
Release FRocksDB 6.20.3-ververica-2.0
| 100% | 31-1-2023 | ||||||||
Checkpoint | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Benchmark Generic Incremental CP + UC + BD | Yuan Mei | |||||||||
| 100% |
| ||||||||
Benchmark | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Improve benchmark stability
| Yuan Mei | 50% | 31-1-2023 | |||||||
API | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Deprecated | 100% | 08-11-2022 | 08-11-2022 | 08-11-2022 | |||||
| Removed | 40% | 08-11-2022 | |||||||
| 31-01-2022 | |||||||||
FLIP Discussion | Need to be postponed to the next release | 10-01-2023 | ||||||||
| Martijn Visser | Removed | 100% | 14-11-2022 | 14-11-2022 | 14-11-2022 | ||||
| 90% | 31-01-2022 | ||||||||
| Ruan Hang | Coding | ||||||||
SQL | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
General | ||||||||||
| Godfrey He ,@Yunhong Zheng | 100% | 2023-01-31 | |||||||
| ||||||||||
| Chesnay Schepler | |||||||||
Table API | ||||||||||
| 2023-01-20 | |||||||||
Calcite Update | ||||||||||
| ||||||||||
| ||||||||||
| Sergey Nuyanzin | waiting for review | ||||||||
Flink Dialect | ||||||||||
ALTER TABLE API
| reviewing doc | 100% | 2023-01-10 | |||||||
DELETE/UPDATE API (FLIP-282) | writing doc | 90% | 2023-01-10 | |||||||
Hive Dialect | ||||||||||
| ||||||||||
| 100% | 2023-01-31 | ||||||||
| 31-01-2022 | |||||||||
SQL Gateway | ||||||||||
| Shengkai Fang , Zelin Yu | writing doc | 100% | 2023-01-10 | ||||||
QE | ||||||||||
| doc under review | 100% | 2023-01-10 | |||||||
QO | ||||||||||
FLINK-27591 - Improve the plan for batch queries when statistics is unavailable OPEN | Godfrey He ,@Yunhong Zheng | Need to be postponed to the next release | 40% | 2023-1-31 | ||||||
API/Python | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| HuangXingbo | 100% | 10-01-2023 | |||||||
| HuangXingbo | 0% | 10-01-2023 | |||||||
| HuangXingbo | 100% | 19-01-2023 | |||||||
| HuangXingbo | 100% | 19-01-2023 | |||||||
| HuangXingbo | 100% | 10-01-2023 | |||||||
Machine Learning | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
CEP | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Web | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| 100% | |||||||||
Status / Follow-ups
2022-11-01
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-11-15
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Number of growing test stability issues with "Exit code 137 errors"
- Number of growing test stability issues with "Exit code 137 errors"
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-11-29
- Build instabilities
- : Anton has some new findings on that issue and will get back on it
- Pulsar-related issues:
- Python-related issues:
- Matthias Pohl pings Xingbo Huang / Dian Fu
flink-table-planner
(Godfrey He might be a person to reach out to about it)
: OOM errors are most likely being caused by - Qingsheng Ren will look into the PR :
- Matthias Pohl will ping contributors on the issue - but it's not that urgent since it's not failing that frequently
- Other topics:
- Externalizing Pulsar connector (test instabilities): Martijn Visser is working on externalizing the code base
- Performance test monitoring: Discussion is happening on the mailing list
- Externalizing connectors in general is work-in-progress but looks good
- Public CI documentation can be improved
- Matthias Pohl will work on a first approval
2022-12-13
- Build instabilities
- → Qingsheng to have a look at the PR
- Pulsar connector has been synced to external connector repository. Martijn Visser to open a PR to remove connector from master
- → Qingsheng to ping Godfrey
- → The PR is still failing for the same issue as this PR should fix. We should ping the author to have a look first
- → Martijn to ping Thomas once more
- → Xingbo is working on this
- How to have monitoring and quality control for the externalized connectors → Need to have a discussion on the Dev mailing list. Martijn Visser to make a proposal and open a discussion thread on this topic.
2022-12-27
- Meeting skipped due to Christmas holiday/sick leaves
- Discussion started on moving the feature freeze from Jan 17 to Jan 31 due to pandemic situation in China (see dev ML discussion thread)
- Pulsar connector has been externalized. Pulsar-related test instabilities were disabled in
release-1.16
andrelease-1.15
(see and parent task)
2023-01-10
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
- Godfrey He and Yunhong Zheng are working on it
- run tests after each other/not in parallel
- don't reuse JVMs
- JUnit has feature to log memory consumption
- Priorities of test instabilities (docs about it)
- Test instabilities are prioritized as Critical and become blocker as soon as we notice that they are newly introduced
- Feature freeze extended until Jan 31, 2023
- Switching to weekly calls as we're getting closer to the feature freeze?
- Yes, switch to weekly will happen. Next meeting will be Jan 17 due to Chinese New Year coming up and the feature freeze happening soon
- Votes on FLIPs are stalled due to outstanding votes
- Qingsheng Ren will reach out to Martijn Visser about it
- Share in the Slack dev channel to get more people to look into it
- leader elections lacks test coverage (FLIP-285,
)
- No issues popped up on the mailing list since 1.16
2023-01-17
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
- Blockers
- Performance regressions
- → In progress
- → In progress
- → In progress
- → Qingsheng will look into this one after FLINK-29427 (see below)
- → Qingsheng currently is looking into this
- Performance regressions
- Martijn Visser to check open critical test stabilities to determine if some need to be assigned / if we're OK in the overall direction
- FLIP-272: Generalized delegation token support has been merged in and a blog post will be written and published about it, prior to the 1.17 release
- https://github.com/apache/flink/pull/21606 has been merged into Flink, but should also be taken into account for externalized connectors - This is tracked under
- Blockers
2023-01-24
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
- Performance regressions
- → In progress
- → In progress
- → In progress
- Blockers
- → In progress, needs a status update
- → In progress, needs a status update
- → Resolved
- → In progress, needs a status update
- Martijn Visser is looking into this one →
- Martijn Visser is looking into this one →
- Performance regressions
2023-01-31
- Today is feature freeze day
- 25 features / improvements are in for Flink 1.17 (47 in at Flink 1.16, 20 for Flink 1.15, 27 for Flink 1.14)
- Blockers:
- → PR open, Qingsheng will take a final pass on this
- → PR to update buffers for the test has been merged, will be continued to monitor. Test downgraded to Critical
- → Should be resolved, pending validation by the benchmarks (related to FLINK-30624). Benchmarks look to be improved, downgraded to Critical.
- → Qingsheng has tested this in own CI, looks to be resolved and he'll create a PR for this shortly.
- → Should be resolved, pending validation by the benchmarks (related to FLINK-30625). Benchmarks look to be improved, downgraded to Critical.