Table of contents
Jira Boards
Flink 1.17 Burndown: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=572
Sync meeting
The sync meeting is happening every second Tuesday starting on the 1st of November 2022 at 9am CEST / 4pm China Standard Time / 8am UTC.
As we are getting closer to the feature freeze we will do the meeting on a weekly base.
Feel free to join on Google Meet. Local dial-in numbers can be found at https://tel.meet/wcx-fjbt-hhz?pin=1940846765126
Timeline
Feature Freeze
January, 17th of 2023, end of business CEST.
Chinese new year is 22 January: 1+ week holiday. Ppl will be back in early February
Release
Mid of March 2023
Highlight features
please feel free to add/suggest.
Features
List of features announced by contributors and committers that are likely to be ready for the feature freeze:n
NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.
Legend
State
symbol | meaning | comment |
---|---|---|
validated | through cross team testing | |
done | well documented with a complete test coverage | |
will make it | there is no reason this effort should not go into 1.17 | |
in danger | there are some concerns the effort could be ready for the feature freeze of 1.17 | |
very unlikely | there are severe concerns the effort could make it to 1.17 | |
won't make it | it was decided against adding this for the 1.17 release. working on the effort has been stopped. | |
state unclear | ||
independent | as the artefact could be released independent of Apache Flink |
X-Team verification
symbol | meaning |
---|---|
done | |
not required |
Feature Stage
Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).
- MVP: Have a look, consider whether this can help you in the future.
- Beta: You can benefit from this, but you should carefully evaluate the feature.
- Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
- Stable: Unrestricted use in production
- Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
- Deprecated: Start looking for alternatives now
Summary
Numbers are based on the items in the list below, not on the tickets
∑ | Remaining weeks | ||||||||
2022-11-01 | 11 | ||||||||
2022-11-15 | 0 | ||||||||
2022-11-29 | 4 | 10 | 0 | 0 | 0 | 23 | 0 | 37 | 7 |
2022-12-13 | 7 | 24 | 1 | 0 | 1 | 9 | 0 | 42 | 5 |
2022-12-27 | 7 | 25 | 1 | 0 | 1 | 8 | 0 | 42 | 3 |
2023-01-03 | 2 | ||||||||
2023-01-10 | 1 | ||||||||
2023-01-17 | 0 |
List
Feel free to add categories.
Runtime | ||||||||||
---|---|---|---|---|---|---|---|---|---|---|
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Shuffle | ||||||||||
| Xintong Song | 100% | 10-01-2022 | |||||||
Adaptive Batch Scheduler supports hybrid shuffle mode | Weijie Guo | Xintong Song | ||||||||
| Yuxin Tan | Xintong Song | 50% | 09-01-2023 | ||||||
Further improvement of production availability of hybrid shuffle | 50% | 10-01-2022 | ||||||||
Deployment & Cluster Coordination | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
REST API | ||||||||||
| 15-11-2022 | 15-11-2022 | 15-11-2022 | |||||||
| The first stage is finished, and the second stage is improvement, it will be finished in the next version. | 100% | 09-01-2023 | 05-01-2023 | 09-01-2023 | |||||
Reactive Mode | ||||||||||
Scheduler | ||||||||||
| 80% | 03-01-2022 | ||||||||
FLIP-281: Speculative execution for sink | Biao Liu | FLIP in discussion | 30% | 03-01-2022 | ||||||
FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs | FLIP in discussion | 30% | 03-01-2023 | |||||||
Misc | ||||||||||
OLAP | ||||||||||
State backend | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
FLIP-263: Improve resolving schema compatibility
| Hangxiang Yu | 80% | 10-1-2022 | |||||||
Improve File Management in State Backend | 5% | 13-12-2022 | ||||||||
Improve the serializer performace of state change of changelog
| 100% | 10-1-2022 | ||||||||
Allow to configure Changelog Storage per program
| 50% | 13-12-2022 | ||||||||
Add a metric for back-pressure from the ChangelogStateBackend
| 40% | 13-12-2022 | ||||||||
Frocksdb cannot run on Apple M1
| 90% | 13-12-2022 | ||||||||
Checkpoint | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Benchmark Generic Incremental CP + UC + BD | Yuan Mei | |||||||||
| 70% | 10-12-2022 | ||||||||
Benchmark | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Improve benchmark stability
| Yuan Mei | 50% | 3-1-2023 | |||||||
API | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Deprecated | 100% | 08-11-2022 | 08-11-2022 | 08-11-2022 | |||||
| Removed | 40% | 08-11-2022 | |||||||
| ||||||||||
FLIP Discussion | 13-12-2022 | |||||||||
| Martijn Visser | Removed | 100% | 14-11-2022 | 14-11-2022 | 14-11-2022 | ||||
| ||||||||||
SQL | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
General | ||||||||||
| Godfrey He ,@Yunhong Zheng | 50% | ||||||||
| ||||||||||
| Chesnay Schepler | |||||||||
Calcite Update | ||||||||||
| ||||||||||
| ||||||||||
| Sergey Nuyanzin | |||||||||
Flink Dialect | ||||||||||
ALTER TABLE API
| 90% | 2023-01-10 | ||||||||
DELETE/UPDATE API (FLIP-282) | voting | 2023-01-10 | ||||||||
Hive Dialect | ||||||||||
| 60% | |||||||||
| ||||||||||
| ||||||||||
SQL Gateway | ||||||||||
| Shengkai Fang , Zelin Yu | 70% | 2023-01-10 | |||||||
QE | ||||||||||
FLIP-280: Introduce a new explain mode to provide SQL advice | voting | 2023-01-10 | ||||||||
QO | ||||||||||
FLINK-27591 - Improve the plan for batch queries when statistics is unavailable OPEN | Godfrey He ,@Yunhong Zheng | 40% | ||||||||
API/Python | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| HuangXingbo | 100% | 12-12-2022 | |||||||
| HuangXingbo | 0% | 27-12-2022 | |||||||
| HuangXingbo | 70% | 27-12-2022 | |||||||
| HuangXingbo | 0% | 27-12-2022 | |||||||
| HuangXingbo | 100% | 12-12-2022 | |||||||
Machine Learning | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
CEP | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
Web | ||||||||||
Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| ||||||||||
Status / Follow-ups
2022-11-01
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-11-15
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Number of growing test stability issues with "Exit code 137 errors"
- Number of growing test stability issues with "Exit code 137 errors"
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-11-29
- Build instabilities
- : Anton has some new findings on that issue and will get back on it
- Pulsar-related issues:
- Python-related issues:
- Matthias Pohl pings Xingbo Huang / Dian Fu
flink-table-planner
(Godfrey He might be a person to reach out to about it)
: OOM errors are most likely being caused by - Qingsheng Ren will look into the PR :
- Matthias Pohl will ping contributors on the issue - but it's not that urgent since it's not failing that frequently
- Other topics:
- Externalizing Pulsar connector (test instabilities): Martijn Visser is working on externalizing the code base
- Performance test monitoring: Discussion is happening on the mailing list
- Externalizing connectors in general is work-in-progress but looks good
- Public CI documentation can be improved
- Matthias Pohl will work on a first approval
2022-12-13
- Build instabilities
- → Qingsheng to have a look at the PR
- Pulsar connector has been synced to external connector repository. Martijn Visser to open a PR to remove connector from master
- → Qingsheng to ping Godfrey
- → The PR is still failing for the same issue as this PR should fix. We should ping the author to have a look first
- → Martijn to ping Thomas once more
- → Xingbo is working on this
- How to have monitoring and quality control for the externalized connectors → Need to have a discussion on the Dev mailing list. Martijn Visser to make a proposal and open a discussion thread on this topic.
2022-12-27
- Meeting skipped due to Christmas holiday/sick leaves
- Discussion started on moving the feature freeze from Jan 17 to Jan 31 due to pandemic situation in China (see dev ML discussion thread)
- Pulsar connector has been externalized. Pulsar-related test instabilities were disabled in
release-1.16
andrelease-1.15
(see and parent task)
2023-01-10
- Build instabilities (all 1.17 test instabilities with a priority >=Major)
- Proposal: Priorities of test instabilities (docs about it)
- Blocker: New test instabilities which were introduced in unreleased versions only
- Critical: New test instabilities that also affect released versions; Test instabilities that pop up more than two times per month (because they affect CI maintenance)
- Major: Test instabilities that popped up after the version was released but happen rarely
- Switching to weekly calls as we're getting closer to the feature freeze?