THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Table of contents
Jira Boards
Flink 1.18 Burndown: 1.18 Burndown - Agile Board - ASF JIRA (apache.org)
Sync meeting
- The release sync will start from April 4th, 2023, at 10am (UTC+2) and 4pm (UTC+8). The release sync happens bi-weekly at first, and will be adjusted to weekly as we approaching the feature freeze date.
- Feel free to join on Zoom meeting (with passcode eSj55R if required). iCalendar file can be downloaded via this link.
- Due to the limit of Zoom, a meeting session is limited to 40 mins. We set up another Zoom meeting (with passcode jhS5w7 if required) as a backup if we cannot finish discussions within 40 mins in the first meeting session.
Timeline
- Feature Freeze
July 11, 2023, end of business CEST- July 24, 2023, 00:00 CEST(UTC+2) (announced in dev ML)
- Release
- End of September 2023
Highlight features
please feel free to add/suggest.
Features
List of features announced by contributors and committers that are likely to be ready for the feature freeze:n
NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.
Legend
State
| symbol | meaning | comment |
|---|---|---|
 | validated | through cross team testing |
 | done | well documented with a complete test coverage |
 | will make it | there is no reason this effort should not go into 1.18 |
 | in danger | there are some concerns the effort could be ready for the feature freeze of 1.18 |
 | very unlikely | there are severe concerns the effort could make it to 1.18 |
 | won't make it | it was decided against adding this for the 1.18 release. working on the effort has been stopped. |
 | state unclear | |
 | independent | as the artifact could be released independent of Apache Flink |
X-Team verification
| symbol | meaning |
|---|---|
| done |
| not required |
Feature Stage
Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).
- MVP: Have a look, consider whether this can help you in the future.
- Beta: You can benefit from this, but you should carefully evaluate the feature.
- Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
- Stable: Unrestricted use in production
- Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
- Deprecated: Start looking for alternatives now
Summary
Numbers are based on the items in the list below, not on the tickets
| | | | | | | | ∑ | Remaining weeks | |
| 05/02/2023 | 11 | 32 | 11 | |||||||
| 05/16/2023 | 11 | 32 | 8 | |||||||
| 06/13/2023 | 3 | 23 | 7 | 2 | 2 | 1 | 35 | 4 | ||
| 06/27/2023 | 3 | 22 | 5 | 4 | 1 | 35 | 2 | |||
| 07/25/2023 | 18 | 14 | 1 | 11 | 1 | 45 | 0 |
List
Feel free to add categories.
| Runtime | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Shuffle | ||||||||||
| Xintong Song | done | | 100% | 24-07-2023 | |||||
| Quantify metrics of shuffle(i.e. network) memory | | 13-06-2023 | ||||||||
| Coordination | ||||||||||
| | 100% | 17-07-2023 |
| ||||||
| Supports deciding the parallelism of sources according to runtime information (e.g. DPP result) | | 19-06-2023 | ||||||||
| Supports JobMaster status recovery after JobMaster failover | | 19-06-2023 | ||||||||
|
| | 100% | 26-07-2023 | No documentation needed |
| ||||
| State & Checkpoint | ||||||||||
| yue ma | Yun Tang / ？ | | 17-07-2023 | ||||||
FLIP-306: Unified File Merging Mechanism for Checkpoints
| | 25-07-2023 | ||||||||
FLIP-263: schema compatibility
| Hangxiang Yu | | 80% | 25-07-2023 | ||||||
| @Rui xia | | 13-06-2023 | |||||||
| | 30-05-2023 | ||||||||
| | 25-07-2023 | Documented as Metric | |||||||
| | 25-07-2023 | No documentation needed | |||||||
| | 25-07-2023 | No documentation needed | |||||||
REST | ||||||||||
Support CPU profiler on taskmanager | Yu Chen | | 13-06-2023 | |||||||
| SQL | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Calcite Update | ||||||||||
| Timo Walther | done | | 100% | 25-07-2023 | No documentation needed | ||||
| done | | 100% | 25-07-2023 | No documentation needed | |||||
| done | | 100% | 25-07-2023 | No documentation needed | |||||
| Godfrey He | | 50% | 25-07-2023 | Unclear if it needed. I asked on the ticket. | |||||
| done | | 100% | 25-07-2023 | No documentation needed | |||||
FLIP-300 | done | | 100% | 25-07-2023 |
| |||||
| done | | 100% | 25-07-2023 |
| ||||||
| done, docs missing |
| 90% | 01-08-2023 |
| ||||||
| luoyuxia | done, docs missing |
| 90% | 01-08-2023 |
| |||||
| done | | 100% | 01-08-2023 | |||||||
| done, docs missing |
| 90% | 01-08-2023 | |||||||
| luoyuxia | done, docs missing |
| 90% | 01-08-2023 |
| |||||
| ron liu | luoyuxia | done | | 100% | 25-07-2023 | ||||
FLIP-296 | Yun Tang | done | | 100% | 25-07-2023 | See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/concepts/time_attributes/#advanced-watermark-features | ||||
| Benchao Li | done | | 100% | 25-07-2023 | https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/jdbcdriver/ | |||||
| Jark Wu | done | | 100% | 25-07-2023 | No documentation needed | |||||
| Connectors | ||||||||||
| done, some follow ups next release, docs missing |
| 20% | 25-07-2023 |
| ||||||
| done | | 100% | 25-07-2023 | https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/kafka/ & https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/kafka/ | ||||||
| Leonard Xu | done |
| 100% | 25-07-2023 | ||||||
| under discussion |
| 10% | 25-07-2023 | |||||||
| lincoln lee | done | | 100% | 25-07-2023 | No documentation needed | ||||
| done | | 100% | 25-07-2023 |
| |||||
| done | | 100% | 25-07-2023 | see row above | |||||
| Resource Management | ||||||||||
| David Morávek | done, documentation missing | | 80% | 25-07-2023 |
| ||||
| | 90% | 25-07-2023 | No documentation needed | ||||||
| | 60% | 25-07-2023 | |||||||
Runtime | ||||||||||
| done, some follow ups next release, docs missing | | 90% | 25-07-2023 |
| |||||
| 100% | 25-07-2023 | No documentation needed |
| ||||||
Misc | ||||||||||
| | 100% | 25-07-2023 | Checking with the contributor | ||||||
| Martijn Visser | | 100% | 25-07-2023 | No documentation needed |
| ||||
Status / Follow-ups
04/04/2023
- Is Zoom OK for everyone?
- Looks good for now, except the 40 min limit. One possible solution is to create two meetings as a backup. Usually release syncs can finish within 40 mins.
- Retrospective of 1.17
- Has been summarized in 1.17 Release, at the end of the page.
- Daily work divisions
- JIRA ticket (Mainly focus on new blockers, progress of existing blockers each week) (Qingsheng Ren )
- CI (Failures reported in #build channel and not reported) (Sergey Nuyanzin )
- Benchmark (Check if there's regression in the #flink-dev-benchmarks channel) Konstantin Knauf
- Mailing list monitoring (Jing Ge )
- This division doesn't mean that other RMs are now allowed to do the work. All RMs should take responsibilities on all of these daily issues if they are available.
- Feature freezing date is set to July 11, 2023.
04/18/2023
- CI instabilities
- - PR is under review
- - Lijie Wang will take a look
- Konstantin Knauf will ping David again
- - wait if there are more occurrences
- Konstantin Knauf will ping David, Chesnay
- - Xingbo will take a look. Seems like a CI environment issue
- - wait if this happens more often ( Martijn Visser will check)
- - Hangxiang Yu will take a look
- Konstantin Knauf will ping Ferenc
- - Pinged Francesco on JIRA
- Azure CI failing with no space left on device → Robert Metzger will check if cleanup script needs to be udpated →
- Version management
- Qingsheng Ren will remove 1.15 related resources within this week.
05/30/2023
- Remind in the mailing list about updating the release wiki page.
06/13/2023
- Progress checking
- yue ma Yun Tang
- Support CPU profiler on taskmanager Yu Chen Yun Tang
- Kui Yuan Yun Tang
- David Morávek
- Panagiotis Garefalakis Piotr Nowojski
- Matthias Pohl Chesnay Schepler
- Blockers
- Konstantin Knauf to ping Tzu-Li Tai
06/27/2023
- Feature freezing date
- We are still collecting progress of features, and will decide whether to postpone the feature freezing date in the next week. Currently we will stick to and push the original date.
- Akka 2.6 will not receive any critical security updates after Sep 2023, so we might need to get rid of it in 1.18. Chesnay Schepler is working on it and might affect the feature freezing date.
- Blockers and critical issues
- Sergey Nuyanzin to ping Matthias Pohl
-
Konstantin Knauf to ping Stefan Richter
- PR by Stefan is already opened.
-
Konstantin Knauf to ping Roman Khachatryan or Anton Kalashnikov
- I've pinged Roman.
-
Konstantin Knauf
- I've pinged Ferenc.
- Qingsheng Ren
- Qingsheng Ren to ping Fang Yong
- Qingsheng Ren to ping Junhan Yang
- Yunhong Zheng is working on it
- Qingsheng Ren to ping liudalong
- Lijie Wang will try to fix it before feature freeze
- Qingsheng Ren to ping Yanfei Lei
-
Konstantin Knauf will ask Chesnay Schepler if he could take it over
- Chesnay will do another round of reviews today. It was actually updated by the contributor.
- Qingsheng Ren
07/11/2023
- Feature freeze date is extended until July 24, 2023, which is a Monday
- Blocker / Critical issues
07/25/2023
- Deprecate APIs in 1.18
Purely adding annotations / JavaDocs: go directly because they are not new featuresWorks requiring changes to code baseCreate a thread in ML for receiving applications including:FLIP ID and current status (in discussion / vote passed / developing)Why this deprecation must be done in 1.18
All 2.0 unrelated requests will be deniedTo be discussed: deadline of these deprecation works. 1.17 release testing lasted 3 weeks, so what about 2 weeks for feature freeze of these deprecations?Get 2.0 RMs involved in evaluating cases
- Decided not to give more extension, as we couldn't see a lot of benefits to 2.0 giving another 2 weeks
- Cross-team testing
Goes in parallel with additional deprecation works above- will go normally, for about 3 weeks
- Add a reminder about documentation before x-team test
- Cutting branch
will be delayed until at least 2 weeks laterwe will target at the next Tuesday. - Blocker / Critical issues
08/01/2023
- ical event doesn't work anymore
- Requests to merge into
master:- Deprecating SourceFunction API
- Blockers/Critical (last 7 days)
- Performance regression, PR has been approved and waiting for CI
- Performance regression, Weijie Guo is looking into it
- 1.18 API deprecations, maybe downgrade?
- Release testing issue
- Commit merged to master after feature freeze
- : PR provided (test-code issue)
- : PR provided (test-code issue)
- : Test-code issue. Lijie will try to fix it
- : Looks like a test instability caused by the underlying OS (Stefan Richter is looking into it)
08/08/2023
- Blockers / Critical issues (last 7 days)
- blocker, Junrui is working on it
- blocker, Dalong is working on it
- blocker, PR is ready and Chesnay is reviewing it.
- critical, looks like an old one
- Benchmark: TPC-DS result? Qingsheng Ren
- Documentation: Remind contributors to finish docs
- Release announcement
- Cutting the branch: we'll wait for the status of blockers
08/15/2023
- Blockers / Critical issues (last 7 days)
- blocker, Dalong is working on it, waiting for Lincoln Lee to review
- blocker, PR is ready and Chesnay is reviewing it.
- Benchmark: TPC-DS result? Qingsheng Ren
- Cutting the branch: we'll wait for the status of blockers
08/22/2023
- Blocker / Critical issues (last 7 days)
- new CI instability. Qingsheng Ren to ping the author
- Cutting 1.18 branch
- The branch will be cut today in order to unblock the master branch Jing Ge
- Release testing
- Issues are mistakenly assigned to the original author of new features. Qingsheng Ren will correct them. The original author should provide instruction / docs to help volunteers to do the verification
- We target to finish all tests in 2 weeks. We'll make sure that all tickets are assigned on next Tuesday, and close them all before the next Tuesday after.
- Release Announcement
- All contributors with the link can edit / comment now
- Release Managing
- Sergey Nuyanzin will clone the umbrella issue we used in 1.17 for tracking task assignees and the entire progress of promoting the release.
08/29/2023
- Release Testing
- 2 unassigned issues:
- No descriptions
- Progress 50%, with 15 open / 30 total
- Try to close all before next Tuesday
- 2 unassigned issues:
- Blocker / Critical issues (last 7 days)
- Blocker
- Critical
- Blocker
- CI status
- release-1.18 cron build?
09/05/2023
- Release Testing:
- Blocker / Critical issues (last 7 days)
- Blocker
- Critical
- Major
-
- CI status
last ci with nightly (20230903.4) is green https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=52961&view=results
09/12/2023
- Release Testing:
-
- Blocker / Critical issues (last 7 days)
- Blocker
- Blocker
- CI status
- last green was on 3rd of September, after that blocked because of blocker above
09/19/2023
- Blocker / Critical issues (last 7 days)
- Critical
- Critical
- CI status
- last green nightly was on 16th of September https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53296&view=results