THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Table of contents
Jira Boards
Flink 1.16 Burndown: https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=548
Sync meeting
The sync meeting is happening every second Tuesday starting on the 17th of May at 9am CEST / 3pm China Standard Time / 7am UTC.
As we are getting closer to the feature freeze we will do the meeting on a weekly base.
Feel free to join on Google Meet. Local dial-in numbers can be found at https://tel.meet/out-cobp-hmp?pin=1681248614050
Timeline
Feature FreezeMonday, 25th of July 2022, end of business CEST.
Tuesday, 9th of August 2022, end of business CEST.
Release
End of September 2022
Highlight features
please feel free to add/suggest.
Features
List of features announced by contributors and committers that are likely to be ready for the feature freeze:n
NOTICE: It's preferred if only new features end up there and not all bugs/tasks separately, so that the page is not over bloated. Of course, unless fixing a bug is a really big or important one equivalent to implementing a completely new feature. A good rule of thumb would be that each entry in the page could (but does not have to) be later on included in a release blog post.
Legend
State
| symbol | meaning | comment |
|---|---|---|
 | validated | through cross team testing |
 | done | well documented with a complete test coverage |
 | will make it | there is no reason this effort should not go into 1.16 |
 | in danger | there are some concerns the effort could be ready for the feature freeze of 1.16 |
 | very unlikely | there are severe concerns the effort could make it to 1.16 |
 | won't make it | it was decided against adding this for the 1.16 release. working on the effort has been stopped. |
 | state unclear | |
 | independent | as the artefact could be released independent of Apache Flink |
X-Team verification
| symbol | meaning |
|---|---|
| done |
| not required |
Feature Stage
Please align with the list on the Apache Flink Roadmap (https://flink.apache.org/roadmap.html).
- MVP: Have a look, consider whether this can help you in the future.
- Beta: You can benefit from this, but you should carefully evaluate the feature.
- Ready and Evolving: Ready to use in production, but be aware you may need to make some adjustments to your application and setup in the future, when you upgrade Flink.
- Stable: Unrestricted use in production
- Reaching End-of-Life: Stable, still feel free to use, but think about alternatives. Not a good match for new long-lived projects.
- Deprecated: Start looking for alternatives now
Summary
Numbers are based on the items in the list below, not on the tickets
| | | | | | | | ∑ | Remaining weeks | |
| 2022-05-17 | 10 | ||||||||
| 2022-05-31 | 1 | 48 | 2 | 0 | 0 | 14 | 0 | 65 | 8 |
| 2022-06-14 | 3 | 41 | 3 | 0 | 1 | 16 | 0 | 64 | 6 |
| 2022-06-28 | 3 | 45 | 4 | 0 | 1 | 13 | 0 | 66 | 4 |
| 2022-07-05 | 8 | 44 | 4 | 0 | 4 | 7 | 0 | 67 | 3 |
| 2022-07-12 | 9 | 43 | 3 | 1 | 5 | 7 | 0 | 68 | |
| 2022-07-19 | 12 | 41 | 3 | 1 | 7 | 7 | 0 | 71 | |
| 2022-07-26 | 17 | 35 | 2 | 1 | 13 | 2 | 1 | 71 | |
| 2022-08-02 | 19 | 33 | 2 | 1 | 13 | 2 | 1 | 71 | 1 |
| 2022-08-09 | 47 | 0 | 0 | 0 | 25 | 0 | 1 | 73 | 0 |
List
Feel free to add categories.
| Runtime | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| | 100% | 2022-7-19 | self-test | ||||||
| Yun Gao | Will not be fully finished during 1.16 | | 10% | 2022-07-05 | |||||
| Xuannan Su | Yun Gao | Beta | | 100% | 2022-8-8 | ||||
| Shuffle | ||||||||||
(Single Flink Cluster Multiple Shuffle Plugin) | Yingjie Cao | PoC ready (Moved to 1.17) | | 30% | 09-08-2022 | |||||
(Sort Blocking Shuffle Optimization) | Merged | | 100% | 09-08-2022 | ||||||
(Blocking Shuffle Data Deduplication) | Merged | | 100% | 09-08-2022 | ||||||
(Introduce Shuffle Data Compression Algorithm with Higher Compression Ratio) | Merged | | 100% | 09-08-2022 | ||||||
| Deployment & Cluster Coordination | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | Updated | Implemented | Documented | X-team verified | |
| REST API | ||||||||||
| Reactive Mode | ||||||||||
| Scheduler | ||||||||||
| Speculative Execution | ||||||||||
| 100% | 08-08-2022 | ||||||||
| 100% | 25-07-2022 | ||||||||
| 100% | 01-08-2022 | ||||||||
| @Gen Luo | 100% | 03-08-2022 | |||||||
| Misc | ||||||||||
| MartijnVisser | Will not be fully finished during 1.16 | 20% | 09-08-2022 | ||||||
| | 100% | 19-07-2022 | |||||||
| OLAP | ||||||||||
| | 100% | 08-08-2022 | |||||||
| Will not be fully finished during 1.16 | | 19-07-2022 | |||||||
| State backend | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
changelog v2 | Yuan Mei Roman Khanchych Yun Tang Hangxiang Yu @Yanfei Lei | | 100% | 09-08-2022 | ||||||
RocksDB rescaling improvement & rescaling benchmark | @Yanfel LeiYuan Mei | | 100% | 30-05-2022 | ||||||
state metrics & monitor improvement | | 100% | 01-08-2022 | Yes | Yes | |||||
heap state-backend incremental checkpoint | Will not be fully finished during 1.16 | | 20% | 09-08-2022 | ||||||
| Support third-party state handle to support fully third-party state-backend | Might not be included in 1.16 | | 28-06-2022 | |||||||
| Checkpoint | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Piotr Nowojski | | 100% | 29-06-2022 | ||||||
| Rui Fan | | 100% | 29-06-2022 | ||||||
| Rui Fan | Piotr Nowojski | Will not be fully finished during 1.16 | | 60% | 09-08-2022 | ||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| maosuhan | | 100% | 28-06-2022 | ||||||
| | 10 | 2022-7-19 | |||||||
WAL Sink | Yun Gao | Will not be included in 1.16 | | 0% | 05-07-2022 | |||||
| | 10% | 2022-08-09 | |||||||
| Yufei Zhang | Pulsar needs to be stable before new features can be merged | | 10% | 2022-08-09 | |||||
| Yufei Zhang | Pulsar needs to be stable before new features can be merged | | 10% | 2022-08-09 | |||||
Python Wrappers for KDS/KDF sinks | | 100% | 2022-07-05 | |||||||
Introduce configurable RateLimitingStrategy for Async Sink | 100% | 2022-08-09 | ||||||||
| API | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| SQL | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| Flink Dialect | ||||||||||
| Mang Zhang | FLIP Accepted | Partially finished (non-atomic CTAS). Docs preparing. | | 50% | 2022-08-09 | ||||
| FLIP Accepted | Docs preparing. | | 80% | 2022-08-09 | |||||
| Hive Dialect | ||||||||||
| | 100% | 2022-08-09 | |||||||
| Will not be fully finished during 1.16 | | 25% | 2022-08-09 | ||||||
| The main part is merged. Docs preparing. | | 100% | 2022-08-09 | ||||||
| SQL Gateway | ||||||||||
| Shengkai Fang | FLIP in Accepted | The main part is merged. | | 100% | 2022-08-09 | ||||
| Shengkai Fang | FLIP in Accepted | Last PR reviewing. Docs preparing. | | 100% | 2022-08-09 | ||||
| QE | ||||||||||
FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems | lincoln lee | reviewing | | 100% | 2022-8-9 | |||||
| lincoln lee | main parts merged | | 100% | 2022-8-9 | |||||
| lincoln lee | | 100% | 2022-7-19 | ||||||
| lincoln lee | the error handling behaviors of stream operators need to be discussed futher, pending the pr for 1.16 | | 2022-8-9 | ||||||
| ron liu | | 100% | 2022-08-09 | ||||||
| Last PR is reviewing. | | 100% | 2022-08-09 | ||||||
bulit-in function enhance | | 10% | 2022-7-19 | |||||||
| | 0% | 2022-7-19 | |||||||
| FLIP-221 Abstraction for lookup source cache and metric | Qingsheng Ren | FLIP Accepted | Last PR is reviewing. | | 100% | 2022-08-09 | ||||
| QO | ||||||||||
| xuyang | FLIP Accepted | doc preparing | | 100% | 2022-8-2 |
| |||
| FLIP-???: Support SortBy & ClusterBy & DistributedBy | | 2022-8-2 | ||||||||
| FLIP Accepted | | 100% | 2022-8-9 | ||||||
| FLIP-240: Introduce ANALYZE TABLE Syntax | FLIP in Accepted | doc reviwing | | 100% | 2022-8-9 | |||||
| FLIP-248: Introduce dynamic partition pruning | Godfrey He | FLIP in Accepted | waiting CI | | 100% | 2022-8-9 | ||||
| Godfrey He | | 0% | |||||||
| | 0% | ||||||||
| | 0% | ||||||||
| | 0% | ||||||||
| @luoyuxia | | 60% | 2022-08-09 | ||||||
| API/Python | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| HuangXingbo | Dian Fu | Beta | Some cases supported won't be in release 1.16 | | 100% | 01-08-2022 | Yes | Yes |
|
| Dian Fu | Stable | | 0% | 26-07-2022 | |||||
| @Juntao Hu @zhangjingcun | Dian Fu | Stable | | 100% | 01-08-2022 | Yes | Yes |
| |
| @Juntao Hu | Stable | | 100 | 27-06-2022 | Yes | Yes | self-test
| ||
| @Juntao Hu | HuangXingbo | Stable | | 100% | 26-07-2022 | Yes | Yes | self-test
| |
| Shuiqiang Chen | Stable | | 100% | 11-07-2022 | Yes | Not needed | | ||
| Machine Learning | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| CEP | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| @Nicholas Jiang | | 100% | 14-06-2022 | ||||||
| @Nicholas Jiang | | 100% | 11-07-2022 | ||||||
| @Yue Ma | developing | | 50% | 11-07-2022 | |||||
FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP) | @Nicholas Jiang | discussing | | 20% | 18-07-2022 | |||||
| Web | ||||||||||
| Name/JIRA Issue | Responsible Contributor | Reviewer / committer available | Feature Stage | Note | State | % | Updated | Implemented | Documented | X-team verified |
| @Junhan Yang | | 100% | 19-07-2022 | ||||||
| @Junhan Yang | | 100% | 19-07-2022 | ||||||
| @Junhan Yang | | 100% | 19-07-2022 | ||||||
| @Junhan Yang | | 100% | 19-07-2022 | ||||||
| @Junhan Yang | reviewing | | 100% | 02-08-2022 | |||||
Status / Follow-ups
2022-05-17
Agenda
- Kickoff
- Keeping the state of features updated (ideally before the sync)
- Blockers
- Build stability
- Total test instabilities for currently supported versions: 171
- Please have a look at the test instabilities for components that you maintain or think you can help out with
- Most common failing tests
- Problems with build infrastructure
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-05-31
Agenda
- Overview on the effort.
- 65 features collected (1 done, 48 green, 2 in danger, 14 state unclear)
- Once more the request to update the state of all features before the sync
- Blockers
- There were no blockers, only critical tickets
- Build stability
- Godfrey He no more reported, watching
- Zhu Zhu
- Yang Wang
- Xingbo Huang
- lincoln lee local environment cannot reproduce and the testing itself is deterministic
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
- No help request was brought forward
- We briefly discussed if a community sync would be interesting. This will be brought up to the Dev mailing list
2022-06-14
Agenda
- Overview on the effort.
- Martijn Visser to ping Danny Cranmer on the status of AWS tickets
- Once more the request to update the state of all features before the sync
- Blockers
Martijn Visser to ping Atri Sharma for a status update
- Matthias Pohl
- Build stability
- @ferenc-csaky
- Xingbo Huang
- Martijn Visser to contact engineers from Cloudera with the help of Márton Balassi
- Chesnay Schepler
- Martijn Visser to contact the StreamNative engineers
- Pinged Qingsheng Ren
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
- No help request was brought forward
2022-06-28
Agenda
- Overview on the effort
- Blockers
- → https://github.com/apache/flink/pull/19473 should help here, but needs work from someone to improve that PR overall
- Build stability
- Martijn Visser
- Martijn Visser
- @masteryhx
- Yun Gao
- Godfrey He no more reported, watching
- @masteryhx
- Setting up Slack channel with build pipeline status
- Looking for someone who has the right access to help here (Maybe Matthias Pohl ?)
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-07-05
Agenda
- Overview on the effort
Blockers
- Critical test stabilities, some highlights. We have multiple critical test stabilities overall. What do we want to do with them?
- → Needs a reviewer
- → Needs a volunteer
- → WIP
- → WIP
- → WIP
- → Needs a status update / volunteer
- → WIP
- → Needs a volunteer
- → WIP
- → Needs a volunteer
- Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?
2022-07-11
Agenda
- Overview on the effort
Blockers
- → Is pending on a review of / https://github.com/apache/flink/pull/19972
- Critical test stabilities: multiple are pending, only listing those that are not yet assigned and therefore need a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
2022-07-19
Agenda
- Overview on the effort
- Blockers
- Tison is helping review
- Critical test stabilities: multiple are pending, only listing those that are not yet assigned and therefore need a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
- → Needs a volunteer
2022-07-26
Agenda
- Overview on the effort
- Blockers
- Still being reviewed
- Robert will reply in the ticket
2022-08-02
- Overview on the effort
- Blockers
- Tison is helping review
2022-08-09
- Cut branch tomorrow
- Will ask for cross team testers after meeting
- Overview on the effort (waiting to merge into release-1.16)
- Waiting CI
- Waiting CI
- Waiting CI
- Waiting CI
- Waiting CI
- Reviewing
- Reviewing
- Reviewing
- Blockers
- Tison is helping review
2022-08-16
- X-team testing:
- 51 test stability issues: https://issues.apache.org/jira/issues/?filter=12352149
- Blockers
- Start moving unrelated features to 1.17 ?
2022-08-23
- X-team testing:
- 55 test stability issues: https://issues.apache.org/jira/issues/?filter=12352149
- Blockers
- (Qingsheng will fix this ASAP)
- (the pr is ready)
- unrelated issues cleanup:
- Improvement and "New Feature" remaining in 1.16: https://issues.apache.org/jira/issues/?filter=12352190
All unresolved issues in 1.16: https://issues.apache.org/jira/issues/?filter=12352193
2022-08-30
- X-team testing: (6 - finished, 2 - 90% , 5 - 30%, 4 - will start in these two days)
- 20 critical test stability issues: https://issues.apache.org/jira/issues/?filter=12352149 (Thanks Chesnay Schepler for cleaning up many test issues)
- Blockers
- (xuyang has prepared a PR to fix it.)
- (Qingsheng Ren has locate on the root cause, but needs some discussions to decide how to fix it.)
- when to cut branch ? September 1st?
2022-09-06
- X-team testing (14 - finished, 3 -90% (found some bugs need to be fixed))
- 17 critical test stability issues: https://issues.apache.org/jira/issues/?filter=12352149
- Blockers
- (xuyang https://github.com/apache/flink/pull/20697 revewing)
- (Qingsheng Ren https://github.com/apache/flink/pull/20760 need a reviewer - Asked Piotr Nowojski for a review)
- (@Yunfeng Zhou https://github.com/apache/flink/pull/20752 reviewing)
- (HuangXingbo https://github.com/apache/flink/pull/20758 need a reviewer)
- has cut branch release-1.16 (when to prepare rc0)
- license check (Thanks Chesnay Schepler for checking, only found flink-python NOTICE problem. )
2022-09-13
- X-team testing （done)
- 6 critical test stability issues (After cleanup the issues that haven't appreared in months)
- (@zelin is helping fix it, which is expected to be fixed in the next two day)
- (@yanfei is helping fix it, which is expected to be fixed in this week)
- (https://github.com/apache/flink/pull/20734 is ready, Qingsheng Ren is helping review)
- (need to be familiar with unaligned checkpoint)
- https://issues.apache.org/jira/browse/FLINK-28352?filter=12352149 (Pulsar test stability umbrella, maybe we can downgrade the priority)
- (maybe ping @Jing Ge)
- Blockers
- (Weijie Guo https://github.com/apache/flink/pull/20796 reviewing. It is expected to be fixed in this week, but it won't block rc0)
- release announcement (Godfrey He 50% Chinese doc, he will share the English doc this week)
- Others
2022-09-20
- critical test stability issues (https://issues.apache.org/jira/issues/?filter=12352149 )
- Blockers
- release note https://github.com/apache/flink/pull/20859
- release announcement https://docs.google.com/document/d/1rIBNpzJulqEKJCuYtWtG-vDmSsGpN9sip_ewpzMequ0/edit#
- Others
2022-09-27
- critical test stability issues (https://issues.apache.org/jira/issues/?filter=12352149 )
- Blockers
- (Yun Gao is helping fix it, which is expected to be fixed today)
- release note: https://github.com/apache/flink/pull/20859
- release announcement https://docs.google.com/document/d/1rIBNpzJulqEKJCuYtWtG-vDmSsGpN9sip_ewpzMequ0/edit#
- Others
2022-10-04
- critical test stability issues (https://issues.apache.org/jira/issues/?filter=12352149 )
- Blockers
- There were no blockers, only critical tickets
- rc1 is in vote
- Others
2022-10-11
- critical test stability issues (https://issues.apache.org/jira/issues/?filter=12352149 )
- Blockers
- （Qingsheng Ren is fixing it, which is expected to be fixed before Thursday)
- Others
2022-10-18
- critical test stability issues (https://issues.apache.org/jira/issues/?filter=12352149 )
- Blockers
- Others
- Temporary infrastructure issue:
2022-10-25
- critical test stability issues (https://issues.apache.org/jira/issues/?filter=12352149 )
- rc2 is in vote