...

Feel free to join on Google Meet. Local dial-in numbers can be found at https://tel.meet/wcx-fjbt-hhz?pin=1940846765126

Timeline

• Feature Freeze
  • January, 17th of 2023, end of business CEST.

...

• • January 31st, 2023, end of business CEST (ML discussion on change)
• Release
  • Mid of March 2023
  • End of March 2023

Chinese new year is 22 January: 1+ week holiday. Ppl will be back in early FebruaryRelease
Mid of March 2023

Highlight features

please feel free to add/suggest.

...

symbol	meaning	comment
	validated	through cross team testing
	done	well documented with a complete test coverage
	will make it	there is no reason this effort should not go into 1.17
	in danger	there are some concerns the effort could be ready for the feature freeze of 1.17
	very unlikely	there are severe concerns the effort could make it to 1.17
	won't make it	it was decided against adding this for the 1.17 release. working on the effort has been stopped.
	state unclear
	independent	as the artefact artifact could be released independent of Apache Flink

...

Numbers are based on the items in the list below, not on the tickets 

110753031

									∑	Remaining weeks
2022-11-01										13
2022-11-15										11
2022-11-29		4	10	0	0	0	23	0	37	9
2022-12-13		7	24	1	0	1	9	0	42	7
2022-12-27		7	25	1	0	1	8	0	42	5
2023-01-	2	2023-01-10	10		9	27	0	0	3	5	0	44	3
2023-01-17		13	23	0

List

List

Feel Feel free to add categories.

Runtime
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified

Shuffle

Jira server ASF JIRA

columnIdsissuekey,summary

serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29801	Hang Ruan			FLIP in voting
Shuffle
Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29766	Weijie Guo	Xintong Song

100%

10-01-2023

Adaptive Batch Scheduler supports hybrid shuffle modeWeijie Guo Xintong Song 

10-01-2023

14-02-2023

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

30469

30938
AdaptiveBatchScheduler should supports early consumption for dynamic graph.	Weijie Guo

Yuxin Tan

Xintong Song

50%

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30469	Yuxin Tan	Xintong Song				100%	17-01-2023	17

09

-01-2023	10-02-2023	self-test
Further improvement of production availability of hybrid shuffle	Weijie Guo	Xintong Song

50%

100%

17-01-2023

17

10

-01-2023		self-test
Deployment & Cluster Coordination
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
REST API
Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-27060	ConradJam

100%

15

20-

11

02-

2022

2023	15-11-2022	15-11-2022	self-test
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30583	Rui Fan	Xintong Song		The first stage is finished, and the second stage is improvement, it will be finished in the next version.		100%	09-01-2023	05-01-2023	09-01-2023
Reactive Mode
n/a
Scheduler
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29663	Lijie Wang	Zhu Zhu

80%

100%

03

17-01-2022

FLIP-281: Speculative execution for sink

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30725	Biao Liu	Zhu Zhu

FLIP in discussion30%

100%

31

03

-01-2022

FLIP-283: Use adaptive batch scheduler as default scheduler for batch jobs

Junrui Lee 

Zhu Zhu 

FLIP in discussion30%

		Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31005
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30682	Junrui Lee	Zhu Zhu				100%	31

03

-01-2023
Misc
n/a
OLAP
n/a
State backend
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
FLIP-263: Improve resolving schema compatibility Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29844	Hangxiang Yu	Yuan Mei

80%

10

31-1-2023
Improve File Management in State Backend	Zakelly Lan  Yanfei Lei  Han Yin	Yuan Mei

5%

13

31-

12

1-

2022

2023
Improve the serializer performace of state change of changelog Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30345	Hangxiang Yu	Roman Khachatryan Yuan Mei

100%

10

31-1-2023			self-test
Allow to configure Changelog Storage per program Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26372	Roman Khachatryan Yanfei Lei   Hangxiang Yu	Yuan Mei

70%

10

31-1-2023
Add a metric for back-pressure from the ChangelogStateBackend Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-24402	Yanfei Lei	Yuan Mei

40%

13

31-

12

1-

2022

2023
Frocksdb cannot run on Apple M1 Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-24932	Yun Tang  Yanfei Lei	Yuan Mei  Yun Tang

90%

100%

13

30-

12

1-

2022CheckpointName/JIRA IssueResponsible ContributorReviewer / committer availableFeature StageNoteState%UpdatedImplementedDocumentedX-team verifiedBenchmark Generic Incremental CP + UC + BD

Rui xia

Yanfei Lei 

Hangxiang Yu 

Yuan Mei  JiraserverASF JIRA

2023			Tested  by Martijn Visser
Release FRocksDB 6.20.3-ververica-2.0 Jira server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution

serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

26803

30836

Rui Fan

Yanfei Lei

Piotr Nowojski

Yuan Mei

70%10-12-2022

Yun Tang				100%	31-1-2023			Tested by Martijn Visser
Checkpoint

Benchmark

Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified

Improve benchmark stability

Benchmark Generic Incremental CP + UC + BD	Rui xia Yanfei Lei  Hangxiang Yu	Yuan Mei
Jira server ASF JIRA

columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolutioncolumnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolutionserverId5aa69414-a9e9-3523-82ec-879b028fb15bkeyFLINK-29825

Yanfei Lei 

Yuan Mei 50%3-1-2023

serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26803	Rui Fan	Piotr Nowojski				100%	29 Jan 2023
Benchmark

API JiraserverASF JIRA

Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
Improve benchmark stability Jira server ASF JIRA

serverId5aa69414-a9e9-3523-82ec-879b028fb15bkeyFLINK-29740

Martijn Visser 

Deprecated100%08-11-202208-11-202208-11-2022

columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution

serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

28641

29825

Martijn Visser

Yanfei Lei

Removed

Yuan Mei

40%

50%

08-11-2022

31-1-2023
API
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

24456

29740	Martijn Visser		Deprecated			100%	08-11-2022	08-11-2022	08-11-2022	Self-tested
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

18647

28641	Martijn Visser		Removed			40%	31-01-2023
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-24456	Dawid Wysakowicz					90%	31-01-2022
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18647	Yun Gao		FLIP Discussion	Need to be postponed to the next release			10-01-2023
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29668	Martijn Visser		Removed

		100%	14-11-2022	14-11-2022	14-11-2022	Self-tested
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30050

						90%	31-01-2022			Self-tested
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-25509	Hang Ruan	Dong Lin	Coding
SQL
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
General
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29942	Godfrey He ,@Yunhong Zheng

70%

100%

2023-01-

10

31
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-28016	Chesnay Schepler

					31-01-2023
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29281	Chesnay Schepler

						31-01-2023
Table API

Calcite Update

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

20873

30025	Jing Ge						2023-01-20
Calcite Update
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-20873	Sergey Nuyanzin
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-21239	Sergey Nuyanzin
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29932	Sergey Nuyanzin

						31-01-2023
Flink Dialect
ALTER TABLE API Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-21634	Shengkai Fang

90%

100%

2023-

01Hive Dialect

02-

10DELETE/UPDATE API (FLIP-282) 

luoyuxia 

voting2023-01-10

14

self-tested

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

29635

30648

luoyuxia

60%

					100%	2023-02-14			self-tested
Hive Dialect
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

29717

29635

ron liu 

80%

luoyuxia

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30951

2023-01-10

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

26603

29717

luoyuxia

ron liu

100%	2023-01-31

SQL Gateway

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

29941

26603

Shengkai Fang

luoyuxia

, Zelin Yu202370%

31-01-

10

2022

QEFLIP-280: Introduce a new explain mode to provide SQL advice

Jane Chan 

voting2023-01-10QO

Image RemovedFLINK-27591 - Improve the plan for batch queries when statistics is unavailable OPEN

Godfrey He ,@Yunhong Zheng

40%API/PythonName/JIRA IssueResponsible ContributorReviewer / committer availableFeature StageNoteState%UpdatedImplementedDocumentedX-team verified Jira

SQL Gateway
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29941	Shengkai Fang  , Zelin Yu			writing doc		100%	2023-01-10			Jira

server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

29155HuangXingbo 100%10-01-2023

30936

QE

Jira server ASF JIRA

serverId5aa69414-a9e9-3523-82ec-879b028fb15bkeyFLINK-29833HuangXingbo 0%10-01-2023

columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution

JiraserverASF JIRA

serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

28957

30650

HuangXingbo

Jane Chan

70%

100%

10

2023-01-

2023

10

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

29421

31025

HuangXingbo 0%10-01-2023

QO
Image AddedFLINK-27591 - Improve the plan for batch queries when statistics is unavailable OPEN	Godfrey He ,@Yunhong Zheng			Need to be postponed to the next release		40%	2023-1-31
API/Python

Jira
server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-21223

HuangXingbo 100%10-01-2023Machine Learning

Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified

CEPName/JIRA IssueResponsible ContributorReviewer / committer availableFeature StageNoteState%UpdatedImplementedDocumentedX-team verifiedWebName/JIRA IssueResponsible ContributorReviewer / committer availableFeature StageNoteState%UpdatedImplementedDocumentedX-team verified

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29155	HuangXingbo					100%	10-01-2023			self-test
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29833	HuangXingbo					0%	10-01-2023
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

29995

Yu Chen

Yun Tang 

Status / Follow-ups

2022-11-01

Agenda

...

28957	HuangXingbo					100%	19-01-2023			self-test
Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29421	HuangXingbo					100%	19-01-2023			self-test

...

...

21223	HuangXingbo					100%	10-01-2023			self-test
Machine Learning
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
n/a
CEP
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified
n/a
Web
Name/JIRA Issue	Responsible Contributor	Reviewer / committer available	Feature Stage	Note	State	%	Updated	Implemented	Documented	X-team verified

2022-11-15

Agenda

...

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29387

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29315

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29818

...

Jira server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29995	Yu Chen	Yun Tang				100%

Status / Follow-ups

2022-11-01

Agenda

• Kickoff
• Keeping the state of features updated (ideally before the sync)
• Blockers
  -18356

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

    29899

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26674

  24119

• Build stability
• Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?

2022-11-

...

15

Agenda

• Kickoff
• Keeping the state of features updated (ideally before the sync)
• Blockers

...

• Build instabilities

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-28766

    : Anton has some new findings on that issue and will get back on it

    29387

  • Pulsar-related issues:

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

    29835

    29315

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

    29755

    29818

• Build stability
  • Number of growing test stability issues with "Exit code 137 errors"
    • Jiraserver

      Jira
      server

      ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

      29836
    Python-related issues:

    • 18356

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29461

      Matthias Pohl pings Xingbo Huang / Dian Fu 

      29899

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26974-26674

• Are there any (new) contributors who need a PR reviewed or merged? And if yes, who can help out?

2022-11-29

• Build instabilities

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18356

    : OOM errors are most likely being caused by flink-table-planner (Godfrey He might be a person to reach out to about it)

    28766

    : Anton has some new findings on that issue and will get back on it
  • Pulsar-related issues:

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

    29427: Qingsheng Ren will look into the PR

    • 29835

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

    27916 Matthias Pohl will ping contributors on the issue - but it's not that urgent since it's not failing that frequently 
• Other topics:
  • Externalizing Pulsar connector (test instabilities): Martijn Visser  is working on externalizing the code base
  • Performance test monitoring: Discussion is happening on the mailing list
  • Externalizing connectors in general is work-in-progress but looks good
  • Public CI documentation can be improved
    • Matthias Pohl will work on a first approval

2022-12-13

...

• • • 29755

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29836

  • Python-related issues:

  • • • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

  ...

  • • • 29461

        Matthias Pohl pings Xingbo Huang / Dian Fu 

  • • • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

  ...

  • • • 26974

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18356

  ...

  • • : OOM errors are most likely being caused by flink-table-planner (Godfrey He might be a person to reach out to about it)

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

  ...

  • • 29427

      : Qingsheng Ren will look into the PR

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK

  ...

  2022-12-27

  ...

  • • -27916

      Matthias Pohl will ping contributors on the issue - but it's not that urgent since it's not failing that frequently 
  • Other topics:
    • Externalizing Pulsar connector (test instabilities): Martijn Visser  is working on externalizing the code base
    • Performance test monitoring: Discussion is happening on the mailing list
    • Externalizing connectors in general is work-in-progress but looks good
    • Public CI documentation can be improved
      • Matthias Pohl will work on a first approval

  2022-12-13

  ...

  • Build instabilities

    • Jira

  • • server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

  ...

  • • 29405

      → Qingsheng to have a look at the PR
    • Pulsar connector has been synced to external connector repository. Martijn Visser to open a PR to remove connector from master

  2023-01-10

  ...

  • • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18356

      → Qingsheng to ping Godfrey

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

  ...

  • • 18356

      → The PR is still failing for the same issue as this PR should fix. We should ping the author to have a look first

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-

  ...

  • Blocker: New test instabilities which were introduced in unreleased versions only
  • Critical: New test instabilities that also affect released versions; Test instabilities that pop up more than two times per month (because they affect CI maintenance)
  • Major: Test instabilities that popped up after the version was released but happen rarely

  • • 27916

      → Martijn to ping Thomas once more

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26974

      → Xingbo is working on this
  • How to have monitoring and quality control for the externalized connectors → Need to have a discussion on the Dev mailing list. Martijn Visser to make a proposal and open a discussion thread on this topic.

  2022-12-27

  • Meeting skipped due to Christmas holiday/sick leaves
  • Discussion started on moving the feature freeze from Jan 17 to Jan 31 due to pandemic situation in China (see dev ML discussion thread)
  • Pulsar connector has been externalized. Pulsar-related test instabilities were disabled in release-1.16  and release-1.15  (see

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30351

    and parent task)

  2023-01-10

  • Build instabilities (all 1.17 test instabilities with a priority >=Major)

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18356

      
      
      • Godfrey He and Yunhong Zheng are working on it
      • run tests after each other/not in parallel
      • don't reuse JVMs
      • JUnit has feature to log memory consumption

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26974

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29427

  • Priorities of test instabilities (docs about it)
    • Test instabilities are prioritized as Critical and become blocker as soon as we notice that they are newly introduced
  • Feature freeze extended until Jan 31, 2023
  • Switching to weekly calls as we're getting closer to the feature freeze?
    • Yes, switch to weekly will happen. Next meeting will be Jan 17 due to Chinese New Year coming up and the feature freeze happening soon  
  • Votes on FLIPs are stalled due to outstanding votes
    • Qingsheng Ren will reach out to Martijn Visser  about it
    • Share in the Slack dev channel to get more people to look into it
  • leader elections lacks test coverage (FLIP-285,

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-26522

    )
    • No issues popped up on the mailing list since 1.16

  2023-01-17

  • Build instabilities (all 1.17 test instabilities with a priority >=Major)
    • Blockers
      • Performance regressions

        • Jira
          server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30623

          → In progress

        • Jira
          server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30625

          → In progress

        • Jira
          server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30624

          → In progress

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29405

        → Qingsheng will look into this one after FLINK-29427 (see below)

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29427

        → Qingsheng currently is looking into this 
    • Martijn Visser to check open critical test stabilities to determine if some need to be assigned / if we're OK in the overall direction
    • FLIP-272: Generalized delegation token support has been merged in and a blog post will be written and published about it, prior to the 1.17 release
    • https://github.com/apache/flink/pull/21606 has been merged into Flink, but should also be taken into account for externalized connectors - This is tracked under

      Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30639

  2023-01-24

  
  

  • Build instabilities (all 1.17 test instabilities with a priority >=Major)
    • Performance regressions

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30623

        → In progress

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30625

        → In progress

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30624

        → In progress
    • Blockers

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29405

        → In progress, needs a status update

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30727

        → In progress, needs a status update

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30328

        → Resolved

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29427

        → In progress, needs a status update

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30618

      → Martijn Visser is looking into this one

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30733

      → Martijn Visser is looking into this one

  2023-01-31

  • Today is feature freeze day
  • 26 features / improvements are in for Flink 1.17 (47 in at Flink 1.16, 20 for Flink 1.15, 27 for Flink 1.14)
    • 5 features are still listed as expected to be completed, but are not yet in, 4 of them have been merged and are writing documentation,  Martijn Visser to check/update for the status of these items. 
  • Blockers:
    

    • • Jira
        server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,customfield_12311032,customfield_12311037,customfield_12311022,customfield_12311027,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,Priority,Priority,Priority,Priority,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30881

        → Matthias will look into this 

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29427

        → Leonard review PASS, waiting CI green

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-29405

        → Fixed by Qingsheng

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30625

        → Should be resolved, pending validation by the benchmarks (related to FLINK-30624). Benchmarks look to be improved, downgraded to Critical. 

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30624

        → Should be resolved, pending validation by the benchmarks (related to FLINK-30625). Benchmarks look to be improved, downgraded to Critical. 

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30623

        → Martijn to reach out to Dong, Rui Fan, Piotr. We're planning to give them until Friday the 3rd of February to come to a conclusion on this ticket; if no consensus is achieved, then the original commit that introduced the regression should be reverted. 

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30826

        → Matthias to check if this has already been resolved via another ticket (multiple related tickets)

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30727

        →  [Critical]  PR to update buffers for the test has been merged, will be continued to monitor. Test downgraded to Critical

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30846

        →  [Critical]  Downgraded to Critical as it's only a test-related issue and doesn't indicate a bug in production.

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30844

        →  [Major] Test downgraded to Major, If the test fail again contributor will increase the waiting interval for this test as a solution.

      • Jira
        server ASF JIRA columnIds issuekey,summary,issuetype,created,updated,duedate,assignee,reporter,customfield_12311032,customfield_12311037,customfield_12311022,customfield_12311027,priority,status,resolution columns key,summary,type,created,updated,due,assignee,reporter,Priority,Priority,Priority,Priority,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30870

        →  [Major]  Leonard downgrade the issue priority to Major as it's a known slack plugin issue
  • Martijn Visser to communicate to the Flink community this evening that the feature freeze has started and we plan to cut the release branch at the end of this week (Friday 3rd of February). 

  2023-02-07

  • master  is stabilized enough to cut the release-1.17  branch
    • FLINK-30921 - The Azure apt mirror instabilities seem to have been resolved for now. 
      
    • FLINK-30908 - The issue turned out to be a problem that existed in previous releases
  • Release branch is going to be cut today by Leonard Xu 
  • Release testing will be announced: 2 weeks will be planned for this

  2023-02-14

  • Status update on release testing efforts

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30926

    • End of cross-team testing date is 21st of February 2023. We will monitor the status throughout the week and hopefully conclude everything next week. 
    • When a feature is cross-team tested, the icon needs to be changed from to to indicate that the testing has been completed
    • We always look for volunteers: picking up a cross-team testing task is much appreciated. 
  • Proposal: Create Jira issues for release management tasks to document what was done to improve review-ability (alternatively, add expected output to release documentation)
    • Do this from now on for the next steps in release management (create release candidate etc.)
  • Test instabilities:

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31036

      : Blocker 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18356

      : Starts failing more regularly again. Qingsheng Ren to ping Godfrey He and Yunhong Zheng 
    • FLINK-30972 (openssl version update necessary) continues to fail because ofFLINK-30965 (repo-sync doesn't pick up 1.15 changes anymore)

  2023-02-21

  • Open Blocker issues
  • Example Release Jira issues:

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31146

     >

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31154

  • Release testing not finished, yet
  • Decision on RC creation moved to next week's Flink release sync call
  • Other issues:

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31133

      : Issue in 1.15.3 unexplained

  • • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31092

      : 1.17 issue in Hive

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30733

      : no updates on the Slack bot instability so far

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31134

      : OOM issue - waiting for Gabor's response

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31145

      (Kafka infrastructure umbrella ticket)

  2023-02-28

  • Blockers: are these really blockers or do we want to move the release forward?

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31092

       → Qingsheng Ren to check the status with Shengkai Fang 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31104

      → Qingsheng Ren to check the status with Shengkai Fang 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30978

       → Qingsheng Ren to check the status with Shengkai Fang 
  • Other issues:

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31134

      : Revisit OOM in Kafka e2e test in 1.15.3 - we're keeping it as critical for now. There are no other artifacts that we can investigate. The test doesn't involve table-runner code (therefore, isn't connected to

      Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-18356

      ). → Martijn Visser to check internally
    • Dependabot alerts are affecting all commiters now
  • Release managers to review draft of 1.17 announcement
  • Release 1.17 preparation:

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31146

  2023-03-07

  • Ubuntu mirror instabilities: 

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30921

    Matthias Pohl will follow-up on that one
  • API backwards compatibility:

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31167

    Do we have additional documentation on API compatibility? Leonard Xu will do another pass over the ticket comments to confirm the findings
  • Announcement: FLIPs missing: FLIP-272: Generalized delegation token support, FLIP-217: Support watermark alignment of source splits Leonard Xu reaches out to contributors to add something related to the announcement
  • Instabilities (blocker query):

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31351

      :Leonard Xu has pinged luoyuxia  to look into this issue. 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31278

      : Downgraded→  [Critical] PR ready, flink-runtime tests will be made sequential ,

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31339

      : Downgraded→  [Critical] Leonard Xu has pinged Godfrey He to look into this issue.

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31342

      : Downgraded→  [Critical] 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31341

      : Downgraded→  [Critical] 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31354

      :  Downgraded→  [Critical] Matthias Pohl 

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31355

      : Downgraded→  [Critical]  Leonard Xu has pinged Dian Fu to look into this issue. 

  2023-03-14

  • Flink RC2 for 1.17 is available, needs checkers

  2023-03-21

  • RC3 for 1.17.0 was created and is in voting stage (1 binding vote miss)
  • Robert Metzger agreed to assist with the license check

  2023-03-23

  • Flink 1.17.0 is officially released today!

  Retrospective:

  From Qingsheng Ren 

  • As discussed in the mailing list, we need to trigger a final patch version for 1.15 after releasing 1.17. Some cleanup steps need to be reviewed and changed, such as removing 1.15 data from svn, CI, flink-docker etc. See

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31570

  • I like the idea made by Matthias Pohl that we track TODOs for releasing on JIRA 👍 I used it as a checklist to make sure we don't miss anything. Also it helps collaborating, as we can divide works across RMs easily by assigning JIRA tickets.

  From Matthias Pohl 

  • Google Meet might not be the best choice for the release sync. We need to be able to invite attendees even if the creator of the meeting isn't available (maybe try Zoom or even Jitsi as an OpenSource alternative instead?)

  • Release sync every 2 weeks and a switch to weekly after feature freeze felt reasonable

  • Slack worked well as a collaboration tool to document the monitoring tasks (#builds, #flink-dev-benchmarks) in a team with multiple release managers

  • The Slack Azure Pipeline bot seems to be buggy. It swallows some build failures. It's not a severe issue, though. We created #builds-debug to monitor whether it's happening consistently. The issue is covered in

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-30733

  • We experienced occasional issues in the manual steps of the release creation in the past (e.g. japicmp config was not properly pushed). Creating Jira issues for the release helped to make the release creation more transparent and made the steps more reviewable. Additionally, it helped to distribute subtasks to different people with Jira being the tool for documentation and synchronization. That's especially helpful when there is more than one person in charge of creating the release.

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31146

    • RCs

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31154

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31578

      • Jira
        server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31583

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31562

    • Jira
      server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31567

  • We had backports/merges without PRs happening by committers occasionally during the 1.17 release which broke master/release branches (probably, changes were done locally before merging which were not part of the PR to have a faster backport experience). It might make sense to remind everyone that this should be avoided. Not sure whether we want/can restrict that.

  • We observed a good response on fixing test instabilities by the end of the release cycle but had some long running issues earlier in the cycle which caused extra efforts on the release managers due to reoccurring test failures.

  • Release testing picked up “slowly”: Initially, we planned 2 weeks for release testing. But there was not really any progress (tickets being created and worked on) in the first week. In the end, we had to extend the phase by another week resulting in 3 instead of 2 weeks of release testing. I guess we could encourage the community to create release testing tasks earlier and label them properly to be able to monitor the effort. That would even enable us to do release testing for a certain feature after the feature is done and not necessarily only at the end of the release cycle.

  • Manual test data generation is tedious (

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-31593

    ). But this should be fixed in 1.18 with

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key FLINK-27518

    being almost done.
  • We started creating documentation for release management. The goal is to collect what tasks are there to help support a Flink release to encourage newcomers to pick up the task.

  From Leonard Xu 

  • We can keep RC0 (a non-votable one) in future releases, as an initial version for developers to validate, so that some issues could be found earlier and avoid repeatedly canceling and re-creating RCs. 

  From Martijn Visser 

  • We should be more careful for commits without a PR / green CI, which brought some problem at the end of 1.17 release cycle. There might not be possible to totally ban this, but we could give an reminder to committers. 

  ...