You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 625 Next »

https://lists.apache.org/thread/59mj7nwkjn00f140nfftomfqmtrzdpg5

This page describes the Flink Improvement Proposal (FLIP) process for proposing a major change to Flink.

To create your own FLIP, click on "Create" on the header and choose "FLIP-Template" other than "Blank page".

Purpose

The purpose of FLIPs is to have a central place to collect and document planned major enhancements to Apache Flink. While JIRA is still the tool to track tasks, bugs, and progress, the FLIPs give an accessible high level overview of the result of design discussions and proposals. Think of FLIPs as collections of major design documents for user-relevant changes.

We want to make Flink a core architectural component for users. We also support a large number of integrations with other tools, systems, and clients. Keeping this kind of usage healthy requires a high level of compatibility between releases — core architectural elements can't break compatibility or shift functionality from release to release. As a result each new major feature or public API has to be done in a way that we can stick with it going forward.

This means when making this kind of change we need to think through what we are doing as best we can prior to release. And as we go forward we need to stick to our decisions as much as possible. All technical decisions have pros and cons so it is important we capture the thought process that lead to a decision or design to avoid flip-flopping needlessly.

Hopefully we can make these proportional in effort to their magnitude — small changes should just need a couple brief paragraphs, whereas large changes need detailed design discussions.

This process also isn't meant to discourage incompatible changes — proposing an incompatible change is totally legitimate. Sometimes we will have made a mistake and the best path forward is a clean break that cleans things up and gives us a good foundation going forward. Rather this is intended to avoid accidentally introducing half thought-out interfaces and protocols that cause needless heartburn when changed. Likewise the definition of "compatible" is itself squishy: small details like which errors are thrown when are clearly part of the contract but may need to change in some circumstances, likewise performance isn't part of the public contract but dramatic changes may break use cases. So we just need to use good judgement about how big the impact of an incompatibility will be and how big the payoff is.

What is considered a "major change" that needs a FLIP?

Any of the following should be considered a major change:

  • Any major new feature, subsystem, or piece of functionality
  • Any change that impacts the public interfaces of the project

What are the "public interfaces" of the project?


All of the following are public interfaces that people build around:

  • DataStream and DataSet API, including classes related to that, such as StreamExecutionEnvironment
  • Classes marked with the @Public annotation
  • On-disk binary formats, such as checkpoints/savepoints
  • User-facing scripts/command-line tools, i.e. bin/flink, Yarn scripts, Mesos scripts
  • Configuration settings
  • Exposed monitoring information


Not all compatibility commitments are the same. We need to spend significantly more time on public APIs as these can break code for users. They cause people to rebuild code and lead to compatibility issues in large multi-dependency projects (which end up requiring multiple incompatible versions). Configuration, monitoring, and command line tools can be faster and looser — changes here will break monitoring dashboards and require a bit of care during upgrades but aren't a huge burden.

For the most part monitoring, command line tool changes, and configs are added with new features so these can be done with a single FLIP.

What should be included in a FLIP?

A FLIP should contain the following sections:

  • Motivation: describe the problem to be solved
  • Proposed Change: describe the new thing you want to do. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences, depending on the scope of the change.
  • New or Changed Public Interfaces: impact to any of the "compatibility commitments" described above. We want to call these out in particular so everyone thinks about them.
  • Migration Plan and Compatibility: if this feature requires additional support for a no-downtime upgrade describe how that will work
  • Rejected Alternatives: What are the other alternatives you considered and why are they worse? The goal of this section is to help people understand why this is the best solution now, and also to prevent churn in the future when old alternatives are reconsidered.

Who should initiate the FLIP?

Anyone can initiate a FLIP but you shouldn't do it unless you have an intention of getting the work done to implement it (otherwise it is silly).

Process

Here is the process for making a FLIP:

  1. Create a page which is a child of this one. Take the next available FLIP number (below) and give your proposal a descriptive heading. e.g. "FLIP 42: Enable Flink Streaming Jobs to stop gracefully". If you don't have the necessary permissions for creating a new page, please ask on the development mailing list.
  2. Fill in the sections as described above
  3. Start a [DISCUSS] thread on the Apache mailing list. Please ensure that the subject of the thread is of the format [DISCUSS] FLIP-{your FLIP number} {your FLIP heading} The discussion should happen on the mailing list not on the wiki since the wiki comment system doesn't work well for larger discussions. In the process of the discussion you may update the proposal. You should let people know the changes you are making.
  4. Once the proposal is finalized call a [VOTE] to have the proposal adopted. These proposals are more serious than code changes and more serious even than release votes. The criteria for acceptance is consensus.
  5. Please update the FLIP wiki page, and the index below, to reflect the current stage of the FLIP after a vote. This acts as the permanent record indicating the result of the FLIP (e.g., Accepted or Rejected). Also report the result of the FLIP vote to the voting thread on the mailing list so the conclusion is clear.

FLIP round-up

Next FLIP Number: 261

Other free numbers: 178,230

Use this number as the identifier for your FLIP and increment this value.

Adopted/Accepted but unreleased FLIPs

FLIPTarget ReleaseLink to Discussion Thread
FLIP-50: Spill-able Heap Keyed State Backend1.10https://lists.apache.org/thread.html/a10cae910f92936783e59505d7b9fe71c5f66ceea4c1c287c87164ae@%3Cdev.flink.apache.org%3E
FLIP-99: Make Max Exception Configurable1.11

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-99-Make-Max-Exception-Configurable-td37895.html

FLIP-100: Add Attempt Information1.11

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-75-Flink-Web-UI-Improvement-Proposal-td33540.html

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-100-NEW-Add-Attempt-Information-td38453.html

FLIP-128: Enhanced Fan Out for AWS Kinesis Consumers1.12

discuss: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-128-Enhanced-Fan-Out-for-AWS-Kinesis-Consumers-td42720.html

vote: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-128-Enhanced-Fan-Out-for-AWS-Kinesis-Consumers-td42846.html

FLIP-130: Support Python DataStream API (Stateless part)1.12

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-td43035.html

Vote thread:  http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-130-Support-for-Python-DataStream-API-Stateless-Part-tt43424.html

FLIP-133: Rework PyFlink Documentation

1.11

Discussion thread: https://lists.apache.org/thread.html/redebc9d1281edaa4a1fbf0d8c76a69fcff574b0496e78519840a5a61%40%3Cdev.flink.apache.org%3E

Vote thread: https://lists.apache.org/thread.html/rf21ef9bd45c675b836370165a21feaa5b55e9c93ea94c50eb8e3aae7%40%3Cdev.flink.apache.org%3E

FLIP-137: Support Pandas UDAF in PyFlink1.12

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-137-Support-Pandas-UDAF-in-PyFlink-tt44060.html

Vote thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-137-Support-Pandas-UDAF-in-PyFlink-tc44384.html

FLIP-139: General Python User-Defined Aggregate Function Support on Table API1.12

Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-139-General-Python-User-Defined-Aggregate-Function-on-Table-API-td44139.html

Vote thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-139-General-Python-User-Defined-Aggregate-Function-on-Table-API-td44421.html

FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)1.12

DISCUSS: https://lists.apache.org/thread.html/r4f24c4312cef7270a1349c39b89fb1184c84065944b43aedf9cfba6a%40%3Cdev.flink.apache.org%3E

VOTE: https://lists.apache.org/thread.html/r09c17db5599b01faba33c2fa112ef0961d72d78788426b7302db4f59%40%3Cdev.flink.apache.org%3E

FLIP-145: Support SQL windowing table-valued function1.13

DISCUSS: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-145-Support-SQL-windowing-table-valued-function-td45269.html

VOTE: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-145-Support-SQL-windowing-table-valued-function-2nd-td46452.html

FLIP-164: Improve Schema Handling in Catalogs1.13

discuss: https://lists.apache.org/thread.html/r79064a448d38d5b3d091dfb4703ae3eb94d5c6dd3970f120be7d5b13%40%3Cdev.flink.apache.org%3E

vote: https://lists.apache.org/thread.html/r3054e1f7758a442f27c2a9a4d6b7bcf513ae7e44a6412c3a6791cb51%40%3Cdev.flink.apache.org%3E

FLIP-152: Hive Query Syntax Compatibility1.13

DISCUSS: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-152-Hive-Query-Syntax-Compatibility-td46928.html

VOTE: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-152-Hive-Query-Syntax-Compatibility-td48634.html

FLIP-169: DataStream API for Fine-Grained Resource Requirements1.14

discuss: https://lists.apache.org/thread.html/redb8917e0eb2c916589f11870050291b25cfdefa64872e63ab1aa939%40%3Cdev.flink.apache.org%3E

vote: https://lists.apache.org/thread.html/r8c73fc1081bd56979e3a5b1e28d80c1956eabccc080014d32241d169%40%3Cdev.flink.apache.org%3E

FLIP-173: Support DAG of algorithms (Flink ML)TBD

Discuss: https://lists.apache.org/thread.html/r6729f351fb1bc13a93754c199d5fee1051cc8146e22374737c578779%40%3Cdev.flink.apache.org%3E

Vote: https://lists.apache.org/thread/2087m6t1d58lw3xngtwhws6xbd9fm30r

FLIP-174: Improve the WithParam interface (Flink ML)TBD

Discuss: https://lists.apache.org/thread.html/re56b61ec0a4a20dadea8f4fe757b8baf94f2eb42b1044e9dd003bf23%40%3Cdev.flink.apache.org%3E

Vote: https://lists.apache.org/thread/pq1ot6wj1j87jxm4tqydl4vf6klqsy4l

FLIP-175: Compose Estimator/Model/AlgoOperator from DAG of Estimator/Model/AlgoOperator (Flink ML)TBD

Discuss: https://lists.apache.org/thread.html/r3bb7c2c145635f98ae22c5a917c7f0ce80265dd00ffecd754d8bedf8%40%3Cdev.flink.apache.org%3E

Vote: https://lists.apache.org/thread/k6zol3fp59q16ogsg9xmjx5j9yp98dj9

FLIP-176: Unified Iteration to Support Algorithms (Flink ML)TBD

Discuss: https://lists.apache.org/thread.html/r72e87a71b14baac3d7d16268346f5fc7c65f1de989e00b4ab2aab9ab%40%3Cdev.flink.apache.org%3E

Vote: https://lists.apache.org/thread/899vt2momfqpn65zmx6cq74o3qn41yf1

FLIP-189: SQL Client Usability Improvements

TBDDiscuss: https://lists.apache.org/thread/8d580jcqzpcbmfwqvhjso82hdd2x0461
Vote: https://lists.apache.org/thread/f14jjhrscrdv7h6zw6h1k72nfx232qxs 
FLIP-190: Support Version Upgrades for Table API & SQL Programs1.15

Discuss: https://lists.apache.org/thread/n8v32j6o3d50mpblxydbz82q1q436ob4

Vote: https://lists.apache.org/thread/rxxxq0q9v9pmzd2ht9nybpg5vrzyhwx7

FLIP-193: Snapshots ownership1.15

discuss: https://lists.apache.org/thread/zw2crf0c7t7t4cb5cwcwjpvsb3r1ovz2

vote: https://lists.apache.org/thread/tpyros2fl0howbtcb3fc54f7b7pjn1fw

FLIP-197: API stability graduation process1.15

discuss: https://lists.apache.org/thread/gjgr3b3w2c379ny7on3khjgwyjp2gyq1

vote: https://lists.apache.org/thread/vsn21hmmv1km2ybl3hq00p7nhrp1dnsw

FLIP-203: Incremental savepoints1.15

discuss: https://lists.apache.org/thread/k19bn7gq5jth3jn9h3zkcyrk44hv5nyn

vote: https://lists.apache.org/thread/zvf6410wvhlw13mx04kqvgvnnz68kt3x

FLIP-205: Support Cache in DataStream for Batch Processing1.16discuss: https://lists.apache.org/thread/sqxyb5bh99f904g8zpj8tv60m75dmyox
vote: https://lists.apache.org/thread/gtvjl293s7mbm48h1nd47bhv4oqqjto5
FLIP-212: Introduce Flink Kubernetes OperatorTBD

discuss: https://lists.apache.org/thread/1z78t6rf70h45v7fbd2m93rm2y1bvh0z

vote: https://lists.apache.org/thread/xxfpg6sgdc223nj90fsx16b8f89g8kzy 

FLIP-214: Support Advanced Function DDL

1.16

discuss: https://lists.apache.org/thread/7m5md150qgodgz1wllp5plx15j1nowx8

vote: https://lists.apache.org/thread/ht8cbh9r2gpt2lt611d61j2qhr79pznz

FLIP-222: Support full job lifecycle statements in SQL clientTBD

discuss: https://lists.apache.org/thread/qkvh9p5w9b12s7ykh3l7lv7m9dbgnf1g

vote: https://lists.apache.org/thread/yzg9gz3w3fmq2x58tdbmsjthl7obg6lt

FLIP-229: Introduces Join Hint for Flink SQL Batch JobTBD

discuss: https://lists.apache.org/thread/y668bxyjz66ggtjypfz9t571m0tyvv9h

vote: https://lists.apache.org/thread/s70cjbbr5565m44f4mfqo9w7xdq09cf1

FLIP-232: Add Retry Support For Async I/O In DataStream API 1.16

discuss:  https://lists.apache.org/thread/pgm3bf8vd5vqchlm29n6cro0gz4pbd3g

vote: https://lists.apache.org/thread/qp9y1k0gldxymzmrso0xgsrwh15n6clc

FLIP-234: Support Retryable Lookup Join To Solve Delayed Updates Issue In External Systems1.16

discuss: https://lists.apache.org/thread/9k1sl2519kh2n3yttwqc00p07xdfns3h

vote: https://lists.apache.org/thread/bb0kqjs8co3hhmtklmwptws4fc4rz810

FLIP-247: Bulk fetch of table and column statistics for given partitions1.16

discuss:  https://lists.apache.org/thread/sgd36d8s8crc822xt57jxvb6m1k6t07o

vote: https://lists.apache.org/thread/o54ljccbjl4c2vpykpszk5yo825gvjmg

FLIP-249: Flink Web UI Enhancement for Speculative Execution

1.16

discuss: https://lists.apache.org/thread/832tk3zvysg45vrqrv5tgbdqx974pm3m

vote: https://lists.apache.org/thread/0xonn4y8lkj49comors0z86tpyzkvvqg

FLIP-252: Amazon DynamoDB Sink ConnectorTBD

Discuss: https://lists.apache.org/thread/ssmf2c86n3xyd5qqmcdft22sqn4qw8mw

Vote: https://lists.apache.org/thread/4qq33hxfjf5ms21pbo4mk2q7xn4s2l0h 




Implemented and Released FLIPs

FLIPFirst Release VersionLink to Discussion Thread

FLIP-29: Support map/flatMap/aggregate/flatAggregate on TableAPI

1.9http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Table-API-Enhancement-Outline-td25070.html
FLIP-31: Pluggable Shuffle Manager1.9https://lists.apache.org/thread.html/73d7c62a6c6dc7d3fddda79701c71b921e635766737c1b33b84df234@%3Cdev.flink.apache.org%3E
FLIP-34: Terminate/Suspend Job with Savepoint1.9https://lists.apache.org/thread.html/b8d2f3209e7ca7467af6037383ade6c14c35276f7acb2bbbc9a50c0f@%3Cdev.flink.apache.org%3E
FLIP-38: Python Table API1.9https://lists.apache.org/thread.html/08aad344e43a409c318d4bda67bac1fa7beecc6abd8539c53021d310@%3Cdev.flink.apache.org%3E
FLIP-49: Unified Memory Configuration for TaskExecutors1.10http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-49-Unified-Memory-Configuration-for-TaskExecutors-td31436.html
FLIP-52: Remove legacy Program interface.1.10 https://lists.apache.org/thread.html/0dbd0a4adf9ad00d6ad869dffc8820f6ce4c1969e1ea4aafb1dd0aa4@%3Cdev.flink.apache.org%3E
FLIP-53: Fine Grained Operator Resource Management1.10http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-53-Fine-Grained-Resource-Management-td31831.html
FLIP-56: Dynamic Slot Allocation1.13http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-56-Dynamic-Slot-Allocation-td31960.html
FLIP-57: Rework FunctionCatalog1.10http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-57-Rework-FunctionCatalog-td32291.html#a32613

FLIP-58: Flink Python User-Defined Stateless Function for Table

1.10http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Python-User-Defined-Function-for-Table-API-td31673.html
FLIP-59: Enable execution configuration from Configuration object1.10http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-59-Enable-execution-configuration-from-Configuration-object-td32359.html
FLIP-61: Simplify Flink's cluster level RestartStrategy configuration1.10https://lists.apache.org/thread.html/e206390127bcbd9b24d9c41a838faa75157e468e01552ad241e3e24b@%3Cdev.flink.apache.org%3E
FLIP-62: Set default restart delay for FixedDelay- and FailureRateRestartStrategy to 1s1.10https://lists.apache.org/thread.html/9602b342602a0181fcb618581f3b12e692ed2fad98c59fd6c1caeabd@%3Cdev.flink.apache.org%3E
FLIP-63: Rework table partition support1.10

http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-63-Rework-table-partition-support-td32770.html

FLIP-66: Support time attribute in SQL DDL1.10http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-66-Support-time-attribute-in-SQL-DDL-tt32766.html
FLIP-76: Unaligned Checkpoints1.13 (MVP in 1.11)http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-76-Unaligned-checkpoints-td33651.html

Under discussion

Accepted

Released

Archive

Discarded


  • No labels