This document outlines common rules for connectors that are developed & released separately from Flink (otherwise known as "externalized").
Versioning
Source releases:
<major>.<minor>.<patch>
Jar artifacts:
<major>.<minor>.<patch>-<flink-major>.<flink-minor>
For example, 1.0.0-1.15
This may imply releasing the exact same connector jar multiple times under different versions.
Branch model
The default branch is called main
and is used for the next major iteration.
Remaining branches are called v<major>.<minor>
, for example v3.2
.
Branches are not specific to a Flink version, i.e., no v3.2-1.15
.
Flink compatibility
The Flink versions supported by the project (at the time of writing the last 2 minor Flink versions) must be supported.
How this is achieved is left to the connector, as long as it conforms to the rest of the proposal.
Since branches may not be specific to a particular Flink version this may require the creation of dedicated modules for each supported Flink version.
The architecture of such modules is up to the connector.
For example, there could be:
- a base connector with version-specific extension modules ("shims") that inject the version-specific behavior
- a common utility module that is used by version-specific connector modules.
Support
The last 2 major connector releases are supported with only the latter receiving additional features, with the following exceptions:
- If the older major connector version does not support any currently supported Flink version, then it is no longer supported.
- If the last 2 major versions do not cover all supported Flink versions, then the latest connector version that supports the older Flink version /additionally /gets patch support.
For a given major connector version only the latest minor version is supported.
This means if 1.1.x is released there will be no more 1.0.x release
Examples
Change | Initial state | Final state | |||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
New minor Connector version |
|
| |||||||||||||||||||||
New major Connector version |
|
| |||||||||||||||||||||
New major Connector version The last 2 major version versions do not cover all supported Flink versions. |
|
| |||||||||||||||||||||
New minor Flink version An older connector to not support any supported Flink version. | -
|
|
Externalization guide
https://github.com/apache/flink-connector-elasticsearch/ is the most complete example of an externalized connector.
Git history
When moving a connector out of the Flink repo the git history should be preserved.
Use the git-filter-repo tool to extract the relevant commits.
As an example, the externalization of the Cassandra connector required these commands to be run (in a fresh copy of the Flink repository!!!):
python3 git-filter-repo --path docs/content/docs/connectors/datastream/cassandra.md --path docs/content.zh/docs/connectors/datastream/cassandra.md --path flink-connectors/flink-connector-cassandra/ python3 git-filter-repo --path-rename flink-connectors/flink-connector-cassandra:flink-connector-cassandra
The result should be that only the desired modules related to the connector exist in your local branch.
Then rebase this branch on top of the bootstrapped externalized connector repo, then apply changes to make things actually work.
Parent pom
We have a parent pom that connectors should use.
<parent> <!-- This will be migrated to org.apache.flink at a later time. --> <groupId>io.github.zentol.flink</groupId> <artifactId>flink-connector-parent</artifactId> <version>1.0</version> </parent>
It handles various things; from setting up essential modules (like the compiler plugin), to QA (including license checks!), testing and Java 11/17 support.
(Almost) everything is opt-in, requiring the project to put a plugin into the <build>
section.
See the bottom of the <properties>
for properties that sub-projects should define.
Release utilities
We have a collection of release scripts that connectors should use.
https://github.com/apache/flink-connector-shared-utils/tree/release_utils
See the contained README.md
for details.
Documentation integration
The documentation should follow this structure:
<root>/docs/content/<english_content> <root>/docs/content.zh/<chinese_content>
See https://github.com/apache/flink/tree/master/docs#include-externally-hosted-documentation for more information on how to integrate the docs into Flink.
For generating a Maven dependency pom snippet, use the connector_artifact
shortcode instead of artifact
. This allows the Flink docs to inject the Flink version suffix.