Page History

...

RELEASE_VERSION="1.5.0"
SHORT_RELEASE_VERSION="1.5"
CURRENT_SNAPSHOT_VERSION="$SHORT_RELEASE_VERSION-SNAPSHOT"
NEXT_SNAPSHOT_VERSION="1.6-SNAPSHOT"
SHORT_NEXT_SNAPSHOT_VERSION="1.6"

Most of the following commands have to be executed in the tools directory, we will prefix the command prompt to make this explicit.

...

$ git checkout release-$SHORT_RELEASE_VERSION  

The rest of this guide assumes that commands are run in the root (or tools directory) of a repository on the branch of the release version with the above environment variables set.

Checklist to proceed to the next step

...

Build a release candidate

The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized.

Build and stage Java and Python artifacts

Set up a few environment variables to simplify Maven commands that follow. This identifies the release candidate being built. Start with RC_NUM equal to 1 and increment it for each candidate.

RC_NUM="1"
TAG="release-${RELEASE_VERSION}-rc${RC_NUM}"

Now, create a release branch:

$ cd tools
tools $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh

Tag the release commit:

git tag -s ${TAG} -m "${TAG}"

We now need to do several things:

• Create the source release archive
• Deploy jar artefacts to the Apache Nexus Repository, which is the staging area for deploying the jars to Maven Central
• Create binary convenience releases for different Hadoop versions

You might want to create a directory on your local machine for collecting the various source and binary releases before uploading them. Creating the binary releases is a lengthy process but you can do this on a another machine (for example, in the "cloud"). When doing this, you can skip signing the release files on the remote machine, download them to your local machine and sign them there.

First, we build the source release:

tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
 

Next, we stage the maven artefacts:

tools $ releasing/deploy_staging_jars.sh

Review all staged artifacts (https://repository.apache.org/). They should contain all relevant parts for each module, including pom.xml, jar, test jar, source, test source, javadoc, etc. Carefully review any new artifacts.

Close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Flink, version X, release candidate Y”.

Finally, we create the binary convenience release files:

tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_binary_release.sh

If you want to run this step in parallel on a remote machine you have to make the release commit available there (for example by pushing to a repository). This is important: the commit inside the binary builds has to match the commit of the source builds and the tagged release commit. When building remotely, you can skip gpg signing by setting  SKIP_GPG=true. You would then sign the files manually after downloading them to your machine:

for f in flink-*-bin*.tgz; do gpg --armor --detach-sig $f; done

gpg --armor --detach-sig apache-flink-*.tar.gz

Stage source and binary releases on dist.apache.org

Copy the source release to the dev repository of dist.apache.org.

If you have not already, check out the Flink section of the dev repository on dist.apache.org via Subversion. In a fresh directory:

svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates

Make a directory for the new release:

mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}

Copy Flink source/binary distributions, hashes, and GPG signature:

mv <flink-dir>/tools/releasing/release/* flink/flink-${RELEASE_VERSION}-rc${RC_NUM}

Add and commit all the files.

cd flink 
svn add flink-${RELEASE_VERSION}-rc${RC_NUM}
svn commit

...

Verify that files are present

If this is a major release, the newly created branch needs to be pushed to the official repository.

Next, for major releases, some configuration for our documentation builds needs to be manually updated, in the docs/_config.yml file:, as listed below.

version: $SHORT_RELEASE_VERSION
version_title: $SHORT_RELEASE_VERSION
github_branch: release-$SHORT_RELEASE_VERSION
base_url: //ci.apache.org/projects/flink/flink-docs-release-$SHORT_RELEASE_VERSION
stable_base_url: //ci.apache.org/projects/flink/flink-docs-release-$SHORT_RELEASE_VERSION
javadocs_base_url: //ci.apache.org/projects/flink/flink-docs-release-$SHORT_RELEASE_VERSION
pythondocs_base_url: //ci.apache.org/projects/flink/flink-docs-release-$SHORT_RELEASE_VERSION
is_stable: true

After updating the docs configuration and pushing the new major release branch, as the last step you should also update the documentation build bot to also build the documentation for the new release branch. Check Managing Flink Documentation on details on how to do that. You may also want to manually trigger a build to make the changes visible as soon as possible.

The rest of this guide assumes that commands are run in the root (or tools directory) of a repository on the branch of the release version with the above environment variables set.

Checklist to proceed to the next step

1. Release Manager’s GPG key is published to dist.apache.org
2. Release Manager’s GPG key is configured in git configuration
3. Release Manager has org.apache.flink listed under Staging Profiles in Nexus
4. Release Manager’s Nexus User Token is configured in settings.xml
5. JIRA release item for the subsequent release has been created
6. There are no release blocking JIRA issues
7. Release Notes in JIRA have been audited and adjusted
8. Release branch has been created and pushed if it is a major release.
9. Originating branch has the version information updated to the new version
10. (major/minor only) Jenkins deployment updated to create snapshot artifacts for release branch (see here)
11. (major/minor only) Cron end-to-end-tests branch setup for release branch
12. (major/minor only) Update upgrade compatibility table (docs/ops/upgrading.md.
13. docs/_config.yml has been updated appropriately.
14. The new documentation for major releases is visible under https://ci.apache.org/projects/flink/flink-docs-release-$SHORT_RELEASE_VERSION (after at least one doc build finishes).
15. The new documentation for major releases do not contain "-SNAPSHOT" in its version title, and all links refer to the corresponding version docs instead of master.

Build a release candidate

The core of the release process is the build-vote-fix cycle. Each cycle produces one release candidate. The Release Manager repeats this cycle until the community approves one release candidate, which is then finalized.

Build and stage Java and Python artifacts

Set up a few environment variables to simplify Maven commands that follow. This identifies the release candidate being built. Start with RC_NUM equal to 1 and increment it for each candidate.

RC_NUM="1"
TAG="release-${RELEASE_VERSION}-rc${RC_NUM}"

Now, create a release branch:

$ cd tools
tools $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh

Tag the release commit:

git tag -s ${TAG} -m "${TAG}"

We now need to do several things:

• Create the source release archive
• Deploy jar artefacts to the Apache Nexus Repository, which is the staging area for deploying the jars to Maven Central
• Create binary convenience releases for different Hadoop versions

You might want to create a directory on your local machine for collecting the various source and binary releases before uploading them. Creating the binary releases is a lengthy process but you can do this on a another machine (for example, in the "cloud"). When doing this, you can skip signing the release files on the remote machine, download them to your local machine and sign them there.

First, we build the source release:

tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
 

Next, we stage the maven artifacts:

tools $ releasing/deploy_staging_jars.sh

Review all staged artifacts (https://repository.apache.org/). They should contain all relevant parts for each module, including pom.xml, jar, test jar, source, test source, javadoc, etc. Carefully review any new artifacts.

Close the staging repository on Apache Nexus. When prompted for a description, enter “Apache Flink, version X, release candidate Y”.

Finally, we create the binary convenience release files:

tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_binary_release.sh

If you want to run this step in parallel on a remote machine you have to make the release commit available there (for example by pushing to a repository). This is important: the commit inside the binary builds has to match the commit of the source builds and the tagged release commit. When building remotely, you can skip gpg signing by setting  SKIP_GPG=true. You would then sign the files manually after downloading them to your machine:

for f in flink-*-bin*.tgz; do gpg --armor --detach-sig $f; done

gpg --armor --detach-sig apache-flink-*.tar.gz

Stage source and binary releases on dist.apache.org

Copy the source release to the dev repository of dist.apache.org.

1. If you have not already, check out the Flink section of the dev repository on dist.apache.org via Subversion. In a fresh directory:

   Code Block
   language bash
   svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates

   
   

2. Make a directory for the new release:

   Code Block
   language bash
   mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}

   Copy Flink source/binary distributions, hashes, and GPG signature:

   Code Block
   mv <flink-dir>/tools/releasing/release/* flink/flink-${RELEASE_VERSION}-rc${RC_NUM}

   
   

3. Add and commit all the files.

   Code Block
   language bash
   cd flink svn add flink-${RELEASE_VERSION}-rc${RC_NUM} svn commit

   
   

4. Verify that files are present

Propose a pull request for website updates

The final step of building the candidate is to propose a website pull request.

Start by updating the variables for the latest released version in the top-level _config.yml, and list the new release in downloads.md, linking to the source code download and the Release Notes in JIRA. Also add a new blog entry announcing the release in _posts.

Finally, propose a pull request with these changes. (Don’t merge before finalizing the release.)

Checklist to proceed to the next step

1. Maven artifacts deployed to the staging repository of repository.apache.org
2. Source distribution deployed to the dev repository of dist.apache.org
3. Website pull request proposed to list the release
4. Check docs/_config.yml to ensure that
   • the version constants refer to the new version
   • the baseurl does not point to flink-docs-master  but flink-docs-release-X.Y instead

You can (optionally) also do additional verification by:

1. Check hashes (e.g. shasum *.sha512 > checklist.chk; shasum -c checklist.chk)

2. Check signatures (e.g. gpg --verify flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz)
3. grep for legal headers in each file.

Vote on the release candidate

Once you have built and individually reviewed the release candidate, please share it for the community-wide review. Please review foundation-wide voting guidelines for more information.

Start the review-and-vote thread on the dev@ mailing list. Here’s an email

Propose a pull request for website updates

The final step of building the candidate is to propose a website pull request.

Start by updating the variables for the latest released version in the top-level _config.yml, and list the new release in downloads.md, linking to the source code download and the Release Notes in JIRA. Also add a new blog entry announcing the release in _posts.

Finally, propose a pull request with these changes. (Don’t merge before finalizing the release.)

Checklist to proceed to the next step

1. Maven artifacts deployed to the staging repository of repository.apache.org
2. Source distribution deployed to the dev repository of dist.apache.org
3. Website pull request proposed to list the release
4. Check docs/_config.yml to ensure that
   • the version constants refer to the new version
   • the baseurl does not point to flink-docs-master  but flink-docs-release-X.Y instead

You can (optionally) also do additional verification by:

1. Check hashes (e.g. shasum *.sha512 > checklist.chk; shasum -c checklist.chk)

2. Check signatures (e.g. gpg --verify flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz)
3. grep for legal headers in each file.

Vote on the release candidate

Once you have built and individually reviewed the release candidate, please share it for the community-wide review. Please review foundation-wide voting guidelines for more information.

Start the review-and-vote thread on the dev@ mailing list. Here’s an email template; please adjust as you see fit.

From: Release Manager
To: dev@flink.apache.org
Subject: [VOTE] Release 1.2.3, release candidate #3

Hi everyone,
Please review and vote on the release candidate #3 for the version 1.2.3, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint FFFFFFFF [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.2.3-rc3" [5],
* website pull request listing the new release and adding announcement blog post [6]. 

The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] link
[2] link
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] link
[5] link
[6] link

If there are any issues found in the release candidate, reply on the vote thread to cancel the vote. There’s no need to wait 72 hours. Proceed to the Fix Issues step below and address the problem. However, some issues don’t require cancellation. For example, if an issue is found in the website pull request, just correct it on the spot and the vote can continue as-is.

If there are no issues, reply on the vote thread to close the voting. Then, tally the votes in a separate email. Here’s an email template; please adjust as you see fit.

From: Release Manager
To: dev@flink.apache.org
Subject: [RESULT] [VOTEVOTE] Release 1.2.3, release candidate #3

I'm happy to announce that we have unanimously approved this release.

There are XXX approving votes, XXX of which are binding:
* approver 1
* approver 2
* approver 3
* approver 4

There are no disapproving votes.

Thanks everyone!

Checklist to proceed to the finalization step

1. Community votes to release the proposed candidate, with at least three approving PMC votes

Fix any issues

Any issues identified during the community review and vote should be fixed in this step.

Code changes should be proposed as standard pull requests to the master branch and reviewed using the normal contributing process. Then, relevant changes should be cherry-picked into the release branch. The cherry-pick commits should then be proposed as the pull requests against the release branch, again reviewed and merged using the normal contributing process.

Once all issues have been resolved, you should go back and build a new release candidate with these changes.

Checklist to proceed to the next step

1. Issues identified during vote have been resolved, with fixes committed to the release branch.

Finalize the release

Once the release candidate has been reviewed and approved by the community, the release should be finalized. This involves the final deployment of the release candidate to the release repositories, merging of the website changes, etc.

Deploy Python artifacts to PyPI

Release manager should get the pypi account and password from PMC to deploy the Python artifacts to PyPI. The artifacts could be uploaded using twine(https://pypi.org/project/twine/). To install twine, just run:

pip install --upgrade twine==1.12.0

Download the python artifacts from dist.apache.org and upload it to pypi.org:

svn checkout https://dist.apache.org/repos/dist/dev/flink/ --depth=immediates
cd flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
twine upload --repository-url https://upload.pypi.org/legacy/ apache-flink-${RELEASE_VERSION}.tar.gz apache-flink-${RELEASE_VERSION}.tar.gz.asc

If upload failed or incorrect for some reason(e.g. network transmission problem), you need to delete the uploaded release package of the same version(if exists) and rename the artifact to apache-flink-${RELEASE_VERSION}.post0.tar.gz, then re-upload.

Note: re-uploading to pypi.org must be avoided as much as possible because it will cause some irreparable problems. If that happens, users cannot install the apache-flink package by explicitly specifying the package version, i.e. the following command "pip install apache-flink==${RELEASE_VERSION}" will fail. Instead they have to run "pip install apache-flink" or "pip install apache-flink==${RELEASE_VERSION}.post0" to install the apache-flink package.

Deploy artifacts to Maven Central Repository

Use the Apache Nexus repository to release the staged binary artifacts to the Maven Central repository. In the Staging Repositories section, find the relevant release candidate orgapacheflink-XXX entry and click Release. Drop all other release candidates that are not being released.

Deploy source and binary releases to dist.apache.org

Copy the source and binary releases from the dev repository to the release repository at dist.apache.org using Subversion.

Hi everyone,
Please review and vote on the release candidate #3 for the version 1.2.3, as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint FFFFFFFF [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.2.3-rc3" [5],
* website pull request listing the new release and adding announcement blog post [6]. 

The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] link
[2] link
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] link
[5] link
[6] link

If there are any issues found in the release candidate, reply on the vote thread to cancel the vote. There’s no need to wait 72 hours. Proceed to the Fix Issues step below and address the problem. However, some issues don’t require cancellation. For example, if an issue is found in the website pull request, just correct it on the spot and the vote can continue as-is.

If there are no issues, reply on the vote thread to close the voting. Then, tally the votes in a separate email. Here’s an email template; please adjust as you see fit.

From: Release Manager
To: dev@flink.apache.org
Subject: [RESULT] [VOTE] Release 1.2.3, release candidate #3

I'm happy to announce that we have unanimously approved this release.

There are XXX approving votes, XXX of which are binding:
* approver 1
* approver 2
* approver 3
* approver 4

There are no disapproving votes.

Thanks everyone!

Checklist to proceed to the finalization step

1. Community votes to release the proposed candidate, with at least three approving PMC votes

Fix any issues

Any issues identified during the community review and vote should be fixed in this step.

Code changes should be proposed as standard pull requests to the master branch and reviewed using the normal contributing process. Then, relevant changes should be cherry-picked into the release branch. The cherry-pick commits should then be proposed as the pull requests against the release branch, again reviewed and merged using the normal contributing process.

Once all issues have been resolved, you should go back and build a new release candidate with these changes.

Checklist to proceed to the next step

1. Issues identified during vote have been resolved, with fixes committed to the release branch.

Finalize the release

Once the release candidate has been reviewed and approved by the community, the release should be finalized. This involves the final deployment of the release candidate to the release repositories, merging of the website changes, etc.

Deploy Python artifacts to PyPI

Release manager should get the pypi account and password from PMC to deploy the Python artifacts to PyPI. The artifacts could be uploaded using twine(https://pypi.org/project/twine/). To install twine, just run:

pip install --upgrade twine==1.12.0

Download the python artifacts from dist.apache.org and upload it to pypi.org:

svn checkout https://dist.apache.org/repos/dist/dev/flink/ --depth=immediates
cd flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
twine upload --repository-url https://upload.pypi.org/legacy/ apache-flink-${RELEASE_VERSION}.tar.gz apache-flink-${RELEASE_VERSION}.tar.gz.asc

If upload failed or incorrect for some reason(e.g. network transmission problem), you need to delete the uploaded release package of the same version(if exists) and rename the artifact to apache-flink-${RELEASE_VERSION}.post0.tar.gz, then re-upload.

Note: re-uploading to pypi.org must be avoided as much as possible because it will cause some irreparable problems. If that happens, users cannot install the apache-flink package by explicitly specifying the package version, i.e. the following command "pip install apache-flink==${RELEASE_VERSION}" will fail. Instead they have to run "pip install apache-flink" or "pip install apache-flink==${RELEASE_VERSION}.post0" to install the apache-flink package.

Deploy artifacts to Maven Central Repository

Use the Apache Nexus repository to release the staged binary artifacts to the Maven Central repository. In the Staging Repositories section, find the relevant release candidate orgapacheflink-XXX entry and click Release. Drop all other release candidates that are not being released.

Deploy source and binary releases to dist.apache.org

Copy the source and binary releases from the dev repository to the release repository at dist.apache.org using Subversion.

svn move -m "Release Flink ${RELEASE_VERSION}" https://dist.apache.org/repos/dist/dev/flink/flink-${RELEASE_VERSION}-rc${RC_NUM} https://dist.apache.org/repos/dist/release/flink/flink-${RELEASE_VERSION}

Remove old release candidates from dist.apache.org

Remove the old release candidates from https://dist.apache.org/repos/dist/dev/flink using Subversion.

svn checkout 

svn move -m "Release Flink ${RELEASE_VERSION}" https://dist.apache.org/repos/dist/dev/flink/ --depth=immediates
cd flink
svn remove flink-${RELEASE_VERSION}-rc${RC_NUM} https://dist.apache.org/repos/dist/release/flink/flink-rc*
svn commit -m "Remove old release candidates for Apache Flink ${RELEASE_VERSION}

Remove old release candidates from dist.apache.org

...

"

Remove outdated versions from dist.apache.org

For a new major release remove all release files older than 2 versions, e.g., when releasing 1.7, remove all releases <= 1.5.

For a new bugfix version remove all release files for previous bugfix releases in the same series, e.g., when releasing 1.7.1, remove the 1.7.0 release.

1. If you have not already, check out the Flink section of the release repository on dist.apache.org via Subversion. In a fresh directory:

   Code Block
   language bash

...

1. svn checkout https://dist.apache.org/repos/dist/

...

1. release/flink --depth=immediates

...

1. cd flink

   
   

2. Remove files for outdated releases and commit the changes.

   Code Block
   language bash
   svn remove flink-

...

1. <version_to_remove> svn commit

   
   

2. Verify that files are removed

Git tag

Create a new Git tag for the released version by copying the tag for the final release candidate, as follows:

git tag -s "release-${RELEASE_VERSION}" ${TAG}

Mark the version as released in JIRA

In JIRA, inside version management, hover over the current release and a settings menu will appear. Click Release, and select today’s date.

Checklist to proceed to the next step

Remove outdated versions from dist.apache.org

For a new major release remove all release files older than 2 versions, e.g., when releasing 1.7, remove all releases <= 1.5.

For a new bugfix version remove all release files for previous bugfix releases in the same series, e.g., when releasing 1.7.1, remove the 1.7.0 release.

If you have not already, check out the Flink section of the release repository on dist.apache.org via Subversion. In a fresh directory:

...

...

• Python artifacts released and indexed in the PyPI Repository
• Maven artifacts released and indexed in the Maven Central Repository (usually takes about a day to show up)
• Source & binary distributions available in the release repository of 

• https://dist.apache.org/repos/dist/release/flink

...

Remove files for outdated releases and commit the changes.

 svn remove flink-<version_to_remove>
 svn commit

...

Verify that files are removed

• /
• Dev repository https://dist.apache.org/repos/dist/dev/flink/ is empty
• Release tagged in the source code repository
• Release version finalized in JIRA. (Note: Not all committers have administrator access to JIRA. If you end up getting permissions errors ask on the mailing list for assistance.)

Update website to point to new stable release documentation (for major releases only)

In our website repository flink-web, for major releases we need to update the website to point to the new stable release.

In the _config.yml file under the root directory, update the following settings:

FLINK_VERSION_STABLE: $RELEASE_VERSION
FLINK_VERSION_STABLE_SHORT: $SHORT_RELEASE_VERSION
stable: $SHORT_RELEASE_VERSION
snapshot: $SHORT_NEXT_SNAPSHOT_VERSION
docs-stable: "https://ci.apache.org/projects/flink/flink-docs-release-$SHORT_RELEASE_VERSION"

Finally, rebuild the website and push.

Add download links for the new release to the website

In the _config.yml file under the root directory of flink-web, you will find a list of download links for previous releases.

Follow the same format and add a new entry for the release binaries and sources. Please pay notice to the ids assigned to the download entries. They should be unique and reflect their corresponding version number.

Git tag

Create a new Git tag for the released version by copying the tag for the final release candidate, as follows:

git tag -s "release-${RELEASE_VERSION}" ${TAG}

Mark the version as released in JIRA

In JIRA, inside version management, hover over the current release and a settings menu will appear. Click Release, and select today’s date.

Checklist to proceed to the next step

...

Promote the release

Once the release has been finalized, the last step of the process is to promote the release within the project and beyond. Please wait for 24h after finalizing the release in accordance with the ASF release policy.

...