Page History

Building an RC

The Release Candidate artefacts artifacts we vote upon should be the exact ones we vote against, without any modification than renaming – i.e. the contents of the files must be the same between voted release canidate and final release. Because of this the version in the built artefacts artifacts that will become the official Apache releases must not include the rcN suffix.

• Set environment variables 

  Code Block
  language bash theme Midnight
  # Set Version export VERSION=1.10.2rc12rc3 # Set AIRFLOW_REPO_ROOT to the path of your git repo export AIRFLOW_REPO_ROOT=$(pwd) # Example after cloning git clone https://github.com/apache/airflow.git airflow cd airflow export AIRFLOW_REPO_ROOT=$(pwd)

  
  

• set your version to 1.10.2 in airflow/version.py (without the RC tag)
• Commit the version change.

• Tag your release

  Code Block
  language bash theme Midnight
  git tag ${VERSION}

  
  

• Clean the checkout: the sdist step below will

  Code Block
  language bash theme Midnight
  git clean -fxd

  
  

• Tarball the repo


  Code Block
  language bash theme Midnight
  git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz

  
  

• Generate sdist
  NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.

  Code Block
  language bash theme Midnight
  python setup.py compile_assets sdist bdist_wheel

  
  

• Rename the sdist

  Code Block
  language bash theme Midnight
  mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl

  
  

• Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)

  Code Block
  language bash theme Midnight
  ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz && ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz ${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl

  


• Push TagsPush Tags

  Code Block
  language bash theme Midnight
  git push --tags

  
  

• Push the artefacts artifacts to ASF dev dist repo

  Code Block
  language bash theme Midnight
  # First clone the repo svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev # Create new folder for the release cd airflow-dev svn mkdir ${VERSION} # Move the artefactsartifacts to svn folder & commit mv ${AIRFLOW_REPO_ROOT}/apache{-,_}airflow-${VERSION}* airflow-dev/${VERSION}/ cd ${VERSION} svn add * svn commit -m "Add artefactsartifacts for Airflow ${VERSION}"

  
  

PyPI Snapshots (Optional)

At this point we have the artefact that we vote on, but as a convenience to developers we also want to publish "snapshots" of the RC builds to pypi for installing via pip. To do this we need to

• Edit the airflow/version.py to include the RC suffix.
• python setup.py compile_assets sdist bdist_wheel
• Follow the steps in Releasing Airflow 85460999
• Throw away the change - we don't want to commit this: git checkout airflow/version.py

It is important to stress that this snapshot is not intended for users.

Make sure your public key is on id.apache.org and in KEYS

You will need to sign the release artifacts with your pgp key. After you have created a key, make sure you

...

http://www.apache.org/dev/release-signing.html#basic-facts

Voting on an RC

• Once the RC is built (both source and binary), put them in the dev SVN repository:
  https://dist.apache.org/repos/dist/dev/airflow/

• Use the dev/airflow-jira script to generate a list of Airflow JIRAs that were closed in the release.

• Send out a vote to the dev@airflow.incubator.apache.org mailing list:

  Code Block
  language text title \[VOTE\] Airflow 1.910.0rc72rc3 collapse true
  Hey all, I have cut Airflow 1.910.02 RC7RC3. This email is calling a vote on the release, which will last for 72 hours. Consider this my (binding) +1. Airflow 1.910.02 RC7RC3 is available at: https://dist.apache.org/repos/dist/dev/airflow/1.910.0rc72rc3/ *apache-airflow-1.910.0rc72rc3-source.tar.gz* is a source release that comes with INSTALL instructions. *apache-airflow-1.910.0rc72rc3-bin.tar.gz* is the binary Python "sdist" release. Public keys are available at: https://dist.apache.org/repos/dist/release/airflow/KEYS The release containsOnly votes from PMC members are binding, but members of the followingcommunity JIRAs: https://issues.apache.org/jira/browse/AIRFLOW-1 https://issues.apache.org/jira/browse/AIRFLOW-2 https://issues.apache.org/jira/browse/AIRFLOW-3 https://issues.apache.org/jira/browse/AIRFLOW-4 ... Cheers, <your name>

  Once the vote has been passed, you will need to send a result vote to dev@airflow.incubator.apache.org:

  Code Block
  title \[RESULT\]\[VOTE\] Airflow 1.9.0rc8 collapse true
  Hello, Apache Airflow (incubating) 1.9.0 (based on RC8) has been accepted. 5 “+1” binding votes received: - Maxime Beauchemin (binding) - Chris Riccomini (binding) - Bolke de Bruin (binding) - Joy Gao (binding) - Fokko Driesprong (binding) My next step is to open a thread with the IPMCare encouraged to test the release and vote with "(non-binding)". Please note that the version number excludes the `rcX` string, so it's now simply 1.10.2. This will allow us to rename the artifact without modifying the artifact checksums when we actually release. Changes since 1.10.2rc2: *Bugs*: [AIRFLOW-3732] Fix issue when trying to edit connection in RBAC UI [AIRFLOW-2866] Fix missing CSRF token head when using RBAC UI (#3804) ... *Improvements*: [AIRFLOW-3302] Small CSS fixes (#4140) [Airflow-2766] Respect shared datetime across tabs ... *New features*: [AIRFLOW-2874] Enables FAB's theme support (#3719) [AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182) ... *Doc-only Change*: [AIRFLOW-XXX] Fix BashOperator Docstring (#4052) [AIRFLOW-3018] Fix Minor issues in Documentation ... Cheers, <your name>

  
  

• And Once the vote has been passed, you will also need to send a result vote request to general@incubatorto dev@airflow.apache.org:

  Code Block
  language text title \[RESULT\]\[VOTE\] Release Airflow 1.910.02rc3 collapse true
  Hello Incubator PMC’ers, The Apache Airflow community has voted and approved the proposal to release Apache Airflow 1.910.02 (incubating) based on 1.9.0 Release Candidate 8. We now kindly request the Incubator PMC members to review and vote on this incubator release. Airflow is a platform to programmatically author, schedule, and monitor workflows. Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative. Artifacts are available at: https://dist.apache.org/repos/dist/dev/airflow Public keys are available at: https://dist.apache.org/repos/dist/release/airflow apache-airflow-1.9.0rc8-source.tar.gz <httpsRC3) has been accepted. 4 “+1” binding votes received: - Kaxil Naik (binding) - Bolke de Bruin (binding) - Ash Berlin-Taylor (binding) - Tao Feng (binding) 4 "+1" non-binding votes received: - Deng Xiaodong (non-binding) - Stefan Seelmann (non-binding) - Joshua Patchus (non-binding) - Felix Uellendall (non-binding) Vote thread: https://lists.apache.org/thread.html/736404ca3d2b2143b296d0910630b9bd0f8b56a0c54e3a05f4c8b5fe@%3Cdev.airflow.apache.org%3E I'll continue with the release process and the release announcement will follow shortly. Cheers, <your name>

  
  

Publishing release

• After both votes pass (see above), you need to migrate the RC artifacts that passed to this repository:
  https://dist.apache.org/repos/dist/release/airflow/
  (The migration should including renaming the files so that they no longer have the RC number in their filenames.)
  
  The best way of doing this is to svn cp  between the two repos (this avoids having to upload the binaries again, and gives a clearer history in the svn commit logs):

  Code Block
  language bash theme Midnight
  # First clone the repo export RC=1.10.4rc5 export VERSION=${RC/rc?/} svn checkout https://dist.apache.org/repos/dist/devrelease/airflow/1.9.0rc8/apache-airflow-1.9.0rc8-source.tar.gz> is a source release that comes with INSTALL instructions. Along with it, for convenience, find the binary Python "sdist" as apache-airflow-1.9.0rc8-bin.tar.gz <https://dist.apache.org/repos/dist/dev/airflow/1.9.0rc8/apache-airflow-1.9.0rc8-bin.tar.gz> Vote thread: https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201712.mbox/%3CCABYbY7ecu8F8sp%3DgvZrvW3knj%3DQpdCUZu-Jh02KngjGOezhBKA%40mail.gmail.com%3E Git tag: https://github.com/apache/incubator-airflow/releases/tag/1.9.0rc8 The vote will be open for at least 72 hours and until the necessary number of votes are reached. Members please be sure to indicate "(Binding)" with your vote which will help in tallying the vote(s). * Here is my +1 (non-binding) * Cheers, <your name>

  Lastly, you'll need to send the general@ vote result to both general@incubator.apache.org and dev@airflow.incubator.apache.org:

  Code Block
  title \[RESULT\]\[VOTE\] Release Airflow 1.9.0 collapse true
  The vote to release Airflow 1.9.0-incubating, having been open for 11 days is now closed. There were three binding +1s and no -1 votes. +1 (binding): Chris Douglas Justin Mclean Jakob Homan The release is approved. Thanks to all those who voted. Cheers, <your name>

Publishing release

• After both votes pass (see above), you need to migrate the RC artifacts that passed to this repository:
  https://dist.apache.org/repos/dist/release/airflow/
  (The migration should including renaming the files so that they no longer have the RC number in their filenames.)

Publishing to PyPi

...

Create a ~/.pypirc file:

$ cat ~/.pypirc 
[distutils]
index-servers =
  pypi
  pypitest

[pypi]
username=your-username
password=*********

[pypitest]
repository=https://test.pypi.org/legacy/
username=your-username
password=*********

(more details here.)

• airflow-release # Create new folder for the release cd airflow-release svn mkdir ${VERSION} cd ${VERSION} # Move the artifacts to svn folder & commit for f in ../../airflow-dev/$RC/*; do svn cp $f ${$(basename $f)/rc?/}; done svn commit -m "Release Airflow ${VERSION} from ${RC}" # Remove old release # http://www.apache.org/legal/release-policy.html#when-to-archive cd .. export PREVIOUS_VERSION=1.10.1 svn rm ${PREVIOUS_VERSION} svn commit -m "Remove old release: ${PREVIOUS_VERSION}"

  
  

Publishing to PyPi

• Create a ~/.pypirc file:

  Code Block
  language bash
  $ cat ~/.pypirc [distutils] index-servers = pypi pypitest [pypi] username=your-username password=********* [pypitest] repository=https://test.pypi.org/legacy/ username=your-username password=*********

  (more details here.)

• Set proper permissions for the pypirc file:

  Code Block
  language bash
  $ chmod 600 ~/.pypirc

  
  

• Confirm that airflow/version.py is set properly.

• Install twine if you do not have it already. 

  Code Block
  language bash theme Midnight
  pip install twine

  
  

• Build the package:

  Code Block
  language bash theme Midnight
  python setup.py compile_assets sdist bdist_wheel

  
  

• Verify the artifacts that would be uploaded:

  Code Block
  language bash theme Midnight
  twine check dist/*

  
  

• Upload the package to PyPi's test environment:

  Code Block
  language bash theme Midnight
  twine upload -r pypitest dist/*

  
  

• Verify that the test package looks good by downloading it and installing it into a virtual environment. The package download link is available at:
  https://test.pypi.org/project/apache-airflow/#files

• Upload the package to PyPi's production environment:

  Code Block
  language bash theme Midnight
  twine upload -r pypi dist/*

  
  

• Again, confirm that the package is available here:
  https://pypi.python.org/pypi/apache-airflow

Updating CHANGELOG.md

...

...

...

Install the package in PyPi's test environment:

python setup.py compile_assets sdist upload -r pypitest

...

Install the package in PyPi's production environment:

python setup.py compile_assets sdist upload -r pypi

...

Again, confirm that the package is available here:
https://pypi.python.org/pypi/apache-airflow

Updating CHANGELOG.md

$ git log 1.8.0..1.9.0 --pretty=oneline

Dear Airflow community,

Airflow 1.9.0 was just released.

The source release as well as the binary "sdist" release are available here:

https://dist.apache.org/repos/dist/release/airflow/1.9.0-incubating/

We also made this version available on PyPi for convenience (`pip install apache-airflow`):

https://pypi.python.org/pypi/apache-airflow

Find the CHANGELOG here for more details:

https://github.com/apache/incubator-airflow/blob/master/CHANGELOG.txt

Cheers,
<your name>