Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Get a diff between the last version and the current version:

    Code Block
    languagebash
    $ git log 1.8.0..1.9.0 --pretty=oneline


  • Update CHANGELOG.md with the details, and commit it.

Notifying developers of release

  • Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org and announce@apache.org) that the artifacts have been published:

    Code Block
    titleAirflow 1.10.2 is released
    collapsetrue
    Dear Airflow community,
    
    I'm happy to announce that Airflow 1.10.2 was just released.
    
    The source release, as well as the binary "sdist" release, are available
    here:
    
    https://dist.apache.org/repos/dist/release/airflow/1.10.2/
    
    We also made this version available on PyPi for convenience (`pip install apache-airflow`):
    
    https://pypi.python.org/pypi/apache-airflow
    
    The documentation is available on:
    https://airflow.apache.org/
    https://airflow.apache.org/1.10.2/
    https://airflow.readthedocs.io/en/1.10.2/
    https://airflow.readthedocs.io/en/stable/
    
    Find the CHANGELOG here for more details:
    
    https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
    
    Cheers,
    <your name>


  • Update the Announce page on the wiki

Publishing documentation

Old stuff that doesn't apply anymore

Expand
titleView legacy instructions

Committers should use the following commands and steps to create an Apache Airflow release:

  1. Run dev/airflow-jira on the test branch
    dev/airflow-jira compare 1.8.1

  2. Commit RCs here
    svn co https://dist.apache.org/repos/dist/dev/airflow

    Make sure CHANGELOG and UPDATING.md have been updated.

  3. Remove previous versions from  https://dist.apache.org/repos/dist/dev/airflow (`svn rm` -> `svn commit`)

  4. Run a travis build, confirm it passes (duh.)
  5. Build airflow
    python setup.py sdist

  6. Generate MD5
    gpg --print-md MD5 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5

  7. Generate the SHA512
    gpg --print-md SHA512 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha

  8. Upload your GPG public key to https://pgp.mit.edu

  9. Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of the file

  10. Sign Airflow tar.gz
    gpg --armor --output apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc --detach-sig apache-airflow-1.8.1rc0+apache.incubating.tar.gz

  11. Do votes in dev@airflow.incubator.apache.org and general@incubator.apache.org
    https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRdcn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
    https://mail-archives.apache.org/mod_mbox/incubator-general/201705.mbox/%3CCABYbY7cnACrrsrFtdwtq_-4AokMAe8CBOcWSvNz%2BhgB9DfVEvQ%40mail.gmail.com%3E

  12. Commit final release here
    svn co https://dist.apache.org/repos/dist/release/airflow

  13. Merge the test branch into the stable branch
    git pull apache v1-8-test
    git pull apache v1-8-stable
    git co v1-8-stable
    git merge v1-8-test
    git tag 1.8.1
    git push apache v1-8-stable
    git push apache 1.8.1

  14. Follow these instructions to publish to PyPi
    Note: You must remove the +incubating from the airflow/version.py file.

Source releases:
  • ...
  • Generate MD5s
    gpg --print-md MD5 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5

  • Generate the SHA512
    gpg --print-md SHA512 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha

  • Upload your GPG public key to https://pgp.mit.edu

  • Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of the file

  • Sign Airflow tar.gz
    gpg --armor --output apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc --detach-sig apache-airflow-1.8.1rc0+apache.incubating.tar.gz

Bulk migrate JIRAs

Table of Contents
maxLevel3

Building an RC

...

  • .

...

Set environment variables 

Code Block
languagebash
themeMidnight
# Set Version
export VERSION=1.10.2rc3


# Set AIRFLOW_REPO_ROOT to the path of your git repo
export AIRFLOW_REPO_ROOT=$(pwd)


# Example after cloning
git clone https://github.com/apache/airflow.git airflow
cd airflow
export AIRFLOW_REPO_ROOT=$(pwd)

...

Tag your release

Code Block
languagebash
themeMidnight
git tag ${VERSION}

Clean the checkout: the sdist step below will

Code Block
languagebash
themeMidnight
git clean -fxd

Tarball the repo

Code Block
languagebash
themeMidnight
git archive --format=tar.gz ${VERSION} --prefix=apache-airflow-${VERSION}/ -o apache-airflow-${VERSION}-source.tar.gz

Generate sdist
NOTE: Make sure your checkout is clean at this stage - any untracked or changed files will otherwise be included in the file produced.

Code Block
languagebash
themeMidnight
python setup.py compile_assets sdist bdist_wheel

Rename the sdist

Code Block
languagebash
themeMidnight
mv dist/apache-airflow-${VERSION%rc?}.tar.gz apache-airflow-${VERSION}-bin.tar.gz
mv dist/apache_airflow-${VERSION%rc?}-py2.py3-none-any.whl apache_airflow-${VERSION}-py2.py3-none-any.whl

Generate SHA512/ASC (If you have not generated a key yet, generate it by following instructions on http://www.apache.org/dev/openpgp.html#key-gen-generate-key)

Code Block
languagebash
themeMidnight
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-source.tar.gz
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache-airflow-${VERSION}-bin.tar.gz
${AIRFLOW_REPO_ROOT}/dev/sign.sh apache_airflow-${VERSION}-py2.py3-none-any.whl

Push Tags

Code Block
languagebash
themeMidnight
git push --tags

Push the artifacts to ASF dev dist repo

Code Block
languagebash
themeMidnight
# First clone the repo
svn checkout https://dist.apache.org/repos/dist/dev/airflow airflow-dev

# Create new folder for the release
cd airflow-dev
svn mkdir ${VERSION}

# Move the artifacts to svn folder & commit
mv ${AIRFLOW_REPO_ROOT}/apache{-,_}airflow-${VERSION}* airflow-dev/${VERSION}/
cd ${VERSION}
svn add *
svn commit -m "Add artifacts for Airflow ${VERSION}" 

PyPI Snapshots (Optional)

At this point we have the artefact that we vote on, but as a convenience to developers we also want to publish "snapshots" of the RC builds to pypi for installing via pip. To do this we need to

  • Edit the airflow/version.py to include the RC suffix.
  • python setup.py compile_assets sdist bdist_wheel
  • Follow the steps in 85460999
  • Throw away the change - we don't want to commit this: git checkout airflow/version.py

It is important to stress that this snapshot is not intended for users.

Make sure your public key is on id.apache.org and in KEYS

You will need to sign the release artifacts with your pgp key. After you have created a key, make sure you

Code Block
languagebash
themeMidnight
# Create PGP Key
gpg --gen-key

# Checkout ASF dist repo
svn checkout https://dist.apache.org/repos/dist/release/airflow
cd airflow


# Add your GPG pub key to KEYS file. Replace "Kaxil Naik" with your name
(gpg --list-sigs "Kaxil Naik" && gpg --armor --export "Kaxil Naik" ) >> KEYS


# Commit the changes
svn commit -m "Add PGP keys of Airflow developers"

...

http://www.apache.org/dev/release-signing.html#basic-facts

Voting on an RC

...

Once the RC is built (both source and binary), put them in the dev SVN repository:
https://dist.apache.org/repos/dist/dev/airflow/

...

Use the dev/airflow-jira script to generate a list of Airflow JIRAs that were closed in the release.

Send out a vote to the dev@airflow.apache.org mailing list:

Code Block
languagetext
title\[VOTE\] Airflow 1.10.2rc3
collapsetrue
Hey all,

I have cut Airflow 1.10.2 RC3. This email is calling a vote on the release,
which will last for 72 hours. Consider this my (binding) +1.

Airflow 1.10.2 RC3 is available at:
https://dist.apache.org/repos/dist/dev/airflow/1.10.2rc3/

*apache-airflow-1.10.2rc3-source.tar.gz* is a source release that comes
with INSTALL instructions.
*apache-airflow-1.10.2rc3-bin.tar.gz* is the binary Python "sdist" release.

Public keys are available at:
https://dist.apache.org/repos/dist/release/airflow/KEYS

Only votes from PMC members are binding, but members of the community are
encouraged to test the release and vote with "(non-binding)".

Please note that the version number excludes the `rcX` string, so it's now
simply 1.10.2. This will allow us to rename the artifact without modifying
the artifact checksums when we actually release.


Changes since 1.10.2rc2:
*Bugs*:
[AIRFLOW-3732] Fix issue when trying to edit connection in RBAC UI
[AIRFLOW-2866] Fix missing CSRF token head when using RBAC UI (#3804)
...


*Improvements*:
[AIRFLOW-3302] Small CSS fixes (#4140)
[Airflow-2766] Respect shared datetime across tabs
...


*New features*:
[AIRFLOW-2874] Enables FAB's theme support (#3719)
[AIRFLOW-3336] Add new TriggerRule for 0 upstream failures (#4182)
...


*Doc-only Change*:
[AIRFLOW-XXX] Fix BashOperator Docstring (#4052)
[AIRFLOW-3018] Fix Minor issues in Documentation
...

Cheers,
<your name>

Once the vote has been passed, you will need to send a result vote to dev@airflow.apache.org:

Code Block
languagetext
title\[RESULT\]\[VOTE\] Airflow 1.10.2rc3
collapsetrue
Hello,

Apache Airflow 1.10.2 (based on RC3) has been accepted.

4 “+1” binding votes received:
- Kaxil Naik  (binding)
- Bolke de Bruin (binding)
- Ash Berlin-Taylor (binding)
- Tao Feng (binding)


4 "+1" non-binding votes received:

- Deng Xiaodong (non-binding)
- Stefan Seelmann (non-binding)
- Joshua Patchus (non-binding)
- Felix Uellendall (non-binding)

Vote thread:
https://lists.apache.org/thread.html/736404ca3d2b2143b296d0910630b9bd0f8b56a0c54e3a05f4c8b5fe@%3Cdev.airflow.apache.org%3E

I'll continue with the release process and the release announcement will follow shortly.

Cheers,
<your name>

Publishing release

...

Code Block
languagebash
themeMidnight
# First clone the repo
export RC=1.10.4rc5
export VERSION=${RC/rc?/}
svn checkout https://dist.apache.org/repos/dist/release/airflow airflow-release

# Create new folder for the release
cd airflow-release
svn mkdir ${VERSION}
cd ${VERSION}

# Move the artifacts to svn folder & commit
for f in ../../airflow-dev/$RC/*; do svn cp $f ${$(basename $f)/rc?/}; done
svn commit -m "Release Airflow ${VERSION} from ${RC}" 

# Remove old release
# http://www.apache.org/legal/release-policy.html#when-to-archive
cd ..
export PREVIOUS_VERSION=1.10.1
svn rm ${PREVIOUS_VERSION}
svn commit -m "Remove old release: ${PREVIOUS_VERSION}"

Publishing to PyPi

...

Create a ~/.pypirc file:

Code Block
languagebash
$ cat ~/.pypirc 
[distutils]
index-servers =
  pypi
  pypitest

[pypi]
username=your-username
password=*********

[pypitest]
repository=https://test.pypi.org/legacy/
username=your-username
password=*********

(more details here.)

Set proper permissions for the pypirc file:

Code Block
languagebash
$ chmod 600 ~/.pypirc

...

Install twine if you do not have it already. 

Code Block
languagebash
themeMidnight
pip install twine

Build the package:

Code Block
languagebash
themeMidnight
python setup.py compile_assets sdist bdist_wheel

Verify the artifacts that would be uploaded:

Code Block
languagebash
themeMidnight
twine check dist/*

Upload the package to PyPi's test environment:

Code Block
languagebash
themeMidnight
twine upload -r pypitest dist/*

...

Upload the package to PyPi's production environment:

Code Block
languagebash
themeMidnight
twine upload -r pypi dist/*

...

Again, confirm that the package is available here:
https://pypi.python.org/pypi/apache-airflow

Updating CHANGELOG.md

Get a diff between the last version and the current version:

Code Block
languagebash
$ git log 1.8.0..1.9.0 --pretty=oneline
  • Update CHANGELOG.md with the details, and commit it.
  • Notifying developers of release

    Notify users@airflow.apache.org (cc'ing dev@airflow.apache.org) that the artifacts have been published:

    Code Block
    titleAirflow 1.10.2 is released
    collapsetrue
    Dear Airflow community,
    
    I'm happy to announce that Airflow 1.10.2 was just released.
    
    The source release, as well as the binary "sdist" release, are available
    here:
    
    https://dist.apache.org/repos/dist/release/airflow/1.10.2/
    
    We also made this version available on PyPi for convenience (`pip install apache-airflow`):
    
    https://pypi.python.org/pypi/apache-airflow
    
    The documentation is available on:
    https://airflow.apache.org/
    https://airflow.apache.org/1.10.2/
    https://airflow.readthedocs.io/en/1.10.2/
    https://airflow.readthedocs.io/en/stable/
    
    Find the CHANGELOG here for more details:
    
    https://airflow.apache.org/changelog.html#airflow-1-10-2-2019-01-19
    
    Cheers,
    <your name>
  • Update the Announce page on the wiki

  • Publishing documentation

    Bulk migrate JIRAs

    Old stuff that doesn't apply anymore

    Expand
    titleView legacy instructions

    Committers should use the following commands and steps to create an Apache Airflow release:

    1. Run dev/airflow-jira on the test branch
      dev/airflow-jira compare 1.8.1

    2. Commit RCs here
      svn co https://dist.apache.org/repos/dist/dev/airflow

      Make sure CHANGELOG and UPDATING.md have been updated.

    3. Remove previous versions from  https://dist.apache.org/repos/dist/dev/airflow (`svn rm` -> `svn commit`)

    4. Run a travis build, confirm it passes (duh.)
    5. Build airflow
      python setup.py sdist

    6. Generate MD5
      gpg --print-md MD5 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5

    7. Generate the SHA512
      gpg --print-md SHA512 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha

    8. Upload your GPG public key to https://pgp.mit.edu

    9. Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of the file

    10. Sign Airflow tar.gz
      gpg --armor --output apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc --detach-sig apache-airflow-1.8.1rc0+apache.incubating.tar.gz

    11. Do votes in dev@airflow.incubator.apache.org and general@incubator.apache.org
      https://mail-archives.apache.org/mod_mbox/incubator-airflow-dev/201705.mbox/%3CCABYbY7eZk2SYHs3ni31h94uwwRdcn4dkJwumYtUMsfFLq-gwjg%40mail.gmail.com%3E
      https://mail-archives.apache.org/mod_mbox/incubator-general/201705.mbox/%3CCABYbY7cnACrrsrFtdwtq_-4AokMAe8CBOcWSvNz%2BhgB9DfVEvQ%40mail.gmail.com%3E

    12. Commit final release here
      svn co https://dist.apache.org/repos/dist/release/airflow

    13. Merge the test branch into the stable branch
      git pull apache v1-8-test
      git pull apache v1-8-stable
      git co v1-8-stable
      git merge v1-8-test
      git tag 1.8.1
      git push apache v1-8-stable
      git push apache 1.8.1

    14. Follow these instructions to publish to PyPi
      Note: You must remove the +incubating from the airflow/version.py file.

    Source releases:
    • ...
  • Generate MD5s
    gpg --print-md MD5 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.md5

  • Generate the SHA512
    gpg --print-md SHA512 apache-airflow-1.8.1rc0+apache.incubating.tar.gz > apache-airflow-1.8.1rc0+apache.incubating.tar.gz.sha

  • Upload your GPG public key to https://pgp.mit.edu

  • Add your GPG pub key to https://dist.apache.org/repos/dist/release/airflow/KEYS , follow the instructions at the top of the file

  • Sign Airflow tar.gz
    gpg --armor --output apache-airflow-1.8.1rc0+apache.incubating.tar.gz.asc --detach-sig apache-airflow-1.8.1rc0+apache.incubating.tar.gz