Table of Contents |
---|
Introduction
From time to time, Impala creates releases. These are tarballs of the Impala source that follow the rules below.
Decisions about releases are made by four three groups:
- A Release Manager, who must be a committer, does the work of creating the release, signing it, counting votes, and so on.
- The community, who share in the discussion of whether it is the right time to create a release and what that release should contain. The community can also cast non-binding votes on the release.
- The PMC, who cast binding votes on the release.The Incubator PMC, who must approve all Impala releases until Impala graduates to an Apache Top Level Project.
This page describes the procedures that the release manager and voting PMC members take during release time.
How to Manage a Release
Release managers are self-selected from among the set of committers. Release management must happen partially on "hardware owned and controlled by the committer". In particular, release managers are not permitted to store code signing keys on hardware they do not own or control.
- If you have not done so yet, set up your apache.org email address.
- Make a code signing key if you don't have one yet. This must be done on "hardware owned and controlled by the committer".
- Publish your key if you haven't done so yet
- If Optional but reccomended: if you haven't done so yet, add yourself to the Apache Web-of-Trust by meeting face-to-face with a person so they can sign your key
- Publish your signed key
- If you haven't done so before, add your key to the KEYS file in the release staging area and the release distribution area using SVN.
- The staging area is https://dist.apache.org/repos/dist/dev/incubator/impala/. Once graduated, it will presumably be
- The release area is https://dist.apache.org/repos/dist/devrelease/impala/The release area is https://dist.apache.org/repos/dist/release/incubator/impala/, and will presumably be https://dist.apache.org/repos/dist/release/impala/ after graduation.. If you are not a PMC member, you will need to ask a PMC member to edit the file for you.
- Pick a commit in the "master" branch you want to release from and test it.
- Test the licencing using Apache RAT; follow the instructions in
bin/check-rat-report.py
.- You might have to download Apache RAT if you haven't done so previously: https://creadur.apache.org/rat/download_rat.cgi
- Propose Propose a release on the dev@ list. It should start with the string "[DISCUSS]". Explain that this is not a "[VOTE]", and that anyone may participate.
- Make a new branch off of your commit called "branch-x.y.z", where x.y.z is the version you want to release. If you are not on the PMC, ask a PMC member to do this:
Code Block language bash git checkout $COMMIT_HASH_YOU_CHOSE git checkout -b branch-x.y.z git push apache branch-x.y.z
In this branch (but not in master)
update the version number, update IMPALA_VERSION in bin/
saveimpala-
versionconfig.sh
toto x.y.z-RELEASE
. For versions older than 4.1.0, update the version number in
bin/save-version.sh
instead.(Only for 4.1+) Update the version numbers for Java pom.xml files, use this maven command:
Code Block language bash # From $IMPALA_HOME cd java mvn versions:set -DnewVersion=x.y.z-RELEASE
- Also in this branch (but not in master), update the GIT_HASH in bin/save-version.sh to the git hash of the top commit in the branch. Please bear in mind to update this on every RC. (The git hash will of course not reflect that of the commit that includes this change as well, but that's okay)
- Continue testing. If you find bugs, file them. When they are fixed, cherry-pick the fixes from master to your branch that you want to include in the release
Code Block language bash git fetch apache # to make sure you have the latest master git log apache/master # to find the commits you want to cherry-pick git checkout apache/branch-x.y.z git checkout -b x.y.z-patch-foo # the name doesn't matter - it will only be in your local workstation
Code Block language bash git fetch apache # to make sure you have the latest master git log apache/master # to find the commits you want to cherry-pick git checkout apache/branch-x.y.z git checkout -b x.y.z-patch-foo # the name doesn't matter - it will only be in your local workstation git cherry-pick b4d1a3... # or whatever git hashes # then resolve conflicts, if there are any # cherry-pick some more commits: git cherry-pick .b4d1a3... git # or whatever git hashes # then resolve conflicts, if there are any # cherry-pick some more commits: git cherry-pick .... git cherry-pick ..... git push apache HEAD:branch-x.y.z
At that time, tag the tree at the release candidate
Code Block language bash git fetch apache git checkout apache/branch-x.y.z git tag -a x.y.z-rcw -m "x.y.z release candidate w" # when making release candidate w of version x.y.z git push apache x.y.z-rcw
- Make a release tarball:
Make the tarball using git archive. Name it apache-impala-incubating-x.y.z.tar.gz, or apache-impala-x.y.z.tar.gz if Impala has graduated from the incubator..
Code Block language Code Block language bash git archive --prefix=apache-impala-incubating-x.y.z/ -o /tmp/apache-impala-incubating-x.y.z.tar.gz x.y.z-rcw
Make signature, as well as md5 and sha1 sha512 checksums. This must be done on "hardware owned and controlled by the committer".
Code Block language bash cd /tmp # Make the signature: gpg -u YOUR_APACHE_USER_NAME@apache.org --armor --output apache-impala-incubating-x.y.z.tar.gz.asc --detach-sign apache-impala-incubating-x.y.z.tar.gz # Make sure it worked: gpg --verify apache-impala-incubating-x.y.z.tar.gz.asc apache-impala-x.y.z.tar.gz # Make checksums: md5sumsha512sum apache-impala-incubating-x.y.z.tar.gz > apache-impala-incubating-x.y.z.tar.gz.md5 sha1sum apache-impala-incubating-x.y.z.tar.gz > apache-impala-incubating-x.y.z.tar.gz.sha # Make sure sha512 # Make sure they worked: md5sumsha512sum --check apache-impala-incubating-x.y.z.tar.gz.md5 sha1sum --check apache-impala-incubating-x.y.z.tar.gz.shasha512
Before uploading your release candidate, follow the procedure in the Before uploading your release candidate, follow the procedure in the section below on how to vote on a release. Don't upload until you could vote +1.
- Get commitments from at least five three PMC members to vote on the artifact in the time frame for the upcoming vote. While we are incubating, the The list of PMC members is at http://incubator.apache.org/projects/impala.html. Once Impala graduates, that list will presumably be available at available at http://people.apache.org/committers-by-project.html#impala-pmc.
Upload the artifacts. While incubating, the The location is https://dist.apache.org/repos/dist/dev/incubator/impala/. Once graduated, it will presumably be https://dist.apache.org/repos/dist/dev/impala/. Upload all four artifactsUpload all three artifacts (tarball, .asc and .sha512). Do not overwrite old release candidates.
Code Block language bash svn checkout https://dist.apache.org/repos/dist/dev/incubator/impala/ cd impala/ # TheIf directoryyou layoutalready is x.y.z/RCw, where w is the release have a local checked out repo (e.g. if you did release before), run "svn update" to pull the latest updates. # The directory layout is x.y.z/RCw, where w is the release candidate number - RC1 is the first candidate, RC2 the second, and so on. mkdir -p x.y.z/RCw cd x.y.z/RCw cp /tmp/apache-impala-incubating-x.y.z.tar.gz* ./ svncd .. svn add . svn commit --username=YOUR_APACHE_USER_NAME -m "Impala x.y.z release candidate w"
- Prepare a patch to the downloads.html page (on the asf-site branch of the git repo) to point to the latest downloads and to remove the links to any downloads that are subsumed by this release. Do not submit it the patch yet.
- Take a vote on dev@. Your vote email should:
- Have a subject that starts with "[VOTE]"
- Explain what the vote is about
- Explain how to find the artifacts for testing, and include the tag and git tree hash (not release hash!) they correspond to. The tree hash can be viewed with git log --pretty="%T %s".
- Explain what each type of vote means
- Explain the conditions for the vote passing, including how long the vote will remain open for.
- Include a link to this wiki page so voters can read the instructions in it on how to test, verify, and vote.
- Explain how you tested it.
- At your discretion, discuss what dependencies or tools you used to compile or run it, like gcc, hadoop, impala-lzo, and so on.
- Be consistent with the Impala bylaws. For instance, at the moment I am writing this wiki page, votes stay open for 72 hours (not counting weekends), and only PPMC members (and mentors) have binding votes.
When the vote closes, tally up the votes (who votes what) in a thread with the same title as the vote thread, but with "[RESULT] " prepended. That email should include a list of every vote, and reasons for the -1s ala:
No Format nopanel true +1 (binding): Alice Bobopolis Carol Davestein -1 (non-binding): Emily Frankfurt (Build failed)
- If the vote passes, and Impala has yet to graduate, take a vote in the incubator PMC, following current incubator policy.
- Subscribe to general@incubator.apache.org by sending mail to general-subscribe@ and responding to the email it sends back.
- Send an email to that list including:
- Subject "[VOTE] Impala x.y.z release candidate w"
- A link to the archive of the proposal thread from the dev@impala mailing list, available either http://mail-archives.apache.org/mod_mbox/impala-dev/ or https://lists.apache.org/list.html?dev@impala.apache.org
- A link to the archive of the [RESULT] of the vote from the dev@impala mailing list.
- A link to the artifacts for testing
- A link to the KEYS file with the signatures
- A link to the git tag the release candidate tarball was created from
- The tree hash of the git commit the release candidate tarball was created from
- Instructions on using RAT on our code, or a description of where to find that information.
- Build instructions, or a description of where to find that information.
- A list of the IPMC members who have already +1ed. For instance, as of 28 September 2016, all of our mentors are IPMC members, so their +1s are good for either vote - the one with the PPMC or the one with the IPMC.
No Format nopanel true This vote will be open for at least 72 hours, or until the necessary number of votes (3 +1) is reached. [ ] +1 Approve the release [ ] -1 Don't approve the release (please provide specific comments)
Like with the dev@ vote, post a "[RESULT]" thread to general@ with the results.
Once that vote passes, tag the git tree at the release:
Code Block language bash git fetch apache git checkout apache/branch-x.y.z git tag -a x.y.z -m "x.y.z release" git push apache x.y.z
- Prepare a patch to include the changelog for the latest release on the docs site. More help on this here.
- Access the IMPALA Versions page
- Select the proper release version. They are not all in a sanely sorted order.
- On the top, select "Release Notes".
- Select "Configure Release Notes". Ensure HTML and the proper version are set.
- Click Create
- Scroll down to find a TEXTAREA containing the HTML release notes. Copy the contents and save it to a file.
Add HTML tags to make this a standalone page. For example:
Code Block language xml <!DOCTYPE html> <html> <head> <title>Impala 2.8 Change Log</title> </head> <body> <h1>Impala 2.8 Change Log</h1> <!-- contents of Jira-generated release notes go here --> </body> </html>
- Reorder the sections so that the issueType groupings show the more interesting groups, like New Feature, Epic, and Improvement, before less interesting groups like Sub-tasks.
- Provide a link to the changelog on the docs main page.
- Full example of this is here.
- If the full docs are not yet ready for the release, it's OK to leave the link to the previous release's HTML documentation so that a link to HTML documentation always exists. But please coordinate with the people who are working on the docs before you announce the release.
- Take a vote on dev@. Your vote email should:
- Have a subject that starts with "[VOTE]"
- Explain what the vote is about
- Explain how to find the artifacts for testing, and include the tag and git tree hash (not release hash!) they correspond to. The tree hash can be viewed with git log --pretty="%T %s".
- Include apache.org user name of the release manager
- Include the key that the artifacts are signed with, e.g. "Artifacts were signed with the 3BA2A72B key which can be found in https://downloads.apache.org/impala/KEYS"
- Explain what each type of vote means
- Explain the conditions for the vote passing, including how long the vote will remain open for.
- Include a link to this wiki page so voters can read the instructions in it on how to test, verify, and vote.
- Explain how you tested it.
- At your discretion, discuss what dependencies or tools you used to compile or run it, like gcc, hadoop, impala-lzo, and so on.
- Be consistent with the Impala bylaws. For instance, at the moment I am writing this wiki page, votes stay open for 72 hours (not counting weekends), and only PMC members have binding votes.
When the vote closes, tally up the votes (who votes what) in a thread with the same title as the vote thread, but with "[RESULT] " prepended. Link to the vote thread form https://lists.apache.org/list.html?dev@impala.apache.org. The email should include a list of every vote, and reasons for the -1s ala:
No Format nopanel true +1 (binding): Alice Bobopolis Carol Davestein -1 (non-binding): Emily Frankfurt (Build failed)
If that vote passes, tag the git tree at the release:
Code Block language bash git fetch apache git checkout apache/branch-x.y.z git tag -a x.y.z -m "x.y.z release" git push apache x.y.z
Publish the release. The location is https://dist.apache.org/repos/dist/release/impala/. Upload all three artifacts. If you are not a PMC member, you will need a PMC member to do the upload for you.
Code Block language bash svn checkout https://dist.apache.org/repos/dist/release/impala/ cd impala/ # If you already have a local checked out repo (e.g. if you did release before), run "svn update" to pull the latest updates. mkdir x.y.z cp /tmp/apache-impala-x.y.z.tar.gz* x.y.z/ svn add x.y.z svn
Publish the release. While incubating, the location is https://dist.apache.org/repos/dist/release/incubator/impala/. Once graduated, it will presumably be https://dist.apache.org/repos/dist/release/impala/. Upload all four artifacts.
Code Block language bash svn checkout https://dist.apache.org/repos/dist/release/incubator/impala/ cd impala/ mkdir x.y.z cd x.y.z cp /tmp/apache-impala-incubating-x.y.z.tar.gz* ./ svn add . svn commit --username=YOUR_APACHE_USER_NAME # You will be prompted to write a commit message. Make it look like (without the '#'s): # # Impala release x.y.z # # PPMCPMC vote thread: ... # # PPMCPMC vote results thread: ... # # IPMC vote thread: ... # # IPMC vote results thread: ...
- Wait 24 hours for mirrors to catch up.
- Push your patch to the downloads.html pageand changelog patches.In the bug
- tracker, change the target version for bugs targeted at x.y.z to a future release number. Optional but recommended: Add the release to the Apache Reporter Service: https://reporter.apache.org/addrelease.html?impala. If you are not a PMC member, you will need a PMC member to do it for you.
- Announce the release to dev@impala.apache.org, user@impala.apache.org, and announce@apache.org. The email must come from your apache.org email address.
- If you are not already subscribed from your @apache.org address, subscribe to dev@ and user@ by mailing dev-subscribe@ and user-subscribe@.
Give your email a subject like "[ANNOUNCE] Apache Impala (incubating) x.y.z release", and include in the body:
No Format nopanel true The Apache Impala (incubating) team is pleased to announce the release of Impala x.y.z. Impala is a high-performance C++ and Javadistributed SQL query engine for data stored in Apache Hadoop-based clusters. The release is available at: https://impala.incubator.apache.org/downloads.html Thanks, The Apache Impala (incubating) team ===== *Disclaimer* Apache Impala is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the name of Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
- Announce the release on the community links listed on https://impala.apache.org/community.html.
team
- To avoid the email get rejected by announce@apache.org, use "Plain text mode". The default in gmail has MIME Content-Type: 'text/html' which will be rejected.
- Announce the release on the community links listed on https://impala.apache.org/community.html.
- Send a patch review to the master branch to update its version number to "p.q.r-SNAPSHOT" (where p.q.r is greater than x.y.z)
- Update the location in bin/impala-config.sh
To update the version numbers for Java pom.xml files, use this maven command:
Code Block language bash # From $IMPALA_HOME cd java mvn versions:set -DnewVersion=p.q.r-SNAPSHOT
- Update the location in bin/impala-config.sh
- In the issue tracker, change the target version for open tickets targeted at x.y.z to p.q.r. You may need to create a version p.q.r if it does not already exist.
- In the issue tracker, "Release" the version on this screen: https://issues.apache.org/jira/plugins/servlet/project-config/IMPALA/versions. If you are not a PMC member, you will need a PMC member to do it for you.
In the SVN repo where you put the final release artifacts, delete the previous release, assuming that branch is no longer being actively worked on. If there is more than one active branch, leave the latest release from each branch up. If you are not a PMC member, you will need a PMC member do that for you. E.g. for removing the 4.1.1 release:
Send a patch review to the master branch to update its version number to "p.q.r-SNAPSHOT" (where p.q.r is greater than x.y.z)Code Block # In the folder where you checkout https://dist.apache.org/repos/dist/release/impala/ svn update svn delete 4.1.1 svn commit -m "Remove 4.1.1 release"
How to Vote on a Release Candidate
Download the tarball. Check the signature and the checksums, and check that the tarball matches the upstream tag. The script below shows how to do each of those steps. To use it, set your environment variables VERSION, RELEASE_CANDIDATE, TREE_HASH, and RELEASE_MANAGER like:
Code Block language bash VERSION=2.8.0 RELEASE_CANDIDATE=1 TREE_HASH=cc8de358d5c64778d171ad47aa6b513d437ac4b0 RELEASE_MANAGER=jbapple ./release.sh
You can get the RELEASE_MANAGER username from the KEYS file here. The rest of the fields should be evident from the Vote thread.
Committers can run this (along with the pre-commit tests) via the experimental "release-test-ub1604" job on Impala's Jenkins server.
Code Block language bash #!/bin/bash set -euxo pipefail echo "Set up a sandbox for key import and a temporary directory for git" export GNUPGHOME="$(mktemp -d)" IMPALA_GIT="$(mktemp -d)" echo "Delete gpg sandbox and temporary directory when done" function onexit { rm -rf "${GNUPGHOME}" rm -rf "${IMPALA_GIT}" df -m free -m uptime -p } trap onexit EXIT pushd "${GNUPGHOME}" echo "Download the keys of the release managers:" wget https://dist.apache.org/repos/dist/dev/incubator/impala/KEYS gpg --import KEYS echo "If in an interactive shell, At the prompt, enter '5' for 'I trust ultimately', then 'y' for 'yes', then 'q' for 'quit'" if [[ $- == *i* ]]; then gpg --edit-key ${RELEASE_MANAGER} trust fi echo "Download the release artifacts:" for SUFFIX in gz gz.asc gz.md5 gz.shasha512; do wget -q "https://dist.apache.org/repos/dist/dev/incubator/impala/${VERSION}/RC${RELEASE_CANDIDATE}/apache-impala-incubating-${VERSION}.tar.${SUFFIX}" done echo "Check the checksums:" md5sumsha512sum --check "apache-impala-incubating-${VERSION}.tar.gz.md5" sha1sum --check "apache-impala-incubating-${VERSION}.tar.gz.shasha512" echo "Check the signature:" gpg --verify "apache-impala-incubating-${VERSION}.tar.gz.asc" "apache-impala-${VERSION}.tar.gz" echo "Download git to make sure the tarball, git tag, and tree hash all correspond:" pushd "${IMPALA_GIT}" sudo apt-get -q=2 update sudo apt-get -q=2 install git git clone https://git-wip-usgitbox.apache.org/repos/asf/incubator-impala.git cd * git checkout "${VERSION}-rc${RELEASE_CANDIDATE}" echo "Check the tree hash from the release manager is correct:" if ! (git rev-list --pretty=format:"%T" --max-count=1 HEAD | grep "${TREE_HASH}"); then echo "Tree hash ${TREE_HASH} not found" exit 1 fi echo "Remove the .git directory to make tarball and git directories equal:" rm -rf .git IMPALA_GIT="$(pwd)" popd echo "Compare the tarball and the repo:" tar xzf "apache-impala-incubating-${VERSION}.tar.gz" diff -r "apache-impala-incubating-${VERSION}" "${IMPALA_GIT}"
Test the release quality, possibly using
bin/run-all-tests.py.
The ASF requires in its "Release Policy" that: "Before voting +1 PMC members are required to download the signed source code package, compile it as provided, and test the resulting executable on their own platform, along with also verifying that the package meets the requirements of the ASF policy on releases." The ASF interprets "own platform" in this sentence to not require that you own and physically control the machine you are testing on, unlike the procedure for signing a release.Check compliance with ASF release policy. Use Apache RAT and follow the instructions in
bin/check-rat-report.py
to check licence compliance.- If it is an official "[VOTE]" thread, vote +1 or -1. If you are a PMC member, add "(binding)" after your vote; otherwise, add "(non-binding)". If you vote -1, explain why.
...