How To Release Tez
GPG Keys Setup
Follow Apache guidelines on setting up your GPG keys and ensure that your fingerprint is updated at id.apache.org.
| Code Block |
|---|
gpg --armor --fingerprint --list-sigs <keyid>
gpg --armor --export <keyid>
|
Ensure that you publish your key at http://pgp.mit.edu/ and sks-keyservers.net and keyserver.ubuntu.com. While creating a staging artifact, "Signature Validation" process seems to check your key at sks-keyserver.net and keyserver.ubuntu.com.
The snippet below will take care of the important keyservers for you.
| Code Block |
|---|
gpg --keyserver pgp.mit.edu --send-key <keyid>
gpg --keyserver keyserver.ubuntu.com --send-key <keyid>
gpg --keyserver pool.sks-keyservers.net --send-key <keyid>
|
Add PGP Key from above to Tez source tree and distribution repository. KEYS file in Tez distribution repository will be used for release verification in release candidate voting
| Code Block |
|---|
| language | bash |
|---|
| title | Distribution Keys |
|---|
|
# Append PGP Key to KEYS file in https://gitbox.apache.org/repos/asf/tez.git
$ svn co https://dist.apache.org/repos/dist/release/tez/ tez-dist/
# copy KEYS file from gitbox tez/KEYS to tez-dist/KEYS and verify difference
$ svn diff
# Commit the changes (no jira required) with a useful message substituting apache user name as appropriate
$ svn ci -m "Add <apache user name> public key to KEYS file" |
Creating a Release Candidate
Create or Checkout the Release Branch
| Code Block |
|---|
git checkout -b branch-x.y.z
|
Run Basic Release checks
| Code Block |
|---|
mvn clean install -DskipTests
mvn clean apache-rat:check
|
Push branch to remote
| Code Block |
|---|
git push -u origin |
* Copy the full tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez.tar.gz
* Copy the minimal tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez-minimal.tar.gz How To Release Tez
GPG Keys Setup
Follow Apache guidelines on setting up your GPG keys and ensure that your fingerprint is updated at id.apache.org.
| Code Block |
|---|
gpg --armor --fingerprint --list-sigs <keyid>
gpg --armor --export <keyid>
|
Ensure that you publish your key at http://pgp.mit.edu/ and sks-keyservers.net and keyserver.ubuntu.com. While creating a staging artifact, "Signature Validation" process seems to check your key at sks-keyserver.net and keyserver.ubuntu.com.
The snippet below will take care of the important keyservers for you.
| Code Block |
|---|
gpg --keyserver pgp.mit.edu --send-key <keyid>
gpg --keyserver keyserver.ubuntu.com --send-key <keyid>
gpg --keyserver pool.sks-keyservers.net --send-key <keyid>
|
Add PGP Key from above to Tez source tree and distribution repository. KEYS file in Tez distribution repository will be used for release verification in release candidate voting
| Code Block |
|---|
| language | bash |
|---|
| title | Distribution Keys |
|---|
|
# Append PGP Key to KEYS file in https://gitbox.apache.org/repos/asf/tez.git
$ svn co https://dist.apache.org/repos/dist/release/tez/ tez-dist/
# copy KEYS file from gitbox tez/KEYS to tez-dist/KEYS and verify difference
$ svn diff
# Commit the changes (no jira required) with a useful message substituting apache user name as appropriate
$ svn ci -m "Add <apache user name> public key to KEYS file" |
Creating a Release Candidate
Create or Checkout the Release Branch
| Code Block |
|---|
git checkout -b branch-x.y.z
|
Run Basic Release checks
| Code Block |
|---|
mvn clean install -DskipTests
mvn clean apache-rat:check
|
Push branch to remote
| Code Block |
|---|
git push -u origin branch-x.y.z
|
Generate CHANGES.Generate CHANGES.txt
Currently we don't keep CHANGES.txt in repo and only generate it on demand. To find previous example, search in previous release branch.
To generate CHANGES.txt, please pull information from JIRA and git log. To get all JIRAs of this release, use filter 'fixVersion = <version> AND resolution = resolved'. To get all commits of this release, you can find the last commit in last release in corresponding release branch and do a git diff against current release branch. Open a JIRA to commit the newly generated CHANGES.txt to branch-x.y.z. A review of the patch is preferable.
| Code Block |
|---|
git log rel/<previous_release>..# origin points to the tez repo: https://github.com/apache/tez
# a.b.c: last tez release version
# x.z.y: current tez release version
git log rel/release-a.b.c..origin/branch-x.y.z --format="%s" # command to get the commit logs that go in CHANGES.txt
#origin is the git remote for https://gitbox.apache.org/repos/asf/tez.git |
There are cases where information from JIRAs is inconsistent with that from There are cases where information from JIRAs is inconsistent with that from git log (which shouldn't happen but unfortunately happens sometime...). For example, committer may forget to mark JIRA as resolved or set the fix version, or commit message contains wrong JIRA number. So it's better to check both side.
...
| Code Block |
|---|
export RELEASE_VERSION=x.y.z #e.g. 0.8.4
export RC_VERSION=rc0 #define rc version if you haven't defined yet
git tag -a release-$RELEASE_VERSION-rc0$RC_VERSION -m "Tez $RELEASE_VERSION RC0$RC_VERSION"
# push the new tag
git push origin release-$RELEASE_VERSION-rc0
$RC_VERSION |
Deploy the jars to Staging
...
| Code Block |
|---|
export RELEASE_VERSION=x.y.z #e.g. 0.8.4
mvn clean
git clean -f -x -d
cd ..
cp -R tez-git-$RELEASE_VERSION apache-tez-$RELEASE_VERSION-src #where tez-git-$RELEASE_VERSION is the tez code directory
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-$RELEASE_VERSION-src.tar.gz apache-tez-$RELEASE_VERSION-src
|
Sign the Release
| Code Block |
|---|
gpg2 --armor --output
# handle TEZ-4533 temporarily
rm ./tez-runtime-library/src/test/resources/TestIFile_concatenated_compressed.bin
sed -i "" 's/public void testConcatenatedZlibPadding/@org.junit.Ignore public void testConcatenatedZlibPadding/g' ./tez-runtime-library/src/test/java/org/apache/tez/runtime/library/common/sort/impl/TestIFile.java
cd .. # assuming that you're standing in a tez git folder,
cp -R tez-git-$RELEASE_VERSION apache-tez-$RELEASE_VERSION-src.tar.gz.asc --detach-sig apache-tez #where tez-git-$RELEASE_VERSION-src.tar.gz
|
Generate checksums
| Code Block |
|---|
shasum -a 512 is the tez code directory
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-$RELEASE_VERSION-src.tar.gz > apache-tez-$RELEASE_VERSION-src.tar.gz.sha512
|
...
Sign the Release
| Code Block |
|---|
openssl md5 gpg2 --armor --output apache-tez-$RELEASE_VERSION-src.tar.gz >.asc --detach-sig apache-tez-$RELEASE_VERSION-src.tar.gz
|
Generate checksums
| Code Block |
|---|
shasum -a 512.md5
openssl dgst -sha512 apache-tez-$RELEASE_VERSION-src.tar.gz > apache-tez-$RELEASE_VERSION-src.tar.gz.sha512
|
Create Binary Tarball as a helper for Users
...
- Create a top-level directory called apache-tez-{x.y.z}-bin
- Untar the contents of the minimal tarball into this directory.
- Copy the full tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez.tar.gz
Copy the minimal tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez-minimal.tar.gz - Copy the LICENSE* files and NOTICE file from the full tarball and replace the files under apache-tez-{x.y.z}-bin/ which came from the minimal tarball.
- If the version is higher than 0.8.x, use the mvn javadoc command to generate the xml-based config documentation and copy the files under apache-tez-{x.y.z}-bin/conf/ with a .template suffix.
- Tar the top-level directory and do the necessary signing as well as the checksum creation as done for the src tarball.
| Code Block |
|---|
#Define the following environment variables
export |
| Code Block |
|---|
#Define the following environment variables
export TEZ_SRC_DIR=/home/user/tez #e.g. tez-src or the absolute path
export RELEASE_VERSION=x.y.z #e.g. 0.8.4
cd ${TEZ_SRC_DIR};
mvn clean install -DskipTests
mvn site
cd ..
mkdir apache-tez-${RELEASE_VERSION}-bin
tar -C apache-tez-${RELEASE_VERSION}-bin -zxvf ${TEZ_SRC_DIR}=/tez-disthome/targetuser/tez-${RELEASE_VERSION}-minimal.tar.gz
mkdir tmp-tez-full
tar -C tmp-tez-full -zxvf #e.g. tez-src or the absolute path
export RELEASE_VERSION=x.y.z #e.g. 0.8.4
cd ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}.tar.gz
cp tmp-tez-full/LICENSE*;
mvn clean install -DskipTests
mvn site
cd ..
mkdir apache-tez-${RELEASE_VERSION}-bin/
cptar tmp-tez-full/NOTICE*-C apache-tez-${RELEASE_VERSION}-bin/
rm -rf tmp-tez-full
mkdir apache-zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}-bin/conf
# copy over tez templates into conf dir.
cpminimal.tar.gz
mkdir tmp-tez-full
tar -C tmp-tez-full -zxvf ${TEZ_SRC_DIR}/tez-apidist/target/site/apidocs/configs/tez-default-template.xmltez-${RELEASE_VERSION}.tar.gz
cp tmp-tez-full/LICENSE* apache-tez-${RELEASE_VERSION}-bin/conf/
cp ${TEZ_SRC_DIR}/tmp-tez-runtime-library/target/site/apidocs/configs/tez-runtime-default-template.xmlfull/NOTICE* apache-tez-${RELEASE_VERSION}-bin/conf/
# TODO: If TEZ-3322 is not fixed, modify the config files under rm -rf tmp-tez-full
mkdir apache-tez-${RELEASE_VERSION}-bin/conf/
# andcopy addover thetez Apache header, see example below
mkdir apache-tez-${RELEASE_VERSION}-bin/sharetemplates into conf dir.
cp ${TEZ_SRC_DIR}/tez-distapi/target/site/apidocs/configs/tez-${RELEASE_VERSION}.tar.gzdefault-template.xml apache-tez-${RELEASE_VERSION}-bin/shareconf/tez.tar.gz
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-${RELEASE_VERSION}-bin.tar.gz
cp ${TEZ_SRC_DIR}/tez-runtime-library/target/site/apidocs/configs/tez-runtime-default-template.xml apache-tez-${RELEASE_VERSION}-bin/conf/
# SignTODO: releaseIf andTEZ-3322 createis checksumsnot usingfixed, commandsmodify similarthe toconfig thosefiles called out earlier forunder apache-tez-${RELEASE_VERSION}-bin/conf/ and add the source tarball
gpg2 --armor --outputApache header, see example below
mkdir apache-tez-${RELEASE_VERSION}-bin.tar.gz.asc --detach-sig apache-/share
cp ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}-bin.tar.gz
openssl md5 apache-tez-${RELEASE_VERSION}-bin/share/tez.tar.gz > apache-tez-${RELEASE_VERSION}-bin.tar.gz.md5
openssl dgst -sha512
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-${RELEASE_VERSION}-bin.tar.gz > apache-tez-${RELEASE_VERSION}-bin.tar.gz.sha512
# Sign Removerelease unnecessaryand textcreate fromchecksums checksumusing files
# (the commands below are supposed to work with GNU sed)
sed -i"" s/"^.*= "//commands similar to those called out earlier for the source tarball
gpg2 --armor --output apache-tez-${RELEASE_VERSION}-bin.tar.gz.md5
sed -i"" s/"^.*= "// asc --detach-sig apache-tez-${RELEASE_VERSION}-bin.tar.gz.sha512
sedshasum -i"" s/"^.*= "// a 512 apache-tez-${RELEASE_VERSION}-src.tar.gz.md5
sed -i"" s/"^.*= "//$RELEASE_VERSION-src.tar.gz > apache-tez-${RELEASE$RELEASE_VERSION}-src.tar.gz.sha512
#Verify integrity of checksum files
|
...
The artifacts ( source tarball, binary tarball, checksums etc) need to be copied over to https://dist.apache.org/repos/dist/dev/tez/. ( This is an SVN repo where the release artifacts need to be committed to ). The artifacts should be created under a directory with the release number and RC number.
| Code Block |
|---|
$ svn co https://dist.apache.org/repos/dist/dev/tez/svn co https://dist.apache.org/repos/dist/dev/tez/ tez_svn # assuming that you want to store svn files in a folder named "svn" for clarity's sake
# copy contents to "tez_svn" directory, which is the svn working dir
export RC_VERSION=rc0 #define rc version if you haven't defined yet
mkdir tez/apache-tez-${RELEASE_VERSION}-${RC_VERSION}
cp apache-tez-${RELEASE_VERSION}-bin.tar.gz tez haven't defined yet
mkdir tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-bin.tar.gz.asc tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-bin.tar.gz.md5asc tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-bin.tar.gz.sha512 tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-src.tar.gz tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-src.tar.gz.asc tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-src.tar.gz.md5 tez/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-src.tar.gz.sha512 tezsha512 tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
# check contents of "tez_svn" for actual release, you should see something like:
$ ls tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
apache-tez-x.y.z-bin.tar.gz apache-tez-x.y.z-src.tar.gz
apache-tez-x.y.z-bin.tar.gz.asc apache-tez-x.y.z-bin.tar.gz.sha512 apache-tez-x.y.z-src.tar.gz.asc apache-tez-x.y.z-src.tar.gz.sha512
$ svn add tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/
$ # if this is the first time you commit to svn, you might want to define an editor for commit messages, like: export SVN_EDITOR=vi
svn ci tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/ #specify your ASF username with --username=$USER_NAME if it's not the same as your current user |
...
Log on to https://repository.apache.org and look at the Staging Repositories. "Release" the tez repository previously sent out in the vote thread, so that it is available for general consumption.
...
| Code Block |
|---|
svn mv https://dist.apache.org/repos/dist/dev/tez/apache-tez-x.y.z-rc0 https://dist.apache.org/repos/dist/release/tez/x.y.z
|
...
