Page History

...

Follow Apache guidelines on setting up your GPG keys and ensure that your fingerprint is updated at id.apache.org.Also, append your keys to the KEYS file at https://dist.apache.org/repos/dist/release/tez/KEYS as well as the KEYS file at the top of the source tree.

Code Block
gpg --armor --fingerprint --list-sigs <keyid> gpg --armor --export <keyid>

Ensure that you publish your key at http://pgp.mit.edu/

Creating a Release Candidate

Create or Checkout the Release Branch

Code Block
git checkout -b branch-x.y.z

Run Basic Release checks

Code Block
mvn clean install -DskipTests=true mvn clean apache-rat:check

Push branch to remote

Code Block
git push -u origin branch-x.y.z

Generate CHANGES.txt

Currently we don't keep CHANGES.txt in repo and only generate it on demand. To find previous example, search in previous release branch.

To generate CHANGES.txt, please pull information from JIRA and git log. To get all JIRAs of this release, use filter 'fixVersion = <version> AND resolution = resolved'. To get all commits of this release, you can find the last commit in last release in corresponding release branch and do a git diff against current release branch. Open a JIRA to commit the newly generated CHANGES.txt to branch-x.y.z. A review of the patch is preferable.

Code Block
git log rel/<previous_release>..origin/branch-x.y.z --format="%s" # command to get the commit logs that go in CHANGES.txt #origin is the git remote for https://gitbox.apache.org/repos/asf/tez.git

There are cases where information from JIRAs is inconsistent with that from git log (which shouldn't happen but unfortunately happens sometime...). For example, committer may forget to mark JIRA as resolved or set the fix version, or commit message contains wrong JIRA number. So it's better to check both side.

Several things need notice:

Incompatible changes should be found by using filter "Hadoop Flags" = "Incompatible change". Don't search by labels 'incompatible' or something similar.
Umbrella JIRA should be listed in individually (instead of getting mixed in non-umbrella jira)
Changes are grouped by release version. If one patch gets committed into multiple branches, put this change in the oldest one. For example, if a patch is committed in branch 0.9.0 (current release), 0.8.6 (unreleased of 0.8 line), this change should be listed in the section of 0.8.6.

Update Version Number if Required

Code Block

mvn versions:set -DnewVersion="x.y.z"
Modify CHANGES.txt to set the release date to when the vote will likely end ( +3 or 4 days from the vote start ).
If needed, add missing entries in CHANGES.txt for this release's ChangeLog using the output from "git log --pretty=oneline"
Commit and push all the changes.

Create a Release Tag

Code Block
git tag -a release-x.y.z-rc0 -m 'Tez x.y.z RC0' # push the new tag git push origin release-x.y.z-rc0

Deploy the jars to Staging

Your .m2/settings.xml should have something along these lines:

Code Block

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
     <id>apache.staging.https</id>
     <username>XXXXXX</username>
     <password>YYYYYY</password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>gpg</id>
      <properties>
        <gpg.passphrase>XXXXXXX</gpg.passphrase>
        <!-- gpg.executable should be gpg if a lower version is used for creating the key -->
        <gpg.executable>gpg2</gpg.executable>
      </properties>
    </profile>
  </profiles>
</settings>

To deploy jars, run:

Code Block
mvn clean deploy -Psources,javadoc,sign -DskipTests=true

The above command allows one to see the deployed jars in the Staging area. Log on to https://repository.apache.org and look at the Staging Repositories. "Close" the tez jars recently uploaded to get the temporary staging repo URL.

Create a Release tarball for the Source

Code Block

mvn clean
git clean -f -x -d
cd ..
cp -R tez-git-x.y.z apache-tez-{x.y.z}-src (where tez-git-x.y.z is the tez code directory)
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-{x.y.z}-src.tar.gz apache-tez-{x.y.z}-src

Sign the Release

Code Block
gpg2 --armor --output apache-tez-{x.y.z}-src.tar.gz.asc --detach-sig apache-tez-{x.y.z}-src.tar.gz

Generate checksums

Code Block
shasum -a 512 apache-tez-{x.y.z}-src.tar.gz > apache-tez-{x.y.z}-src.tar.gz.sha512

or

Code Block
openssl dgst -sha512 apache-tez-{x.y.z}-src.tar.gz > apache-tez-{x.y.z}-src.tar.gz.sha512

Create Binary Tarball as a helper for Users

Create a final binary tarball using the minimal and full tarballs:

Create a top-level directory called apache-tez-{x.y.z}-bin
Untar the contents of the minimal tarball into this directory.
Copy the full tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez.tar.gz
Copy the LICENSE* files and NOTICE file from the full tarball and replace the files under apache-tez-{x.y.z}-bin/ which came from the minimal tarball.
If the version is higher than 0.8.x, use the mvn javadoc command to generate the xml-based config documentation and copy the files under apache-tez-{x.y.z}-bin/conf/ with a .template suffix.
Tar the top-level directory and do the necessary signing as well as the checksum creation as done for the src tarball.

and sks-keyservers.net and keyserver.ubuntu.com. While creating a staging artifact, "Signature Validation" process seems to check your key at sks-keyserver.net and keyserver.ubuntu.com.

The snippet below will take care of the important keyservers for you.

Code Block
gpg --keyserver pgp.mit.edu --send-key <keyid> gpg --keyserver keyserver.ubuntu.com --send-key <keyid> gpg --keyserver pool.sks-keyservers.net --send-key <keyid>

Add PGP Key from above to Tez source tree and distribution repository. KEYS file in Tez distribution repository will be used for release verification in release candidate voting

Code Block

language	bash
title	Distribution Keys

# Append PGP Key to KEYS file in https://gitbox.apache.org/repos/asf/tez.git
$ svn co https://dist.apache.org/repos/dist/release/tez/ tez-dist/
# copy KEYS file from gitbox tez/KEYS to tez-dist/KEYS and verify difference
$ svn diff
# Commit the changes (no jira required) with a useful message substituting apache user name as appropriate
$ svn ci -m "Add <apache user name> public key to KEYS file"

Creating a Release Candidate

Create or Checkout the Release Branch

Code Block
git checkout -b branch-x.y.z

Run Basic Release checks

Code Block
mvn clean install -DskipTests mvn clean apache-rat:check

Push branch to remote

Code Block
git push -u origin branch-x.y.z

Generate CHANGES.txt

Currently we don't keep CHANGES.txt in repo and only generate it on demand. To find previous example, search in previous release branch.

To generate CHANGES.txt, please pull information from JIRA and git log. To get all JIRAs of this release, use filter 'fixVersion = <version> AND resolution = resolved'. To get all commits of this release, you can find the last commit in last release in corresponding release branch and do a git diff against current release branch. Open a JIRA to commit the newly generated CHANGES.txt to branch-x.y.z. A review of the patch is preferable.

Code Block

# origin points to the tez repo: https://github.com/apache/tez
# a.b.c: last tez release version
# x.z.y: current tez release version
git log rel/release-a.b.c..origin/branch-x.y.z --format="%s" # command to get the commit logs that go in CHANGES.txt

There are cases where information from JIRAs is inconsistent with that from git log (which shouldn't happen but unfortunately happens sometime...). For example, committer may forget to mark JIRA as resolved or set the fix version, or commit message contains wrong JIRA number. So it's better to check both side.

Several things need notice:

Incompatible changes should be found by using filter "Hadoop Flags" = "Incompatible change". Don't search by labels 'incompatible' or something similar.
Umbrella JIRA should be listed in individually (instead of getting mixed in non-umbrella jira)
Changes are grouped by release version. If one patch gets committed into multiple branches, put this change in the oldest one. For example, if a patch is committed in branch 0.9.0 (current release), 0.8.6 (unreleased of 0.8 line), this change should be listed in the section of 0.8.6.

Update Version Number if Required

Code Block

mvn versions:set -DnewVersion="x.y.z"
Modify CHANGES.txt to set the release date to when the vote will likely end ( +3 or 4 days from the vote start ).
If needed, add missing entries in CHANGES.txt for this release's ChangeLog using the output from "git log --pretty=oneline"
Commit and push all the changes.

Create a Release Tag

Code Block

export RELEASE_VERSION=x.y.z #e.g. 0.8.4
export RC_VERSION=rc0 #define rc version if you haven't defined yet
git tag -a release-$RELEASE_VERSION-$RC_VERSION -m "Tez $RELEASE_VERSION $RC_VERSION"
# push the new tag 
git push origin release-$RELEASE_VERSION-$RC_VERSION

Deploy the jars to Staging

Your .m2/settings.xml should have something along these lines:

The full apache guide for publishing artifacts can be found at http://www.apache.org/dev/publishing-maven-artifacts.html

Code Block

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <servers>
    <server>
     <id>apache.staging.https</id>
     <username>XXXXXX</username>
     <password>YYYYYY</password>
    </server>
  </servers>
  <profiles>
    <profile>
      <id>gpg</id>
      <properties>
        <gpg.passphrase>XXXXXXX</gpg.passphrase>
        <!-- gpg.executable should be gpg if a lower version is used for creating the key -->
        <gpg.executable>gpg2</gpg.executable>
      </properties>
    </profile>
  </profiles>
</settings>

To deploy jars, run:

Code Block
mvn clean deploy -Psources,javadoc,sign -DskipTests # if you get this cryptic error: gpg: signing failed: Inappropriate ioctl for device, then just do this below first: export GPG_TTY=$(tty)

The above command allows one to see the deployed jars in the Staging area. Log on to https://repository.apache.org and look at the Staging Repositories. "Close" the tez jars recently uploaded to get the temporary staging repo URL.

Create a Release tarball for the Source

Code Block

export RELEASE_VERSION=x.y.z #e.g. 0.8.4
mvn clean
git clean -f -x -d

# handle TEZ-4533 temporarily
rm ./tez-runtime-library/src/test/resources/TestIFile_concatenated_compressed.bin
sed -i "" 's/public void testConcatenatedZlibPadding/@org.junit.Ignore public void testConcatenatedZlibPadding/g' ./tez-runtime-library/src/test/java/org/apache/tez/runtime/library/common/sort/impl/TestIFile.java


cd .. # assuming that you're standing in a tez git folder, 
cp -R tez-git-$RELEASE_VERSION apache-tez-$RELEASE_VERSION-src #where tez-git-$RELEASE_VERSION is the tez code directory
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-$RELEASE_VERSION-src.tar.gz apache-tez-$RELEASE_VERSION-src

Sign the Release

Code Block
gpg2 --armor --output apache-tez-$RELEASE_VERSION-src.tar.gz.asc --detach-sig apache-tez-$RELEASE_VERSION-src.tar.gz

Generate checksums

Code Block
shasum -a 512 apache-tez-$RELEASE_VERSION-src.tar.gz > apache-tez-$RELEASE_VERSION-src.tar.gz.sha512

Create Binary Tarball as a helper for Users

Create a final binary tarball using the minimal and full tarballs:

Create a top-level directory called apache-tez-{x.y.z}-bin
Untar the contents of the minimal tarball into this directory.
Copy the full tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez.tar.gz

Copy the minimal tarball into the directory under apache-tez-{x.y.z}-bin/share/ and name the file tez-minimal.tar.gz

Copy the LICENSE* files and NOTICE file from the full tarball and replace the files under apache-tez-{x.y.z}-bin/ which came from the minimal tarball.
If the version is higher than 0.8.x, use the mvn javadoc command to generate the xml-based config documentation and copy the files under apache-tez-{x.y.z}-bin/conf/ with a .template suffix.
Tar the top-level directory and do the necessary signing as well as the checksum creation as done for the src tarball.

Code Block

#Define the following environment variables
export TEZ_SRC_DIR=/home/user/tez #e.g. tez-src or the absolute path
export RELEASE_VERSION=x.y.z #e.g. 0.8.4
 
cd ${TEZ_SRC_DIR};
mvn clean install -DskipTests
mvn site
cd ..


mkdir apache-tez-${RELEASE_VERSION}-bin
tar -C apache-tez-${RELEASE_VERSION}-bin -zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}-minimal.tar.gz
mkdir tmp-tez-full
tar -C tmp-tez-full -zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}.tar.gz
cp tmp-tez-full/LICENSE* apache-tez-${RELEASE_VERSION}-bin/
cp tmp-tez-full/NOTICE* apache-tez-${RELEASE_VERSION}-bin/
rm -rf tmp-tez-full


mkdir apache-tez-${RELEASE_VERSION}-bin/conf
# copy over tez templates into conf dir.
cp ${TEZ_SRC_DIR}/tez-api/target/site/apidocs/configs/tez-default-template.xml apache-tez-${RELEASE_VERSION}-bin/conf/
cp ${TEZ_SRC_DIR}/tez-runtime-library/target/site/apidocs/configs/tez-runtime-default-template.xml apache-tez-${RELEASE_VERSION}-bin/conf/


# TODO: If TEZ-3322 is not fixed, modify the config files under apache-tez-${RELEASE_VERSION}-bin/conf/ and add the Apache header, see example below


mkdir apache-tez-${RELEASE_VERSION}-bin/share
cp ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}.tar.gz apache-tez-${RELEASE_VERSION}-bin/share/tez.tar.gz
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf apache-tez-${RELEASE_VERSION}-bin.tar.gz apache-tez-${RELEASE_VERSION}-bin
# Sign release and create checksums using commands similar to those called out earlier for the source tarball
gpg2  --armor --output apache-tez-${RELEASE_VERSION}-bin.tar.gz.asc --detach-sig apache-tez-${RELEASE_VERSION}-bin.tar.gz
shasum -a 512 apache-tez-$RELEASE_VERSION-src.tar.gz > apache-tez-$RELEASE_VERSION-src.tar.gz.sha512


#Verify integrity of checksum files

Apache header example

Code Block

<?xml version="1.0"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

Upload Artifacts for Release Vote

The artifacts ( source tarball, binary tarball, checksums etc) need to be copied over to https://dist.apache.org/repos/dist/dev/tez/. ( This is an SVN repo where the release artifacts need to be committed to ). The artifacts should be created under a directory with the release number and RC number.

Code Block

svn co https://dist.apache.org/repos/dist/dev/tez/ tez_svn # assuming that you want to store svn files in a folder named "svn" for clarity's sake

# copy contents to "tez_svn" directory, which is the svn working dir
export RC_VERSION=rc0 #define rc version if you haven't defined yet
mkdir tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}
cp

Code Block

#Define the following environment variables
TEZ_SRC_DIR (e.g. tez-src or the absolute path)
RELEASE_VERSION (e.g. 0.8.4)
 
cd ${TEZ_SRC_DIR};
mvn clean install -DskipTests
mvn site
cd ..


mkdir apache-tez-${RELEASE_VERSION}-bin
tar -C apache-tez-${RELEASE_VERSION}-bin -zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}-minimal.tar.gz
mkdir tmp-tez-full
tar -C tmp-tez-full -zxvf ${TEZ_SRC_DIR}/tez-dist/target/tez-${RELEASE_VERSION}.tar.gz
cp tmp-tez-full/LICENSE* apache-tez-${RELEASE_VERSION}-bin/
cp tmp-tez-full/NOTICE* apache-tez-${RELEASE_VERSION}-bin/
rm -rf tmp-tez-full


mkdir .tar.gz tez_svn/apache-tez-${RELEASE_VERSION}-bin/conf
# copy over tez templates into conf dir.
cp ${TEZRC_SRC_DIR}/tez-api/target/site/apidocs/configs/tez-default-template.xml VERSION}/
cp apache-tez-${RELEASE_VERSION}-bin.tar.gz.asc tez_svn/apache-tez-${RELEASE_VERSION}-bin/conf${RC_VERSION}/
cp apache-tez-${TEZRELEASE_SRC_DIRVERSION}/tez-runtime-library/target/site/apidocs/configs/tez-runtime-default-template.xml -bin.tar.gz.sha512 tez_svn/apache-tez-${RELEASE_VERSION}-bin/conf/


# TODO: If TEZ-3322 is not fixed, modify the config files under ${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-bin/conf/ and add the Apache header


mkdir src.tar.gz tez_svn/apache-tez-${RELEASE_VERSION}-bin/share
cp ${TEZRC_SRC_DIRVERSION}/tez-dist/target/
cp apache-tez-${RELEASE_VERSION}-src.tar.gz.asc tez_svn/apache-tez-${RELEASE_VERSION}-bin/share/tez.tar.gz
COPYFILE_DISABLE=1 tar --exclude=.git -zcvf ${RC_VERSION}/
cp apache-tez-${RELEASE_VERSION}-binsrc.tar.gz.sha512 tez_svn/apache-tez-${RELEASE_VERSION}-bin${RC_VERSION}/

# Signcheck releasecontents and create checksums using commands similar to those called out earlier for the source tarball
gpg2  --armor --output of "tez_svn" for actual release, you should see something like:
ls tez_svn/apache-tez-${RELEASE_VERSION}-bin.tar.gz.asc --detach-sig apache-tez-${RELEASERC_VERSION}/


apache-tez-x.y.z-bin.tar.gz
openssl md5         apache-tez-${RELEASE_VERSION}-binx.y.z-src.tar.gz >      
apache-tez-${RELEASE_VERSION}x.y.z-bin.tar.gz.md5
openssl dgst -sha512asc     apache-tez-${RELEASE_VERSION}x.y.z-bin.tar.gz.sha512 > apache-tez-${RELEASE_VERSION}-binx.y.z-src.tar.gz.sha512

#Remove unnecessary text from checksum files 
sed -i "" s/"^.*= "// asc     apache-tez-x.y.z-src.tar.gz.sha512

svn add tez_svn/apache-tez-${RELEASE_VERSION}-bin.tar.gz.md5
sed -i "" s/"^.*= "// apache-tez-${RELEASE_VERSION}-bin.tar.gz.sha512
#Verify integrity of checksum files

Upload Artifacts for Release Vote

...

${RC_VERSION}/
# if this is the first time you commit to svn, you might want to define an editor for commit messages, like: export SVN_EDITOR=vi
svn ci tez_svn/apache-tez-${RELEASE_VERSION}-${RC_VERSION}/ #specify your ASF username with --username=$USER_NAME if it's not the same as your current user

Release Voting process

Start a Vote

...

I have created an tez-x.y.zrelease z release candidate rc0.
GIT source tag (r***)

...

Staging site: https://dist.apache.org/repos/dist/dev/tez/tez -release-rc/tez-x.y.z-rc0/ ( svn revision <revision> )
Nexus Staging URL:

...

Log on to https://repository.apache.org and look at the Staging Repositories. "Release" the tez repository previously sent out in the vote thread, so that it is available for general consumption.

...

Code Block
svn mv https://dist.apache.org/repos/dist/dev/tez/apache-tez-x.y.z-rc0 https://dist.apache.org/repos/dist/release/tez/x.y.z

...

git tag -a rel/release-x.y.z -m 'Tez x.y.z'
git push --tags originorigin rel/release-x.y.z

Note that it takes 24 hours for the changes to propagate to the mirrors.
Wait 24 hours and verify that the bits are available in the mirrors before sending an announcement. Sometime it takes longer than 24 hours.

...

The javadocs should be generated under the correct branch so that they are tagged properly. To generate the javadocs, go to each project directory (that has user APIs) and run.

Code Block
mvn clean install javadoc:aggregate -DskipTests mvn site -f tez-api mvn site -f tez-mapreduce mvn site -f tez-runtime-library

The javadocs would be created under target/site/apidocs inside each project directory. Spot check the javadocs for obvious errors. Create a new folder x.y.z under the releases directory in the svn site repo and copy release-notes and apidocs from all the projects into that directory. The website is built from the git repo but the javadocs etc are committed only to svn to reduce the size of the git repo.

...

Send out Announcement to dev@tez.apache.org and user@tez.apache.org
SujectSubject: [ANNOUNCE] Apache Tez x.y.z.

...

The Apache Tez project is aimed at creating a framework to build efficient and scalable data processing applications that can be modelled modeled as data flow graphs.

...

Find all JIRAs that were fixed in this release ( i.e. Fix Version set to the released version ) and do a bulk edit to mark them all as Closed. A useful point to note is that when doing this bulk update, mail notifications should be turned off to avoid triggering a barrage of emails being sent out. Find all JIRAs with fix version x.y.z and not-resolved/not-closed and remove the fix version.

Once this is complete, using the JIRA Administration tab, mark the x.y.z as released and set the appropriate release date.

https://issues.apache.org/jira/plugins/servlet/project-config/TEZ/versions

Create the next version if its it is already not done. Find all JIRAs with target version x.y.z and not-resolved/not-closed and change the target version to the next release following the above bulk update process.

...

Space shortcuts

Child pages

Versions Compared

Old Version 71

New Version Current

Key

Creating a Release Candidate

Create or Checkout the Release Branch

Run Basic Release checks

Push branch to remote

Generate CHANGES.txt

Update Version Number if Required

Create a Release Tag

Deploy the jars to Staging

Create a Release tarball for the Source

Sign the Release

Generate checksums

Create Binary Tarball as a helper for Users

Creating a Release Candidate

Create or Checkout the Release Branch

Run Basic Release checks

Push branch to remote

Generate CHANGES.txt

Update Version Number if Required

Create a Release Tag

Deploy the jars to Staging

Create a Release tarball for the Source

Sign the Release

Generate checksums

Create Binary Tarball as a helper for Users

Apache header example

Upload Artifacts for Release Vote

Upload Artifacts for Release Vote

Start a Vote