Page History

...

• The process of auditing release has been automated via this release audit script.
  • Find the staging repository in Apache Nexus to which the artifacts were uploaded to. 
  • Configure the script by specfiying the version number to audit, the key ID of the signing key, and the URL to staging repository.
  • This script has to be run from the parent directory for the script.
  • Make sure "sbt" is installed.
• The release auditor will test example builds against the staged artifacts, verify signatures, and check for common mistakes made when cutting a release.

Setting up EC2 Instance for Preparing and Creating Releases Candidates

• Either you can use the AMI ami-4c721b7c (available is US-West) which has all the necessary tools installed. Or you can create prepare your own instance by the steps given in the Miscellaneous section (see at the end of this document).

• Transfer your GPG keys from your home machine to the EC2 instance

  Code Block
  language bash
  # == On home machine == gpg --list-keys # Identify the KEY_ID of the key you generated gpg --output pubkey.gpg --export <KEY_ID> gpg --output ---export-secret-key <KEY_ID> | cat pubkey.gpg - | gpg --armor --output keys.asc --symmetric --cipher-algo AES256 # Copy keys.asc to EC2 instance   # == On EC2 machine == gpg --no-use-agent --output - keys.asc | gpg --import rm keys.asc

• Edit ~/.m2/settings.xml and specify your Apache user name and password. 

  Code Block
  language xml
  <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> <servers> <server> <id>apache.snapshots.https</id> <username>APACHE_USERNAME</username> <password>PASSWORD</password> </server> <server> <id>apache.releases.https</id> <username>APACHE_USERNAME</username> <password>PASSWORD</password> </server> </servers> </settings>

• Download appropriate version of Spark that has the right scripts related to the releases.

Calling a Release Vote

• The release voting happens in two stages. First, a vote takes place on the Apache Spark developers list (the podling PMC or PPMC is voting), then one takes place on the general@i.a.o list (the IPMC). I used the same template for both votes. Look at past vote threads to see how this goes. Once the vote is finished you should also send out a summary e-mail with the totals (subject “[RESULT] [VOTE]...”).
• If possible, attach a draft of the release notes with the e-mail
• Attach the CHANGES.txt file in the e-mail
• NOTE: This will change once we graduate and there will be a single vote
  

Cutting the Official Release

Performing the Final Release in Nexus

• Find the staging repository and click "Release" and confirm. 

Uploading Final Source and Binary Artifacts

...

...

Calling a Release Vote

• The release voting happens in two stages. First, a vote takes place on the Apache Spark developers list (the podling PMC or PPMC is voting), then one takes place on the general@i.a.o list (the IPMC). I used the same template for both votes. Look at past vote threads to see how this goes. Once the vote is finished you should also send out a summary e-mail with the totals (subject “[RESULT] [VOTE]...”).
• If possible, attach a draft of the release notes with the e-mail
• Attach the CHANGES.txt file in the e-mail
• NOTE: This will change once we graduate and there will be a single vote
  

Cutting the Official Release

Performing the Final Release in Nexus

• Find the staging repository and click "Release" and confirm. 

Uploading Final Source and Binary Artifacts

# Create SVN folder and add the release artifacts there:
# https://dist.apache.org/repos/dist/dev/incubator/spark/spark-0.9.0-incubating-rc5
$ scp pwendell@people.apache.org:~/public_html/spark-0.9.0-incubating-rc5/* spark-0.9.0-incubating-rc5/
# Verify md5 sums
$ svn add spark-0.9.0-incubating
$ svn commit -m "Adding spark-0.8.1-incubating-rc1" 
$ svn mv https://dist.apache.org/repos/dist/dev/incubator/spark/spark-0.8.1-incubating-rc4 \  
>    https://dist.apache.org/repos/dist/release/incubator/spark/spark-0.8.1-incubating
# Look at http://www.apache.org/dist/incubator/spark/ to make sure it's there.
# This will be mirrored throughout the Apache network.

Packaging and Wrap-Up for the Release

• Update remaining version numbers in the release branch (see this example commit)
• Update the spark-ec2 scripts
  • Upload the binary packages to the spark-related-packages bucket in S3 and make them public
  • Alter the init scripts in amplab/spark-ec2 repository to pull new binaries (see this example commit)
  • You can audit the ec2 set-up by launching a cluster and running this audit script
• Update the Spark website
  • The website repo is at: https://svn.apache.org/repos/asf/incubator/spark
  • Copy new documentation to /site/docs and update the "latest" link
  • NOTE: For the below items, look at how previous releases are documented on the site
  • Create release notes
  • Update documentation page
  • Update downloads page
  • Update the main page with a news item
• Once everything is working (ec2, website docs, website changes) create an announcement on the website and then send an e-mail to the mailing list
• Enjoy an adult beverage of your choice, congrats on making a Spark release

Miscellaneous

Steps to create the AMI useful for making releases

# CreateInstall SVNnecessary foldertools
$ and add the release artifacts there:
# https://dist.apache.org/repos/dist/dev/incubator/spark/spark-0.9.0-incubating-rc5
$ scp pwendell@people.apache.org:~/public_html/spark-0.9.0-incubating-rc5/* spark-0.9.0-incubating-rc5/
# Verify md5 sums
$ svn add spark-0.9.0-incubating
$ svn commit -m "Adding spark-0.8.1-incubating-rc1" 
$ svn mv https://dist.apache.org/repos/dist/dev/incubator/spark/spark-0.8.1-incubating-rc4 \  
>    https://dist.apache.org/repos/dist/release/incubator/spark/spark-0.8.1-incubating
# Look at http://www.apache.org/dist/incubator/spark/ to make sure it's there.
# This will be mirrored throughout the Apache network.

Packaging and Wrap-Up for the Release

• Update remaining version numbers in the release branch (see this example commit)
• Update the spark-ec2 scripts
  • Upload the binary packages to the spark-related-packages bucket in S3 and make them public
  • Alter the init scripts in amplab/spark-ec2 repository to pull new binaries (see this example commit)
  • You can audit the ec2 set-up by launching a cluster and running this audit script
• Update the Spark website
  • The website repo is at: https://svn.apache.org/repos/asf/incubator/spark
  • Copy new documentation to /site/docs and update the "latest" link
  • NOTE: For the below items, look at how previous releases are documented on the site
  • Create release notes
  • Update documentation page
  • Update downloads page
  • Update the main page with a news item
• Once everything is working (ec2, website docs, website changes) create an announcement on the website and then send an e-mail to the mailing list
• Enjoy an adult beverage of your choice, congrats on making a Spark release

Miscellaneous

Steps to create the AMI useful for making releases

# Install necessary tools
$ wget http://apt.typesafe.com/repo-deb-build-0002.deb
$ sudo dpkg -i repo-deb-build-0002.deb   # Add Typesafe APT Repo for sbt
$ sudo apt-get update —fix-missing  
$ sudo apt-get install -y sbt git openjdk-7-jdk maven rubygems python-epydoc gnupg-agent linkchecker
 
# Install Scala 2.10
$ wget http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
$ tar xvzf scala*.tgz
$ ln -s scala-2.10.3 scala

# Add stuff to ~/.bashrcsudo apt-get update —fix-missing  
$ sudo apt-get install -y git openjdk-7-jdk maven rubygems python-epydoc gnupg-agent linkchecker
 
# Install Scala 2.10
$ cd
$ wget http://www.scala-lang.org/files/archive/scala-2.10.3.tgz
$ tar xvzf scala*.tgz
$ ln -s scala-2.10.3 scala

# Instal SBT 0.13.1
$ cd
$ mkdir sbt
$ cd sbt 
$ wget http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.1/sbt-launch.jar
# Create /home/ubuntu/sbt/sbt with the following code
	SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
	java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"
$ chmod u+x /home/ubuntu/sbt/sbt
 
# Add stuff to ~/.bashrc
$ echo "export SCALA_HOME=/home/ubuntu/scala/" >> ~/.bashrc 
$ echo "export SCALA_HOME=/home/ubuntu/scala/MAVEN_OPTS='-Xmx3g -XX:MaxPermSize=1g -XX:ReservedCodeCacheSize=1g'" >> ~/.bashrc 
$ echo "alias scala='$SCALA_HOME/bin/scala' " >> ~/.bashrc
$ echo "exportalias MAVEN_OPTS='-Xmx3g -XX:MaxPermSize=1g -XX:ReservedCodeCacheSize=1g'sbt='/home/ubuntu/sbt/sbt' " >> ~/.bashrc