Page History

...

• Update the Spark Apache repository

  • Checkout the tagged commit for the release candidate and apply the correct version tag

    Code Block
    language bash
    # Apply the correct tag $ git checkout v0.9.1-rc3 # checkout the RC that passed $ git tag v0.9.1 $ git push apache v0.9.1   # Verify on the Apache git repo that the tag has been applied correctly   # Remove the old tag $ git push apache :v0.9.1-rc3

  • Update remaining version numbers in the release branch
    • If you are doing a patch release, see the similar commit made after the previous release in that branch. For example, for branch 1.0, see this example commit.
    • In general, the rule are as follows.  Grep through the repository to find such occurrences.
      • References to just-released version - Upgrade them to next release version. If it is not a documentation related version (e.g. inside spark/docs/ or inside spark/python/epydoc.conf), then make sure you add -SNAPSHOT.
      • References to next version - Make sure that they have -SNAPSHOT at the end.
• Update the spark-ec2 scripts
  • Upload the binary packages to the S3 bucket s3n://spark-related-packages (ask pwendell to do this)
  • Alter the init scripts in mesos/spark-ec2 repository to pull new binaries (see this example commit)
    • Spark 1.0+ : update v3 branch 
    • Spark 0.9: Update v2 branch 
  • You can audit the ec2 set-up by launching a cluster and running this audit script 
    • Make sure you create cluster with default instance type (m1.xlarge)
• Update the Spark website

  • The website repo is at: https://svn.apache.org/repos/asf/spark

    Code Block
    language bash
    $ svn co https://svn.apache.org/repos/asf/spark

    
    

  • Copy new documentation to spark/site/docs and update the "latest" link. Make sure that the docs were generated with PRODUCTION=1 tag and java 7, if it wasnt already generated with it.

    Code Block
    language bash
    $ PRODUCTION=1 jekyll build

    
    

  • Update the rest of the Spark website. See how previous release are documented on the site.
    • Take a look at the changes to *.md files in this commit (all the html file changes are generated by jekyll).
      
    • Create release notes 
      • The following code creates a list of contributors and identifies large patches. Extra care must be taken to make sure commits from previous releases are not counted since git cannot easily associate commits that were back ported into different branches:

      • Code Block

        # Determine PR numbers closed only in the new release. git log v1.1.0-rc4 |grep "Closes #" | cut -d " " -f 5,6 | grep Closes | sort > closed_1.1 git log v1.0.0 |grep "Closes #" | cut -d " " -f 5,6 | grep Closes | sort > closed_1.0 diff --new-line-format="" --unchanged-line-format="" closed_1.1 closed_1.0 > diff.txt # Grep expression with all new patches expr=$(cat diff.txt | awk '{ print "\\("$1" "$2" \\)"; }' | tr "\n" "|" | sed -e "s/|/\\\|/g" | sed "s/\\\|$//") # Contributor list: git shortlog v1.1.0-rc4 --grep "$expr"   # Large patch list (300+ lines): git log v1.1.0-rc4 --grep "$expr" --shortstat --oneline | grep -B 1 -e "[3-9][0-9][0-9] insert" -e "[1-9][1-9][1-9][1-9] insert" | grep SPARK

        
        

    • Update downloads page
    • Update the main page with a news item
• Remove old releases from mirror network. Spark always keeps two releases in the mirror network: the most recent release on the current and previous maintenance branch. To delete older versions simply use "svn rm". The downloads.js file in the website js/ directory must also be updated to reflect the changes.

  • Code Block
    svn rm https://dist.apache.org/repos/dist/release/spark/spark-0.9.2 svn commit -m "Removing Spark 0.9.2 release"

• Once everything is working (ec2, website docs, website changes) create an announcement on the website and then send an e-mail to the mailing list
• Enjoy an adult beverage of your choice, congrats on making a Spark release

...