Page History

...

Prerequisites for Managing A Release

Apache Account

• Must have an Apache account
• Make sure you can access your Apache web space. Try ssh-ing into <USER_NAME>@people.apache.org.
• Set up password-less access by uploading your public key.

Create a GPG Key (https://www.apache.org/dev/release-signing)

...

• Check JIRA for remaining issues tied to the release
  
  • Review and merge any blocking features
  • Bump other remaining features to subsequent releases
• Ensure Spark versions are correct in the codebase
  • See this example commit
  • The places to change are:
    • SBT build: Change version in file 'project/SparkBuild.scala'
    • Maven build: Change version in ALL the pom.xml files in repo. Note that the version should be SPARK-VERSION_SNAPSHOT and it will be changed to SPARK-VERSION automatically by Maven when cutting the release.
      • Exception: Change 'yarn/alpha/pom.xml' to SPARK-VERSION. Note that this is different from the main 'pom.xml' because the YARN alpha module does not get published as an artifact through Maven when cutting the release and so does not get version bumped from SPARK-VERSION_SNAPSHOT to SPARK-VERSION.
    • Spark REPLs
      • Scala REPL: Check inside 'repl/src/main/scala/org/apache/spark/repl/'
      • Python REPL: Check inside 'python/pyspark'
    • Docs: Change in file 'docs/_config.yml'
    • Spark EC2 scripts: Change mapping between Spark and Shark versions and the default Spark version in cluster

Check

...

Out and Run Tests

$ git clone https://git-wip-us.apache.org/repos/asf/spark.git -b branch-0.9
$ cd incubator-spark
$ sbt/sbt assembly
$ export MAVEN_OPTS="-Xmx3g -XX:MaxPermSize=1g -XX:ReservedCodeCacheSize=1g"
$ mvn test

Check for

...

Dead Links in the

...

Docs

$ cd $SPARK_HOME/docs
$ jekyll serve --watch
$ sudo apt-get install linkchecker
$ linkchecker -r 2 http://localhost:4000 --no-status --no-warnings

...

Run License Audit

...

Tool (not necessary for Spark 1.0)

Check whether all the source files have Apache headers.  Note that Spark REPL files and some pyspark files will have other license headers (as they are not Apache licensed) and should be ignored.

$ java -jar /path/to/apache-rat-0.10.jar --dir . --exclude *.md > rat_results.txt
$ vi rat_results.txt
$ # Look for source files that seem to have missing headers
$ cat rat_results.txt  | grep "???" | grep -e \.scala$ -e \.java$ -e \.py$ -e \.sh$
$ # Add missing headers if necessary

Create new CHANGES.txt File

...

...

The new CHANGES.txt can be generated using this script.

• Checkout the Spark release version in a Spark git repository. 
• Download the script to a location within the repo.
• Updated the previous release tag, and other information in the script.

• Set SPARK_HOME environment variable and run the script.

  Code Block
  language bash
  $ export SPARK_HOME="..." $ python -u generate-changelist.py

Cutting a Release Candidate

...