Versions Compared

Old Version 37

changes.mady.by.user Tathagata Das

Saved on

compared with

New Version 38

changes.mady.by.user Tathagata Das

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • The process of cutting a release requires a number of tools to be locally installed (maven, jekyll, etc). Ubuntu users can install those tools via apt-get. However, it may be most convenient to use a EC2 instance based on the AMI ami-4c721b7c (available is US-West, has Scala 2.10.3 and SBT 0.13.1 installed). This has all the necessary tools installed. Mac users are especially recommended to use a EC2 instance instead of attempting to install all the necessary tools. If you want to prepare your own EC2 instance (different version of Scala, SBT, etc.), follow the steps given in the Miscellaneous section (see at the end of this document).
  • Consider using CPU-optimized instances, which may provide better bang for the buck.

  • Transfer your GPG keys from your home machine to the EC2 instance.

    Code Block
    languagebash
    # == On home machine ==
gpg --list-keys  # Identify the KEY_ID of the key you generated
gpg --output pubkey.gpg --export <KEY_ID>
gpg --output ---export-secret-key <KEY_ID> | cat pubkey.gpg - | gpg --armor --output keys.asc --symmetric --cipher-algo AES256
# Copy keys.asc to EC2 instance
 
# == On EC2 machine ==
gpg --no-use-agent --output - keys.asc | gpg --import
rm keys.asc

  • Download appropriate version of Spark that has the right scripts related to the releases.

...

Code Block
languagebash
# Install necessary tools
$ sudo apt-get update —fix-missing  
$ sudo apt-get install -y git openjdk-7-jdk maven rubygems python-epydoc gnupg-agent linkchecker
 
# Install Scala 2.10of the same version as that used by Spark
$ cd
$ wget http://www.scala-lang.org/files/archive/scala-2.10.3.tgz  
$ tar xvzf scala*.tgz
$ ln -s scala-2.10.3 scala

# Instal Install SBT of a version compatible with the SBT of Spark (at least 0.13.1)
$ cd
$ && mkdir sbt
$ cd sbt 
$ wget http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.1/sbt-launch.jar
# Create /home/ubuntu/sbt/sbt with the following code
	SBT_OPTS="-Xms512M -Xmx1536M -Xss1M -XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=256M"
	java $SBT_OPTS -jar `dirname $0`/sbt-launch.jar "$@"
$ chmod u+x /home/ubuntu/sbt/sbt
 
# Add stuff to ~/.bashrc
$ echo "export SCALA_HOME=/home/ubuntu/scala/" >> ~/.bashrc 
$ echo "export SBT_HOME=/home/ubuntu/sbt/" >> ~/.bashrc 
$ echo "export MAVEN_OPTS='-Xmx3g -XX:MaxPermSize=1g -XX:ReservedCodeCacheSize=1g'" >> ~/.bashrc
$ echo "export PATH='$SCALA_HOME/bin/:$SBT_HOME:$PATH'" >> ~/.bashrc

$ source ~/.bashrc
 
# Verify Scala and SBT
sbt sbt-version  # Should force the download of SBT dependencies and finally print SBT version, verify that SBT itversion is >= 0.13.1 .
scala -version   # ShouldVerify print thethat Scala version number,is verifysame thatas itthe matchesone theused version for which Spark release was built.