Page History

...

1. Install Spark (either download pre-built Spark, or build assembly from source).  
   • Install/build a compatible version.  Hive root pom.xml's <spark.version> defines what version of Spark it was built/tested with. 
   • Install/build a compatible distribution.  Each version of Spark has several distributions, corresponding with different versions of Hadoop.
   • Once Spark is installed, find and keep note of the <spark-assembly-*.jar> location.

   • Note that you must have a version of Spark which does not include the Hive jars. Meaning one which was not built with the Hive profile. To remove Hive jars from, simply use the following command under your Spark repository:

     Code Block
     language bash
     ./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4"

2. Start Spark cluster (both standalone and Spark on YARN are supported).
   • Keep note of the <Spark Master URL>.  This can be found in Spark master WebUI.

...