Page History

...

...

1. Install Spark (either download pre-built Spark, or build assembly from source).  
   • Install/build a compatible version.  Hive root pom.xml's <spark.version> defines what version of Spark it was built/tested with. 
   • Install/build a compatible distribution.  Each version of Spark has several distributions, corresponding with different versions of Hadoop.
   • Once Spark is installed, find and keep note of the <spark-assembly-*.jar> location.

   • Note that you must have a version of Spark which does not include the Hive jars. Meaning one which was not built with the Hive profile. If you will use Parquet tables, it's recommended to also enable the "parquet-provided" profile. Otherwise there could be conflicts in Parquet dependency. To remove Hive jars from the installation, simply use the following command under your Spark repository:

     Prior to Spark 2.0.0:

     Code Block
     language bash
     ./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided"

     Since Spark 2.0.0:

     Code Block
     language bash
     ./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided"

     Since Spark 2.3.0:

     Code Block
     language bash
     ./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided,orc-provided"

2. Start Spark cluster
   • Keep note of the <Spark Master URL>.  This can be found in Spark master WebUI.

...