Hive on Spark: Getting Started

Spark Installation

Follow instructions to install latest spark: https://spark.apache.org/docs/latest/spark-standalone.html.  In particular:

1. Install spark (either download pre-built spark, or build assembly from source).  
   • Install/build a compatible version.  To find out what version of Spark that your particular Hive build was built/tested on, check your Hive's root pom.xml for <spark.version>.  
   • Install/build a compatible distribution.  Each version of Spark has several distributions, corresponding with different versions of Hadoop.
   • Once spark is installed, find and keep note of the <spark-assembly-*.jar> location.
2. Start Spark cluster (Master and workers).
   • Keep note of the <Spark Master URL>.  This can be found in Spark master WebUI.

Configuring Hive

1. As Hive on Spark is still in development, only a Hive assembly built from hive/spark development branch works against spark: https://github.com/apache/hive/tree/spark.  Build hive assembly from this branch as described in https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ.

2. Start hive and add the <spark-assembly-*.jar> to the hive auxpath.

   hive --auxpath /location/to/spark-assembly-spark_version-hadoop_version.jar

3. Configure hive execution engine to run on spark:

   hive> set hive.execution.engine=spark;

4. Configure required properties for spark-conf.  See: http://spark.apache.org/docs/latest/configuration.html.  This can be done either by adding a file "spark-defaults.conf" to the hive classpath, or configured as normal properties from hive.

   hive> set spark.master=<Spark Master URL>
   
   hive> set spark.eventLog.enabled=true;             
   
   hive> set spark.executor.memory=512m;              
   
   hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;

Common Issues