Hive on Spark: Getting Started

Spark Installation

Follow instructions to install latest spark: https://spark.apache.org/docs/latest/spark-standalone.html.  In particular:

1. Install spark (either download pre-built spark, or build assembly from source).  Note that Spark has distributions for different versions of Hadoop.  Keep note of the spark-assembly-*.jar location.
2. Start Spark cluster (Master and workers).  Keep note of the Spark master URL.  This can be found in Spark master WebUI.

Configuration Hive

1. As of now, only hive/spark branch works against spark: https://github.com/apache/hive/tree/spark.  Build hive assembly from this branch as described in https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ.

2. Start hive and add the spark-assembly.jar to the hive auxpath.

   hive --auxpath /location/to/spark-assembly-spark_version-hadoop_version.jar

3. Configure hive execution engine to run on spark:

   hive> set hive.execution.engine=spark;

4. Configure required properties for spark-conf.  See: http://spark.apache.org/docs/latest/configuration.html.  This can be done either by adding a file "spark-defaults.conf" to the hive classpath, or configured as normal properties from hive.

   hive> set spark.master=<spark master URL>
   
   hive> set spark.eventLog.enabled=true;             
   
   hive> set spark.executor.memory=512m;              
   
   hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;

Known Issues