Page History

...

1. Install spark (either download pre-built spark, or build assembly from source).  Note that Spark has distributions for  To find out what version of Spark that Hive was built/tested on, check Hive's root pom.xml.  Note each Spark version in turn has several distributions, corresponding with different versions of Hadoop.  Keep  Once spark is installed, find and keep note of the spark-assembly-*.jar location.
2. Start Spark cluster (Master and workers).  Keep note of the Spark master URL.  This can be found in Spark master WebUI.

...

Configuring Hive

1. As of nowHive on Spark is still in development, only a Hive assembly built from hive/spark branch works against spark: https://github.com/apache/hive/tree/spark.  Build hive assembly from this branch as described in https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ.

2. Start hive and add the spark-assembly.jar to the hive auxpath.

   Code Block
   hive --auxpath /location/to/spark-assembly-spark_version-hadoop_version.jar

3. Configure hive execution engine to run on spark:

   Code Block
   hive> set hive.execution.engine=spark;

4. Configure required properties for spark-conf.  See: http://spark.apache.org/docs/latest/configuration.html.  This can be done either by adding a file "spark-defaults.conf" to the hive classpath, or configured as normal properties from hive.

   Code Block
   hive> set spark.master=<spark master URL> hive> set spark.eventLog.enabled=true; hive> set spark.executor.memory=512m; hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;

...

Common Issues

...