Versions Compared

Old Version 22

changes.mady.by.user Brock Noland

Saved on

compared with

New Version 23

changes.mady.by.user Brock Noland

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Install spark (either download pre-built spark, or build assembly from source).  
    • Install/build a compatible version.  Hive root pom.xml's <spark.version> defines what version of spark it was built/tested with. 
    • Install/build a compatible distribution.  Each version of Spark has several distributions, corresponding with different versions of Hadoop.
    • Once spark is installed, find and keep note of the <spark-assembly-*.jar> location.
    • If you download spark pre-build you will need to replace the spark 1.0.x assembly with http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar
  2. Start Spark cluster (Master and workers).
    • Keep note of the <Spark Master URL>.  This can be found in Spark master WebUI.

...

  1. As Hive on Spark is still in development, currently only a Hive assembly built from hive/spark development branch supports Spark execution.  The development branch is located here: https://github.com/apache/hive/tree/spark.  Checkout branch and build hive assembly as described in https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ.
  2. If you download spark, make sure you use a 1.1.x assembly: http://ec2-50-18-79-139.us-west-1.compute.amazonaws.com/data/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar

  3. Start hive with <spark-assembly-*.jar> on the hive auxpath:

    Code Block
    hive --auxpath /location/to/spark-assembly-*.jar

  4. Configure hive execution to Spark:

    Code Block
    hive> set hive.execution.engine=spark;

  5. Configure Spark-application configs for Hive.  See: http://spark.apache.org/docs/latest/configuration.html.  This can be done either by adding a file "spark-defaults.conf" with these properties to the hive classpath, or by setting them on hive configuration:

    Code Block
    hive> set spark.master=<Spark Master URL>

hive> set spark.eventLog.enabled=true;             

hive> set spark.executor.memory=512m;              

hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;

...