Versions Compared

Old Version 13

changes.mady.by.user Szehon Ho

Saved on

compared with

New Version 14

changes.mady.by.user Szehon Ho

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. As Hive on Spark is still in development, currently only a Hive assembly built from hive/spark development branch supports Spark execution.  Branch is located here: https://github.com/apache/hive/tree/spark.  Build hive assembly from this branch as described in https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ.

  2. Start hive with <spark-assembly-*.jar> on the hive auxpath:

    Code Block
    hive --auxpath /location/to/spark-assembly-spark_version-hadoop_version.jar

  3. Configure hive execution engine to run on spark:

    Code Block
    hive> set hive.execution.engine=spark;

  4. Configure Spark-application configs for Hive.  See: http://spark.apache.org/docs/latest/configuration.html.  This can be done either by adding a file "spark-defaults.conf" to the hive classpath, or configured as normal properties from hive:

    Code Block
    hive> set spark.master=<Spark Master URL>

hive> set spark.eventLog.enabled=true;             

hive> set spark.executor.memory=512m;              

hive> set spark.serializer=org.apache.spark.serializer.KryoSerializer;

...