Page History

NOTE: This Wiki is obsolete as of November 2016 and is retained for reference only.

Unit Testing

To run an individual Hive compatibility test:

...

1. Download Hive's 0.13.1 release and set HIVE_HOME (HIVE_DEV_HOME is not needed. See 

   Jira
   server ASF JIRA key SPARK-4119

    for details).

2. Set HADOOP_HOME.
3. Download all 0.13.1a jars from http://mvnrepository.com/artifact/org.spark-project.hive and replace corresponding jars in $HIVE_HOME/lib.
4. Download kryo 2.21.jar (Note: 2.22 jar does not work) and javolution 5.5.1 jar (http://mvnrepository.com/artifact/javolution/javolution/5.5.1) to $HIVE_HOME/lib.
5. You may not need this step. But, if a Hive query fails and you find that Hive tries to talk to HDFS or you find weird runtime NPEs, set the following in your test suite...

...

val testTempDir = Utils.createTempDir()
// We have to use kryo to let Hive correctly serialize some plans.
sql("set hive.plan.serialization.format=kryo")
// Explicitly set fs to local fs.
sql(s"set fs.default.name=file://$testTempDir/")
//sql(s"set mapred.working.dir=${testTempDir}")
// Ask Hive to run jobs in-process as a single map and reduce task.
sql("set mapred.job.tracker=local")

...