Page History

NOTE: This Wiki is obsolete as of November 2016 and is retained for reference only.

Unit Testing

To run an individual Hive compatibility test:

...

To generate golden answer files based on Hive 0.12, you need to setup your development environment as follows according to the "Other dependencies for developers" of this README. To generate golden answer files based on Hive 0.13.1, please follow the following steps.

1. Download Hive's 0.13.1 release and set HIVE_HOME (HIVE_DEV_HOME is not needed. See 

   Jira
   server ASF JIRA key SPARK-4119

    for details).

2. Set HADOOP_HOME.
3. Download all 0.13.1a jars from http://mvnrepository.com/artifact/org.spark-project.hive and replace corresponding jars in $HIVE_HOME/lib.
4. Download kryo 2.21.jar to $HIVE_HOME/lib  (Note: 2.22 jar does not work) and javolution 5.5.1 jar (http://mvnrepository.com/artifact/javolution/javolution/5.5.1) to $HIVE_HOME/lib.
5. You may not need this step. But, if a Hive query fails and you find that Hive tries to talk to HDFS or you find weird runtime NPEs, set the following in your test suite...

...

val testTempDir = Utils.createTempDir()
// We have to use kryo to let Hive correctly serialize some plans.
sql("set hive.plan.serialization.format=kryo")
// Explicitly set fs to local fs.
sql(s"set fs.default.name=file://$testTempDir/")
//sql(s"set mapred.working.dir=${testTempDir}")
// Ask Hive to run jobs in-process as a single map and reduce task.
sql("set mapred.job.tracker=local")

...