• Created by Unknown User (joshrosen), last updated by Unknown User (yhuai) on Apr 23, 2015 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

Unit Testing

To run an individual Hive compatibility test:

sbt/sbt -Phive -Dspark.hive.whitelist="testname.*" "hive/test-only org.apache.spark.sql.hive.execution.HiveCompatibilitySuite"

where testname.* can be a list of comma separated regex patterns that matche the tests that you want to run. You can also use the following command to save some typing:

sbt/sbt -Phive "hive/test-only *.HiveCompatibilitySuite -- -z substring"

Then all tests whose names contain substring will be executed. The -z option comes with ScalaTest, and can be used with any other test suites.

Hive Golden Answer files

For some test suites, Hive golden answer files are generated when test cases are executed for the first time. These files cache results generated by Hive, and Spark SQL testing framework use them to accelerate test execution. For all test suites that sub-classes org.apache.spark.sql.hive.execution.HiveComparisonTest, if a test case is added via HiveComparisonTest.createQueryTest, developers should check and add corresponding golden answer files to the Git repository. In most cases, developers only need to pay attention to the following two test suites:

  • org.apache.spark.sql.hive.execution.HiveCompatibilitySuite
    • Newly whitelisted test cases (listed in HiveCompatibilitySuite.whiteList)
  • org.apache.spark.sql.hive.execution.HiveQuerySuite

    • Test cases created via createQueryTest

To generate golden answer files, you need to setup your development environment as follows.

  1. Download Hive's 0.13.1 release and set HIVE_HOME (HIVE_DEV_HOME is not needed. See  SPARK-4119 - Getting issue details... STATUS  for details).

  2. Download all 0.13.1a jars from http://mvnrepository.com/artifact/org.spark-project.hive and replace corresponding jars in $HIVE_HOME/lib.
  3. Download kryo 2.21.jar to $HIVE_HOME/lib (Note: 2.22 jar does not work).
  4. You may not need this step. But, if a Hive query fails and you find that Hive tries to talk to HDFS or you find weird runtime NPEs, set the following in your test suite...

  

val testTempDir = Utils.createTempDir()
// We have to use kryo to let Hive correctly serialize some plans.
sql("set hive.plan.serialization.format=kryo")
// Explicitly set fs to local fs.
sql(s"set fs.default.name=file://$testTempDir/")
//sql(s"set mapred.working.dir=${testTempDir}")
// Ask Hive to run jobs in-process as a single map and reduce task.
sql("set mapred.job.tracker=local")

 

After running the new test cases for the first time, golden answer files should be found under the sql/hive/src/test/resources/golden folder.

  • No labels