Spark SQL Internals

Unit Testing

To run an individual Hive compatibility test:

sbt/sbt -Phive -Dspark.hive.whitelist="testname.*" "hive/test-only org.apache.spark.sql.hive.execution.HiveCompatibilitySuite"

where testname.* can be a list of comma separated regex patterns that matche the tests that you want to run. You can also use the following command to save some typing:

sbt/sbt -Phive "hive/test-only *.HiveCompatibilitySuite -- -z substring"

Then all tests whose names contain substring will be executed. The -z option comes with ScalaTest, and can be used with any other test suites.

Hive Golden Answer files

For some test suites, Hive golden answer files are generated when test cases are executed for the first time. These files cache results generated by Hive, and Spark SQL testing framework use them to accelerate test execution. For all test suites that sub-classes org.apache.spark.sql.hive.execution.HiveComparisonTest, if a test case is added via HiveComparisonTest.createQueryTest, developers should check and add corresponding golden answer files to the Git repository. In most cases, developers only need to pay attention to the following two test suites:

• org.apache.spark.sql.hive.execution.HiveCompatibilitySuite
  • Newly whitelisted test cases (listed in HiveCompatibilitySuite.whiteList)
• org.apache.spark.sql.hive.execution.HiveQuerySuite

  • Test cases created via createQueryTest

To generate golden answer files, you need to setup your development environment according to this README. After running the new test cases for the first time, golden answer files should be found under the sql/hive/src/test/resources/golden folder.