Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Described challenges more concretely. Mentioned HiveRunner metastore optimisation.

...

  • Defining boundaries between components: How can and how should a problem be decomposed into smaller, testable units. The ability to do this is limited by the set of language features provided by HQL.
  • Harness provision: Providing a local execution environment that seamlessly supports Hive’s features in a local IDE setting (UDFs etc.)). Ideally the harness should have no environmental dependencies such as a local Hive or Hadoop installation. Developers should be able to simply check out a project and run the tests.
  • Speed of execution: The goal is to have large numbers of isolated, small tests. Test isolation requires frequent setup and teardown and the costs incurred are multiplied the number of tests. The Hive CLI is fairly heavy process to repeatedly start and stop and so some Hive test frameworks attempt to optimise this aspect of test execution.

Modularisation

By modularising processes implemented using Hive they become easier to test effectively and more resilient to change. Although Hive provides a number of vectors for modularisation it is not always clear how a large process can be decomposed. Features for encapsulation of query logic into components is separated into two perpendicular concerns: column level logic, and set level logic. Column level logic refers to the expressions applied to individual columns or groups of columns in the query, commonly described as ‘functions’. Set level logic concerns HQL constructs that manipulate groupings of data such as: column projection with SELECT, GROUP BY aggregates, JOINs, ORDER BY sorting, etc. In either case we expect individual components to live in their own source file or deployable artifact and imported as needed by the composition. For HQL based components, the SOURCE command provides this functionality.

...

  • HiveRunner: Test cases are declared using Java, HQL and JUnit and can execute locally in your IDE. This library focuses on ease of use and execution speed. No local Hive/Hadoop installation required. Full Provides full test isolation, fine grained assertions, and seamless UDF integration (they need only be on the project classpath). The metastore is backed by an in-memory database to increase test performance.
  • beetest: Test cases are declared using HQL and 'expected' data files. Test suites are executed using a script on the command line. Apparently requires HDFS to be installed in the environment in which the tests are executed.
  • hive_test: Test cases are declared using Java, HQL and JUnit and can execute locally in your IDE.
  • How to utilise the Hive project's internal test framework.

...