...
Info |
---|
|
Hive is using Maven as its build tool. Versions prior to 0.13 were using Ant. |
Developing
How do I
...
See MiniDriver Tests for information about MiniDriver and Beeline tests.
How do I move some files?
Post a patch for testing purposes which simply does add and deletes. SVN will not understand these patches are actually moves, therefore you should actually upload the following, in order so the last upload is the patch for testing purposes:
...
For general information, see Unit Tests and Debugging in the Developer Guide.
Where is the log output of a test?
Logs are put in a couple locations:
From the root of the source tree: find . -name hive.log
/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
How do I run a single test?
Warning |
---|
|
Note that any test in the itests directory needs to be executed from with the itests directory. The pom is disconnected from the parent project for technical reasons. |
Single test class:
No Format |
---|
mvn test -Dtest=ClassName
|
Single test method:
No Format |
---|
mvn test -Dtest=ClassName#methodName
|
Note that a pattern can also be supplied to -Dtests to run multiple tests matching the pattern:
Code Block |
---|
|
mvn test -Dtest='org.apache.hive.beeline.*' |
For more usage see the documentation for the Maven Surefire Plugin.
Why isn't the itests pom connected to the root pom?
The qfile tests in itests require the packaging phase. The Maven test phase is after compile and before packaging. We could change the qfile tests to run during the integration-test phase using the "failsafe" plugin but the "failsafe" plugin is different than surefire and IMO is hard to use. If you'd like to give that a try, by all means, go ahead.
A test fails with a NullPointerException in MiniDFSCluster
If any test fails with the error below it means you have an inappropriate umask setting. It should be set to 0022.
No Format |
---|
java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:426)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:284)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
|
...
How do I run precommit tests on a patch?
Hive precommit testing is triggered automatically when a file is uploaded to the JIRA ticket:
- Attach the patch file to a JIRA ticket: in the ticket's "More" tab, select "Attach Files" and use "Choose File" to upload the file, then add a descriptive comment.
- Put the patch in the review queue: click the "Submit Patch" button. The button name will change to "Cancel Patch" and the ticket status will change to Patch Available.
See Hive PreCommit Patch Testing for more detail.
How do I rerun precommit tests over the same patch?
For patch updates, our convention is to number them like HIVE-1856.1.patch, HIVE-1856.2.patch, etc. And then click the "Submit Patch" button again when a new one is uploaded; this makes sure it gets back into the review queue.
How do I run a single test?
Warning |
---|
|
Note that any test in the itests directory needs to be executed from within the itests directory. The pom is disconnected from the parent project for technical reasons. |
Single test class:
No Format |
---|
mvn test -Dtest=ClassName
|
Single test method:
No Format |
---|
mvn test -Dtest=ClassName#methodName
|
Note that a pattern can also be supplied to -Dtests to run multiple tests matching the pattern:
Code Block |
---|
|
mvn test -Dtest='org.apache.hive.beeline.*' |
For more usage see the documentation for the Maven Surefire Plugin.
How do I run all of the unit tests?
Code Block |
---|
|
mvn test
cd itests
mvn test
|
Note that you need to have previously built and installed the jars:
Code Block |
---|
|
mvn test
cd itests
mvn test
|
Note that you need to have previously built and installed the jars:
Code Block |
---|
|
mvn clean clean install -DskipTests
cd itests
mvn clean install -DskipTests
|
...
Code Block |
---|
|
cd itests
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' |
How do I
...
run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
Code Block |
---|
ant |
Code Block |
---|
|
ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
test
|
MVN:
Code Block |
language |
---|
text | cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile=
|
To run a single clientnegative test alter1.q
...
Code Block |
---|
ant -Dtestcase=TestNegativeCliDriver |
As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild
Code Block |
---|
|
ant test -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
|
How do I update the results of many test cases?
Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.
Code Block |
---|
|
head -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join |
You can re-generate all those output files in batches of 20 with the command below
MVN:
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q
|
To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests
Code Block |
---|
ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|
MVN:
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.*
|
To run a single contrib test alter1.q and overwrite the result file
Code Block |
---|
|
ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test
|
MVN:
Code Block |
---|
|
cd itests/qtest
mvn test -Dtest=TestContribCliDriver -Dqfile=alter1.q |
Code Block |
---|
|
egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile={} |
How do I run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
To run a single test groupby1.q and output detailed information during execution
Code Block |
---|
|
code |
ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test
|
MVN:
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestCliDriver
|
To run a single clientnegative test alter1.q
As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. For example, run all the partition_wise_fileformat tests
Code Block |
---|
ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|
How do I update the output of a CliDriver testcase?
Code Block |
---|
|
ant test -Dtestcase=TestCliDriver |
Code Block |
---|
ant -Dtestcase=TestNegativeCliDriver -Dqfile=alter1.q test-Doverwrite=true
|
MVN:
Code Block |
---|
|
cd itests/qtest
mvn test -Dtest=TestNegativeCliDriverTestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true
|
To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests
As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild
Code Block |
---|
|
ant test -Dmodule=ql |
Code Block |
---|
ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|
MVN:
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.*
|
To run a single contrib test alter1.q and overwrite the result file
Code Block |
---|
|
ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test
|
MVN:
Code Block |
---|
|
cd itests/qtest
mvn test -Dtest=TestContribCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true
|
To run a single test groupby1.q and output detailed information during execution
Code Block |
---|
|
ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test
|
As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. For example, run all the partition_wise_fileformat tests
Code Block |
---|
ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|
How do I rerun precommit tests over the same patch?
Upload the exact same patch again to the JIRA.
=alter1.q -Doverwrite=true
|
How do I update the results of many test cases?
Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.
Code Block |
---|
|
head -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join |
You can re-generate all those output files in batches of 20 with the command below
Code Block |
---|
|
egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile={} |
Where is the log output of a test?
Logs are put in a couple locations:
From the root of the source tree: find . -name hive.log
/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
Why isn't the itests pom connected to the root pom?
The qfile tests in itests require the packaging phase. The Maven test phase is after compile and before packaging. We could change the qfile tests to run during the integration-test phase using the "failsafe" plugin but the "failsafe" plugin is different than surefire and IMO is hard to use. If you'd like to give that a try, by all means, go ahead.
Why does a test fail with a NullPointerException in MiniDFSCluster?
If any test fails with the error below it means you have an inappropriate umask setting. It should be set to 0022.
No Format |
---|
java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:426)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:284)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
|
Debugging
How do I debug into a single test in Eclipse?
...
At this point, attach the remote debugger as mentioned before to start debugging your queries.