...
Table of Contents |
---|
Info | ||
---|---|---|
| ||
Run the test and generate the output file using the appropriate |
...
In master, only Hadoop 2.x is supported, thus there is no need to specify a Maven profile for most build operations.
In master, MVN:
Code Block |
---|
mvn clean install -DskipTests cd itests mvn clean install -DskipTests |
...
Build and generate Eclipse files (the conservative method):.
For master - without using local maven repository;master – without using a local Maven repository:
Code Block | ||
---|---|---|
| ||
$ mkdir workspace $ cd workspace $ git clone https://github.com/apache/hive.git $ cd hive $ mvn clean package eclipse:clean eclipse:eclipse -Pitests -DskipTests -DdownloadSources -DdownloadJavadocs |
NOTE: The presence of the package target is crucial in sucessfully successfully generating the project files.
NOTE: I would also recommend It is recommended to wipe out or rename the org/apache/hive subtree of the local m2 repo repository prior to generating the project files; and you may want to check after its execution that it does not contain any files - if – if it does contain something; , then your project files may not be fully independent of downloaded maven Maven artifacts.
This will work on all branches; but keep in mind that in this case you are installing the hive Hive artifacts into your local repository.
...
Note that if you use the Hive git base directory as the Eclipse workspace, then it does not pick the right project names (for example, picks 'ant' instead of 'hive-ant'). Therefore it's recommended to have the workspace directory one up from the git directory. For example workspaces/hive-workspace/hive where hive-workspace is the Eclipse workspace and hive is the git base directory.
How to generate tarball?
...
Code Block | ||
---|---|---|
| ||
mvn clean package -DskipTests -Pdist |
...
How to generate protobuf code?
MVN:
Code Block |
---|
cd ql mvn clean install -DskipTests -Pprotobuf |
How to generate Thrift code?
MVN:
Code Block |
---|
mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local |
Warning | ||||||||
---|---|---|---|---|---|---|---|---|
Don’t forget to update |
How to run findbugs after How to run findbugs after a change?
Code Block |
---|
mvn site -Pfindbugs |
Note: Available in Hive 1.1.0 onward (see HIVE-8327).
How to compile ODBC?
...
Code Block |
---|
cd odbc mvn compile -Podbc -Dthrift.home=/usr/local -Dboost.home=/usr/local |
How do I publish Hive artifacts to my local Maven repository?
Code Block | ||
---|---|---|
| ||
ant package
ant -Dmvn.publish.repo=local maven-build
ant -Dmvn.publish.repo=local maven-publish
|
MVN:
Code Block |
---|
mvn clean install -mvn clean install -DskipTests cd itests mvn clean install -DskipTests |
...
Code Block | ||
---|---|---|
| ||
mvn clean install -DskipTests cd itests mvn clean install -DskipTests | ||
Info | ||
|
How do I run all of the unit tests except for a certain few tests?
Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:
Code Block | ||
---|---|---|
| ||
export ANT_OPTS="-XX:MaxPermSize=512m"
|
Code Block | ||
---|---|---|
| ||
ant clean package test |
Note that running ant test
will not work; ant package
does some setup work that is required for the testcases to run successfully.
How do I run all of the unit tests except for a certain few tests?
Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:
cd itests
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' |
How do I run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestCliDriver |
To run a single clientnegative test alter1.q
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q |
To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests
Code Block | ||
---|---|---|
cd itests/qtest | ||
Code Block | ||
| ||
cd itests mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' |
How do I run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
Code Block |
---|
ant -Dtestcase=TestCliDriver test |
MVN:
Code Block |
---|
cd itests/qtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* # Alternatively, you can specify comma separated list with "-Dqfile" argument mvn test -Dtest=TestCliDriver TestMiniLlapLocalCliDriver -Dqfile='vectorization_0.q,vectorization_17.q,vectorization_8.q' |
To run a single clientnegative contrib test alter1.q and overwrite the result file
Code Block | ||
---|---|---|
| ||
cd itests/qtest mvn testant -DtestcaseDtest=TestNegativeCliDriverTestContribCliDriver -Dqfile=alter1.q test |
...
-Dtest.output.overwrite=true |
How do I remote debug a qtest?
Code Block | ||
---|---|---|
| ||
cd itests/qtest mvn -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" test -Dtest=TestNegativeCliDriverTestCliDriver -Dqfile=alter1<test>.q |
To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests
Code Block |
---|
ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test |
MVN:
How do I modify the init script when testing?
The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).
To skip initialization:
Code Block | ||
---|---|---|
| ||
mvn test -Dtest=TestCliDriver -Phadoop-2 -Dqfile=test_to_run.q -DinitScript= |
To supply a custom script:
Code Block | ||
---|---|---|
| ||
Code Block | ||
cd itests/qtest mvn test -Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true -Dqfile_regex=partitiontest_wiseto_fileformatrun.* |
To run a single contrib test alter1.q and overwrite the result file
Code Block | ||
---|---|---|
| ||
ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test |
...
q -DinitScript=custom_script.sql |
How do I update the output of a CliDriver testcase?
Code Block | ||
---|---|---|
| ||
cd itests/qtest mvn test -Dtest=TestContribCliDriverTestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true |
How do I update the results of many test cases?
Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.To run a single test groupby1.q and output detailed information during execution
Code Block | ||
---|---|---|
| ||
ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test |
As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. For example, run all the partition_wise_fileformat tests
Code Block |
---|
ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|
How do I modify the init script when testing?
The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).
To skip initialization:
Code Block | ||
---|---|---|
| ||
mvn test -Dtest=TestCliDriver -Phadoop-2 -Dqfile=test_to_run.q -DinitScript= |
To supply a custom script:
Code Block | ||
---|---|---|
| ||
mvn test -Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true -Dqfile=test_to_run.q -DinitScript=custom_script.sql |
How do I update the output of a CliDriver testcase?
Code Block | ||
---|---|---|
| ||
ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
|
MVN:
head -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join |
You can re-generate all those output files in batches of 20 with the command below
Code Block | ||
---|---|---|
| ||
egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} | ||
Code Block | ||
| ||
cd itests/qtest mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.Dtest.output.overwrite=true -Dqfile={} |
To do the same from the output of a precommit result, with multiple drivers, you can doAs of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild
Code Block | ||
---|---|---|
| ||
antimport test -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true |
How do I update the results of many test cases?
Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.
Code Block | ||
---|---|---|
| ||
head -2 /tmp/failed-TestCliDriver-file-tests re from itertools import groupby s = """ org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency] (batchId=231) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35) org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=141) """ PAT = re.compile("org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join |
You can re-generate all those output files in batches of 20 with the command below
Code Block | ||
---|---|---|
| ||
egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile={} |
Where is the log output of a test?
Logs are put in a couple locations:
From the root of the source tree: find . -name hive.log
/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
See Hive Logging for details about log files, including alternative configurations.
How do I add a test case?
First, add the test case to the qfile test suite:
([^\.]*).*\[([^\]]*).*")
l = [PAT.match(x.strip()) for x in s.split("\n") if x.strip()]
for driver,q in groupby(sorted([a.groups() for a in l if a]), key=lambda a:a[0]):
print """mvn clean test -Dtest=%s '-Dqfile=%s' -Dtest.output.overwrite=true""" % (driver, ",".join(["%s.q" % a[1] for a in q]))
|
Where is the log output of a test?
Logs are put in a couple locations:
From the root of the source tree: find . -name hive.log
/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
See Hive Logging for details about log files, including alternative configurations.
How do I add a test case?
First, add the test case to the qfile test suite:
- Copy the test to a new file under
ql/src/test/queries/clientpositive/<filename>.q
(or/clientnegative
if it is a negative test).- If the new test creates any table, view, function, etc., make sure that the name is unique across tests. For instance, name a table in the test file
foo.q
,foo_t1
instead of simplyt1
. This will help reduce flakiness in the test runs, since Jenkins will run tests and batches, and currently it does not restore to former state after running each of the q files. - If there is any interaction with file system, use unique folders for the test to avoid any collision with other tests
ql/src/test/queries/clientpositive/<filename>.q
(or/clientnegative
if it is a negative test)- .
- If the new test creates any table, view, function, etc., make sure that the name is unique across tests. For instance, name a table in the test file
- Add the
<filename.q>
toitests/src/test/resources/testconfiguration.properties
to the appropriate variable (ex.minimr.query.files
).
...
Run the test and generate the output file using the appropriate
-Dtest
(ex.TestMinimrCliDriver
TestCliDriver
; seeitests/qtest/pom.xml
for the names of other test suites):Code Block language text cd qtest mvn test -Dtest=TestMinimrCliDriverTestCliDriver -Dqfile=<filename>.q -Dtest.output.overwrite=true
...
It would be great to have it connected, but it would make it harder to use mvn test locally. The best option would be to utilize the failsafe plugin for integration testing; but it needs a bit different setup, and its it's harder to use for now.... If you'd like to give that a try, by all means, go ahead.
There is an option to attach all the itest subprojects to the main project by enabling this with -Pitests (HIVE-13490).
There are some good and bad sides using this, it's introduced as a profile to clearly communicate that the integration tests are attached to the main project.
The good side is that you may use freely use -Pitests to run integration tests from the root project without the need of mvn install.
Code Block | ||
---|---|---|
| ||
mvn test -q -Pitests -Dtest=TestCliDriver -Dqtest=sample2.q |
...
The bad thing side is that a simple mvn test -Pitests will start executing all integration tests.
Why does a test fail with a NullPointerException in MiniDFSCluster?
...
At this point, attach the remote debugger as mentioned before to start debugging your queries.