...
Info |
---|
|
Run the test and generate the output file using the appropriate -Dtest Hive is using Maven as its build tool. Versions prior to 0.13 were using Ant. |
...
In master, only Hadoop 2.x is supported, thus there is no need to specify a Maven profile for most build operations.
In master, MVN:
Code Block |
---|
mvn clean install -DskipTests
cd itests
mvn clean install -DskipTests |
...
Note that if you use the Hive git base directory as the Eclipse workspace, then it does not pick the right project names (for example, picks 'ant' instead of 'hive-ant'). Therefore it's recommended to have the workspace directory one up from the git directory. For example workspaces/hive-workspace/hive where hive-workspace is the Eclipse workspace and hive is the git base directory.
How to generate tarball?
MVN:
Code Block |
---|
|
mvn clean package -DskipTests -Pdist
|
...
How to generate protobuf code?
MVN:
Code Block |
---|
cd ql
mvn clean install -DskipTests -Pprotobuf
|
How to generate Thrift code?
MVN:
Code Block |
---|
mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local |
How to run findbugs after a change?
Code Block |
---|
mvn site -Pfindbugs |
Don’t forget to update hive_metastore.proto when changing hive_metastore.thrift Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HIVE-26769 |
---|
|
|
How to run findbugs after a change?
Code Block |
---|
mvn site -Pfindbugs |
Note: Note: Available in Hive 1.1.0 onward (see HIVE-8327).
How to compile ODBC?
MVN:
Code Block |
---|
cd odbc
mvn compile -Podbc -Dthrift.home=/usr/local -Dboost.home=/usr/local
|
How do I publish Hive artifacts to my local Maven repository?
Code Block |
---|
|
ant package
ant -Dmvn.publish.repo=local maven-build
ant -Dmvn.publish.repo=local maven-publish
|
MVN:
Code Block |
---|
mvn clean install -DskipTests
cd itests
mvn clean install -DskipTests
cd itests
mvn clean install -DskipTests
|
Testing
For general information, see Unit Tests and Debugging in the Developer Guide.
...
Code Block |
---|
|
mvn clean install -DskipTests
cd itests
mvn clean install -DskipTests
|
Info |
---|
title | Legacy information for the Ant build |
---|
|
Make sure that your JAVA_HOME is appropriately set (some tests need this), and set ANT_OPTS to increase the size allocated to the Permanent Generation as per the following
How do I run all of the unit tests except for a certain few tests?
Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:
Code Block |
---|
|
export ANT_OPTS="-XX:MaxPermSize=512m"
|
Then, for a clean build, run
Code Block |
---|
|
ant clean package test |
Note that running ant test
will not work; ant package
does some setup work that is required for the testcases to run successfully.
How do I run all of the unit tests except for a certain few tests?
Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:
cd itests
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' |
How do I run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestCliDriver |
To run a single clientnegative test alter1.q
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q |
To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests
Code Block |
---|
cd itests/qtest |
Code Block |
---|
|
cd itests
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' |
How do I run the clientpositive/clientnegative unit tests?
All of the below require that you have previously run ant package
.
To run clientpositive tests
Code Block |
---|
ant -Dtestcase=TestCliDriver test |
MVN:
Code Block |
---|
cd itests/qtest
mvn test -Dtest=TestCliDriver |
To run a single clientnegative test alter1.q
Code Block |
---|
ant -Dtestcase=TestNegativeCliDriver -Dqfile=alter1.q test |
MVN:
Code Block |
---|
cd itests/qtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.*
# Alternatively, you can specify comma separated list with "-Dqfile" argument
mvn test -Dtest=TestNegativeCliDriverTestMiniLlapLocalCliDriver -Dqfile=alter1'vectorization_0.q,vectorization_17.q,vectorization_8.q ' |
To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests
Code Block |
---|
ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test |
MVN:
a single contrib test alter1.q and overwrite the result file
Code Block |
---|
|
code |
cd itests/qtest
mvn test -Dtest=TestCliDriverTestContribCliDriver -Dqfile_regex=partition_wise_fileformat.*
# Alternatively, you can specify comma separated list with "-Dqfile" argument
mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile='vectorization_0.q,vectorization_17.q,vectorization_8.q' |
To run a single contrib test alter1.q and overwrite the result file
Code Block |
---|
|
ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test |
=alter1.q -Dtest.output.overwrite=true |
How do I remote debug a qtest?
Code Block |
---|
|
cd itests/qtest
mvn -Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" test -Dtest=TestCliDriver -Dqfile=<test>.q |
How do I modify the init script when testing?
The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).
To skip initializationMVN:
Code Block |
---|
|
cd itests/qtest
mvn test -Dtest=TestContribCliDriverTestCliDriver -Phadoop-2 -Dqfile=alter1test_to_run.q -Dtest.output.overwrite=true DinitScript= |
To supply a custom script:To run a single test groupby1.q and output detailed information during execution
Code Block |
---|
|
antmvn test -DtestcaseDtest=TestCliDriver -Dqfile=groupby1.q-Phadoop-2 -Dtest.silentoutput.overwrite=falsetrue test |
As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. For example, run all the partition_wise_fileformat tests
-Dqfile=test_to_run.q -DinitScript=custom_script.sql |
How do I update the output of a CliDriver testcase?
Code Block |
---|
|
cd itests/qtest
mvn test -Dtest |
Code Block |
---|
ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test=alter1.q -Dtest.output.overwrite=true
|
How do I
...
update the results of many test cases?
Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.
...
The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).
To skip initialization:
Code Block |
---|
|
mvnhead test -Dtest=TestCliDriver -Phadoop-2 -Dqfile=test_to_run.q -DinitScript=-2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join |
You can re-generate all those output files in batches of 20 with the command belowTo supply a custom script:
Code Block |
---|
|
mvnegrep test -Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true -Dqfile=test_to_run.q -DinitScript=custom_script.sql |
How do I update the output of a CliDriver testcase?
Code Block |
---|
|
ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
|
MVN:
Code Block |
---|
|
cd itests/qtest
'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true
-Dqfile={} |
To do the same from the output of a precommit result, with multiple drivers, you can doAs of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild
Code Block |
---|
|
ant test -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
|
How do I update the results of many test cases?
Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.
Code Block |
---|
|
head -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join |
You can re-generate all those output files in batches of 20 with the command below
Code Block |
---|
|
egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dtest.output.overwrite=true -Dqfile={} |
To do the same from the output of a precommit result, with multiple drivers, you can do
|
import re
from itertools import groupby
s = """
org.apache.hadoop.hive.cli |
Code Block |
---|
|
import re
from itertools import groupby
s = """
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency] (batchId=231)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_if_expr] (batchId=141)
"""
PAT = re.compile("org.apache.hadoop.hive.cli.([^\.]*).*\[([^\]]*).*")
l = [PAT.match(x.strip()) for x in s.split("\n") if x.strip()]
for driver,q in groupby(sorted([a.groups() for a in l if a]), key=lambda a:a[0]):
print """mvn clean test -Dtest=%s '-Dqfile=%s' -Dtest.output.overwrite=true""" % (driver, ",".join(["%s.q" % a[1] for a in q]))
|
...
From the root of the source tree: find . -name hive.log
/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
See Hive Logging for details about log files, including alternative configurations.
How do I add a test case?
First, add the test case to the qfile test suite:
hive.log
/tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)
See Hive Logging for details about log files, including alternative configurations.
How do I add a test case?
First, add the test case to the qfile test suite:
- Copy the test to a new file under
ql/src/test/queries/clientpositive/<filename>.q
(or /clientnegative
if it is a negative test).- If the new test creates any table, view, function, etc., make sure that the name is unique across tests. For instance, name a table in the test file
foo.q
, foo_t1
instead of simply t1
. This will help reduce flakiness in the test runs, since Jenkins will run tests and batches, and currently it does not restore to former state after running each of the q files. - If there is any interaction with file system, use unique folders for the test to avoid any collision with other tests
Copy the test to a new file under ql/src/test/queries/clientpositive/<filename>.q
(or /clientnegative
if it is a negative test) - Add the
<filename.q>
to itests/src/test/resources/testconfiguration.properties
to the appropriate variable (ex. minimr.query.files
).
...
...
At this point, attach the remote debugger as mentioned before to start debugging your queries.