Page History

...

...

In master, only Hadoop 2.x is supported, thus there is no need to specify a Maven profile for most build operations.

In master, MVN:

mvn clean install -DskipTests
cd itests 
mvn clean install -DskipTests

...

Note that if you use the Hive git base directory as the Eclipse workspace, then it does not pick the right project names (for example, picks 'ant' instead of 'hive-ant'). Therefore it's recommended to have the workspace directory one up from the git directory. For example workspaces/hive-workspace/hive where hive-workspace is the Eclipse workspace and hive is the git base directory.

How to generate tarball?

...

Code Block
language text
mvn clean package -DskipTests -Pdist

...

How to generate protobuf code?

MVN:

cd ql
mvn clean install -DskipTests -Pprotobuf

How to generate Thrift code?

MVN:

mvn clean install -Pthriftif -DskipTests -Dthrift.home=/usr/local

How to run findbugs after a change?

mvn site -Pfindbugs 

How to run findbugs after a change?

mvn site -Pfindbugs 

NoteNote:  Available in Hive 1.1.0 onward (see HIVE-8327).

How to compile ODBC?

MVN:

cd odbc
mvn compile -Podbc -Dthrift.home=/usr/local -Dboost.home=/usr/local

How do I publish Hive artifacts to my local Maven repository?

ant package
ant -Dmvn.publish.repo=local maven-build
ant -Dmvn.publish.repo=local maven-publish

MVN:

mvn clean install -DskipTests
cd itests 
mvn clean install -DskipTests
cd itests 
mvn clean install -DskipTests

Testing

For general information, see Unit Tests and Debugging in the Developer Guide.

...

mvn clean install -DskipTests 
cd itests 
mvn clean install -DskipTests 

How do I run all of the unit tests except for a certain few tests?

Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:

Then, for a clean build, run

export ANT_OPTS="-XX:MaxPermSize=512m"

ant clean package test

Note that running ant test will not work; ant package does some setup work that is required for the testcases to run successfully.

How do I run all of the unit tests except for a certain few tests?

Similar to running all tests, but define test.excludes.additional to specify a test/pattern to exclude from the test run. For example the following will run all tests except for the CliDriver tests:

cd itests 
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' 

How do I run the clientpositive/clientnegative unit tests? 

All of the below require that you have previously run ant package.

To run clientpositive tests

cd itests/qtest
mvn test -Dtest=TestCliDriver 

To run a single clientnegative test alter1.q

cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q 

To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests

cd itests/qtest

cd itests 
mvn test -Dtest.excludes.additional='**/Test*CliDriver.java' 

How do I run the clientpositive/clientnegative unit tests? 

All of the below require that you have previously run ant package.

To run clientpositive tests

ant -Dtestcase=TestCliDriver test

MVN:

cd itests/qtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* 

# Alternatively, you can specify comma separated list with "-Dqfile" argument
mvn test -Dtest=TestCliDriver TestMiniLlapLocalCliDriver -Dqfile='vectorization_0.q,vectorization_17.q,vectorization_8.q'

To run a single clientnegative contrib test alter1.q and overwrite the result file

cd itests/qtest
mvn testant -DtestcaseDtest=TestNegativeCliDriverTestContribCliDriver -Dqfile=alter1.q test

...

-Dtest.output.overwrite=true 

How do I remote debug a qtest?

cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q 

To run all of the clientpositive tests that match a regex, for example the partition_wise_fileformat tests

ant -Dtestcase-Dmaven.surefire.debug="-Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=8000 -Xnoagent -Djava.compiler=NONE" test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test

MVN:

=<test>.q

How do I modify the init script when testing?

The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).

To skip initialization:

cd itests/qtest
mvn test -Dtest=TestCliDriver -Phadoop-2 -Dqfile_regex=partitiontest_wiseto_fileformatrun.*q  -DinitScript=

To run a single contrib test alter1.q and overwrite the result filesupply a custom script:

antmvn -Dtestcase=TestContribCliDrivertest -Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true -Dqfile=alter1test_to_run.q  -Doverwrite=true test

...

DinitScript=custom_script.sql

How do I update the output of a CliDriver testcase?

cd itests/qtest
mvn test -Dtest=TestContribCliDriverTestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true 

How do I update the results of many test cases?

Assume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.To run a single test groupby1.q and output detailed information during execution

ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test

As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. For example, run all the partition_wise_fileformat tests

ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test

How do I modify the init script when testing?

The option to skip the init script or supply a custom init script was added in Hive 2.0 (see HIVE-11538).

To skip initialization:

mvn test -Dtest=TestCliDriver -Phadoop-2 -Dqfile=test_to_run.q  -DinitScript=

head -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join

You can re-generate all those output files in batches of 20 with the command belowTo supply a custom script:

mvnegrep test -Dtest=TestCliDriver -Phadoop-2 -Dtest.output.overwrite=true -Dqfile=test_to_run.q  -DinitScript=custom_script.sql

How do I update the output of a CliDriver testcase?

ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true

MVN:

cd itests/qtest
'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true 

As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild

ant test -Dmodule=ql -Dtestcase=TestCliDriver=true -Dqfile=alter1.q -Doverwrite=true

How do I update the results of many test cases?

{}

To do the same from the output of a precommit result, with multiple drivers, you can doAssume that you have a file like below which you'd like to re-generate output files for. Such a file could be created by copying the output from the precommit tests.

import re
from itertools import groupby
s = """
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[drop_with_concurrency] (batchId=231)head -2 /tmp/failed-TestCliDriver-file-tests
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_allcolref_in_udf[comments] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriverTestMiniLlapLocalCliDriver.testCliDriver[vector_annotate_stats_join

You can re-generate all those output files in batches of 20 with the command below

egrep 'TestCliDriver' /tmp/failed-TestCliDriver-file-tests | perl -pe 's@.*testCliDriver_@@g' | awk '{print $1 ".q"}' | xargs -n 30 | perl -pe 's@ @,@g' | xargs -I{} mvn test -Dtest=TestCliDriverif_expr] (batchId=141)
"""
PAT = re.compile("org.apache.hadoop.hive.cli.([^\.]*).*\[([^\]]*).*")
l = [PAT.match(x.strip()) for x in s.split("\n") if x.strip()]
for driver,q in groupby(sorted([a.groups() for a in l if a]), key=lambda a:a[0]):
	print """mvn clean test -Dtest=%s '-Dqfile=%s' -Dtest.output.overwrite=true -Dqfile={}""" % (driver, ",".join(["%s.q" % a[1] for a in q]))

Where is the log output of a test?

...

• From the root of the source tree: find . -name hive.log

• /tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)

See Hive Logging for details about log files, including alternative configurations.

How do I add a test case?

First, add the test case to the qfile test suite:

• .log

• /tmp/$USER/ (Linux) or $TMPDIR/$USER/ (MacOS)

See Hive Logging for details about log files, including alternative configurations.

How do I add a test case?

First, add the test case to the qfile test suite:

• Copy the test to a new file under ql/src/test/queries/clientpositive/<filename>.q (or /clientnegative if it is a negative test).
  • If the new test creates any table, view, function, etc., make sure that the name is unique across tests. For instance, name a table in the test file foo.q, foo_t1 instead of simply t1. This will help reduce flakiness in the test runs, since Jenkins will run tests and batches, and currently it does not restore to former state after running each of the q files.
  • If there is any interaction with file system, use unique folders for the test to avoid any collision with other tests
  Copy the test to a new file under ql/src/test/queries/clientpositive/<filename>.q (or /clientnegative if it is a negative test)
  • .
• Add the <filename.q> to itests/src/test/resources/testconfiguration.properties to the appropriate variable (ex. minimr.query.files).

...

• Run the test and generate the output file using the appropriate -Dtest (ex. TestMinimrCliDriver TestCliDriver; see itests/qtest/pom.xml for the names of other test suites):

  Code Block
  language text
  cd qtest mvn test -Dtest=TestMinimrCliDriverTestCliDriver -Dqfile=<filename>.q -Dtest.output.overwrite=true

  
  

...

At this point, attach the remote debugger as mentioned before to start debugging your queries.