THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Hive Developer FAQ
| Table of Contents |
|---|
| Info | ||
|---|---|---|
| ||
Hive is using Maven as its build tool. Versions prior to 0.13 were using Ant. |
Developing
How do I add a new MiniMR test?
How do I move some files?
Post a patch for testing purposes which simply does add and deletes. SVN will not understand these patches are actually moves, therefore you should actually upload the following, in order so the last upload is the patch for testing purposes:
...
The script should be a set of svn mv commands along with any perl commands required for find/replace. E.g.:
| No Format |
|---|
$ svn mv MyCLass.java MyClass.java
$ perl -i -pe 's<at:var at:name="MyCLass" />MyClass@g' MyClass.java
|
...
How to build all source?
MVN:
| Code Block |
|---|
mvn clean install -DskipTests -Phadoop-1
cd itests
mvn clean install -DskipTests -Phadoop-1
|
...
Build and generate eclipse files (the conservative method):
| Code Block |
|---|
$ mkdir workspace
$ cd workspace
$ git clone https://github.com/apache/hive.git
$ cd hive
$ mvn clean install -DskipTests -Phadoop-1
$ mvn eclipse:clean
$ mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs -Phadoop-1
$ cd itests
$ mvn clean install -DskipTests -Phadoop-1
$ mvn eclipse:clean
$ mvn eclipse:eclipse -DdownloadSources -DdownloadJavadocs -Phadoop-1
|
In eclipse define M2_REPO in Preferences -> Java -> Build Path -> Classpath Variables to either:
Mac Example:
| Code Block |
|---|
/Users/$USER/.m2/repository
|
Linux Example:
| Code Block |
|---|
/home/$USER/.m2/repository
|
Windows Example:
| Code Block |
|---|
C:/users/$USER/.m2/repository
|
...
How to generate tarball?
MVN:
| Code Block |
|---|
mvn clean package -DskipTests -Phadoop-1 -Pdist
|
...
How to generate protobuf code?
MVN:
| Code Block |
|---|
cd ql
mvn clean install -DskipTests -Phadoop-1,protobuf
|
How to generate thrift code?
MVN:
| Code Block |
|---|
mvn clean install -Phadoop-1,thriftif -DskipTests -Dthrift.home=/usr/local
|
How to compile ODBC?
MVN:
| Code Block |
|---|
cd odbc
mvn compile -Phadoop-1,odbc -Dthrift.home=/usr/local -Dboost.home=/usr/local
|
How do I publish hive artifacts to my local maven repository?
| Code Block |
|---|
ant package
ant -Dmvn.publish.repo=local maven-build
ant -Dmvn.publish.repo=local maven-publish
|
MVN:
| Code Block |
|---|
mvn clean install -DskipTests -Phadoop-1
cd itests
mvn clean install -DskipTests -Phadoop-1
|
...
Single test class:
| No Format |
|---|
mvn test -Dtest=ClassName -Phadoop-1
|
Single test method:
| No Format |
|---|
mvn test -Dtest=ClassName#methodName -Phadoop-1
|
...
If any test fails with the error below it means you have an inappropriate umask setting. It should be set to 0022.
| No Format |
|---|
java.lang.NullPointerException: null
at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:426)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:284)
at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:124)
|
...
Make sure that your JAVA_HOME is appropriately set (some tests need this), and set ANT_OPTS to increase the size allocated to the Permanent Generation as per the following:
| Code Block |
|---|
export ANT_OPTS="-XX:MaxPermSize=512m"
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=256M"
|
Then, for a clean build, run
| Code Block |
|---|
ant clean package test
|
MVN:
| Code Block |
|---|
mvn test -Phadoop-1
cd itests
mvn test -Phadoop-1
|
...
MVN: Note that you need to have previously built and installed the jars:
| Code Block |
|---|
mvn clean install -DskipTests -Phadoop-1
cd itests
mvn clean install -DskipTests -Phadoop-1
|
How do update the output of a CliDriver testcase?
| Code Block |
|---|
ant test -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
|
MVN:
| Code Block |
|---|
cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true -Phadoop-1
|
As of Hive 0.11.0+ you can cut this time in half by specifying that only the ql module needs to rebuild
| Code Block |
|---|
ant test -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile=alter1.q -Doverwrite=true
|
...
To run clientpositive tests
| Code Block |
|---|
ant -Dtestcase=TestCliDriver test
|
MVN:
| Code Block |
|---|
cd itests/qtest
mvn test -Dtest=TestCliDriver -Phadoop-1
|
To run a single clientnegative test alter1.q
| Code Block |
|---|
ant -Dtestcase=TestNegativeCliDriver -Dqfile=alter1.q test
|
MVN:
| Code Block |
|---|
cd itests/qtest
mvn test -Dtest=TestNegativeCliDriver -Dqfile=alter1.q -Phadoop-1
|
To run all of the clientpositive tests that match a regex, e.g. the partition_wise_fileformat tests
| Code Block |
|---|
ant -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|
MVN:
| Code Block |
|---|
cd itests/qtest
mvn test -Dtest=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* -Phadoop-1
|
To run a single contrib test alter1.q and overwrite the result file
| Code Block |
|---|
ant -Dtestcase=TestContribCliDriver -Dqfile=alter1.q -Doverwrite=true test
|
MVN:
| Code Block |
|---|
cd itests/qtest
mvn test -Dtest=TestContribCliDriver -Dqfile=alter1.q -Dtest.output.overwrite=true -Phadoop-1
|
To run a single test groupby1.q and output detailed information during execution
| Code Block |
|---|
ant -Dtestcase=TestCliDriver -Dqfile=groupby1.q -Dtest.silent=false test
|
As of Hive 0.11.0+ you can cut down the total build time by specifying that only the ql module needs to rebuild. e.g. run all the partition_wise_fileformat tests
| Code Block |
|---|
ant -Dmodule=ql -Dtestcase=TestCliDriver -Dqfile_regex=partition_wise_fileformat.* test
|