...
Code Block |
---|
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script -Dharness.hadoop.home=hadoop_home_dir test-e2e-deploy
|
Where old_pig
is where you installed the old version of Pig, hadoop_conf_dir
is the directory where your hadoop-site.xml
or mapred-site.xml
file is, and hadoop_script
is where your hadoop
executable is located. For example, if you have installed Pig 0.8.1 in /usr/local/pig/pig-0.8.1
and Hadoop in /usr/local/hadoop
, then your command line would look like:
Code Block |
---|
ant -Dharness.old.pig=/usr/local/pig/pig-0.8.1 -Dharness.cluster.conf=/usr/local/hadoop/conf -Dharness.cluster.bin=/usr/local/hadoop/bin/hadoop -Dharness.hadoop.home=hadoop_home_dir test-e2e-deploy |
This takes a couple of minutes and only needs to be run once. After building Pig itself it will display information on the data it is generating.
...
Code Block |
---|
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script -Dharness.hadoop.home=hadoop_home_dir test-e2e
|
Run with test-e2e-tez
instead of test-e2e
to run tests with Tez as execution engine.
Running the full test suite is rarely what you want, as it takes around 10 hours. If you are running against a cluster with more capacity, you can speedup the execution of the tests by parallelizing it. The fork.factor.conf.file
property tells how many test conf files to run in parallel. The fork.factor.group
property tells how many groups to run in parallel within each test file. Within a group, each tests are run sequentially. For eg: -Dfork.factor.conf.file=2 -Dfork.factor.group=5
will run 2 test files and 5 groups in each totaling 10 tests in parallel.
Running the full test suite is rarely what you want, as it takes around 10 hours. To run only some tests, set the tests.to.run
property. This value can be passed a group of tests (e.g. Checkin), or a single test (e.g. Checkin_1). You can pass multiple tests or groups in this property. Each test or group of tests must be proceeded by a {{-t }}. For example, to run the Checkin tests and the first MergeJoin test, do:
Code Block |
---|
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script -Dharness.hadoop.home=hadoop_home_dir -Dtests.to.run="-t Checkin -t MergeJoin_1" test-e2e
|
...
Code Block |
---|
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script -Dharness.hadoop.home=hadoop_home_dir test-e2e-undeploy
|
There is no need to do this on a regular basis.
...
Code Block |
---|
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script -Dharness.hadoop.home=hadoop_home_dir test-e2e-deploy-local
|
(Yes you still have to give cluster information even though you aren't using a cluster. Pig doesn't use it in this case and you can pass bogus info if you want.)
...
Code Block |
---|
ant -Dharness.old.pig=old_pig -Dharness.cluster.conf=hadoop_conf_dir -Dharness.cluster.bin=hadoop_script -Dharness.hadoop.home=hadoop_home_dir test-e2e-local
|
Running on EC2
...