Page History

...

Unit tests are executed via JUnit. Currently, many "unit tests" are really end-to-end tests. We are in the process of changing this so that all of end-to-end tests will be run by the e2e harness (see below). See PigTestProposal for details.

Preparation

If you want Prior to run running unit testtests, make sure to set umask 0022.

We to 0022 before running unit test (we also see unit test tests fail due to extended acl, so use setfacl -b to remove extened extended acl if applicable).

Running all unit tests

To run the unit tests do ant test in the top level Pig directory. Currently this takes 8 hours to run. We intend to drive this to under five minutes. Until this is done it is not expected that contributors will run all these tests before submitting their patch.

...

Code Block

cd your_path_to_pig_src
scp -i your_private_rsa_key_file test/e2e/pig/whirr/whirr_test_patch.sh your_namenode:~

if you have a patch you want to run
    scp -i your_private_rsa_key_file your_patch your_namenode:~
    
ssh -i your_private_rsa_key_file your_namenode

...

For features that are new, you cannot test against old versions of Pig. For example, macros in 0.9 cannot be tested against 0.8.1. As an alternate to running the same Pig Latin script against an old version, you can run a different script. This script will be run using the current version, not the old one. To specify a different script, you need a key verify_pig_script. For example:

Code Block

       { 
          # simple macro, no args
          'num' => 1,
          'pig' => q#define simple_macro() returns void {
                         a = load ':INPATH:/singlefile/studenttab10k' as (name, age, gpa);
                         b = foreach a generate age, name;
                         store b into ':OUTPATH:';
                     }

                     simple_macro();#,
          'verify_pig_script' => q#a = load ':INPATH:/singlefile/studenttab10k' as (name, age, gpa);
                                   b = foreach a generate age, name;
                                   store b into ':OUTPATH:';#,
        }

...

Key	What it Does	Example	Required?
delimiter	Provides `floatpostprocess` with delimiter to use	`'delimiter' => ':'`	Only with `flostpostprocess`
execonly	This test will only be executed in specified mode; options are `local` and `mapred`	`'execonly' => 'mapred'`	No
expected_err_regex	Checks stderr error for the provided regular expression	`'expected_err_regex' => "Out of bound access."`	No
expected_out_regex	Checks stdout error for the provided regular expression	`'expected_out_regex' => "A: {name: bytearray,age: bytearray,gpa: bytearray}"`	No
floatpostprocess	Run floating point numbers through a post processor, since due to precision issues different runs of the same script will produce slightly different values. All floating point numbers are rounded to 3 decimal places. This must be used in conjunction with `delimiter`	`'floatpostprocess' => 1`	For outputs that include calculated floating point values.
ignore	Do not run this test, used when a test is failing but we don't want to remove it because it will be needed once the issue is fixed. A reason for ignoring the test should be given.	`'ignore' => 'JIRA-19999'`	No
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="0d775cfeea068c43-87be5c87-46c248c3-b5e88607-08be496391c96926fc1003e5"><ac:plain-text-body><![CDATA[	java_params	Values to be passed on the `pig` command line before other Pig parameters; useful for passing properties.	`'java_params' => ['-Dpig.cachedbag.memusage=0']`	No	]]></ac:plain-text-body></ac:structured-macro>
not_expected_err_regex	Checks that stderr does not match the provided regular expression	`'not_expected_err_regex' => "ERROR"`	No
not_expected_out_regex	Checks that stdout does not match the provided regular expression	`'not_expected_out_regex' => "datafile"`	No
notmq	Tells the test harness this is not a multi-query test; only necessary when a test has multiple `store` operators but should not be verified as if it were multi-query.	`'notmq' => 1`	No
num	Test number; must be unique in the test group	`'num' => 1`	Yes
pig	The Pig Latin script to run in the test	`q#a = load ':INPATH:/dir/studenttab10k' as (name, age, gpa); store a into ':OUTPATH:';#`	Yes
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="e42f90552d71a928-f084935c-4de34567-9ee0b238-93562eeeb10b71dfc00e7cbb"><ac:plain-text-body><![CDATA[	pig_params	Command line arguments to pass to `pig` when running this test.	`'pig_params' => ['-p', qq(fname='studenttab10k')]`	No	]]></ac:plain-text-body></ac:structured-macro>
rc	Expected return code	`'rc' => 0`	No
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="c64f821d7879271c-94df374a-45754cd1-9aafbcd6-fed8c8ad201ae23fceaa85ef"><ac:plain-text-body><![CDATA[	sortArgs	Arguments to pass to the Unix `sort` utility. When these are given, sort will be called before data is sorted for comparison with the expected results.	`'sortArgs' => ['-t', ' ', '+0', '-1']`	Only when job output should be sorted	]]></ac:plain-text-body></ac:structured-macro>
verify_pig_script	Alternate Pig Latin script to use to generate the expected results	`'verify_pig_script' => q\A = load ':INPATH:/singlefile/studenttab10k' as (name, age, gpa); store A into ':OUTPATH:';\,`	No

...

Child pages

Versions Compared

Old Version 10

New Version 11

Key

Preparation

Running all unit tests