Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Running the e2e tests requires three things: a cluster to run them on, an old version of Pig to use to generate expected results, and perl and a cpan module IPC::Run (plus a few CPAN modules) on your client machine. The cluster can be quite small. Four machines will enough, one for a Name Node/Job Tracker and three for Data Node/Task TrackersA single machine will enough. Since performance is not the goal it is fine if these are virtual machinesthis is a virtual machine. If you do not have access to a cluster, see below for information on how to run the tests on EC2.For installing the

You will need the following CPAN modules:

  • IPC::Run
  • Parallel::ForkManager
  • DBI

For help installing CPAN modules perl module, see cpan module install instructions .

...

Key

What it Does

Example

Required?

delimiter

Provides floatpostprocess with delimiter to use

'delimiter' => ':'

Only with flostpostprocess

execonly

This test will only be executed in specified mode; options are local and mapred

'execonly' => 'mapred'

No

expected_err_regex

Checks stderr error for the provided regular expression

'expected_err_regex' => "Out of bound access."

No

expected_out_regex

Checks stdout error for the provided regular expression

'expected_out_regex' => "A: {name: bytearray,age: bytearray,gpa: bytearray}"

No

floatpostprocess

Run floating point numbers through a post processor, since due to precision issues different runs of the same script will produce slightly different values. All floating point numbers are rounded to 3 decimal places. This must be used in conjunction with delimiter

'floatpostprocess' => 1

For outputs that include calculated floating point values.

ignore

Do not run this test, used when a test is failing but we don't want to remove it because it will be needed once the issue is fixed. A reason for ignoring the test should be given.

'ignore' => 'JIRA-19999'

No

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="66b66127d81bc661-77089fb9-42724adc-8fc992ca-e9321e092634c2f95050e799"><ac:plain-text-body><![CDATA[

java_params

Values to be passed on the pig command line before other Pig parameters; useful for passing properties.

'java_params' => ['-Dpig.cachedbag.memusage=0']

No

]]></ac:plain-text-body></ac:structured-macro>

not_expected_err_regex

Checks that stderr does not match the provided regular expression

'not_expected_err_regex' => "ERROR"

No

not_expected_out_regex

Checks that stdout does not match the provided regular expression

'not_expected_out_regex' => "datafile"

No

notmq

Tells the test harness this is not a multi-query test; only necessary when a test has multiple store operators but should not be verified as if it were multi-query.

'notmq' => 1

No

num

Test number; must be unique in the test group

'num' => 1

Yes

pig

The Pig Latin script to run in the test

q#a = load ':INPATH:/dir/studenttab10k' as (name, age, gpa); store a into ':OUTPATH:';#

Yes

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="5e078217b6d144bd-e89654ba-43ad4db0-a49987a8-f30aab661c8d7fbc35be78e6"><ac:plain-text-body><![CDATA[

pig_params

Command line arguments to pass to pig when running this test.

'pig_params' => ['-p', qq(fname='studenttab10k')]

No

]]></ac:plain-text-body></ac:structured-macro>

rc

Expected return code

'rc' => 0

No

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="b3b8c641722a1848-c870cdf1-49094b55-9a9ba480-730564a85d374391f5b250e7"><ac:plain-text-body><![CDATA[

sortArgs

Arguments to pass to the Unix sort utility. When these are given, sort will be called before data is sorted for comparison with the expected results.

'sortArgs' => ['-t', ' ', '+0', '-1']

Only when job output should be sorted

]]></ac:plain-text-body></ac:structured-macro>

verify_pig_script

Alternate Pig Latin script to use to generate the expected results

'verify_pig_script' => q\A = load ':INPATH:/singlefile/studenttab10k' as (name, age, gpa); store A into ':OUTPATH:';\,

No

...