Page History

...

The Pig tutorial file (pigtutorial.tar.gz) or the tutorial/pigtutorial.tar.gz file in the pig distribution) includes the Pig JAR file (pig.jar) and the tutorial files (tutorial.jar, Pigs scripts, log files). These files work with Hadoop 0.18 and provide everything you need to run the Pig scripts. To get started, follow these basic steps:

...

The user-defined functions (UDFs) are described here.

...

• Call the NonURLDetector UDF to remove records if the query field is empty or a URL.

  Code Block
  clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);

• Call the ! ToLower UDF to change the query field to lowercase.

  Code Block
  clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;

• Because the log file only contains queries for a single day, we are only interested in the hour. The excite query log timestamp format is YYMMDDHHMMSS. Call the ! ExtractHour UDF to extract the hour (HH) from the time field.

  Code Block
  houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query;

• Call the N!GramGenerator NGramGenerator UDF to compose the n-grams of the query.

  Code Block
  ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;

...

• For each group, identify the hour in which this n-gram is used with a particularly high frequency. Call the ! ScoreGenerator UDF to calculate a "popularity" score for the n-gram.

  Code Block
  uniq_frequency2 = FOREACH uniq_frequency1 GENERATE flatten($0), flatten(org.apache.pig.tutorial.ScoreGenerator($1));

...

• Call the NonURLDetector UDF to remove records if the query field is empty or a URL.

  Code Block
  clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query);

• Call the ! ToLower UDF to change the query field to lowercase.

  Code Block
  clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query;

• Because the log file only contains queries for a single day, we are only interested in the hour. The excite query log timestamp format is YYMMDDHHMMSS. Call the ! ExtractHour UDF to extract the hour from the time field.

  Code Block
  houred = FOREACH clean2 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query;

• Call the N!GramGenerator NGramGenerator UDF to compose the n-grams of the query.

  Code Block
  ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram;

...