• Created by Unknown User (lewismc), last updated on Mar 17, 2026 7 minute read


Running Nutch on Tez

This document merges the Nutch wiki tutorial with the Nutch Tez Migration Plan. It covers why and how to run Nutch on Apache Tez, MapReduce inefficiencies vs Tez benefits, workflow diagrams, configuration, deployment, experiments, and the roadmap to native Tez DAGs.

Introduction

This tutorial covers running Nutch jobs on Apache Tez instead of MapReduce. Tez is an application framework on YARN that executes directed-acyclic graphs (DAGs) of tasks. Using Tez can improve runtime speed while preserving the ability to scale to large data. This document also describes the migration plan to evolve Nutch so that crawl cycles can run as DAGs on YARN.

Background context

This tutorial is a work in progress and should be considered DRAFT. The work to evaluate Tez as an execution engine for Nutch was initiated in December 2020. A blocking issue was discovered (TEZ-4371) and work stopped until a patch was developed (apache/tez#464). Work resumed around March 2026 with the TEZ-4371 patch.

Initial experiments (December 2020):

  • Hadoop: 3.1.4
  • Tez: 0.10.0-SNAPSHOT (commit 849e1d7)
  • Nutch: 1.18-SNAPSHOT (commit 88a17f2)

Updated experiments:

  • Hadoop: 3.4.3
  • Tez: 1.0.0-SNAPSHOT (with TEZ-4371 patch)
  • Nutch: 1.23-SNAPSHOT

Audience

This document is for Nutch administrators who want to improve runtime speed while keeping MapReduce’s scalability. Readers are encouraged to share their experience running Nutch on Tez.

What is Apache Tez?

Apache Tez is an application framework on Apache Hadoop YARN that runs a complex directed-acyclic graph (DAG) of tasks for data processing.

Design themes:

  • Empowering users: expressive dataflow APIs, flexible Input-Processor-Output model, data-type agnostic, simpler deployment.
  • Execution performance: gains over MapReduce, better resource management, runtime plan reconfiguration, dynamic data flow decisions.

Projects like Apache Hive and Apache Pig run multi-job MR workflows as a single Tez job. Nutch can use Tez in two ways:

  1. MR compatibility (Phase 1): Set mapreduce.framework.name=yarn-tez. Each existing Nutch MapReduce job is submitted as a 2-vertex Tez DAG (map → reduce). No Nutch code change.
  2. Native Tez DAG (Phase 3, planned): One Tez DAG per crawl round (generate → fetch → parse → updatedb → invertlinks → dedup → index) as a single YARN application.

MapReduce inefficiencies and Tez benefits

How the crawl script works today

The bin/crawl script runs one MapReduce job per step. For each round it:

  1. Runs generate (MR job) → writes a new segment to HDFS.
  2. Discovers the segment path (e.g. ls / hadoop fs -ls).
  3. Runs fetch (MR job) → reads segment, writes fetched content to HDFS.
  4. Runs parse (MR job) → reads segment, writes parsed data to HDFS.
  5. Runs updatedb (MR job) → reads/writes CrawlDb.
  6. Optionally updatehostdb (MR job).
  7. Runs invertlinks (MR job) → reads segment, writes LinkDb.
  8. Runs dedup (MR job) → reads/writes CrawlDb.
  9. Optionally index (MR job) → reads segments/CrawlDb/LinkDb, writes to indexer.

MapReduce inefficiencies

  • Shuffle and sort in every job: Each MR job does Map → Shuffle/Sort → Reduce. Shuffle moves and sorts all map output even when a step does not need global aggregation.
  • Full disk materialization between steps: Every job reads input from HDFS and writes output to HDFS. The next job cannot start until the previous one has finished and materialized its output. No pipelining.
  • Script-driven orchestration: The script waits for each job to complete, discovers output paths (e.g. segment name), then submits the next job. Multiple YARN application submissions and no single view of the round.

Tez benefits

  • Single YARN application per round (native DAG): One submission per round; Tez schedules the whole DAG.
  • In-memory / pipelined data: Data can flow between vertices without writing to HDFS at every step.
  • Shuffle only where needed: Shuffle occurs only at vertices that perform reduce-style aggregation.
  • No script round-trips: With a native DAG, there is no “submit → wait → discover path → submit” between generate, fetch, parse, etc.

With Phase 1 (MR compatibility) you keep the same script and commands; each Nutch job simply runs as a 2-vertex Tez DAG on YARN, so you still have one job per step but gain Tez’s runtime and container reuse. Phase 3 (native DAG) would implement one DAG per round and remove the inefficiencies above.

Workflow diagrams

The following diagrams match the exact order of steps in src/bin/crawl.

MapReduce workflow (current)

Script order (simplified):

Script order (simplified)

MapReduce jobs and inefficiencies:

MapReduce jobs and inefficiencies

Tez DAG workflow (target)

One Tez DAG per crawl round; data can flow between vertices with less disk I/O and shuffle only where needed.

Tez DAG workflow (target)

Migration plan summary

Current state

  • Nutch uses org.apache.hadoop.mapreduce.Job throughout (~20+ job-invoking classes).
  • bin/crawl runs generate → fetch → parse → updatedb → (optional updatehostdb, invertlinks, dedup, index) as separate hadoop jar invocations per step.
  • Build produces a single job JAR; dependencies include hadoop-mapreduce-client-core and hadoop-mapreduce-client-jobclient (Hadoop 3.4.x).

Phase 1: Tez as execution engine (drop-in)

  • Goal: Run existing Nutch jobs on Tez as 2-vertex DAGs per job; no Nutch code change.
  • Actions: Set mapreduce.framework.name=yarn-tez (and cluster Tez config, e.g. tez.lib.uris). Optionally add Tez client dependencies (e.g. tez-api, tez-mapreduce) to the build so the job JAR has Tez on the classpath when submitting to a Tez cluster.
  • Outcome: Same bin/nutch and bin/crawl; each command becomes a Tez application on YARN.

Phase 2: Execution abstraction (optional)

  • Goal: Abstract “run this job” so either MapReduce or Tez (or later a native DAG) can be used.
  • Actions: Introduce a JobRunner/ExecutionEngine interface and MR/Tez implementations; refactor tools to use it.
  • Outcome: Single place to plug in Tez or native DAG.

Phase 3: Native Tez DAG per round

  • Goal: One Tez DAG per crawl round (generate → fetch → parse → updatedb → invertlinks → dedup → index).
  • Actions: Implement a DAG builder (e.g. CrawlCycleTezDAG), new main class (e.g. CrawlCycleTez), and optionally --engine tez in bin/crawl.
  • Outcome: One YARN application per round; data flow and shuffle as in the Tez diagram above.

Backward compatibility

  • Default remains MapReduce. Tez is enabled by configuration (Phase 1) or opt-in command/option (Phase 3). Existing crawl workflow is unchanged unless Tez is configured or used.

Configuring and deploying Hadoop

This section assumes a pseudo-distributed (or larger) cluster. Configure YARN and MapReduce to use Tez.

yarn-site.xml (excerpt; add or adjust for your cluster):

<configuration>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
  </property>
  <!-- ... other YARN properties, e.g. resource, timeline service ... -->
</configuration>

mapred-site.xml (required for Tez):

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn-tez</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>

hadoop-env.sh (add Tez to classpath for client):

export JAVA_HOME=/path/to/JDK
export HADOOP_HOME=/path/to/hadoop
export TEZ_JARS=/path/to/tez/tez-dist/target/tez-0.10.1-SNAPSHOT
export TEZ_CONF_DIR=/path/to/tez/conf
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$TEZ_CONF_DIR:$TEZ_JARS/*:$TEZ_JARS/lib/*

Start services:

$HADOOP_HOME/sbin/start-dfs.sh
$HADOOP_HOME/sbin/start-yarn.sh
$HADOOP_HOME/bin/yarn --daemon start timelineserver

Configuring and deploying Tez

  1. Install Tez and run the Tez examples to verify the cluster.
  2. Copy the Tez distribution to HDFS (adjust paths and version as needed):
hadoop fs -copyFromLocal tez/tez-dist/target/tez-0.10.1-SNAPSHOT.tar.gz /apps/tez-0.10.1-SNAPSHOT/
  1. tez-site.xml (example; adjust tez.lib.uris and classpath for your Nutch/Tez versions):
<configuration>
  <property>
    <name>tez.lib.uris</name>
    <value>${fs.defaultFS}/apps/tez-0.10.1-SNAPSHOT/tez-0.10.1-SNAPSHOT.tar.gz#tez,${fs.defaultFS}/apps/nutch/apache-nutch-1.23-SNAPSHOT-bin.tar.gz#nutch</value>
  </property>
  <property>
    <name>tez.lib.uris.classpath</name>
    <value>./tez/tez-0.10.1-SNAPSHOT/*:./tez/tez-0.10.1-SNAPSHOT/lib/*:./nutch/apache-nutch-1.23-SNAPSHOT/*:./nutch/apache-nutch-1.23-SNAPSHOT/conf/*:./nutch/apache-nutch-1.23-SNAPSHOT/lib/*:./nutch/apache-nutch-1.23-SNAPSHOT/plugins/*/*</value>
  </property>
  <property>
    <name>tez.use.cluster.hadoop-libs</name>
    <value>true</value>
  </property>
  <property>
    <name>plugin.folders</name>
    <value>nutch/apache-nutch-1.23-SNAPSHOT/plugins</value>
  </property>
  <property>
    <name>tez.history.logging.service.class</name>
    <value>org.apache.tez.dag.history.logging.ats.ATSHistoryLoggingService</value>
  </property>
  <property>
    <name>tez.tez-ui.history-url.base</name>
    <value>http://localhost:8080/tez-ui-0.10.1-SNAPSHOT</value>
  </property>
  <property>
    <name>tez.runtime.convert.user-payload.to.history-text</name>
    <value>true</value>
  </property>
</configuration>
  1. Deploy Tez UI (optional) to Tomcat or another servlet container:
cp $TEZ_HOME/tez-ui/target/tez-ui-0.10.1-SNAPSHOT.war $TOMCAT_HOME/webapps

Configuring and deploying Nutch

Build Nutch and copy the distribution to HDFS (adjust version/paths):

cd $NUTCH_HOME && ant clean tar-bin
hadoop fs -copyFromLocal dist/apache-nutch-1.23-SNAPSHOT-bin.tar.gz /apps/nutch-1.23-SNAPSHOT

Run Nutch as usual, e.g. nutch inject crawldb urls. Jobs will be submitted to YARN and run on Tez. Use the Tez UI (e.g. http://localhost:8080/tez-ui-0.10.1-SNAPSHOT/) to inspect Tez DAGs.

Experiments

Community experiments comparing MapReduce and Tez (from the wiki).

Injector job

Run #YARN engine# of URLsElapsed time
1–3MapReduce11,523~00:00:34
4–6Tez11,52300:00:42, 00:00:13, 00:00:14
7–11MapReduce15,763,469~00:02:37–00:03:21
12–14Tez15,763,46900:02:10–00:02:14

After warm-up, Tez shows better runtimes. Short tasks benefit from tez.am.container.reuse.enabled=true (container reuse reduces JVM startup cost).

Generator job

Early runs (Dec 2020) showed the Generator job having issues on Tez (e.g. no records selected, counters not populated). With the TEZ-4371 patch and Tez 1.0.0-SNAPSHOT / Nutch 1.23-SNAPSHOT, experiments were resumed; see NUTCH-2838 and the wiki for latest results.

Observed issues

  1. Counters: With Tez, MapReduce-style counters may not be populated as with classic MR. This affects crawl inspection and debugging. The org.apache.tez.common.counters package may provide equivalents; integration is under evaluation.
  2. Generator on Tez: Historically the Generator job had compatibility issues on Tez (e.g. 0 records selected, no counters). The TEZ-4371 patch and updated Tez/Nutch versions address the blocking bug; further validation is ongoing.


Tez UI Screenshots