Overview

Tez is a new application framework built on Hadoop Yarn that can execute complex directed acyclic graphs of general data processing tasks. In many ways it can be thought of as a more flexible and powerful successor of the map-reduce framework.

...

More information about Tez can be found here:

• Tez apache pagewebsite: http://incubatortez.apache.org/projects/tez.html
• Tez design doc: https://issues.apache.org/jira/browse/TEZ-65
• Tez presentation: http://www.youtube.com/watch?v=9ZLLzlsz7h8
• Tez slides: http://www.slideshare.net/Hadoop_Summit/murhty-saha-june26255pmroom212

...

Whenever a query has multiple reduce sinks (that cannot be combined, i.e.: no correlation between the partition keys), hive Hive will break the plan apart and submit one MR job per sink. All of the MR jobs in this chain need to be scheduled one-by-one and each one has to re-read the output of the previous job from HDFS and shuffle it. In Tez several reduce sinks can be linked directly and data can be pipelined without the need of temporary HDFS files. This pattern is referred to as MRR (Map - reduce - reduce*).

...

All sorting in map-reduce happens using the same binary sort, regardless of the data type. Hive might for instance choose to use a more effective integer-only sort when possible. Tez makes that available.

Since hive Hive uses map-reduce to compute aggregations, processing will always boil down to a sort-merge even though we’re not actually interested in the sort order. Tez will allow for more efficient hash-based algorithms to do the same.

...

• Hive continues to work as is on clusters that do not have TEZ.
  • MR revisions 20, 20S, 23 continue to work unchanged.
• Hive can optionally submit MR jobs to TEZ without any additional improvements.
  • Hive can treat TEZ like just another hadoop Hadoop 23 instance.
• Hive can optionally detect chains of MR jobs and optimize them to a single DAG of the form MR* and submit it to TEZ.
• Hive can optionally detect when a join has multiple parent tasks and combine them into a single DAG of a tree shape.
• Hive will display the MRR optimization in explain plans.
• Hive will give appropriate feedback to the user about progress and completion status of the query when running MRR queries.
• The user will be able to get statistics and diagnostic information as before (counters, logs, debug info on the console).
• Hive has unit tests to cover all new functionality.

The following things are out of scope for the first phase:

• Local tasks will still run as MR only.
• Only Map and Reduce Tez tasks with SimpleEdges will be used (i.e.: no new tasks, new input/output/processors, no new edge types).
• No multi-output task optimizations will be introduced.

One new configuration variable will be introduced:

...

Note: It is possible to execute an MR plan against TEZ. In order to do so, one simply has to change the following variable (assuming tez Tez is installed on the cluster):

...

Here’s a TPC-DS query and plans with and without tez Tez optimizations enabled:

The query (rewritten for hiveHive):

select
  i_item_desc
  ,i_category
  ,i_class
  ,i_current_price
  ,i_item_id
  ,itemrevenue
  ,itemrevenue*100/sum(itemrevenue) over
    (partition by i_class) as revenueratio
from
  (select
     i_item_desc
     ,i_category
     ,i_class
     ,i_current_price
     ,i_item_id
     ,sum(ws_ext_sales_price) as itemrevenue
   from
     web_sales
     join item on (web_sales.ws_item_sk = item.i_item_sk)
     join date_dim on (web_sales.ws_sold_date_sk = date_dim.d_date_sk)
   where
     i_category in ('1', '2', '3')
     and year(d_date) = 2001 and month(d_date) = 10
   group by
     i_item_id
     ,i_item_desc
     ,i_category
     ,i_class
     ,i_current_price) tmp
order by
  i_category
  ,i_class
  ,i_item_id
  ,i_item_desc
  ,revenueratio;

...

Changes that impact current hive Hive code paths:

• Split MR compilation from SemanticAnalyzer (simple)
• Break MapRedWork into MapWork and ReduceWork (straight forward but a number of changes)
• Move execution specific classes from exec to exec.mr package (simple)
• Move some functions from ExecDriver to exec.Utilities to share between Tez and MR
• Build system: Add tez Tez dependencies and tez Tez testing

I believe that all of these are valuable by themselves and make the code cleaner and easier to maintain. Especially the second item will touch quite a few places in the code though. None of them change functionality.

...

The following outlines the changes across the various hive Hive components:

Execution layer

We’ve initially investigated to add Tez as a simple shim option to the code base. This didn’t work out mostly because Tez’ API is very different from the MR api. It does not make much sense to move the entire “execute” infrastructure to the shim layer. That would require large code changes with little benefit. Instead there will be separate “Task” implementations for MR and TEZ and hive Hive will decide at runtime which implementation to use.

...

In order to limit the impact of Tez, we will provide a separate implementation: TezCompiler. The tez Tez compiler will attempt to perform most physical optimizations at the plan level, leaving the breakdown of the plan into Tez jobs for a second round of optimizations.

...

In the short term Tez will not support a “LocalTezDagRunner”. That means that hive Hive will always have to submit MR jobs when executing locally. In order to avoid replanning the query after execution has started in Tez mode some optimizations for converting stages to local jobs will not be available.

...

Some MR jobs have a predetermined number of reducers. This happens for order by (numReducers = 1) and scenarios where bucketing is used (numReducers = numBuckets). The user can also set the number of reducers manually. The same numbers will be used for each reduce tasks. Initially there will be no way for the user to set different numbers of reducers for each of the separate reduce stages. There is already a ticket (HIVE-3946) to address this shortcoming which can be used for both Tez and MR.

In most cases hive Hive will determine the number of reducers by looking at the input size of a particular MR job. Hive will then guess the correct number of reducers. The same guess will be used for subsequent reduce phases in a tez Tez plan. Ultimately, this number will have to be determined using statistics which is out of scope, but applies equally to MR and Tez.

...

Hive variables

The “set” mechanism for hive Hive variables will not change. The variables will be passed through to the execution engine as before. However, hive Hive will not shim or map any mapreduce variables. If a variable is not supported in hive Hive it will be silently ignored.

...

There will be a new “ql” dependency on Tez. The jars will be handled the same way hadoop Hadoop jars are handled, i.e.: They will be used during compile, but not included in the final distribution. Rather we will depend on them being installed separately. The jars will only have to be present to run Tez jobs, they are not needed for regular MR execution.

...

Mini Tez Cluster will initially be the only way to run tez Tez during unit tests. LocalRunner is not yet available. If mr.rev is set to tez all MiniMr tests will run against Tez.

Installation and Configuration

For information about how to set up Tez on a Hadoop 2 cluster, see https://github.com/apache/incubator-tez/blob/branch-0.2.0/INSTALL.txt. 

For information about how to configure Hive 0.13.0+ for Tez, see the release notes for HIVE-6098, Merge Tez branch into trunk.  Also see Configuration Properties: Tez for descriptions of all the Tez parameters.

Hive-Tez Compatibility

For a list of Hive and Tez releases that are compatible with each other, see Hive-Tez Compatibility.