Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • SimpleHistoryLoggingService 
    • This writes the history into a log-file. The location is configurable. It can be logged as a file which will become part of the YARN Application's aggregated logs ( i.e. within the Application Master's container logs ) or to a defined location on a Distributed FileSystem such as HDFS.
    • This is more of a prototype ( useful for quick testing/analysis ) and not fully supported.
  • ATSHistoryLoggingService and ATSV15HistoryLoggingService
    • These impls make use of YARN Timeline to store the history data
    • The ATSV15 impl ( or to be more clear v1.5 ) makes use of enhancements done as part of YARN-4233. The enhancements are mainly around the use of a DistributedFileSystem for most of the data being written and stored instead of using LevelDB based storage.
    • There is no current support for YARN Timeline v2 (being developed currently and potentially available in a future version of hadoop-3.x)

The Tez UI currently only works with the YARN Timeline based stores. It will not be able to display any data for Tez DAGs that were configured to use the SimpleHistoryLoggingService. 

...

From a relationship perspective, you can think of TaskAttempt being a child of Task, Task a child of Vertex and Vertex a child of DAG. This relationship allows for pulling down all the necessary info for a given DAG.

Tez History Data in YARN Timeline v1.

 

Securing Access to Tez History Data

...