Versions Compared

Old Version 3

changes.mady.by.user Suma Shivaprasad

Saved on

compared with

New Version Current

changes.mady.by.user Suma Shivaprasad

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


 

Table of Contents

This document tracks on-going efforts to upgrade from Hadoop 2.x to Hadoop 3.x  - Refer Refer Umbrella Jira HADOOP-15501 for current status on this.

Upgrade Tests for HDFS/YARN

The following scenarios were tested while upgrading from Hadoop 2.8.4 to Hadoop 3.1.0


TypeComponentScenarioIssues FoundResolutionOverall Status
EXPRESS/ROLLING UPGRADEHDFSStarting 3.1.0 NameNode/DataNode with custom MetricsPlugin configured in hadoop2-metrics.properties

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyHADOOP-15502

Workaround is applicable only for EXPRESS UPGRADE -

Replace MetricsPlugin implementation jars( eg: HadoopTimelineMetricsSink) with recompiled jars which use package "org.apache.commons.configuration2"

(tick)
EXPRESS UPGRADEYARNStarting Hadoop 3.1.0 YARN daemons

(tick)
ROLLING UPGRADEHDFS3.1.0 NN is started with rollingUpgrade with default policy configured for Erasure coding

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyHDFS-13596

Workaround Not known(error)
ROLLING UPGRADEYARNStart 3.1.0 NM in batches after starting RM.

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyYARN-8346

Fixed(tick)

EXPRESS/ROLLING

UPGRADE

YARNRM started with recovery enabled

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyYARN-8068

Fixed(tick)


Workloads


Application TypeUpgrade TypeIssues FoundStatusOverall Status
MREXPRESS/ROLLING UPGRADE

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyYARN-8346

Fixed(tick)
HIVE on TEZ
Hive with older versions of Tez (0.7, 0.8.x) with Hadoop 2 client ran into UT failures

Tez 0.10.0 will support Hadoop 3

  • TEZ-3923 Move master to Hadoop 3+ and create separate 0.9.x line

  • TEZ-3252 - [Umbrella] Enable support for Hadoop-3.x



(tick)

Spark 2.2/2.3

Spark 2.2/2.3 has a fork of older version of Hive (1.2) which does not work with Hadoop 3


Ongoing efforts in community to build/validate Spark with Hadoop 3 Libraries

    • SPARK-23534  Umbrella jira to Build/test with Hadoop 3

    • SPARK-23710  Upgrade to Hive 2.x ( which builds with Hadoop 3)







IN-PROGRESS
PIG


Support for Hadoop 3 In-Progress in the community - targeted for PIG 0.18.0

PIG-5253    Pig Hadoop 3 support


IN-PROGRESS
OOZIE
Dependent on PIG support for Hadoop 3

Support for Hadoop 3 In-Progress in the community - Targeted for OOZIE-5.1.0

OOZIE-2973  Make sure Oozie works with Hadoop 3


IN-PROGRESS

MR with Native Task Optimization

Validation Pending
MR with Shared Cache Manager

Validation Pending