Hadoop 2.x to 3.x Upgrade Efforts

This document tracks on-going efforts to upgrade from Hadoop 2.x to Hadoop 3.x - Refer Umbrella Jira HADOOP-15501 for current status on this.

Upgrade Tests for HDFS/YARN

The following scenarios were tested while upgrading from Hadoop 2.8.4 to Hadoop 3.1.0

Type	Component	Scenario	Issues Found	Resolution
EXPRESS/ROLLING UPGRADE	HDFS	Starting 3.1.0 NameNode/DataNode with custom MetricsPlugin configured in hadoop2-metrics.properties	Unable to render Jira issues macro, execution error.	Workaround is applicable only for EXPRESS UPGRADE - Replace MetricsPlugin implementation jars( eg: HadoopTimelineMetricsSink) with recompiled jars which use package "org.apache.commons.configuration2"
EXPRESS UPGRADE	YARN	Starting Hadoop 3.1.0 YARN daemons
ROLLING UPGRADE	HDFS	3.1.0 NN is started with rollingUpgrade with default policy configured for Erasure coding	Unable to render Jira issues macro, execution error.	Workaround Not known
ROLLING UPGRADE	YARN	Start 3.1.0 NM in batches after starting RM.	Unable to render Jira issues macro, execution error.	Fixed
EXPRESS/ROLLING UPGRADE	YARN	RM started with recovery enabled	Unable to render Jira issues macro, execution error.	Fixed

Upgrade Tests for YARN via Distributed Cache

Application Type	Upgrade Type	Issues Found
MR	EXPRESS/ROLLING UPGRADE	Unable to render Jira issues macro, execution error.
TEZ	EXPRESS UPGRADE
SPARK	EXPRESS UPGRADE