2017-09-07

Slightly early update since I'll be out tomorrow. We're one week out, and focus is on blocker burndown.

Highlights:

3.1.0 release planning is underway, led by Wangda. Target release date is in January.

Red flags:

YARN native services merge vote got a -1 for beta1, I recommended we drop it from beta1 and retarget for a later release.
11 blockers on the dashboard, one more than last week

Previously tracked beta1 blockers that have been resolved or dropped:

HADOOP-14826 was duped to HADOOP-14738.
YARN-5536 (Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout): Downgraded in priority in favor of YARN-7162 which Robert has posted a patch for.
MAPREDUCE-6941 (The default setting doesn't work for MapReduce job): I resolved this and Junping confirmed this is fine.

beta1 blockers:

HADOOP-14738 (Remove S3N and obsolete bits of S3A; rework docs): Steve has been actively revving this with our new committer Aaron Fabbri ready to review. The scope has expanded from HADOOP-14826, so it's not just a doc update.
HADOOP-14284 (Shade Guava everywhere): No change since last week. This is an umbrella JIRA.
HADOOP-14771 (hadoop-client does not include hadoop-yarn-client): Patch up, needs review, still waiting on Busbey. Bharat gave it a review.
YARN-7162 (Remove XML excludes file format): Robert has posted a patch and is waiting for a review.
HADOOP-14238 (Rechecking Guava's object is not exposed to user-facing API): Bharat took this up and turned it into an umbrella.
- HADOOP-14847 (Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc) Bharat posted a patch on a subtask to fix the known Guava Supplier issue in AMRMClient. Needs a review.
HADOOP-14835 (mvn site build throws SAX errors): I'm working on this. Debugged it and have a proposed patch up, discussing with Allen.
HDFS-12218 (Rename split EC / replicated block metrics in BlockManager): I'm working on this, just need to commit it, already have a +1 from Eddy.

beta1 features:

Erasure coding
- There are three must-dos, all being actively worked on.
- HDFS-7859 is being actively reviewed and revved by Sammi and Kai and Eddy.
- HDFS-12395 was split out of HDFS-7859 to do the edit log changes.
- HDFS-12218 is discussed above.
Addressing incompatible changes (YARN-6142 and HDFS-11096)
- Ray and Allen reviewed Sean's HDFS rolling upgrade scripts.
- Sean did a run through of the HDFS JACC report and it looked fine.
Classpath isolation (HADOOP-11656)
- Sean has retriaged the subtasks and has been posting patches.
Compat guide (HADOOP-13714)
- Daniel has been collecting feedback on dev lists, but still needs a detailed review of the patch.
YARN native services
- Jian sent out the merge vote, but it's been -1'd for beta1 by Allen. I propose we drop this from beta1 scope and retarget.
TSv2 alpha 2
- This was merged, no problems thus far

GA features:

Resource profiles (Wangda Tan)
- Merge vote was sent out. Since branch-3.0 has been cut, this can be merged to trunk (3.1.0) and then backported once we've completed testing.
HDFS router-based federation (Chris Douglas)
- This is like YARN federation, very separate and doesn't add new APIs, run in production at MSFT.
- If it passes Cloudera internal integration testing, I'm fine putting this in for GA.
API-based scheduler configuration (Jonathan Hung)
- Jonathan mentioned that his main goal is to get this in for 2.9.0, which seems likely to go out after 3.0.0 GA since there hasn't been any serious release planning yet. Jonathan said that delaying this until 3.1.0 is fine.

2017-09-01

We're two weeks out from beta1, focus is on blocker burndown.

...