2017-09-01

We're two weeks out from beta1, focus is on blocker burndown.

Highlights:

• S3Guard merged!
• TSv2 alpha2 merged!
• branch-3.0 has been cut after discussion on dev lists.

Red flags:

• 10 blockers on the dashboard, closed and bumped some but new ones appeared.
• Still need to land YARN native services and fix some S3Guard doc issues for beta1.
• Rolling upgrade JIRAs for YARN and HDFS are not making any visible progress

Previously tracked beta1 blockers that have been resolved:

• HADOOP-13363 (Upgrade to protobuf 3): I dropped this from beta1 since it's simply not going to happen in time.
• YARN-7076: This was quickly resolved! Thanks Jian, Junping, Jason for the action.
• YARN-7094 (Document that server-side graceful decom is currently not recommended): Patch committed!

beta1 blockers:

• HADOOP-14826 (review S3 docs prior to 3.0.0-beta1): New blocker with S3Guard merged. Should just be a quick doc update.
• HADOOP-14284 (Shade Guava everywhere): Agreement to shade yarn-client at at HADOOP-14771. Shading hadoop-hdfs is still being discussed?
• HADOOP-14771 (hadoop-client does not include hadoop-yarn-client): Patch up, needs review, waiting on Busbey
• YARN-5536 (Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout): We're waiting on input from Junping.
• MAPREDUCE-6941 (The default setting doesn't work for MapReduce job): Ray thinks this is a Won't Fix, waiting on Junping to confirm.
• HADOOP-14238 (Rechecking Guava's object is not exposed to user-facing API): This relates to HADOOP-14771, I left a JIRA comment.

beta1 features:

• Erasure coding
  • There are three must-dos. Two have patches, one might not be a must-do.
  • HDFS-11882 has been revved and reviewed, seems close
  • HDFS-11467 and HDFS-7859 are related, Sammi/Eddy/Kai are discussing, Sammi thinks we can still make beta1.
• Addressing incompatible changes (YARN-6142 and HDFS-11096)
  • Sean has HDFS rolling upgrade scripts up, waiting on Ray to add some YARN/MR coverage too.
  • Need to do a final runthrough of the JACC reports for YARN and HDFS.
• Classpath isolation (HADOOP-11656)
  • Sean has retriaged the subtasks and has been posting patches.
• Compat guide (HADOOP-13714)
  • New patch is up, but needs review. Daniel asked Chris Douglas and Steve Loughran.
• YARN native services
  • Jian sent out the merge vote
• TSv2 alpha 2
  • This was merged, no problems thus far 

GA features:

• Resource profiles (Wangda Tan)
  • Merge vote was sent out. Since branch-3.0 has been cut, this can be merged to trunk (3.1.0) and then backported once we've completed testing.
• HDFS router-based federation (Chris Douglas)
  • This is like YARN federation, very separate and doesn't add new APIs, run in production at MSFT.
  • If it passes Cloudera internal integration testing, I'm fine putting this in for GA.
• API-based scheduler configuration (Jonathan Hung)
  • Jonathan mentioned that his main goal is to get this in for 2.9.0, which seems likely to go out after 3.0.0 GA since there hasn't been any serious release planning yet. Jonathan said that delaying this until 3.1.0 is fine.

2017-08-25

Another month flew by without an update. This is a big one.

Red flags:

• 11 blockers still on the dashboard, with some filed recently. Need to burn these down.
• There are many branch merges proposals flying around for features that were not originally being tracked for beta1 and GA. Introducing new code always comes with risk, so I'm working with the different contributors involved to discuss target versions, confirm readiness, and define quality bars for merge.

Miscellaneous blockers:

• HADOOP-14284 (Shade Guava everywhere): We have agreement to shade the yarn client JAR. Shading hadoop-hdfs is still being discussed. 
• HADOOP-13363 (Upgrade to protobuf 3): Waiting on the Guava shading first.
• YARN-7076: New blocker, we need an assignee.
• YARN-7094 (Document that server-side graceful decom is currently not recommended): Robert has a patch up, needs review. This is a stopgap for the old blocker YARN-5464.
• YARN-5536 (Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout): Robert has a proposal that needs to be pushed on.

beta1 features:

• Erasure coding
  • There are three must-dos. Two have patches, one might not be a must-do.
  • I pinged the pluggable policy JIRA to see if metadata and API compatibility is complete.
• Addressing incompatible changes (YARN-6142 and HDFS-11096)
  • Sean has HDFS rolling upgrade scripts up, waiting on Ray to add some YARN/MR coverage too.
  • Need to do a final runthrough of the JACC reports for YARN and HDFS.
• Classpath isolation (HADOOP-11656)
  • We're down to the wire on this, I pinged Sean for an update.
• Compat guide (HADOOP-13714)
  • I pinged the JIRA on this too, no updated patch since May

Features under discussion:

I discussed with a number of lead contributors on these features that were previously not on my radar.

3.0.0-beta1:

3.0.0 GA:

3.1.0:

2017-07-28

Long time since the previous update. In the meanwhile, we released alpha4! Onward to beta1.

Red flags:

• Beta1 is targeted for 09/15, and there are a number of blockers we kicked down the road that need to be resolved.

Miscellaneous blockers

• HADOOP-14284 (Shade Guava everywhere): This one is stalled, concerning.
• HADOOP-13363 (Upgrade to protobuf 3): Waiting on the Guava shading first.

beta1 features:

• Erasure coding
  • Need to wrap up pluggable EC work and other must-dos, reasonable confidence since these are actively being worked on
• Addressing incompatible changes
  • YARN-6142 (compatibility with 2.x): Still need automated testing for this
  • HDFS-11096 (compatibility with 2.x): Sean has revived efforts here and has a rolling upgrade script under review, closing on some incompats we found too.
• Classpath isolation (HADOOP-11656)
  • The subtasks were not completed for alpha4. Sean has revived efforts and is committed to finishing this for beta1.
• Compat guide (HADOOP-13714)
  • Daniel posted an initial patch for discussion, but stalled out

2017-04-21

Red flags:

• Less than a month out and some new blockers just surfaced, need to keep pushing.

Miscellaneous blockers

• HADOOP-14284 (Shade Guava everywhere): Progressing, though trying to resolve Curator issues
• HADOOP-13363 (Upgrade to protobuf 3): Waiting on the Guava shading first
  
• HADOOP-14330 (Kerby breaks multiple SPN support): uncovered from secure testing with Oozie, unknown if other Kerby bugs are lurking
• YARN-5894 (fst licensing): no assignee yet

alpha3 features:

• Erasure coding
  • Good burndown of remaining blockers for alpha3, seems on track
  • HDFS-11643 (Balancer fencing fails when writing erasure coded lock file): Close to commit
  • HDFS-11644 (DFSStripedOutputStream should not implement Syncable): API-related issue that affects YARN and HBase

beta1 features:

• Addressing incompatible changes
  • YARN-6142 (compatibility with 2.x): Still need automated testing for this
  • HDFS-11096 (compatibility with 2.x): Still need automated testing for this
• Classpath isolation (HADOOP-11656)
  • Sean has not yet resumed work, but still plans to complete API-related subtasks before alpha3
• Compat guide (HADOOP-13714)
  • Daniel has started working on this

2017-02-21

Red flags:

• Not much movement on EC blockers since last update. Moved out planned alpha3 date as a result from mid-April to mid-May.

alpha3 features:

• Erasure coding
  • Some discussion around configuration of enabled EC policies.
  • Manoj and Andrew are working on EC CLI improvements (HDFS-11405, HDFS-11426, HDFS-11427, HDFS-11428)
  • Manoj continues to work on OIV improvements (HDFS-10983)
  • Kai Sasaki revving patch on HDFS-8196 for webUI additions after reviews from Takanobu Asanuma and Andrew
• Tomcat to Jetty conversions
  
  • HDFS-10860 (HttpFS): This was committed! Will remove section for next update.

beta1 features:

• Addressing incompatible changes
  • YARN-6142 (compatibility with 2.x): Subtask at YARN-6143 has made good progress thanks to Sunil and Wangda, seems close to commit.
  • HDFS-11096 (compatibility with 2.x): Nothing further pending on this JIRA, but would like automated test runs setup to catch future breakages.
• Classpath isolation (HADOOP-11656)
  • Sean plans to work on this in March/April, which lines up with the new alpha3 schedule

2017-02-03

3.0.0-alpha2 was released last week, onward to alpha3!

Notable release blockers:

• Since alpha3 is planned as the last alpha before beta, the goal is to be feature complete by alpha3. beta1 can be used to stage any remaining incompatible changes, or nice-to-have feature-related work.
• No other release blockers are filed.

alpha3 features:

• Erasure coding
  
  • All must-do JIRAs are now marked as blockers for alpha3, goal is for EC to be feature complete and API stable by alpha3.
  • Light development activity. Andrew did some reviews, Manoj is working on OIV improvements.
• Tomcat to Jetty conversions
  
  • HDFS-10860 (HttpFS): Patch has gone through multiple revs and has a +1 pending from Xiao

beta1 features:

• Addressing incompatible changes
  • YARN-6142 (compatibility with 2.x): Tracking JIRA filed and assigned to Ray
  • HDFS-11096 (compatibility with 2.x): Nothing further pending on this JIRA, but would like automated test runs setup to catch future breakages.
• Classpath isolation (HADOOP-11656)
  • Andrew has asked Sean for an ETA on completing the remaining subtasks, ideally by alpha3 but latest beta1

2017-01-19

branch-3.0.0-alpha2 has been created in git. Still on track for planned alpha2 release at the end of January.

Notable release blockers:

• YARN-5667 (Move HBase backend code in ATS v2 into its separate module): One subtask left at YARN-5928, looks close to commit.
• Also resolved blockers like HADOOP-13996, YARN-6071, YARN-5646, YARN-6072, HADOOP-13961, HDFS-11312, YARN-6068, HADOOP-13780 since last update. Wow!

Development activity

• Addressing incompatible changes
  • HDFS-11096 (compatibility with 2.x): Sean filed and fixed HDFS-11312, which was a PB incompatibility. Seems like this work is complete on the HDFS side, but we still need YARN and MR help.
• Erasure coding
  • Andrew went through and re-triaged all the JIRAs on HDFS-8031, using the "hdfs-ec-3.0-must-do" and "hdfs-ec-3.0-nice-to-have" labels. HDFS-8031 more closely reflects what we want to get done for 3.0, and the labels are now tracked on the JIRA dashboard.
• Classpath isolation (HADOOP-11565)
  • Arun added a Maven variable to skip shading for faster build times. No progress on subtasks required for beta1.
• Tomcat to Jetty conversions
  • HADOOP-13597 (KMS): this patch went in
  • HDFS-10860 (HttpFS): Patch up and gone through initial review

2017-01-04

Notable release blockers:

Down to just four release blockers, three of which have patches posted and seem to be close.

• HADOOP-13780 (L&N updates): Patch is posted and +1'd, likely will be committed soon.
• HADOOP-13896 (distribution tarball missing jars): Tiny patch is posted, pending review and precommit.
• YARN-6022 (Revert changes of AbstractResourceRequest): Patch is posted and been reviewed, looks close.
• YARN-5667 (Move HBase backend code in ATS v2 into its separate module): One subtask left at YARN-5928, PA but no progress recently.
  

Development activity

• Addressing incompatible changes
  • MAPREDUCE-6704: resolved!
  • YARN-5882 (issuing YARN delegation tokens even in insecure deployments): Jian reverted YARN-4126 pending further discussion, so no longer a blocker.
  • HDFS-11096 (compatibility with 2.x): Sean reviewed the JACC output and posted a big comment on possible source/binary incompatibilities.
• Erasure coding
  • Still slowly burning down must-dos from the JIRA query.
• Classpath isolation (HADOOP-11565)
  • HADOOP-11804 (shaded client jars): Initial patch has been committed, Sean also filed follow-on JIRAs for related work we need to clean up before beta1.
• Tomcat to Jetty conversions
  • HADOOP-13597 (KMS): +1 pending a few nits
  • HDFS-10860 (HttpFS): Patch up and gone through initial review

2016-12-09

Notable release blockers:

• HADOOP-13780: L&N updates from Jetty bump. Xiao has picked this up, no patch yet though.

Development activity

• Addressing incompatible changes
  • MAPREDUCE-6704: still seems close after examination of Docker issue
  • YARN-5882 (issuing YARN delegation tokens even in insecure deployments): Andrew proposed we revert YARN-4126 to unblock the release while discussion continues
  • YARN-5559: Resolved!
  • YARN-5184: Resolved!
  • HDFS-11096 (compatibility with 2.x): Discussion still underway. Kihwal and Eric worked to resolve HDFS-11207 which reverted some incompatible DN/NN changes that broke rolling upgrade.
• Erasure coding
  • Still slowly burning down must-dos from the JIRA query.
• Classpath isolation (HADOOP-11565)
  • HADOOP-11804 (shaded client jars): Sean added integration tests for WebHDFS and things seem close.
• Tomcat to Jetty conversion (HDFS-11096, HDFS-10860)
  • HDFS-11096 (KMS): patch is under review
  • HDFS-10860 (HttpFS): no patch yet

2016-11-28

Notable release blockers:

• org.json:json: legal-discuss has allowed grandfathering in releases until April 30, 2017 for usages of org.json:json. HADOOP-13050 bumping the problematic AWS SDK dependency has been committed. We're in the clear.
• HADOOP-13780: L&N updates from Jetty bump. Still unassigned.

Development activity

• Addressing incompatible changes
  • MAPREDUCE-6704: still seems close after examination of Docker issue
  • YARN-5882: issuing YARN delegation tokens even in insecure deployments, discussion underway
  • YARN-5559: found via JDiff analysis, fix some accidental incompats
  • YARN-5184: found via downstream testing. New abstract methods added to public class.
  • HDFS-11096 (compatibility with 2.x): Summary of a discussion between Andrew and Karthik was posted, seeking additional community feedback.
• Erasure coding
  • Slowly burning down the must-dos from the JIRA query.
• Classpath isolation (HADOOP-11565)
  • HADOOP-11804 (shaded client jars): POC testing with Avro and HBase seem to work, waiting on additional testing and reviews.
• Tomcat to Jetty conversion (HDFS-11096, HDFS-10860)
  • Initial patch posted on HDFS-11096 for KMS conversion.

2016-11-16

Notable release blockers:

• HADOOP-11694: Need to upgrade AWS SDK dependency, org.json:json fallout
• HADOOP-13780: L&N updates from Jetty bump

Development activity

• Addressing incompatible changes
  • MAPREDUCE-6704: make the default pseudo-cluster work without additional env var configuration, seems close?
  • YARN-5882: issuing YARN delegation tokens even in insecure deployments, discussion underway
  • YARN-5559: found via JDiff analysis, fix some accidental incompats
  • YARN-5184: found via downstream testing. New abstract methods added to public class.
• Erasure coding
  • Work continues on API refinements and supportability-related JIRAs.
• Classpath isolation (HADOOP-11565)
  • HADOOP-11804 (shaded client jars): Multiple revs have been posted based on review feedback.