2017-08-25

Another month flew by without an update. This is a big one.

Red flags:

11 blockers still on the dashboard, with some filed recently. Need to burn these down.
There are many branch merges proposals flying around for features that were not originally being tracked for beta1 and GA. Introducing new code always comes with risk, so I'm working with the different contributors involved to discuss target versions, confirm readiness, and define quality bars for merge.

Miscellaneous blockers:

HADOOP-14284 (Shade Guava everywhere): We have agreement to shade the yarn client JAR. Shading hadoop-hdfs is still being discussed.
HADOOP-13363 (Upgrade to protobuf 3): Waiting on the Guava shading first.
YARN-7076: New blocker, we need an assignee.
YARN-7094 (Document that server-side graceful decom is currently not recommended): Robert has a patch up, needs review. This is a stopgap for the old blocker YARN-5464.
YARN-5536 (Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout): Robert has a proposal that needs to be pushed on.
HDFS-11096 and YARN-6142 (Rolling upgrade JIRAs): We need a final run through of the JACC reports. Need to get Sean's HDFS scripts committed, and Ray to add some additional YARN coverage.

beta1 features:

Erasure coding
- Need to wrap up pluggable EC work and other must-dos, reasonable confidence since these are actively being worked on
Addressing incompatible changes
- YARN-6142 (compatibility with 2.x): Still need automated testing for this
- HDFS-11096 (compatibility with 2.x): Sean has revived efforts here and has a rolling upgrade script under review, closing on some incompats we found too.
Classpath isolation (HADOOP-11656)
- We're down to the wire on this, I pinged Sean for an update.
Compat guide (HADOOP-13714)
- I pinged the JIRA on this too, no updated patch since May

Features under discussion:

I discussed with a number of lead contributors on these features that were previously not on my radar.

3.0.0-beta1:

YARN native services (Jian He)
- I was convinced that this is very separate from the core. I'll get someone from Cloudera to run it through our integration tests to verify it doesn't break anything downstream, then happy to merge.
TSv2 alpha 2 (Vrushali C)
- Despite being called "alpha 2", this is more like "beta" in terms of readiness. Twitter is planning to roll it out to production. Seems quite done.
- I double checked with Haibo, and he successfully ran it through our internal integration testing.

3.0.0 GA:

Resource profiles (Wangda Tan)
- Alpha feature, APIs are not stable yet. Has some compatible PB changes, will verify rolling upgrade from branch-2. Touches some core parts of YARN.
- Decided that it's too close to beta1 for this, we're going to test it a lot and make sure it's ready for 3.0.0 GA.
HDFS router-based federation (Chris Douglas)
- This is like YARN federation, very separate and doesn't add new APIs, run in production at MSFT.
- If it passes Cloudera internal integration testing, I'm fine putting this in for GA.

3.1.0:

Storage Policy Satisfier (Uma Gangumalla)
- We're resolving some design discussions on JIRA. Plan is to do some MVP work on the API to get this into 3.1, and if we're happy with the second phase, consider for 3.0 GA.
HDFS tiered storage (Chris Douglas):
- This touches some core stuff, and the write path is still being worked on. Still somewhat useful with just the read path. Targeting at 3.1.0 gives enough time to wrap this up.

2017-07-28

Long time since the previous update. In the meanwhile, we released alpha4! Onward to beta1.

...