2017-12-14

Hadoop 3.0.0 has been released! Thanks to our many contributors, and congratulations to the community on this milestone.

Thus ends our 3.0.0 release updates. The 3.0.x release series will continue to be maintained, and 3.1.0 is planned for the first half of 2018.

2017-12-01

Haven't written one of these in a month. I had high hopes for RC0, but it failed due to 

We're back to blocker burndown. My new (realistic) goal is to get 3.0.0 out before Christmas. We could always use more help with reviews; most things are patch available.

Highlights:

Red flags:

Previously tracked blockers that have been resolved or dropped:

GA blockers:

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12840

  : Has gone through several rounds of review, looks close.

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-15080

  : New issue, waiting on LEGAL but we might need to pull this entire feature.

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-15059

  : Has gone through some review and has a +1 from Daryn, could use confirmation from Vinod and others

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-15058

  : Needs review, asked Allen but might need someone else to help.

GA criticals:

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12872

  : Patch needs review

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7381

  : Has gone through some review and Wangda +1'd, could use confirmation from Ray and others

Features merged for GA:

• Erasure coding
  • Testing is still ongoing at Cloudera, which resulted in 

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12840

     and 

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12872

    .
• Classpath isolation (HADOOP-11656)
  
  • No change.
• Compat guide (HADOOP-13714)
  • We slid a couple more changes into 3.0.0 after RC0 was cancelled, making this work more complete.
• TSv2 alpha 2
  • No change.
• API-based scheduler configuration 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-5734

  • No change.
• HDFS router-based configuration 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-10467

  • No change.
• Resource types 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-3926

  • Had some post-merge issues that were resolved, nothing outstanding.

2017-10-31

Lots of progress towards GA, we look on track for cutting RC0 this week. I ran the versions script to check the branch matches up with JIRA and fixed things up, and also checked that the changelog and release notes look reasonable.

Highlights:

• Resource types vote has passed and will be merged with branch-3.0 shortly.
• Down to three blockers on the dashboard, all being actively revved.

Red flags:

• Still need to validate that resource types is ready to go once it's merged.

Previous tracked GA blockers that have been resolved or dropped:

• Change of ExecutionType

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7178

    : Arun got the patch in with reviews from Wangda and Haibo.
• ReservationSystem

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-4827

    : Yufei and Subru got this in.
• Rolling upgrade

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-6142

    : Ray resolved this since we think it's sufficiently complete.
• Erasure coding

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12686

    : Resolved this one to incorporate it in HDFS-12682

GA blockers:

• Rolling upgrade

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-11096

    : I asked Sean if we can downgrade this from blocker
• Erasure coding

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12682

    : Actively being worked on and reviewed, should be in soon

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-11467

    : Waiting on HDFS-12682, I asked if we can work concurrently

Features merged for GA:

• Erasure coding
  • Testing is still ongoing at Cloudera, no new bugs found recently
  • Closing on remaining blockers for GA
• Classpath isolation (HADOOP-11656)
  

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-13916

    : Seems unlikely to make it
• Compat guide (HADOOP-13714)

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-14876

    : Patch is being actively revved and reviewed, Robert +1'd, Anu posted a big review

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-14875

    : No patch yet
• TSv2 alpha 2
  • This was merged, no problems thus far 
• API-based scheduler configuration 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-5734

  • Merged, no problems thus far 
• HDFS router-based configuration 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-10467

  • Merged, no problems thus far 
• Resource types 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-3926

  • Vote has passed, Daniel is currently doing the mechanics of merging
  • Need to also perform final validation post-merge

Dropping the "unmerged features" section since we're not letting in anything else at this point.

2017-10-20

Apologies for skipping the update last week. Here's how we're tracking for GA.

...

• We're making a last-minute push to get resource types (but not resource profiles in). Coming this late, it's a risk, but we decided it's worthwhile for this feature. See Daniel's commonyarn-dev email for the full rationale.
• Still uncovering EC bugs from testing

...