Versions Compared

Old Version 17

changes.mady.by.user Andrew Wang

Saved on

compared with

New Version Current

changes.mady.by.user Andrew Wang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

2017-12-14

Hadoop 3.0.0 has been released! Thanks to our many contributors, and congratulations to the community on this milestone.

Thus ends our 3.0.0 release updates. The 3.0.x release series will continue to be maintained, and 3.1.0 is planned for the first half of 2018.

2017-12-01

Haven't written one of these in a month. I had high hopes for RC0, but it failed due to 

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyHADOOP-15058
 which Sangjin found, and then a number of other blockers were found shortly after that.

We're back to blocker burndown. My new (realistic) goal is to get 3.0.0 out before Christmas. We could always use more help with reviews; most things are patch available.

 

Highlights:

Red flags:

Previously tracked blockers that have been resolved or dropped:

GA blockers:

  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHDFS-12840
    : Has gone through several rounds of review, looks close.
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHADOOP-15080
    : New issue, waiting on LEGAL but we might need to pull this entire feature.
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHADOOP-15059
    : Has gone through some review and has a +1 from Daryn, could use confirmation from Vinod and others
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHADOOP-15058
    : Needs review, asked Allen but might need someone else to help.

GA criticals:

  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHDFS-12872
    : Patch needs review
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-7381
    : Has gone through some review and Wangda +1'd, could use confirmation from Ray and others

Features merged for GA:

  • Erasure coding
    • Testing is still ongoing at Cloudera, which resulted in 
      Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12840
       and 
      Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12872
      .
  • Classpath isolation (HADOOP-11656)
    • No change.
  • Compat guide (HADOOP-13714)
    • We slid a couple more changes into 3.0.0 after RC0 was cancelled, making this work more complete.
  • TSv2 alpha 2
    • No change.
  • API-based scheduler configuration 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-5734
    • No change.
  • HDFS router-based configuration 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHDFS-10467
    • No change.
  • Resource types 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-3926
    • Had some post-merge issues that were resolved, nothing outstanding.

2017-10-31

Lots of progress towards GA, we look on track for cutting RC0 this week. I ran the versions script to check the branch matches up with JIRA and fixed things up, and also checked that the changelog and release notes look reasonable.

Highlights:

  • Resource types vote has passed and will be merged with branch-3.0 shortly.
  • Down to three blockers on the dashboard, all being actively revved.

Red flags:

  • Still need to validate that resource types is ready to go once it's merged.

Previous tracked GA blockers that have been resolved or dropped:

  • Change of ExecutionType
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7178
      : Arun got the patch in with reviews from Wangda and Haibo.
  • ReservationSystem
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-4827
      : Yufei and Subru got this in.
  • Rolling upgrade
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-6142
      : Ray resolved this since we think it's sufficiently complete.
  • Erasure coding
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12686
      : Resolved this one to incorporate it in HDFS-12682

GA blockers:

  • Rolling upgrade
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-11096
      : I asked Sean if we can downgrade this from blocker
  • Erasure coding
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12682
      : Actively being worked on and reviewed, should be in soon
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-11467
      : Waiting on HDFS-12682, I asked if we can work concurrently

Features merged for GA:

  • Erasure coding
    • Testing is still ongoing at Cloudera, no new bugs found recently
    • Closing on remaining blockers for GA
  • Classpath isolation (HADOOP-11656)
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHADOOP-13916
      : Seems unlikely to make it
  • Compat guide (HADOOP-13714)
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHADOOP-14876
      : Patch is being actively revved and reviewed, Robert +1'd, Anu posted a big review
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHADOOP-14875
      : No patch yet
  • TSv2 alpha 2
    • This was merged, no problems thus far (smile)
  • API-based scheduler configuration 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-5734
    • Merged, no problems thus far (smile)
  • HDFS router-based configuration 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHDFS-10467
    • Merged, no problems thus far (smile)
  • Resource types 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-3926
    • Vote has passed, Daniel is currently doing the mechanics of merging
    • Need to also perform final validation post-merge

Dropping the "unmerged features" section since we're not letting in anything else at this point.

2017-10-20

Apologies for skipping the update last week. Here's how we're tracking for GA.

Highlights:

  • Merge of HDFS router-based federation and API-based scheduler configuration with no reported problems. Kudos to the contributors involved!

Red flags:

  • We're making a last-minute push to get resource types (but not resource profiles in). Coming this late, it's a risk, but we decided it's worthwhile for this feature. See Daniel's yarn-dev email for the full rationale.
  • Still uncovering EC bugs from testing

Previously tracked GA blockers that have been resolved or dropped:

  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-6623
    : Committed and resolved
  • Change of ExecutionType
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7275
      : Patch committed, resolved.
  • ReservationSystem
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-4859
      : Yufei tested this and found things mostly worked, filed two not-blocker followons: 
      Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7347
       and 
      Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7348

GA blockers:

  • Change of ExecutionType
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7178
      : Still no update from Arun, I pinged it.
  • ReservationSystem
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-4827
      : Yufei said he'd work on it as of 2 days ago
  • Rolling upgrade
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-6142
      : I pinged this and asked for a status update
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-11096
      : I pinged this and asked for a status update
  • Erasure coding
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12682
      : New blocker filed this week, Xiao is working on it
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12686
      : New blocker filed this week, Sammi is on it
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-12686
      : Old blocker, Huafeng is on it, waiting on review from Wei-Chiu or Sammi

Features merged for GA:

  • Erasure coding
    • Continued bug reporting and fixing based on testing at Cloudera.
    • Two new blockers filed this week, mentioned above.
    • Huafeng completed patch to reenable disabled EC tests
  • Classpath isolation (HADOOP-11656)
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHADOOP-13916
      : I pinged it
  • Compat guide (HADOOP-13714)
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHADOOP-14876
      : Daniel has a patch up, revved based on Steve's review feedback, waiting on Steve's reply
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHADOOP-14875
      : No patch yet
  • TSv2 alpha 2
    • This was merged, no problems thus far (smile)
  • API-based scheduler configuration 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-5734
    • Merged, no problems thus far (smile)
  • HDFS router-based configuration 
    Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyHDFS-10467
    • Merged, no problems thus far (smile)

Unmerged features:

  • Resource types / profiles (YARN-3926 and YARN-7069) (Wangda Tan)
    • We're going to try and get in resource types for 3.0.0 and leave resource profiles for 3.1.0. Daniel is spearheading this and other major contributors like Wangda and Sunil are onboard with the plan. Branch has been created and undergone testing, I expect a merge vote ASAP.
    • This is our biggest remaining risk.
  • YARN native services (YARN-5079) (Jian He)
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7351
       is blocking
    • Allen is reviewing YARN-7127 with some design-level questions about the API and architecture
    • This doesn't look like this is getting in based on YARN-7127 discussion

2017-10-06

The beta1 RC0 vote passed, and beta1 is out! Now tracking GA features.

Highlights:

  • 3.0.0-beta1 has been released!
  • Router-based federation merge vote should be about to pass
  • API-based scheduler configuration merge vote is out, has the votes so far

Red flags:

  • Still need to nail down whether we're going to try and merge resource profiles. I've been emailing with Wangda and Daniel about this, we need to reach a decision ASAP (might already be too late).
  • Still waiting on Allen to review YARN native services feature.

Previously tracked GA blockers that have been resolved or dropped:

  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-7134
    :  Wangda downgraded this to "Major", dropping from list.

GA blockers:

  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-6623
    : Actively being reviewed
  • Change of ExecutionType
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7275
      : Kartheek has posted a patch, waiting for review
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-7178
      : No update from Arun, though it's just a docs patch
  • ReservationSystem
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-4859
      : Yufei has picked this up
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-4827
      : Yufei has picked this up, just a docs patch
  • Rolling upgrade
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyYARN-6142
      : Ray is still going through JACC and proto output
    • Jira
      serverASF JIRA
      serverId5aa69414-a9e9-3523-82ec-879b028fb15b
      keyHDFS-11096
      : Sean has revved the patch and is waiting on reviews from Ray, Allen

Features merged for GA:

  • Erasure coding
    • Continued bug reporting and fixing based on testing at Cloudera.
    • Still need to finish the 3.0 must-do's
  • Classpath isolation (HADOOP-11656)
    • HADOOP-14771 is still floating, along with adding documentation.
  • Compat guide (HADOOP-13714)
    • Synced with Daniel, he plans to wrap up the remaining  stuff next week
  • TSv2 alpha 2
    • This was merged, no problems thus far (smile)

Unmerged features:

  • Resource types / profiles (YARN-3926 and YARN-7069) (Wangda Tan)
    • This has been merged for 3.1.0, YARN-7069 tracks follow on work
    • Wangda said that he's okay waiting for 3.1.0 for this, we're waiting on Daniel. I synced with Daniel earlier this week, and he wants to try and get some of it into 3.0.0. Waiting on an update.
    • I still need a JIRA query for tracking the state of this.
  • HDFS router-based federation (HDFS-10467) (Inigo Goiri and Chris Douglas)
    • Merge vote should close any minute now
  • API-based scheduler configuration (Jonathan Hung)
    • Merge vote is out, will close next week
  • YARN native services (YARN-5079) (Jian He)
    • Subtasks were filed to address Allen's review comments from the previous merge vote, only one pending 
    • We need to confirm with Allen that this is ready to go, he hasn't been reviewing

2017-09-29

After about a month of slip, RC0 has been sent out for a VOTE. Focus now turns to GA, where we will attempt to keep the original beta1 target date (early November).

Highlights:

  • RC0 vote was sent out on Thursday, two binding +1's so far.

Red flags:

  • Resource profiles still has a number of pending subtasks, which is concerning from a schedule perspective. I emailed Wangda about this, and we need to discuss with other key contributors.
  • Native services has one pending subtask but we haven't gotten follow-on reviews from Allen (who -1'd the earlier merge vote). Need to confirm that we've satisfied his feedback.

Previously tracked beta1 blockers that have been resolved or dropped:

  • YARN-6623 was pushed out of beta1 to GA, has been committed so we can drop it from tracking.
  • HADOOP-14897 (Loosen compatibility guidelines for native dependencies): Patch committed!

beta1 blockers:

  • None, RC0 is out

GA blockers:

  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-7134
     : this one popped out of nowhere, I don't have an update yet.
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-7178
    : this also popped out of nowhere, no update yet.
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-7275
    : Ditto
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-4859
    : Ditto
  • Jira
    serverASF JIRA
    serverId5aa69414-a9e9-3523-82ec-879b028fb15b
    keyYARN-4827
    : Ditto

Features merged for GA:

  • Erasure coding
    • People are looking more at the flaky tests and nice-to-haves
    • Some bugs reported and being fixed based on testing at Cloudera
    • Need to finish the 3.0 must-do's.
  • Addressing incompatible changes (YARN-6142 and HDFS-11096)
    • Sean has posted a new rev of the rolling upgrade script
    • Some YARN PB backward compat issues that we decided weren't blockers and are scheduled for GA
  • Classpath isolation (HADOOP-11656)
    • HADOOP-13917 (Ensure nightly builds run the integration tests for the shaded client): Resolved, Sean retriggered and determined that this works.
    • HADOOP-14771 is still floating, along with adding documentation.
  • Compat guide (HADOOP-13714)
    • A few subtasks are targeted at GA
  • TSv2 alpha 2
    • This was merged, no problems thus far (smile)

Unmerged features:

  • Resource profiles (YARN-3926 and YARN-7069) (Wangda Tan)
    • This has been merged for 3.1.0, YARN-7069 tracks follow on work
    • ~7 patch available subtasks, I asked Wangda to set up a JIRA query for tracking this
  • HDFS router-based federation (HDFS-10467) (Inigo Goiri and Chris Douglas)
    • Inigo sent out the merge vote
  • API-based scheduler configuration (Jonathan Hung)
    • Jonathan sent out a discuss thread for merge, thinking is early next week. Larry did a security-oriented review.
  • YARN native services (YARN-5079) (Jian He)
    • Subtasks were filed to address Allen's review comments from the previous merge vote, only one pending 
    • We need to confirm with Allen that this is ready to go, he hasn't been reviewing

2017-09-22

We've had some late breaking blockers related to Docker support that are delaying the release. We're on a day-by-day slip at this point.

...