2017-12-14
Hadoop 3.0.0 has been released! Thanks to our many contributors, and congratulations to the community on this milestone.
Thus ends our 3.0.0 release updates. The 3.0.x release series will continue to be maintained, and 3.1.0 is planned for the first half of 2018.
2017-12-01
Haven't written one of these in a month. I had high hopes for RC0, but it failed due to
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HADOOP-15058 |
---|
|
which Sangjin found, and then a number of other blockers were found shortly after that.We're back to blocker burndown. My new (realistic) goal is to get 3.0.0 out before Christmas. We could always use more help with reviews; most things are patch available.
Highlights:
Red flags:
Previously tracked blockers that have been resolved or dropped:
GA blockers:
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12840 |
---|
|
: Has gone through several rounds of review, looks close. Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HADOOP-15080 |
---|
|
: New issue, waiting on LEGAL but we might need to pull this entire feature. Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HADOOP-15059 |
---|
|
: Has gone through some review and has a +1 from Daryn, could use confirmation from Vinod and others Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HADOOP-15058 |
---|
|
: Needs review, asked Allen but might need someone else to help.
GA criticals:
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12872 |
---|
|
: Patch needs review Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7381 |
---|
|
: Has gone through some review and Wangda +1'd, could use confirmation from Ray and others
Features merged for GA:
- Erasure coding
- Testing is still ongoing at Cloudera, which resulted in
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12840 |
---|
|
and Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12872 |
---|
|
.
- Classpath isolation (HADOOP-11656)
- Compat guide (HADOOP-13714)
- We slid a couple more changes into 3.0.0 after RC0 was cancelled, making this work more complete.
TSv2 alpha 2
- API-based scheduler configuration
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-5734 |
---|
|
- HDFS router-based configuration
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-10467 |
---|
|
- Resource types
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-3926 |
---|
|
- Had some post-merge issues that were resolved, nothing outstanding.
2017-10-31
Lots of progress towards GA, we look on track for cutting RC0 this week. I ran the versions script to check the branch matches up with JIRA and fixed things up, and also checked that the changelog and release notes look reasonable.
Highlights:
- Resource types vote has passed and will be merged with branch-3.0 shortly.
- Down to three blockers on the dashboard, all being actively revved.
Red flags:
- Still need to validate that resource types is ready to go once it's merged.
Previous tracked GA blockers that have been resolved or dropped:
- Change of ExecutionType
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7178 |
---|
|
: Arun got the patch in with reviews from Wangda and Haibo.
- ReservationSystem
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-4827 |
---|
|
: Yufei and Subru got this in.
- Rolling upgrade
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-6142 |
---|
|
: Ray resolved this since we think it's sufficiently complete.
- Erasure coding
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12686 |
---|
|
: Resolved this one to incorporate it in HDFS-12682
GA blockers:
- Rolling upgrade
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-11096 |
---|
|
: I asked Sean if we can downgrade this from blocker
- Erasure coding
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12682 |
---|
|
: Actively being worked on and reviewed, should be in soon Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-11467 |
---|
|
: Waiting on HDFS-12682, I asked if we can work concurrently
Features merged for GA:
Dropping the "unmerged features" section since we're not letting in anything else at this point.
2017-10-20
Apologies for skipping the update last week. Here's how we're tracking for GA.
Highlights:
- Merge of HDFS router-based federation and API-based scheduler configuration with no reported problems. Kudos to the contributors involved!
Red flags:
- We're making a last-minute push to get resource types (but not resource profiles in). Coming this late, it's a risk, but we decided it's worthwhile for this feature. See Daniel's yarn-dev email for the full rationale.
- Still uncovering EC bugs from testing
Previously tracked GA blockers that have been resolved or dropped:
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-6623 |
---|
|
: Committed and resolved- Change of ExecutionType
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7275 |
---|
|
: Patch committed, resolved.
- ReservationSystem
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-4859 |
---|
|
: Yufei tested this and found things mostly worked, filed two not-blocker followons: Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7347 |
---|
|
and Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7348 |
---|
|
GA blockers:
- Change of ExecutionType
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7178 |
---|
|
: Still no update from Arun, I pinged it.
- ReservationSystem
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-4827 |
---|
|
: Yufei said he'd work on it as of 2 days ago
- Rolling upgrade
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-6142 |
---|
|
: I pinged this and asked for a status update Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-11096 |
---|
|
: I pinged this and asked for a status update
- Erasure coding
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12682 |
---|
|
: New blocker filed this week, Xiao is working on it Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12686 |
---|
|
: New blocker filed this week, Sammi is on it Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | HDFS-12686 |
---|
|
: Old blocker, Huafeng is on it, waiting on review from Wei-Chiu or Sammi
Features merged for GA:
Unmerged features:
- Resource types / profiles (YARN-3926 and YARN-7069) (Wangda Tan)
- We're going to try and get in resource types for 3.0.0 and leave resource profiles for 3.1.0. Daniel is spearheading this and other major contributors like Wangda and Sunil are onboard with the plan. Branch has been created and undergone testing, I expect a merge vote ASAP.
- This is our biggest remaining risk.
- YARN native services (YARN-5079) (Jian He)
Jira |
---|
server | ASF JIRA |
---|
serverId | 5aa69414-a9e9-3523-82ec-879b028fb15b |
---|
key | YARN-7351 |
---|
|
is blocking- Allen is reviewing YARN-7127 with some design-level questions about the API and architecture
- This doesn't look like this is getting in based on YARN-7127 discussion
2017-10-06
The beta1 RC0 vote passed, and beta1 is out! Now tracking GA features.
...