2017-10-20

Apologies for skipping the update last week. Here's how we're tracking for GA.

Highlights:

• Merge of HDFS router-based federation and API-based scheduler configuration with no reported problems. Kudos to the contributors involved!

Red flags:

• We're making a last-minute push to get resource types (but not resource profiles in). Coming this late, it's a risk, but we decided it's worthwhile for this feature. See Daniel's common-dev email for the full rationale.
• Still uncovering EC bugs from testing

Previously tracked GA blockers that have been resolved or dropped:

• Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-6623

  : Committed and resolved
• Change of ExecutionType

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7275

    : Patch committed, resolved.
• ReservationSystem

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-4859

    : Yufei tested this and found things mostly worked, filed two not-blocker followons: 

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7347

     and 

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7348

GA blockers:

• Change of ExecutionType

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7178

    : Still no update from Arun, I pinged it.
• ReservationSystem

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-4827

    : Yufei said he'd work on it as of 2 days ago
• Rolling upgrade

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-6142

    : I pinged this and asked for a status update

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-11096

    : I pinged this and asked for a status update
• Erasure coding

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12682

    : New blocker filed this week, Xiao is working on it

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12686

    : New blocker filed this week, Sammi is on it

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12686

    : Old blocker, Huafeng is on it, waiting on review from Wei-Chiu or Sammi

Features merged for GA:

• Erasure coding
  • Continued bug reporting and fixing based on testing at Cloudera.
  • Two new blockers filed this week, mentioned above.
  • Huafeng completed patch to reenable disabled EC tests
• Classpath isolation (HADOOP-11656)
  

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-13916

    : I pinged it
• Compat guide (HADOOP-13714)

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-14876

    : Daniel has a patch up, revved based on Steve's review feedback, waiting on Steve's reply

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-14875

    : No patch yet
• TSv2 alpha 2
  • This was merged, no problems thus far 
• API-based scheduler configuration 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-5734

  • Merged, no problems thus far 
• HDFS router-based configuration 

  Jira
  server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-10467

  • Merged, no problems thus far 

Unmerged features:

• Resource types / profiles (YARN-3926 and YARN-7069) (Wangda Tan)
  • We're going to try and get in resource types for 3.0.0 and leave resource profiles for 3.1.0. Daniel is spearheading this and other major contributors like Wangda and Sunil are onboard with the plan. Branch has been created and undergone testing, I expect a merge vote ASAP.
  • This is our biggest remaining risk.
• YARN native services (YARN-5079) (Jian He)

  • Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key YARN-7351

     is blocking
  • Allen is reviewing YARN-7127 with some design-level questions about the API and architecture
  • This doesn't look like this is getting in based on YARN-7127 discussion

2017-10-06

The beta1 RC0 vote passed, and beta1 is out! Now tracking GA features.

...