Page History

Scratch space to brainstorm ideas for improving HDFS test stability

Unit tests

• Update tests using build/test/data, /tmp, etc. to use correct temp
• Flaky tests
  • TestDFSStripedOutputStreamWithFailure (

    Jira
    server ASF JIRA serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-11600

    )
  • TestWebHdfsTimeouts
  • TestErasureCodingPoliciesWithRandomECPolicy
  • TestErasureCodingPolicies
  • TestDFSStripedOutputStream
  • TestDFSStripedInputStream
  • TestDecommission
  • TestUnsetAndChangeDirectoryEcPolicy
  • TestMaintenanceState
  • TestDataNodeErasureCodingMetrics
  • TestReconstructStripedFile
  • TestReadStripedFileWithDNFailure
• MiniDFSCluster improvements
  • Reduce the number of threads that get started by MiniDFSCluster

    • Jira
      server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-13265

      
      

    • Jira
      server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-13262

      
      
• Over-reliance on MiniDFSCluster
• Ignored or skipped tests
  •  TestGlobPaths

  • TestHdfsNativeCodeLoader

  • TestSymlinkHdfsFileSystem

  • TestAppendDifferentChecksum

  • TestDecommission

  • TestDFSUpgrade

  • TestFileConcurrentReader

  • TestParallelReadUtil

  • TestWebHDFSAcl

Branch-2 nightly DNF https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/

Build

• Jira
  server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12711

• Jira
  server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-14696

  , 

  Jira
  server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-12220

  (Windows parallel builds)
• Windows build (stopped running?)
• Upgrade surefire, unless there's a good reason not to in branch-2

  • Jira
    server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-15251

  • Jira
    server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-15153

• branch-2 (see HADOOP-15153)+ JDK7 qbt build fails due to hanging hdfs tests (see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/158/console)
• JDK9: qbt build weekly to run hdfs unit tests on JDK9 (now compilation fails due to the two issues)

  • Jira
    server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HADOOP-12760

  • Jira
    server ASF JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId 5aa69414-a9e9-3523-82ec-879b028fb15b key HDFS-11610