Scratch space to brainstorm ideas for improving HDFS test stability

Unit tests

• Update tests using build/test/data, /tmp, etc. to use correct temp
• Flaky tests
  • TestDFSStripedOutputStreamWithFailure ()
  • TestWebHdfsTimeouts
  • TestErasureCodingPoliciesWithRandomECPolicy
  • TestErasureCodingPolicies
  • TestDFSStripedOutputStream
  • TestDFSStripedInputStream
  • TestDecommission
  • TestUnsetAndChangeDirectoryEcPolicy
  • TestMaintenanceState
  • TestDataNodeErasureCodingMetrics
  • TestReconstructStripedFile
  • TestReadStripedFileWithDNFailure
• MiniDFSCluster improvements
  • Reduce the number of threads that get started by MiniDFSCluster
    • 
      
    • 
      
• Over-reliance on MiniDFSCluster
• Ignored or skipped tests
  •  TestGlobPaths

  • TestHdfsNativeCodeLoader

  • TestSymlinkHdfsFileSystem

  • TestAppendDifferentChecksum

  • TestDecommission

  • TestDFSUpgrade

  • TestFileConcurrentReader

  • TestParallelReadUtil

  • TestWebHDFSAcl

Branch-2 nightly DNF https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/

Build

• 
• ,  (Windows parallel builds)
• Windows build (stopped running?)
• Upgrade surefire, unless there's a good reason not to in branch-2
  • 
  • 
• branch-2 + JDK7 qbt build fails due to hanging hdfs tests (see https://builds.apache.org/job/hadoop-qbt-branch2-java7-linux-x86/158/console)
• JDK9: qbt build weekly to run hdfs unit tests on JDK9 (now compilation fails due to the two issues)
  • 
  •