Versions Compared

Old Version 1

changes.mady.by.user Kirk Lund

Saved on

compared with

New Version 2

changes.mady.by.user Mark Hanson

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Distributed Test Recommendations


The goals of the Green Team

  • Improve green consistency of distributed tests
  • Mass test runs determined which tests we worked on
    • Prioritized any tests that failed more than once in our mass test runs
  • Fixed about 35 issues (working on more)
  • Spent about a quarter of our time fixing product bugs
  • Anecdotal improvements for Geode developers
    • “CIO board is cleaner”
    • “Seeing more Green pull requests”


Green Team Progress

Image Added


If I had a nickel for every thread sleep related failure...

What was helpful vs unhelpful

Helpful

  • Clean test code
  • All callback and domain classes are specific to test and organized as inner classes within the test class
  • Use of non-deprecated APIs
  • Use of AssertJ instead of Assert

Unhelpful

  • Asynchronous activity
  • Thread.sleep or Wait.pause calls
  • Major refactorings such as modularization
  • Non-thread-safe static product test hooks
  • TestHelper classes and custom startup rules that configure processes or perform actions
  • Generalized callback and domain classes reused by more than one test
  • Tests that use dunit working directories instead of TemporaryFolder Rule
  • Catching unexpected exceptions
  • Instantiating a dunit class again from a static method


What you can do to make things better?

Avoid thread sleeping

  • Awaitility -- await for some state to change
  • CountDownLatch -- coordinate between VMs and/or threads
  • Mockito Spy and Verify
    • Wrap anything with spy or use it to implement a callback such as CacheListener
    • Verify is used to validate number of invocations with optional timeout
      • Verify with timeout can be used instead of Awaitility
  • All timeouts should use GeodeAwaitility.getTimeout()
  • Possible exceptions to the rule
    • Busy waiting in loops (Awaitility is usually a better option than custom looping)
    • Slow receiver/listener type testing (there’s probably a better way to do it)


It’s not polite to eat all the exceptions...


Exception handling

  • Expected vs unexpected exceptions in tests
  • Never catch unexpected exceptions. That is bad... way bad...
    • Special case: Asynchronous actions may need to catch checked exceptions and use ErrorCollector
  • Use method throws clause for unexpected exceptions 
    • The alternative serves to make failures more opaque. 
    • Fixing test failures (as you will have to do at some point) is all about visibility into the problems of the test. 
  • Use AssertJ catchThrowable or assertThatThrown for expected exceptions
    • Avoid using JUnit expected exception support such as ExpectedException Rule or Test annotation expected element. Example: @Test(expected = Exception.class) 
    • Remember to catch specific exceptions, not whole parent classes. If someone adds an additional exception, will it break your test?