A DistributedTest in Geode is a test involving multiple members of a distributed system.

• Should use JUnit 4 syntax
• File name ends with *DistributedTest or *DUnitTest
• Should use Category annotation DistributedTest
• Should generally complete in seconds
• May test interactions between multiple JVMs in a Geode cluster or via networking or via file system
• Typically uses black-box testing but may include some white-box testing and helps guarantee external quality (feature delivers value to User)
• A DistributedTest may do any of the following:
  • communicate across the network
  • access the file system
  • require anything special in the environment (such as editing config files or running an external process

Additional notes

If you write an DistributedTest to expose a bug and then confirm its fix, then you should name it with the suffix *RegressionTest. Otherwise you should use the suffix *DistributedTest or *DUnitTest.

Preferred Testing Frameworks and Libraries for Geode DistributedTests

Overall Framework

• JUnit 4 (including Rules)
• Paramerized – JUnit 4.12 has a bug which prevents Parameterized and Category from working together properly
  • Please use CategoryWithParameterizedRunnerFactory from Geode when using Parameterized (see GEODE-1350)
• TemporaryFolder – provides isolated file system that will be cleaned up after the test
• TestName – provides name of the test method currently being tested
• ErrorCollector – use this when performing assertions in callbacks such as CacheListener

Assertion Library

• AssertJ – richer API with better failure messages than JUnit 4 Assert

Mocking Library

• Mockito
• PowerMock – use this as a last resort – refactoring code to facilitate better testing is preferred

Expected Exceptions

• Use AssertJ or Catch-Exception instead of JUnit 4 Rule ExpectedException
• AssertJ (assertThatThrownBy) – this is the cleanest and preferred way to test for expected exception
• Catch-Exception – better support for complex nesting of exceptions or exceptions with unusual APIs

Additional 3rd Party Libraries for UnitTests

• JUnitParams – provides test method parameterization (usually better to use than JUnit 4 Parameterized)
• System Rules – powerful set of JUnit 4 Rules
  • RestoreSystemProperties
• Awaitility – useful for awaiting asynchronous conditions

Custom Geode JUnit 4 Rules and Testing Tools

• ExecutorServiceRule – provides ExecutorService for concurrent testing
• ExpectedTimeoutRule – verifies that an API times out and potentially throws TimeoutException
• JarFileRule – creates a Jar file in an internal TemporaryFolder containing a dynamically generated class
• RequiresGeodeHome – asserts precondition that GEODE_HOME system property is specified
• RestoreLocaleRule – restores JVM to original Locale
• RestoreTCCLRule – restores Thread Context Class Loader
• RetryRule – retries a test until it passes or exceeds specified number of retries
• TemporaryFileRule – destroys specified files or directories that aren't created in a TemporaryFolder
• UseJacksonForJsonPathRule – specifies that JsonPath should use Jackson instead of its default JSON library
• GfshRule – allows use of Gfsh script string to fork a JVM process in an IntegrationTest

Examples of actual tests

Examples of some good DistributedTests:

• ClientProxyWithDeltaDistributedTest

• ClientWithInterestFailoverDistributedTest
• FunctionExecution_ExceptionDUnitTest
• PartitionedRegionCloseDistributedTest
• PRCacheListenerDistributedTest
• PRCacheListenerWithInterestPolicyAllDistributedTest
• PRCreationTotalNumBucketsDistributedTest
• PREntryIdleExpirationDistributedTest
• RegisterInterestServerMetaDataDistributedTest
• ReplicateCacheListenerDistributedTest
• ReplicateEntryIdleExpirationDistributedTest

Examples of good RegressionTests which are DistributedTests:

• BucketRebalanceStatRegressionTest
• ClientFunctionTimeoutRegressionTest
• CreateAndLocalDestroyInTXRegressionTest
• EntriesDoNotExpireDuringGiiRegressionTest (good example of SerializableCountersRule and SharedErrorCollector usage)
• GatewayLegacyAuthenticationRegressionTest

General form of a DistributedTest:

package org.apache.geode.internal.cache.backup;


import static org.apache.geode.test.dunit.Disconnect.disconnectAllFromDS;
import static org.apache.geode.test.dunit.Disconnect.disconnectFromDS;
import static org.apache.geode.test.dunit.Host.getHost;
import static org.apache.geode.test.dunit.IgnoredException.addIgnoredException;
import static org.apache.geode.test.dunit.Invoke.invokeInEveryVM;
import static org.assertj.core.api.Assertions.assertThat;
import static org.awaitility.Awaitility.await;
 
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
 
import org.apache.geode.test.dunit.VM;
import org.apache.geode.test.dunit.rules.CacheRule;
import org.apache.geode.test.dunit.rules.DistributedRestoreSystemProperties;
import org.apache.geode.test.dunit.rules.DistributedTestRule;
import org.apache.geode.test.dunit.rules.SharedCountersRule;
import org.apache.geode.test.dunit.rules.SharedErrorCollector;
import org.apache.geode.test.junit.categories.DistributedTest;


/**
 * Use DistributedTestRule to launch additional JVMs to use in the test.<br>
 * Use CacheRule to keep references to Cache in each JVM which will be cleaned up.<br>
 * Use DistributedRestoreSystemProperties to restore properties across all JVMs.<br>
 * Use SharedCountersRule for blackboard style counters in all JVMs.<br>
 * Use SharedErrorCollector to perform assertions in callbacks/listeners.<br>
 * Use SerializableTemporaryFolder to provide temporary directories for all JVMs.<br>
 * <p>
 * Above static imports show the most commonly used DUnit utilities, AssertJ and 
 * Awaitility.
 */ 
@Category(DistributedTest.class)
@SuppressWarnings("serial")
public class OpsWithChangingMembershipDistributedTest implements Serializable {
   
  private VM vm0;
  private VM vm1;
  private VM vm2;
  private VM vm3;


  @ClassRule
  public static DistributedTestRule distributedTestRule = new DistributedTestRule();

  @Rule
  public CacheRule cacheRule = new CacheRule();
 
  @Rule
  public DistributedRestoreSystemProperties restoreSystemProperties =
      new DistributedRestoreSystemProperties();

  @Rule
  public SharedErrorCollector errorCollector = new SharedErrorCollector();

  @Rule
  public SerializableTemporaryFolder temporaryFolder = new SerializableTemporaryFolder();
 
  @Before
  public void setUp() {
    // arrange: perform as much of your setUp as possible in @Before
 
    vm0.invoke(() -> {
      Cache cache = cacheRule.getOrCreateCache();
      // additional setup
    });
  }
 
  @Test
  public void incrementalBackupWithMissingBaselineThrows() {
    // manipulate VMs with invoke or invokeAsync
  }
}