Note: Please see The Green Report for latest recommendations in writing DistributedTests.

A DistributedTest in Geode is a test written with Geode's DUnit framework involving multiple members of a distributed system.

• Should use JUnit 4 syntax
• File name ends with *DistributedTest or *DUnitTest
• Should be part of the distributedTest source set folder (<geode-module-dir>/distributedTest/java)
• Should generally complete in seconds
• May test interactions between multiple JVMs in a Geode cluster or via networking or via file system
• Typically uses black-box testing but may include some white-box testing and helps guarantee external quality (feature delivers value to User)
• A DistributedTest may do any of the following:
  • communicate across the network
  • access the file system
  • require anything special in the environment (such as editing config files or running an external process

Most DistributedTests use lambdas to invoke SerializableRunnables or SerializableCallables on additional JVMs. Depending on the exact syntax of the lambda, the test class itself may be serialized, so you need to have the test implement Serializable. Non-serializable fields on the test will need to either be transient or serializable.

If you write an DistributedTest to expose a bug and confirm its fix, then you should name it with the suffix *RegressionTest. Otherwise you should use the suffix *DistributedTest or *DUnitTest.

Preferred Testing Frameworks and Libraries for Geode DistributedTests

Overall Framework

• JUnit 4 (including Rules)
• Paramerized – JUnit 4.12 has a bug which prevents Parameterized and Category from working together properly
  • Please use CategoryWithParameterizedRunnerFactory from Geode when using Parameterized (see GEODE-1350)
• TemporaryFolder – provides isolated file system that will be cleaned up after the test
• TestName – provides name of the test method currently being tested
• ErrorCollector – use this when performing assertions in callbacks such as CacheListener

Assertion Library

• AssertJ – richer API with better failure messages than JUnit 4 Assert

Mocking Library

• Mockito
• PowerMock – use this as a last resort – refactoring code to facilitate better testing is preferred

Expected Exceptions

• Use AssertJ or Catch-Exception instead of JUnit 4 Rule ExpectedException
• AssertJ (assertThatThrownBy) – this is the cleanest and preferred way to test for expected exception
• Catch-Exception – better support for complex nesting of exceptions or exceptions with unusual APIs

Additional 3rd Party Libraries for UnitTests

• JUnitParams – provides test method parameterization (usually better to use than JUnit 4 Parameterized)
• System Rules – powerful set of JUnit 4 Rules
  • RestoreSystemProperties
• Awaitility – useful for awaiting asynchronous conditions

Custom Geode JUnit 4 Rules

• ExpectedTimeoutRule – verifies that an API times out and potentially throws TimeoutException
• JarFileRule – creates a Jar file in an internal TemporaryFolder containing a dynamically generated class
• RequiresGeodeHome – asserts precondition that GEODE_HOME system property is specified
• RestoreLocaleRule – restores JVM to original Locale
• RestoreTCCLRule – restores Thread Context Class Loader
• RetryRule – retries a test until it passes or exceeds specified number of retries
• TemporaryFileRule – destroys specified files or directories that aren't created in a TemporaryFolder
• UseJacksonForJsonPathRule – specifies that JsonPath should use Jackson instead of its default JSON library
• GfshRule – allows use of Gfsh script string to fork a JVM process in an IntegrationTest

Custom Geode DUnit Rules

• CacheXmlRule – uses CacheCreation to generate cache.xml
• DistributedRule – launches the DUnit VMs
• CleanupDUnitVMsRule – bounces all DUnit VMs
• DistributedExecutorServiceRule – provides ExecutorService for concurrent testing
• DistributedRestoreSystemProperties – extends RestoreSystemProperties for all DUnit VMs
• DistributedUseJacksonForJsonPathRule – extends UseJacksonForJsonPathRule for all DUnit VMs
• SerializableTestName – extends TestName to be usable within all DUnit VMs
• SerializableTemporaryFolder – extends TemporaryFolder to be usable within all DUnit VMs
• SharedCountersRule – shared counter API for all DUnit VMs
• SharedErrorCollector – extends ErrorCollector for all DUnit VMs

Custom Geode API DUnit Rules

• CacheRule – simple Cache creation and reference for all DUnit VMs
• ClientCacheRule – simple ClientCache creation and reference for all DUnit VMs
• ClusterStartupRule – starts cluster and creates Server or Locator in any DUnit VM

Custom Geode DUnit API Classes

• AsyncInvocation – return type from VM#invokeAsync and implements Future
• Disconnect – disconnect in all or specified VM
• Host – provides the DUnit VMs
• VM – provides invocation API for a DUnit VM

Custom Geode DUnit TestCases

• DistributedTestCase – extend this class for an old-school DUnit test
• CacheTestCase – adds Cache API to DistributedTestCase

Examples of actual tests

Examples of some good DistributedTests:

• ClientProxyWithDeltaDistributedTest

• ClientWithInterestFailoverDistributedTest
• FunctionExecution_ExceptionDUnitTest
• PartitionedRegionCloseDistributedTest
• PRCacheListenerDistributedTest
• PRCacheListenerWithInterestPolicyAllDistributedTest
• PRCreationTotalNumBucketsDistributedTest
• PREntryIdleExpirationDistributedTest
• RegisterInterestServerMetaDataDistributedTest
• ReplicateCacheListenerDistributedTest
• ReplicateEntryIdleExpirationDistributedTest
• RegionExpirationDistributedTest (example of using Mockito Spy CacheListener instead of rolling your own CacheListener impl for testing)

Examples of good RegressionTests which are DistributedTests:

• BucketRebalanceStatRegressionTest
• ClientFunctionTimeoutRegressionTest
• CreateAndLocalDestroyInTXRegressionTest
• EntriesDoNotExpireDuringGiiRegressionTest (good example of SerializableCountersRule and SharedErrorCollector usage)
• GatewayLegacyAuthenticationRegressionTest

General form of a DistributedTest:

package org.apache.geode.internal.cache.backup;

import static org.apache.geode.test.dunit.Disconnect.disconnectAllFromDS;
import static org.apache.geode.test.dunit.Disconnect.disconnectFromDS;
import static org.apache.geode.test.dunit.Host.getHost;
import static org.apache.geode.test.dunit.IgnoredException.addIgnoredException;
import static org.apache.geode.test.dunit.Invoke.invokeInEveryVM;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;
import static org.awaitility.Awaitility.await;
 
import org.junit.After;
import org.junit.Before;
import org.junit.ClassRule;
import org.junit.Rule;
import org.junit.Test;
import org.junit.experimental.categories.Category;
 
import org.apache.geode.test.dunit.VM;
import org.apache.geode.test.dunit.rules.CacheRule;
import org.apache.geode.test.dunit.rules.DistributedRestoreSystemProperties;
import org.apache.geode.test.dunit.rules.DistributedTestRule;
import org.apache.geode.test.dunit.rules.SharedCountersRule;
import org.apache.geode.test.dunit.rules.SharedErrorCollector;
import org.apache.geode.test.junit.categories.DistributedTest;

/**
 * Use DistributedRule to launch additional JVMs to use in the test.<br>
 * Use CacheRule to keep references to Cache in each JVM which will be cleaned up.<br>
 * Use DistributedRestoreSystemProperties to restore properties across all JVMs.<br>
 * Use SharedCountersRule for blackboard style counters in all JVMs.<br>
 * Use SharedErrorCollector to perform assertions in callbacks/listeners.<br>
 * Use SerializableTemporaryFolder to provide temporary directories for all JVMs.<br>
 * <p>
 * Above static imports show the most commonly used DUnit utilities, AssertJ and 
 * Awaitility.
 */ 
@SuppressWarnings("serial")
public class OpsWithChangingMembershipDistributedTest implements Serializable {
   
  private VM vm0;
  private VM vm1;
  private VM vm2;
  private VM vm3;

  @Rule
  public DistributedRule distributedRule = new DistributedRule();

  @Rule
  public CacheRule cacheRule = new CacheRule();
 
  @Rule
  public DistributedRestoreSystemProperties restoreSystemProperties =
      new DistributedRestoreSystemProperties();

  @Rule
  public SharedErrorCollector errorCollector = new SharedErrorCollector();

  @Rule
  public SerializableTemporaryFolder temporaryFolder = new SerializableTemporaryFolder();
 
  @Before
  public void setUp() {
    // arrange: perform as much of your setUp as possible in @Before
 
    vm0.invoke(() -> {
      Cache cache = cacheRule.getOrCreateCache();
      // additional setup
    });
  }
 
  @Test
  public void incrementalBackupWithMissingBaselineThrows() {
    // manipulate VMs with invoke or invokeAsync
  }
}