DUnit tests are JUnit tests that extend CacheTestCase or DistributedTestCase and run code in more than one VM. Because these tests tend to involve multiple members in a distributed system, these tests are more prone to potential race conditions. Here are some tips for tracking down failures and fixing these tests.

Running a single junit test

Panel

./gradlew --no-daemon geode-core:test -Dtest.single=BucketRegionJUnitTest

Running a single dunit test

You can use the standard Gradle properties to run a single test. Or, better yet, run the test in your IDE to help you debug it.

Panel
./gradlew gemfiregeode-core:distributedTest -DdistributedTest.single=PartitionedRegionTestUtilsDUnitTest

...

To specify a logLevl:

./gradlew geode-core:distributedTest -DlogLevel=debug -DdistributedTest.single=PartitionedRegionTestUtilsDUnitTest

Running a test multiple times

One way to run a test a lot of times is to add a new method to your case test that runs a problematic method many times. Here's an example

Code Block
public void testLoop() throws Exception { for(int i=0; i < 200; i++) { testGetBucketOwners(); tearDown(); setUp(); } }

...

Running a list of tests

...

To save time, the JVMs are reused between test cases. This can cause issues if previous tests leave the JVM in a bad state. You can see the list of previously executed tests as the first step in the stdout of a test, if you are looking at a failure from Jenkins:

Panel

Previously run tests: [PartitionedRegionHAFailureAndRecoveryDUnitTestPartitionedRegionBucketCreationDistributionDUnitTest, NetSearchMessagingDUnitTest, EvictionObjectSizerDUnitTest, PartitionedRegionBucketCreationDistributionDUnitTest, InterruptsConserveSocketsFalseDUnitTest, PartitionedRegionRedundancyZoneDUnitTest, DeltaFaultInDUnitTest, PutAllGlobalDUnitTest, DeltaPropagationStatsDUnitTest, P2PDeltaPropagationDUnitTest, PartitionedRegionPRIDDUnitTest, TransactionsWithDeltaDUnitTest, Bug33726DUnitTest, PartitionedRegionTestUtilsDUnitTest]

PartitionedRegionTestUtilsDUnitTest, DistTXDebugDUnitTest]

To run the same tests in order, you can add a suite to your JUnit test

Code Block

public static Test suite() {
  final TestSuite Class[]suite classes = new Class[] {TestSuite();
  suite.addTestSuite(PartitionedRegionBucketCreationDistributionDUnitTest.class, InterruptsConserveSocketsFalseDUnitTest.class, PartitionedRegionRedundancyZoneDUnitTest.class, );
  suite.addTestSuite(PartitionedRegionTestUtilsDUnitTest.class});
  //Use the JUnit4TestAdapter  return new TestSuite(classesone of your test has been converted to Junit4 syntax.
  suite.addTest(new JUnit4TestAdapter(DistTXDebugDUnitTest.class));
  return suite;
}

Fixing suspect strings

Distributed tests check the log output from each test for "suspect strings" which include any warnings, errors, or exceptions. If your test is supposed to log one of these strings, you can add an expected exception to the beginning of your test method, or even in your setUp method if all test cases are expected to log this message. There is no need to remove the expected exception, it is automatically removed in tearDown().

Code Block
public void testClientPutWithInterrupt() throws Throwable { addExpectedException("InterruptedException"); //... }

Turn on debug level in dunit tests

There are many ways to run a test case with a specific debug level; the easiest way is to temporarily add the following code to the test program.

Code Block
@Override public Properties getDistributedSystemProperties() { Properties props = new Properties(); props.setProperty("log-level", "debug"); return props; }

Turn on trace level using log4j2.xml

Product code contains some trace code similar to the following:

Code Block

A complete example: Note, it mixed dunit tests and junit tests.
 
package org.apache.geode.internal.cache.wan;
import org.apache.geode.internal.cache.DistributedRegionJUnitTest;
import org.apache.geode.internal.cache.PartitionedRegionRedundancyZoneDUnitTest;
import org.apache.geode.internal.cache.PartitionedRegionTestUtilsDUnitTest;
import org.apache.geode.internal.cache.wan.misc.ShutdownAllPersistentGatewaySenderDUnitTest;
import junit.framework.Test;
import junit.framework.TestCase;
import junit.framework.TestSuite;
@Category(UnitTest.class)
public class MyTestCase extends TestCase {
 
  public static Test suite() {
    Class[] classes = new Class[] {
        CacheClientNotifierDUnitTest.class,

Code Block

      if (logger.isTraceEnabled(LogMarker.TOMBSTONE)) {
        logger.trace(LogMarker.TOMBSTONE, "Destroyed entries sweeper starting with default sleep interval={}", this.expiryTime)ShutdownAllPersistentGatewaySenderDUnitTest.class, 
        PartitionedRegionRedundancyZoneDUnitTest.class, 
        PartitionedRegionTestUtilsDUnitTest.class,
        DistributedRegionJUnitTest.class
        };
    return new TestSuite(classes);
  }

This will not be displayed in a debug level log. To specify a customized log4j2.xml, there are also several ways, but the easiest one is to copy your log4j2.xml into open/gemfire-core/src/main/resources/log4j2.xml. Here is an example of log4j2.xml that turns on LogMarker.TOMBSTONE, LogMarker.DISTRIBUTION, LogMarker.DISK_STORE_MONITOR:

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR" shutdownHook="disable" packages="com.gemstone.gemfire.internal.logging.log4j">
  <Properties>
    <Property name="gemfire-pattern">[%level{lowerCase=true} %date{yyyy/MM/dd HH:mm:ss.SSS z} &lt;%thread&gt; tid=%tid] %message%n%throwable%n</Property>
  </Properties>
  <filters>
    <MarkerFilter marker="DISTRIBUTION" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
    <MarkerFilter marker="DISK_STORE_MONITOR" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
    <MarkerFilter marker="TOMBSTONE" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
    <!--MarkerFilter marker="PERSIST_RECOVERY" onMatch="ACCEPT" onMismatch="NEUTRAL"/-->
    <!--MarkerFilter marker="PERSIST_WRITES" onMatch="ACCEPT" onMismatch="NEUTRAL"/-->
    <!-- have to explicitly DENY all the markers' log, then explicitly add back one by one -->
    <MarkerFilter marker="GEMFIRE_MARKER" onMatch="DENY" onMismatch="NEUTRAL"/>
  </filters>
  <Appenders>
    <Console name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout pattern="${gemfire-pattern}"/>
    </Console>
    <Console name="STDERR" target="SYSTEM_ERR">
      <PatternLayout pattern="${gemfire-pattern}"/>
    </Console>
    <!--RollingFile name="RollingFile" fileName="${filename}"-->
    <File name="Log" fileName="system.log" bufferedIO="true">
      <PatternLayout pattern="${gemfire-pattern}"/>
    </File>
  </Appenders>
  <Loggers>
    <Logger name="com.gemstone" level="INFO" additivity="false">
      <AppenderRef ref="Log"/>
    </Logger>
    <Logger name="com.gemstone.gemfire.distributed.internal" level="TRACE" additivity="false">
      <AppenderRef ref="STDOUT" level="DEBUG"/>
    </Logger>
    <Logger name="com.gemstone.gemfire.cache.client.internal" level="TRACE" additivity="false">
      <AppenderRef ref="STDOUT" level="DEBUG"/>
    </Logger>
    <Logger name="com.gemstone.gemfire.internal.cache" level="TRACE" additivity="false">
      <AppenderRef ref="STDOUT" level="TRACE"/>
    </Logger>
    <!--Logger name="com.gemstone.gemfire.distributed.internal.DistributionManager" level="TRACE" additivity="false">
      <AppenderRef ref="Log" level="DEBUG"/>
    </Logger-->
    <!--Logger name="com.gemstone.gemfire.distributed.internal.DistributionMessage" level="TRACE" additivity="false">
      <AppenderRef ref="Log" level="DEBUG"/>
    </Logger-->
    <Root level="ERROR">
      <!--AppenderRef ref="STDOUT"/-->
    </Root>
  </Loggers>
</Configuration>

Specify bigger heap size for dunit tests

The default heap size for a DUnit test is 768m. We can modify open/build.gradle to use a larger heap size in case the DUnit test runs out of memory.

Code Block

        maxHeapSize '768m'
        jvmArgs = ['-XX:+HeapDumpOnOutOfMemoryError', '-ea']
        systemProperties = [
          'gemfire.DEFAULT_MAX_OPLOG_SIZE' : '10',
          'gemfire.disallowMcastDefaults'  : 'true',
          'jline.terminal'                 : 'jline.UnsupportedTerminal',
        ]

Use new await() to wait for an async event to finish

We've introduced com.jayway.awaitility.Awaitility.await in Geode. There's a better way to replace our old WaitCriterion. await() can also be used in product code, while WaitCriterion is only in DistributedTestCase.


}
 
Or a new version:
@RunWith(Suite.class)
@Suite.SuiteClasses({ CacheClientNotifierDUnitTest.class, DistributedRegionJUnitTest.class })
//@Category(UnitTest.class)
public class MyTestCase2 extends TestCase {
}

Fixing suspect strings

Distributed tests check the log output from each test for "suspect strings" which include any warnings, errors, or exceptions. If your test is supposed to log one of these strings, you can add an expected exception to the beginning of your test method, or even in your setUp method if all test cases are expected to log this message. There is no need to remove the expected exception, it is automatically removed in tearDown().

Code Block
public void testClientPutWithInterrupt() throws Throwable { addExpectedException("InterruptedException"); //... }

Turn on debug level in dunit tests

There are many ways to run a test case with a specific debug level; the easiest way is to temporarily add the following code to the test program.

Code Block
@Override public Properties getDistributedSystemProperties() { Properties props = new Properties(); props.setProperty("log-level", "debug"); return props; }

Turn on trace level using log4j2.xml

Product code contains some trace code similar to the following:

Code Block
if (logger.isTraceEnabled(LogMarker.TOMBSTONE)) { logger.trace(LogMarker.TOMBSTONE, "Destroyed entries sweeper starting with default sleep interval={}", this.expiryTime); }

This will not be displayed in a debug level log. To specify a customized log4j2.xml, let's say log4me.xml:

Code Block
./gradlew --no-daemon geode-${section}:distributedTest -Dlog4j.configurationFile=c:\\cygwin64\\home\\zhouxh\\log4me.xml -DdistributedTest.single=$case

Here is an example of log4me.xml that turns on LogMarker.TOMBSTONE, LogMarker.DISTRIBUTION, LogMarker.DISK_STORE_MONITOR:

Code Block

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="ERROR" shutdownHook="disable" packages=org.apache.geode.internal.logging.log4j">
  <Properties>
    <Property name="gemfire-pattern">[%level{lowerCase=true} %date{yyyy/MM/dd HH:mm:ss.SSS z} &lt;%thread&gt; tid=%tid] %message%n%throwable%n</Property>
  </Properties>
  <filters>
    <MarkerFilter marker="DISTRIBUTION" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
    <MarkerFilter marker="DISK_STORE_MONITOR" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
    <MarkerFilter marker="TOMBSTONE" onMatch="ACCEPT" onMismatch="NEUTRAL"/>
    <!--MarkerFilter marker="PERSIST_RECOVERY" onMatch="ACCEPT" onMismatch="NEUTRAL"/-->
    <!--MarkerFilter marker="PERSIST_WRITES" onMatch="ACCEPT" onMismatch="NEUTRAL"/-->
    <!-- have to explicitly DENY all the markers' log, then explicitly add back one by one -->
    <MarkerFilter marker="GEMFIRE_MARKER" onMatch="DENY" onMismatch="NEUTRAL"/>
  </filters>
  <Appenders>
    <Console name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout pattern="${gemfire-pattern}"/>
    </Console>
    <Console name="STDERR" target="SYSTEM_ERR">
      <PatternLayout pattern="${gemfire-pattern}"/>
    </Console>
    <!--RollingFile name="RollingFile" fileName="${filename}"-->
    <File name="Log" fileName="system.log" bufferedIO="true">
      <PatternLayout pattern="${gemfire-pattern}"/>
    </File>
  </Appenders>
  <Loggers>
    <Logger name="org.apache.geode" level="INFO" additivity="false">
      <AppenderRef ref="Log"/>
    </Logger>
    <Logger name="org.apache.geode.distributed.internal" level="TRACE" additivity="false">
      <AppenderRef ref="STDOUT" level="DEBUG"/>
    </Logger>
    <Logger name="org.apache.geode.cache.client.internal" level="TRACE" additivity="false">
      <AppenderRef ref="STDOUT" level="DEBUG"/>
    </Logger>
    <Logger name="org.apache.geode.internal.cache" level="TRACE" additivity="false">
      <AppenderRef ref="STDOUT" level="TRACE"/>
    </Logger>
    <!--Logger name="org.apache.geode.distributed.internal.DistributionManager" level="TRACE" additivity="false">
      <AppenderRef ref="Log" level="DEBUG"/>
    </Logger-->
    <!--Logger name="org.apache.geode.distributed.internal.DistributionMessage" level="TRACE" additivity="false">
      <AppenderRef ref="Log" level="DEBUG"/>
    </Logger-->
    <Root level="ERROR">
      <!--AppenderRef ref="STDOUT"/-->
    </Root>
  </Loggers>
</Configuration>

Specify bigger heap size for dunit tests

The default heap size for a DUnit test is 768m. We can modify open/build.gradle to use a larger heap size in case the DUnit test runs out of memory.

Code Block

        maxHeapSize '768m'
        jvmArgs = ['-XX:+HeapDumpOnOutOfMemoryError', '-ea']
        systemProperties = [
          'gemfire.DEFAULT_MAX_OPLOG_SIZE' : '10',

Code Block

              Cache cache = getCache();
              DistributedTestCase.disconnectFromDS();
              await().atMost(30, SECONDS).until(() -> {return (cache == null || cache.isClosed());});

is equivalent to:            'gemfire.disallowMcastDefaults'  : 'true',
          'jline.terminal'    WaitCriterion waitForCacheClose = new WaitCriterion() {
                @Override: 'jline.UnsupportedTerminal',
        ]

Use new await() to wait for an async event to finish

We've introduced com.jayway.awaitility.Awaitility.await in Geode. There's a better way to replace our old WaitCriterion. await() can also be used in product code, while WaitCriterion is only in DistributedTestCase.

Code Block

         public boolean done() {
             Cache cache = getCache();
   return cache == null || cache.isClosed           DistributedTestCase.disconnectFromDS();
              await().atMost(30, SECONDS).until(() -> {return (cache == null || cache.isClosed());});

is equivalent to:               @Override
              WaitCriterion waitForCacheClose public= Stringnew descriptionWaitCriterion() {
                @Override
  return "Wait for Cache to be closed";
              public boolean done() {
                  return cache ==  }null || cache.isClosed();
                };

                @Override
              DistributedTestCase.waitForCriterion(waitForCacheClose, 30000, 500, true);

Using Mockito to test region class

public String description() {
                  return "Wait for Cache to be closed";
                }
              };
              
              DistributedTestCase.waitForCriterion(waitForCacheClose, 30000, 500, true);

Using Mockito to test region class

Our region classes (LocalRegion, DistributedRegion, PartitionedRegion, BucketRegion, etc) and some other legacy gemfire classes heaily rely on cache, distribute system, distribution manager and other modules to function. Our region classes (LocalRegion, DistributedRegion, PartitionedRegion, BucketRegion, etc) and some other legacy gemfire classes heaily rely on cache, distribute system, distribution manager and other modules to function. Thus it's difficult to mock the class and do the junit test.

...

create a region
create an event with version tag
call the methods of the region
verify the distribution is triggered or not

The challenge is how to mock the required components to create region, event, version tag.

The comTheorg.gemstoneapache.gemfiregeode.test.fake.Fakes class has done the major mocking for us and also defined the expected behaviors between these mock objects. You only need to use it:

Code Block
GemFireCacheImpl cache = Fakes.cache();

In DistributedRegionJUnitTest, it gives an example on how to create a DistributedRegion using the mocking.

Code Block

  protected DistributedRegion prepare(boolean isConcurrencyChecksEnabled) {
    GemFireCacheImpl cache = Fakes.cache(); // using the Fakes to define mock objects and their behavior
    
    // create region attributes and internal region arguments
    RegionAttributes ra = createRegionAttributes(isConcurrencyChecksEnabled);
    InternalRegionArguments ira = new InternalRegionArguments();
    
    doPRSpecific(ira); // for BucketRegion, we need to config some PR attributes
    
    // create a region object
    DistributedRegion region = createAndDefineRegion(isConcurrencyChecksEnabled, ra, ira, cache);
    if (isConcurrencyChecksEnabled) {
      region.enableConcurrencyChecks();
    }
    
    // some additional behaviors to be define for the test
    doNothing().when(region).notifyGatewaySender(any(), any());
    doReturn(true).when(region).hasSeenEvent(any(EntryEventImpl.class));

    return region;
  }

The region created here is not a mock object, but a real region. However, we need to monitor its behavior in mocking.

Code Block

    DistributedRegion region = new DistributedRegion("testRegion", ra, null, cache, ira);
    if (isConcurrencyChecksEnabled) {
      region.enableConcurrencyChecks();
    }
    
    // since it is a real region object, we need to tell mockito to monitor it
    region = Mockito.spy(region);

    doNothing().when(region).distributeUpdate(any(), anyLong(), anyBoolean(), anyBoolean(), any(), anyBoolean());
    doNothing().when(region).distributeDestroy(any(), any());
    doNothing().when(region).distributeInvalidate(any());
    doNothing().when(region).distributeUpdateEntryVersion(any());

BucketRegionJUnitTest is a subclass of DistributedRegionJUnitTest to reuse most of the logic. However it needs to implement the doPRSpecific() for more mocking.

Code Block

  protected void doPRSpecific(InternalRegionArguments ira) {
    // PR specific
    PartitionedRegion pr = mock(PartitionedRegion.class);
    BucketAdvisor ba = mock(BucketAdvisor.class);
    ReadWriteLock primaryMoveLock = new ReentrantReadWriteLock();
    Lock activeWriteLock = primaryMoveLock.readLock();
    when(ba.getActiveWriteLock()).thenReturn(activeWriteLock);
    when(ba.isPrimary()).thenReturn(true);
    
    ira.setPartitionedRegion(pr)
      .setPartitionedRegionBucketRedundancy(1)
      .setBucketAdvisor(ba);
  }

Overall, the Fakes.java, DistributedRegionJUnitTest, BucketRegionJUnitTest are a good start to build a shareable template to write mock-based test cases for gemfire legacy components.

Summary of a junit test for BucketRegion:

Code Block

    GemFireCacheImpl cache = Fakes.cache();
 
    RegionAttributes ra = createRegionAttributes(isConcurrencyChecksEnabled);
    InternalRegionArguments ira = new InternalRegionArguments();
 
    // specify more mock behaviors for ParitionedRegion and BucketRegion
    PartitionedRegion pr = mock(PartitionedRegion.class);
    BucketAdvisor ba = mock(BucketAdvisor.class);
    ReadWriteLock primaryMoveLock = new ReentrantReadWriteLock();
    Lock activeWriteLock = primaryMoveLock.readLock();
    when(ba.getActiveWriteLock()).thenReturn(activeWriteLock);
    when(ba.isPrimary()).thenReturn(true);
    ira.setPartitionedRegion(pr)
      .setPartitionedRegionBucketRedundancy(1)
      .setBucketAdvisor(ba);
 
    // create Bucket Region
    BucketRegion br = new BucketRegion("testRegion", ra, null, cache, ira);
    // since br is a real bucket region object, we need to tell mockito to monitor it
    br = Mockito.spy(br);
 
    doNothing().when(br).distributeUpdateOperation(any(), anyLong());
    doNothing().when(br).distributeDestroyOperation(any());
    doNothing().when(br).distributeInvalidateOperation(any());
    doNothing().when(br).distributeUpdateEntryVersionOperation(any());
    doNothing().when(br).checkForPrimary();
    doNothing().when(br).handleWANEvent(any());
    doReturn(false).when(br).needWriteLock(any());
    
    if (isConcurrencyChecksEnabled) {
      region.enableConcurrencyChecks();
    }
    
    doNothing().when(region).notifyGatewaySender(any(), any());
    doReturn(true).when(region).hasSeenEvent(any(EntryEventImpl.class));
 
    EntryEventImpl event = createDummyEvent(region);
    VersionTag tag = createVersionTag(false);
    event.setVersionTag(tag);
 
    br.virtualPut(event, false, false, null, false, 12345L, false);
    // verify the result
    if (cnt > 0) {
      verify(br, times(cnt)).distributeUpdateOperation(eq(event), eq(12345L));
    } else {
      verify(br, never()).distributeUpdateOperation(eq(event), eq(12345L));
    }

Running a single hydra test, continously if necessary

We often need to reproduce a hydra test failure. The following scripts are very useful in practice. It can keep running one specific hydra test until it failed. I added another parameter to only stop when a specific failure happened.

There're following files: runbt.sh, local.conf, testforever.sh

runbt.sh

Code Block

#! /bin/sh
exec 1>&1
exec 2>&1
exec 0>&1
set -v
date
##
rm batterytest.log
rm oneliner.txt
rm summ*.txt
################################################################################
# edit this section as desired to choose
#   1) java version
#   2) checkout directory
#   3) an open or closed build
JAVA_HOME=/export/gcm/where/jdk/1.8.0_45/x86_64.linux
checkoutDir=/export/aruba2/users/zhouxh/wan_develop/git3/gemfire
buildRoot=$checkoutDir     #  when build is redirected
 
# choose to run open or closed
# open
#buildType=open
#GEMFIRE=$buildRoot/$buildType/gemfire-assembly/build/install/geode
# closed
buildType=closed
GEMFIRE=$buildRoot/$buildType/pivotalgf-assembly/build/install/pivotal-gemfire
################################################################################
PATH=$JAVA_HOME/bin:$PATH
JTESTS=$buildRoot/closed/gemfire-test/build/resources/test
CLASSPATH=$GEMFIRE/lib/geode-core-9.0.0-SNAPSHOT.jar:$JTESTS:$JTESTS/../../classes/test:$JTESTS/../../classes/hydraClasses:$GEMFIRE/lib/gemfire-core-dependencies.jar:$GEMFIRE/lib/log4j-api-2.5.jar:$GEMFIRE/lib/log4j-core-2.5.jar:$JTESTS/../extraJars/groovy-all-2.4.3.jar
BT=batt.bt
if [ "$1" != "" ]; then
  BT="$1"
fi
echo "BT=$BT"
CONF=./local.conf
if [ "$2" != "" ]; then
  CONF="$2"
fi
echo "local.conf=$CONF"
java -cp $CLASSPATH \
  -DRELEASE_DIR=/export/gcm/where/gemfire/releases \
  -DprovideBugReportTemplate=true \
  -DprovideRegressionSummary=true \
  -DlocalConf=$CONF \
  -DmoveRemoteDirs=true \
  -DremovePassedTest=true \
  -DnukeHungTest=true \
  -DJTESTS=$JTESTS \
  -DGEMFIRE=$GEMFIRE \
  -DtestFileName=$BT \
  -DnumTimesToRun=1 batterytest.BatteryTest

2. local.conf

Code Block

// hydra.GemFirePrms-logLevel=fine;
// hydra.VmPrms-extraVMArgs = "-DDistributionManager.VERBOSE=true -ea";
// hydra.GemFirePrms-lockMemory = ONEOF true false FOENO;
// hydra.VmPrms-extraClassPaths += "$GEMFIRE/lib/jna-4.0.0.jar";
// hydra.GemFirePrms-extraLocatorClassPath += "$GEMFIRE/lib/jna-4.0.0.jar";
// hydra.Prms-maxResultWaitSec=1200;
// hydra.Prms-randomSeed=1427554905641;
// newWan.WANOperationsClientPrms-randModValueForDebug = true;
// hydra.RegionPrms-concurrencyChecksEnabled = true;

An advanced local.conf to run on multiple hosts using hostdirs.

Code Block

hydra.HostPrms-hostNames =
  fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\",  1, 4)" ncf
  fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\",  1, 4)" ncf
  fcn "newWan.WanTestConfigFcns.pool(\"w2-2013-lin-03 w2-2013-lin-04 w2-2013-lin-14 w2-2013-lin-15\",  1, 4)" ncf
  ;
hydra.HostPrms-resourceDirBaseMapFileName = /home/xzhou/bin/hostdirs.prop;

3. testforever.sh

Code Block

#!/bin/bash
testfile=$1
testcase=$2
greppattern=$3
if [ "$testcase" = "" ]; then
    echo "Usage: ./testforever.sh testX.bt testcase [greppattern]"
    exit 1
fi
while [ 1 -eq 1 ]
do
./runbt.sh ${testfile}
if [ -f ${testcase}*/errors.txt ]; then
  date >> merged_errors.txt
  cat ${testcase}*/errors.txt >> merged_errors.txt
  if [ "$greppattern" != "" ]; then
    grep "${greppattern}" ${testcase}*/errors.txt
    if [ $? -eq 0 ]; then
      ${testcase}*/nukerun.sh
      break
    else
      ${testcase}*/nukerun.sh
      sleep 30
      rm -rf ${testcase}*
    fi
  else
    ${testcase}*/nukerun.sh
    break
  fi
else
    ${testcase}*/nukerun.sh
    sleep 30
    rm -rf ${testcase}*
fi
done

An example of using the scripts:

If only to run it once:

./runbt.sh ./test52028.bt

To reproduce bug 52028, run:

./testforever.sh ./test52028.bt p2pEventTransportFilterHA

It will run until the hydra test fails.

However, it might fail due to other reasons, while I only care about reproducing NullPointerException. So run following command instead:

./testforever.sh test52028.bt p2pEventTransportFilterHA NullPointerException

Under current directory, you will find a merged_errors.txt file, which contains all the errors ever happened before the script detected a NullPointerException.


    // since it is a real region object, we need to tell mockito to monitor it
    region = Mockito.spy(region);

    doNothing().when(region).distributeUpdate(any(), anyLong(), anyBoolean(), anyBoolean(), any(), anyBoolean());
    doNothing().when(region).distributeDestroy(any(), any());
    doNothing().when(region).distributeInvalidate(any());
    doNothing().when(region).distributeUpdateEntryVersion(any());

BucketRegionJUnitTest is a subclass of DistributedRegionJUnitTest to reuse most of the logic. However it needs to implement the doPRSpecific() for more mocking.

Code Block

  protected void doPRSpecific(InternalRegionArguments ira) {
    // PR specific
    PartitionedRegion pr = mock(PartitionedRegion.class);
    BucketAdvisor ba = mock(BucketAdvisor.class);
    ReadWriteLock primaryMoveLock = new ReentrantReadWriteLock();
    Lock activeWriteLock = primaryMoveLock.readLock();
    when(ba.getActiveWriteLock()).thenReturn(activeWriteLock);
    when(ba.isPrimary()).thenReturn(true);
    
    ira.setPartitionedRegion(pr)
      .setPartitionedRegionBucketRedundancy(1)
      .setBucketAdvisor(ba);
  }

Overall, the Fakes.java, DistributedRegionJUnitTest, BucketRegionJUnitTest are a good start to build a shareable template to write mock-based test cases for gemfire legacy components.

Summary of a junit test for BucketRegion:

Code Block

    GemFireCacheImpl cache = Fakes.cache();
 
    RegionAttributes ra = createRegionAttributes(isConcurrencyChecksEnabled);
    InternalRegionArguments ira = new InternalRegionArguments();
 
    // specify more mock behaviors for ParitionedRegion and BucketRegion
    PartitionedRegion pr = mock(PartitionedRegion.class);
    BucketAdvisor ba = mock(BucketAdvisor.class);
    ReadWriteLock primaryMoveLock = new ReentrantReadWriteLock();
    Lock activeWriteLock = primaryMoveLock.readLock();
    when(ba.getActiveWriteLock()).thenReturn(activeWriteLock);
    when(ba.isPrimary()).thenReturn(true);
    ira.setPartitionedRegion(pr)
      .setPartitionedRegionBucketRedundancy(1)
      .setBucketAdvisor(ba);
 
    // create Bucket Region
    BucketRegion br = new BucketRegion("testRegion", ra, null, cache, ira);
    // since br is a real bucket region object, we need to tell mockito to monitor it
    br = Mockito.spy(br);
 
    doNothing().when(br).distributeUpdateOperation(any(), anyLong());
    doNothing().when(br).distributeDestroyOperation(any());
    doNothing().when(br).distributeInvalidateOperation(any());
    doNothing().when(br).distributeUpdateEntryVersionOperation(any());
    doNothing().when(br).checkForPrimary();
    doNothing().when(br).handleWANEvent(any());
    doReturn(false).when(br).needWriteLock(any());
    
    if (isConcurrencyChecksEnabled) {
      region.enableConcurrencyChecks();
    }
    
    doNothing().when(region).notifyGatewaySender(any(), any());
    doReturn(true).when(region).hasSeenEvent(any(EntryEventImpl.class));
 
    EntryEventImpl event = createDummyEvent(region);
    VersionTag tag = createVersionTag(false);
    event.setVersionTag(tag);
 
    br.virtualPut(event, false, false, null, false, 12345L, false);
    // verify the result
    if (cnt > 0) {
      verify(br, times(cnt)).distributeUpdateOperation(eq(event), eq(12345L));
    } else {
      verify(br, never()).distributeUpdateOperation(eq(event), eq(12345L));
    }

...

Space shortcuts

Page tree

Versions Compared

Old Version 10

New Version Current

Key

Running a single junit test

Running a single dunit test

Running a test multiple times

Running a list of tests

Fixing suspect strings

Turn on debug level in dunit tests

Turn on trace level using log4j2.xml

Specify bigger heap size for dunit tests

Use new await() to wait for an async event to finish

Fixing suspect strings

Turn on debug level in dunit tests

Turn on trace level using log4j2.xml

Specify bigger heap size for dunit tests

Use new await() to wait for an async event to finish

Using Mockito to test region class

Using Mockito to test region class

Running a single hydra test, continously if necessary

Space shortcuts

Page tree

Page History

Versions Compared

Old Version 10

New Version Current

Key

Running a single junit test

Running a single dunit test

Running a test multiple times

Running a list of tests

Fixing suspect strings

Turn on debug level in dunit tests

Turn on trace level using log4j2.xml

Specify bigger heap size for dunit tests

Use new await() to wait for an async event to finish

Fixing suspect strings

Turn on debug level in dunit tests

Turn on trace level using log4j2.xml

Specify bigger heap size for dunit tests

Use new await() to wait for an async event to finish

Using Mockito to test region class

Using Mockito to test region class

Running a single hydra test, continously if necessary