...
DUnit tests are JUnit tests that extend CacheTestCase or DistributedTestCase and run code in more than one VM. Because these tests tend to involve multiple members in a distributed system, these tests are more prone to potential race conditions. Here are some tips for tracking down failures and fixing these tests.
Running a single junit test
Panel |
---|
./gradlew --no-daemon geode-core:test -Dtest.single=BucketRegionJUnitTest
|
Running a single dunit test
You can use the standard Gradle properties to run a single test. Or, better yet, run the test in your IDE to help you debug it.
...
Panel |
---|
./gradlew geode-core:distributedTest -DdistributedTest.single=PartitionedRegionTestUtilsDUnitTest
To specify a logLevl: ./gradlew geode-core:distributedTest -DlogLevel=debug -DdistributedTest.single=PartitionedRegionTestUtilsDUnitTest |
Running a test multiple times
One way to run a test a lot of times is to add a new method to your case test that runs a problematic method many times. Here's an example
...
Code Block |
---|
public void testLoop() throws Exception { for(int i=0; i < 200; i++) { testGetBucketOwners(); tearDown(); setUp(); } } |
Running a list of tests
To save time, the JVMs are reused between test cases. This can cause issues if previous tests leave the JVM in a bad state. You can see the list of previously executed tests as the first step in the stdout of a test, if you are looking at a failure from Jenkins:
...
Code Block |
---|
A complete example: Note, it mixed dunit tests and junit tests. package com.gemstone.gemfire.internal.cache.wan; import com.gemstone.gemfire.internal.cache.DistributedRegionJUnitTest; import com.gemstone.gemfire.internal.cache.PartitionedRegionRedundancyZoneDUnitTest; import com.gemstone.gemfire.internal.cache.PartitionedRegionTestUtilsDUnitTest; import com.gemstone.gemfire.internal.cache.wan.misc.ShutdownAllPersistentGatewaySenderDUnitTest; import junit.framework.Test; import junit.framework.TestCase; import junit.framework.TestSuite; @Category(UnitTest.class) public class MyTestCase extends TestCase { public static Test suite() { Class[] classes = new Class[] { CacheClientNotifierDUnitTest.class, ShutdownAllPersistentGatewaySenderDUnitTest.class, PartitionedRegionRedundancyZoneDUnitTest.class, PartitionedRegionTestUtilsDUnitTest.class, DistributedRegionJUnitTest.class }; return new TestSuite(classes); } } Or a new version: @RunWith(Suite.class) @Suite.SuiteClasses({ CacheClientNotifierDUnitTest.class, DistributedRegionJUnitTest.class }) //@Category(UnitTest.class) public class MyTestCase2 extends TestCase { } |
Fixing suspect strings
Distributed tests check the log output from each test for "suspect strings" which include any warnings, errors, or exceptions. If your test is supposed to log one of these strings, you can add an expected exception to the beginning of your test method, or even in your setUp method if all test cases are expected to log this message. There is no need to remove the expected exception, it is automatically removed in tearDown().
...
Code Block |
---|
public void testClientPutWithInterrupt() throws Throwable { addExpectedException("InterruptedException"); //... } |
Turn on debug level in dunit tests
There are many ways to run a test case with a specific debug level; the easiest way is to temporarily add the following code to the test program.
...
Code Block |
---|
@Override public Properties getDistributedSystemProperties() { Properties props = new Properties(); props.setProperty("log-level", "debug"); return props; } |
Turn on trace level using log4j2.xml
Product code contains some trace code similar to the following:
...
Code Block |
---|
<?xml version="1.0" encoding="UTF-8"?> <Configuration status="ERROR" shutdownHook="disable" packages="com.gemstone.gemfire.internal.logging.log4j"> <Properties> <Property name="gemfire-pattern">[%level{lowerCase=true} %date{yyyy/MM/dd HH:mm:ss.SSS z} <%thread> tid=%tid] %message%n%throwable%n</Property> </Properties> <filters> <MarkerFilter marker="DISTRIBUTION" onMatch="ACCEPT" onMismatch="NEUTRAL"/> <MarkerFilter marker="DISK_STORE_MONITOR" onMatch="ACCEPT" onMismatch="NEUTRAL"/> <MarkerFilter marker="TOMBSTONE" onMatch="ACCEPT" onMismatch="NEUTRAL"/> <!--MarkerFilter marker="PERSIST_RECOVERY" onMatch="ACCEPT" onMismatch="NEUTRAL"/--> <!--MarkerFilter marker="PERSIST_WRITES" onMatch="ACCEPT" onMismatch="NEUTRAL"/--> <!-- have to explicitly DENY all the markers' log, then explicitly add back one by one --> <MarkerFilter marker="GEMFIRE_MARKER" onMatch="DENY" onMismatch="NEUTRAL"/> </filters> <Appenders> <Console name="STDOUT" target="SYSTEM_OUT"> <PatternLayout pattern="${gemfire-pattern}"/> </Console> <Console name="STDERR" target="SYSTEM_ERR"> <PatternLayout pattern="${gemfire-pattern}"/> </Console> <!--RollingFile name="RollingFile" fileName="${filename}"--> <File name="Log" fileName="system.log" bufferedIO="true"> <PatternLayout pattern="${gemfire-pattern}"/> </File> </Appenders> <Loggers> <Logger name="com.gemstone" level="INFO" additivity="false"> <AppenderRef ref="Log"/> </Logger> <Logger name="com.gemstone.gemfire.distributed.internal" level="TRACE" additivity="false"> <AppenderRef ref="STDOUT" level="DEBUG"/> </Logger> <Logger name="com.gemstone.gemfire.cache.client.internal" level="TRACE" additivity="false"> <AppenderRef ref="STDOUT" level="DEBUG"/> </Logger> <Logger name="com.gemstone.gemfire.internal.cache" level="TRACE" additivity="false"> <AppenderRef ref="STDOUT" level="TRACE"/> </Logger> <!--Logger name="com.gemstone.gemfire.distributed.internal.DistributionManager" level="TRACE" additivity="false"> <AppenderRef ref="Log" level="DEBUG"/> </Logger--> <!--Logger name="com.gemstone.gemfire.distributed.internal.DistributionMessage" level="TRACE" additivity="false"> <AppenderRef ref="Log" level="DEBUG"/> </Logger--> <Root level="ERROR"> <!--AppenderRef ref="STDOUT"/--> </Root> </Loggers> </Configuration> |
Specify bigger heap size for dunit tests
The default heap size for a DUnit test is 768m. We can modify open/build.gradle to use a larger heap size in case the DUnit test runs out of memory.
...
Code Block |
---|
maxHeapSize '768m' jvmArgs = ['-XX:+HeapDumpOnOutOfMemoryError', '-ea'] systemProperties = [ 'gemfire.DEFAULT_MAX_OPLOG_SIZE' : '10', 'gemfire.disallowMcastDefaults' : 'true', 'jline.terminal' : 'jline.UnsupportedTerminal', ] |
Use new await() to wait for an async event to finish
We've introduced com.jayway.awaitility.Awaitility.await in Geode. There's a better way to replace our old WaitCriterion. await() can also be used in product code, while WaitCriterion is only in DistributedTestCase.
...
Code Block |
---|
Cache cache = getCache(); DistributedTestCase.disconnectFromDS(); await().atMost(30, SECONDS).until(() -> {return (cache == null || cache.isClosed());}); is equivalent to: WaitCriterion waitForCacheClose = new WaitCriterion() { @Override public boolean done() { return cache == null || cache.isClosed(); } @Override public String description() { return "Wait for Cache to be closed"; } }; DistributedTestCase.waitForCriterion(waitForCacheClose, 30000, 500, true); |
Using Mockito to test region class
Our region classes (LocalRegion, DistributedRegion, PartitionedRegion, BucketRegion, etc) and some other legacy gemfire classes heaily rely on cache, distribute system, distribution manager and other modules to function. Thus it's difficult to mock the class and do the junit test.
...
Code Block |
---|
GemFireCacheImpl cache = Fakes.cache(); RegionAttributes ra = createRegionAttributes(isConcurrencyChecksEnabled); InternalRegionArguments ira = new InternalRegionArguments(); // specify more mock behaviors for ParitionedRegion and BucketRegion PartitionedRegion pr = mock(PartitionedRegion.class); BucketAdvisor ba = mock(BucketAdvisor.class); ReadWriteLock primaryMoveLock = new ReentrantReadWriteLock(); Lock activeWriteLock = primaryMoveLock.readLock(); when(ba.getActiveWriteLock()).thenReturn(activeWriteLock); when(ba.isPrimary()).thenReturn(true); ira.setPartitionedRegion(pr) .setPartitionedRegionBucketRedundancy(1) .setBucketAdvisor(ba); // create Bucket Region BucketRegion br = new BucketRegion("testRegion", ra, null, cache, ira); // since br is a real bucket region object, we need to tell mockito to monitor it br = Mockito.spy(br); doNothing().when(br).distributeUpdateOperation(any(), anyLong()); doNothing().when(br).distributeDestroyOperation(any()); doNothing().when(br).distributeInvalidateOperation(any()); doNothing().when(br).distributeUpdateEntryVersionOperation(any()); doNothing().when(br).checkForPrimary(); doNothing().when(br).handleWANEvent(any()); doReturn(false).when(br).needWriteLock(any()); if (isConcurrencyChecksEnabled) { region.enableConcurrencyChecks(); } doNothing().when(region).notifyGatewaySender(any(), any()); doReturn(true).when(region).hasSeenEvent(any(EntryEventImpl.class)); EntryEventImpl event = createDummyEvent(region); VersionTag tag = createVersionTag(false); event.setVersionTag(tag); br.virtualPut(event, false, false, null, false, 12345L, false); // verify the result if (cnt > 0) { verify(br, times(cnt)).distributeUpdateOperation(eq(event), eq(12345L)); } else { verify(br, never()).distributeUpdateOperation(eq(event), eq(12345L)); } |
Running a single hydra test, continously if necessary
https://sites.google.com/a/pivotal.io/gem-engineering/process/issue-tracking/run-a-single-hydra-test
...