...
So, it looks like a newer version server with Lucene 7 puts a File into the fileAndChunkRegion, and an older version server with Lucene 6 gets it out and attempts to read it unsuccessfully.
I added the luceneQueryReturnsCorrectResultsAfterClientAndServersAreRolledOverAllBucketsCreated test to LuceneSearchWithRollingUpgradeDUnit that reproduces this issue every time.
The main sequence in this test that causes the failure is:
- start two servers with old version using Lucene 6
- roll one server to new version server using Lucene 7
- do puts into primary buckets in new server which creates entries in the fileAndChunk region with Lucene 7 format
- stop the new version server which causes the old version server to become primary for those buckets
- do a query which causes the IndexFormatTooNewException to be thrown
Hydra Test Execution
The lucene.bt and backwardCompatibilityLucene.bt had only 1 hung test which I don't think was related to Lucene 7.x.
Code Block |
---|
lucene/serialParReg.conf P 00:11:14 closed/pivotalgf-assembly/build/hydraTest/serialParReg-1023-192645 lucene/concParReg.conf P 00:24:33 closed/pivotalgf-assembly/build/hydraTest/concParReg-1023-193804 lucene/serialParRegPersist.conf P 00:15:20 closed/pivotalgf-assembly/build/hydraTest/serialParRegPersist-1023-200244 lucene/serialParRegIncremental.conf P 00:10:49 closed/pivotalgf-assembly/build/hydraTest/serialParRegIncremental-1023-201809 lucene/concParRegPersist.conf P 00:22:47 closed/pivotalgf-assembly/build/hydraTest/concParRegPersist-1023-202902 lucene/concParRegIncremental.conf P 00:12:30 closed/pivotalgf-assembly/build/hydraTest/concParRegIncremental-1023-205153 lucene/concParRegBridge.conf P 00:25:06 closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-210430 lucene/concParRegBridge.conf P 00:25:05 closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-212942 lucene/concParRegBridge.conf P 00:25:46 closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-215452 lucene/concParRegBridge.conf P 00:23:51 closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-222043 lucene/concParRegBridgePersist.conf P 00:24:18 closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-224441 lucene/concParRegBridgePersist.conf P 00:23:09 closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-230905 lucene/concParRegBridgePersist.conf P 00:22:24 closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-233220 lucene/concParRegBridgePersist.conf P 00:22:15 closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-235449 lucene/serialParRegHA.conf P 00:10:56 closed/pivotalgf-assembly/build/hydraTest/serialParRegHA-1024-001710 lucene/concParRegHA.conf P 00:32:02 closed/pivotalgf-assembly/build/hydraTest/concParRegHA-1024-002811 lucene/serialParRegHA.conf P 00:11:25 closed/pivotalgf-assembly/build/hydraTest/serialParRegHA-1024-010017 lucene/concParRegHA.conf P 00:32:37 closed/pivotalgf-assembly/build/hydraTest/concParRegHA-1024-011147 lucene/serialParRegHAPersist.conf P 00:13:38 closed/pivotalgf-assembly/build/hydraTest/serialParRegHAPersist-1024-014428 lucene/serialParRegHAPersist.conf P 00:13:36 closed/pivotalgf-assembly/build/hydraTest/serialParRegHAPersist-1024-015811 lucene/concParRegHAPersist.conf P 00:19:42 closed/pivotalgf-assembly/build/hydraTest/concParRegHAPersist-1024-021152 lucene/concParRegHAPersist.conf P 00:20:06 closed/pivotalgf-assembly/build/hydraTest/concParRegHAPersist-1024-023138 lucene/concParRegHABridge.conf P 00:34:45 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-025150 lucene/concParRegHABridge.conf P 00:33:29 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-032640 lucene/concParRegHABridge.conf P 00:33:07 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-040015 lucene/concParRegHABridge.conf P 00:33:54 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-043328 lucene/concParRegHABridgePersist.conf P 00:21:58 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridgePersist-1024-050729 lucene/concParRegHABridgePersist.conf P 00:21:55 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridgePersist-1024-052931 lucene/concParRegRecover.conf P 00:13:42 closed/pivotalgf-assembly/build/hydraTest/concParRegRecover-1024-055135 lucene/concParRegRecover.conf P 00:15:49 closed/pivotalgf-assembly/build/hydraTest/concParRegRecover-1024-060521 lucene/concParRegRecover.conf P 00:14:59 closed/pivotalgf-assembly/build/hydraTest/concParRegRecover-1024-062115 lucene/luceneQueriesWithRebalance.conf P 00:06:02 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalance-1024-063618 lucene/luceneQueriesWithRebalance.conf P 00:06:53 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalance-1024-064226 lucene/luceneQueriesWithRebalancePersist.conf P 00:11:01 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalancePersist-1024-064923 lucene/luceneQueriesWithRebalancePersist.conf P 00:09:16 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalancePersist-1024-070030 lucene/luceneQueriesWithHA.conf P 00:12:03 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-070951 lucene/luceneQueriesWithHA.conf H 00:16:37 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-072200 lucene/luceneQueriesWithHA.conf P 00:12:38 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-073844 lucene/luceneQueriesWithHA.conf P 00:12:56 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-075128 lucene/luceneQueriesWithHA.conf P 00:13:05 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-080428 lucene/luceneQueriesWithHAPersist.conf P 00:14:38 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHAPersist-1024-081740 lucene/luceneQueriesWithHAPersist.conf P 00:15:38 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHAPersist-1024-083223 lucene/luceneQueriesWithHAPersist.conf P 00:15:51 closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHAPersist-1024-084807 lucene/concDestroyIndex.conf P 00:12:10 closed/pivotalgf-assembly/build/hydraTest/concDestroyIndex-1024-090403 lucene/backwardCompatibility/concParRegBridge.conf P 00:25:50 closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1024-091618 lucene/backwardCompatibility/concParRegBridge.conf P 00:25:09 closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1024-094213 lucene/backwardCompatibility/concParRegBridgePersist.conf P 00:23:35 closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1024-100726 lucene/backwardCompatibility/concParRegBridgePersist.conf P 00:24:43 closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1024-103106 lucene/backwardCompatibility/concParRegHABridge.conf P 00:33:06 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-105556 lucene/backwardCompatibility/concParRegHABridgePersist.conf P 00:21:18 closed/pivotalgf-assembly/build/hydraTest/concParRegHABridgePersist-1024-112907 |
The errors.txt file contained:
Code Block |
---|
THREAD vm_0_thr_7_accessor1_mclaren_11724 Subthread Dynamic Client VM Stopper HANG Timeout during dynamic action: Failed to stop vm_3 within 300 seconds hydra.HydraTimeoutException: Failed to stop vm_3 within 300 seconds at hydra.ClientMgr.killClientVm(ClientMgr.java:969) at hydra.ClientMgr.stopClientVm(ClientMgr.java:819) at hydra.ClientMgr._stopClientVm(ClientMgr.java:761) at hydra.ClientMgr$2.run(ClientMgr.java:720) at java.lang.Thread.run(Thread.java:745) -------------------------------------------------------------------------------- |
...