Versions Compared

Old Version 4

changes.mady.by.user Barry Oglesby

Saved on

compared with

New Version Current

changes.mady.by.user Barry Oglesby

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

So, it looks like a newer version server with Lucene 7 puts a File into the fileAndChunkRegion, and an older version server with Lucene 6 gets it out and attempts to read it unsuccessfully.

I added the luceneQueryReturnsCorrectResultsAfterClientAndServersAreRolledOverAllBucketsCreated test to LuceneSearchWithRollingUpgradeDUnit that reproduces this issue every time.

The main sequence in this test that causes the failure is:

  1. start two servers with old version using Lucene 6
  2. roll one server to new version server using Lucene 7
  3. do puts into primary buckets in new server which creates entries in the fileAndChunk region with Lucene 7 format
  4. stop the new version server which causes the old version server to become primary for those buckets
  5. do a query which causes the IndexFormatTooNewException to be thrown

Hydra Test Execution

The lucene.bt and backwardCompatibilityLucene.bt had only 1 hung test which I don't think was related to Lucene 7.x.

Code Block
lucene/serialParReg.conf                                       P    00:11:14    closed/pivotalgf-assembly/build/hydraTest/serialParReg-1023-192645
lucene/concParReg.conf                                         P    00:24:33    closed/pivotalgf-assembly/build/hydraTest/concParReg-1023-193804
lucene/serialParRegPersist.conf                                P    00:15:20    closed/pivotalgf-assembly/build/hydraTest/serialParRegPersist-1023-200244
lucene/serialParRegIncremental.conf                            P    00:10:49    closed/pivotalgf-assembly/build/hydraTest/serialParRegIncremental-1023-201809
lucene/concParRegPersist.conf                                  P    00:22:47    closed/pivotalgf-assembly/build/hydraTest/concParRegPersist-1023-202902
lucene/concParRegIncremental.conf                              P    00:12:30    closed/pivotalgf-assembly/build/hydraTest/concParRegIncremental-1023-205153
lucene/concParRegBridge.conf                                   P    00:25:06    closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-210430
lucene/concParRegBridge.conf                                   P    00:25:05    closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-212942
lucene/concParRegBridge.conf                                   P    00:25:46    closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-215452
lucene/concParRegBridge.conf                                   P    00:23:51    closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1023-222043
lucene/concParRegBridgePersist.conf                            P    00:24:18    closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-224441
lucene/concParRegBridgePersist.conf                            P    00:23:09    closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-230905
lucene/concParRegBridgePersist.conf                            P    00:22:24    closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-233220
lucene/concParRegBridgePersist.conf                            P    00:22:15    closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1023-235449
lucene/serialParRegHA.conf                                     P    00:10:56    closed/pivotalgf-assembly/build/hydraTest/serialParRegHA-1024-001710
lucene/concParRegHA.conf                                       P    00:32:02    closed/pivotalgf-assembly/build/hydraTest/concParRegHA-1024-002811
lucene/serialParRegHA.conf                                     P    00:11:25    closed/pivotalgf-assembly/build/hydraTest/serialParRegHA-1024-010017
lucene/concParRegHA.conf                                       P    00:32:37    closed/pivotalgf-assembly/build/hydraTest/concParRegHA-1024-011147
lucene/serialParRegHAPersist.conf                              P    00:13:38    closed/pivotalgf-assembly/build/hydraTest/serialParRegHAPersist-1024-014428
lucene/serialParRegHAPersist.conf                              P    00:13:36    closed/pivotalgf-assembly/build/hydraTest/serialParRegHAPersist-1024-015811
lucene/concParRegHAPersist.conf                                P    00:19:42    closed/pivotalgf-assembly/build/hydraTest/concParRegHAPersist-1024-021152
lucene/concParRegHAPersist.conf                                P    00:20:06    closed/pivotalgf-assembly/build/hydraTest/concParRegHAPersist-1024-023138
lucene/concParRegHABridge.conf                                 P    00:34:45    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-025150
lucene/concParRegHABridge.conf                                 P    00:33:29    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-032640
lucene/concParRegHABridge.conf                                 P    00:33:07    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-040015
lucene/concParRegHABridge.conf                                 P    00:33:54    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-043328
lucene/concParRegHABridgePersist.conf                          P    00:21:58    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridgePersist-1024-050729
lucene/concParRegHABridgePersist.conf                          P    00:21:55    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridgePersist-1024-052931
lucene/concParRegRecover.conf                                  P    00:13:42    closed/pivotalgf-assembly/build/hydraTest/concParRegRecover-1024-055135
lucene/concParRegRecover.conf                                  P    00:15:49    closed/pivotalgf-assembly/build/hydraTest/concParRegRecover-1024-060521
lucene/concParRegRecover.conf                                  P    00:14:59    closed/pivotalgf-assembly/build/hydraTest/concParRegRecover-1024-062115
lucene/luceneQueriesWithRebalance.conf                         P    00:06:02    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalance-1024-063618
lucene/luceneQueriesWithRebalance.conf                         P    00:06:53    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalance-1024-064226
lucene/luceneQueriesWithRebalancePersist.conf                  P    00:11:01    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalancePersist-1024-064923
lucene/luceneQueriesWithRebalancePersist.conf                  P    00:09:16    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithRebalancePersist-1024-070030
lucene/luceneQueriesWithHA.conf                                P    00:12:03    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-070951
lucene/luceneQueriesWithHA.conf                                H    00:16:37    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-072200
lucene/luceneQueriesWithHA.conf                                P    00:12:38    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-073844
lucene/luceneQueriesWithHA.conf                                P    00:12:56    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-075128
lucene/luceneQueriesWithHA.conf                                P    00:13:05    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHA-1024-080428
lucene/luceneQueriesWithHAPersist.conf                         P    00:14:38    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHAPersist-1024-081740
lucene/luceneQueriesWithHAPersist.conf                         P    00:15:38    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHAPersist-1024-083223
lucene/luceneQueriesWithHAPersist.conf                         P    00:15:51    closed/pivotalgf-assembly/build/hydraTest/luceneQueriesWithHAPersist-1024-084807
lucene/concDestroyIndex.conf                                   P    00:12:10    closed/pivotalgf-assembly/build/hydraTest/concDestroyIndex-1024-090403
lucene/backwardCompatibility/concParRegBridge.conf             P    00:25:50    closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1024-091618
lucene/backwardCompatibility/concParRegBridge.conf             P    00:25:09    closed/pivotalgf-assembly/build/hydraTest/concParRegBridge-1024-094213
lucene/backwardCompatibility/concParRegBridgePersist.conf      P    00:23:35    closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1024-100726
lucene/backwardCompatibility/concParRegBridgePersist.conf      P    00:24:43    closed/pivotalgf-assembly/build/hydraTest/concParRegBridgePersist-1024-103106
lucene/backwardCompatibility/concParRegHABridge.conf           P    00:33:06    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridge-1024-105556
lucene/backwardCompatibility/concParRegHABridgePersist.conf    P    00:21:18    closed/pivotalgf-assembly/build/hydraTest/concParRegHABridgePersist-1024-112907

The errors.txt file contained:

Code Block
THREAD vm_0_thr_7_accessor1_mclaren_11724 Subthread Dynamic Client VM Stopper
HANG Timeout during dynamic action: Failed to stop vm_3 within 300 seconds
hydra.HydraTimeoutException: Failed to stop vm_3 within 300 seconds
	at hydra.ClientMgr.killClientVm(ClientMgr.java:969)
	at hydra.ClientMgr.stopClientVm(ClientMgr.java:819)
	at hydra.ClientMgr._stopClientVm(ClientMgr.java:761)
	at hydra.ClientMgr$2.run(ClientMgr.java:720)
	at java.lang.Thread.run(Thread.java:745)
--------------------------------------------------------------------------------

...