Initial Changes
The lucene.version in gradle/dependency-versions.properties was changed:
From:
lucene.version = 6.4.1
To:
lucene.version = 7.1.0
Compilation
With this change, compilation failed with:
Code Block |
---|
geode/open/geode-lucene/src/main/java/org/apache/geode/cache/lucene/internal/repository/IndexRepositoryImpl.java:125: error: incompatible types: possible lossy conversion from long to int
totalHits = docs.totalHits; |
org.apache.lucene.search.TopDocs totalHits changed from int to long.
That required a few changes.
org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImpl query was changed:
From:
int totalHits = 0;
To:
long totalHits = 0;
org.apache.geode.cache.lucene.internal.LuceneIndexStats endRepositoryQuery signature was changed:
From:
public void endRepositoryQuery(long start, final int totalHits)
To:
public void endRepositoryQuery(long start, final long totalHits) {
Test Execution
Issue
org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImplJUnitTest queryShouldUpdateStats failed with:
Code Block |
---|
org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImplJUnitTest > queryShouldUpdateStats FAILED
Argument(s) are different! Wanted:
luceneIndexStats.endRepositoryQuery(
<any long>,
(Integer) 2
);
-> at org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImplJUnitTest.queryShouldUpdateStats(IndexRepositoryImplJUnitTest.java:163)
Actual invocation has different arguments:
luceneIndexStats.endRepositoryQuery(
0L,
(Long) 2L
);
-> at org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImpl.query(IndexRepositoryImpl.java:136)
at org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImplJUnitTest.queryShouldUpdateStats(IndexRepositoryImplJUnitTest.java:163) |
Resolution
The org.apache.geode.cache.lucene.internal.repository.IndexRepositoryImplJUnitTest queryShouldUpdateStats test was changed:
From:
verify(stats, times(1)).endRepositoryQuery(anyLong(), eq(2));
To:
verify(stats, times(1)).endRepositoryQuery(anyLong(), eq(2l));
Issue
Several org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit tests failed with the exception below including:
- luceneQueryReturnsCorrectResultsAfterClientAndServersAreRolledOver
- luceneQueryReturnsCorrectResultAfterTwoLocatorsWithTwoServersAreRolled
luceneQueryReturnsCorrectResultsAfterServersRollOverOnPartitionRegion
Code Block | ||
---|---|---|
| ||
org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit > luceneQueryReturnsCorrectResultsAfterClientAndServersAreRolledOver[0] FAILED
org.apache.geode.test.dunit.RMIException: While invoking org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit$$Lambda$204/1077331008.run in VM 2 running on Host 192.168.2.6 with 4 VMs
at org.apache.geode.test.dunit.VM.invoke(VM.java:393)
at org.apache.geode.test.dunit.VM.invoke(VM.java:363)
at org.apache.geode.test.dunit.VM.invoke(VM.java:308)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.verifyLuceneQueryResultInEachVM(LuceneSearchWithRollingUpgradeDUnit.java:553)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.putSerializableObjectAndVerifyLuceneQueryResult(LuceneSearchWithRollingUpgradeDUnit.java:500)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.luceneQueryReturnsCorrectResultsAfterClientAndServersAreRolledOver(LuceneSearchWithRollingUpgradeDUnit.java:264)
Caused by:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.verifyLuceneQueryResults(LuceneSearchWithRollingUpgradeDUnit.java:536)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.lambda$verifyLuceneQueryResultInEachVM$b83f705c$2(LuceneSearchWithRollingUpgradeDUnit.java:553)
Caused by:
java.lang.IllegalArgumentException: Could not load codec 'Lucene62'. Did you forget to add lucene-backward-codecs.jar?
Caused by:
java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene62' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current classpath supports the following names: [Lucene70, Asserting, CheapBastard, FastCompressingStoredFields, FastDecompressionCompressingStoredFields, HighCompressionCompressingStoredFields, DummyCompressingStoredFields, SimpleText] |
Resolution
The following line was added to geode-lucene/build.gradle:
testRuntime 'org.apache.lucene:lucene-backward-codecs:' + project.'lucene.version'
Issue
org.apache.geode.cache.lucene.internal.distributed.DistributedScoringJUnitTest uniformDistributionProducesComparableScores failed with:
Code Block |
---|
org.apache.geode.cache.lucene.internal.distributed.DistributedScoringJUnitTest > uniformDistributionProducesComparableScores FAILED
org.junit.ComparisonFailure: expected:<key-[6]> but was:<key-[5]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at org.apache.geode.cache.lucene.internal.distributed.DistributedScoringJUnitTest.uniformDistributionProducesComparableScores(DistributedScoringJUnitTest.java:119) |
Analysis
This test compares the scores for query results from one IndexRepositoryImpl vs the scores for query results from three IndexRepositoryImpls. The test expects the ordering to be the same but its not.
In Lucene 6.4.1, logging shows the scores like:
Code Block |
---|
Single IndexRepositoryImpl Result:
singleQueryResult=[key=key-0 score=1.8132632, key=key-3 score=1.2614386, key=key-6 score=1.0530728, key=key-5 score=1.0299048, key=key-7 score=0.63357]
Three IndexRepositoryImpl Result:
collector1QueryResult=[key=key-0 score=1.8866812]
collector2QueryResult=[key=key-3 score=1.1131656, key=key-5 score=0.5000039]
collector3QueryResult=[key=key-6 score=0.59086174, key=key-7 score=0.33355096]
mergedResult=[key=key-0 score=1.8866812, key=key-3 score=1.1131656, key=key-6 score=0.59086174, key=key-5 score=0.5000039, key=key-7 score=0.33355096] |
In Lucene 7.1.0, logging shows the scores like:
Code Block |
---|
Single IndexRepositoryImpl Result:
singleQueryResult=[key=key-0 score=1.9968563, key=key-3 score=1.2946333, key=key-5 score=1.1341828, key=key-6 score=1.0530728, key=key-7 score=0.73058176]
Three IndexRepositoryImpl Result:
collector1QueryResult=[key=key-0 score=2.0834162]
collector2QueryResult=[key=key-3 score=1.13994, key=key-5 score=0.5442147]
collector3QueryResult=[key=key-6 score=0.59086174, key=key-7 score=0.3901917]
mergedResult=[key=key-0 score=2.0834162, key=key-3 score=1.13994, key=key-6 score=0.59086174, key=key-5 score=0.5442147, key=key-7 score=0.3901917] |
The difference is key-5 and key-6 are in different orders in the single IndexRepositoryImpl query vs the multiple IndexRepositoryImpl query.
Issue
org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit luceneQueryReturnsCorrectResultsAfterServersRollOverOnPartitionRegion failed with:
Code Block | ||
---|---|---|
| ||
[info 2017/10/18 16:14:35.167 PDT <RMI TCP Connection(1)-192.168.2.6> tid=0x13] Got result: EXCEPTION_OCCURRED
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.verifyLuceneQueryResults(LuceneSearchWithRollingUpgradeDUnit.java:536)
at org.apache.geode.cache.lucene.LuceneSearchWithRollingUpgradeDUnit.lambda$verifyLuceneQueryResultInEachVM$b83f705c$2(LuceneSearchWithRollingUpgradeDUnit.java:553)
Caused by: org.apache.geode.InternalGemFireError: Unable to create index repository
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.lambda$computeRepository$0(AbstractPartitionedRepositoryManager.java:118)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager$$Lambda$35/248730969.apply(Unknown Source)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.computeRepository(AbstractPartitionedRepositoryManager.java:108)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.getRepository(AbstractPartitionedRepositoryManager.java:137)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.getRepositories(AbstractPartitionedRepositoryManager.java:91)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:105)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2911)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:98)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.lang.Thread.run(Thread.java:745)
at org.apache.geode.distributed.internal.ReplyException.handleAsUnexpected(ReplyException.java:89)
at org.apache.geode.internal.cache.execute.FunctionStreamingResultCollector.waitForCacheOrFunctionException(FunctionStreamingResultCollector.java:465)
at org.apache.geode.internal.cache.partitioned.PRFunctionStreamingResultCollector.getResult(PRFunctionStreamingResultCollector.java:91)
at org.apache.geode.cache.lucene.internal.LuceneQueryImpl.findTopEntries(LuceneQueryImpl.java:121)
at org.apache.geode.cache.lucene.internal.LuceneQueryImpl.findKeys(LuceneQueryImpl.java:69)
... 31 more
Caused by: org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource BufferedChecksumIndexInput(segments_2)): 7 (needs to be between 4 and 6)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:216)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:302)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:286)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.IndexRepositoryFactory.computeIndexRepository(IndexRepositoryFactory.java:84)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.PartitionedRepositoryManager.computeRepository(PartitionedRepositoryManager.java:42)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.lambda$computeRepository$0(AbstractPartitionedRepositoryManager.java:116)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager$$Lambda$35/248730969.apply(Unknown Source)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.computeRepository(AbstractPartitionedRepositoryManager.java:108)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.getRepository(AbstractPartitionedRepositoryManager.java:137)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.AbstractPartitionedRepositoryManager.getRepositories(AbstractPartitionedRepositoryManager.java:91)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.cache.lucene.internal.distributed.LuceneQueryFunction.execute(LuceneQueryFunction.java:105)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.internal.cache.PartitionedRegionDataStore.executeOnDataStore(PartitionedRegionDataStore.java:2911)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.internal.cache.partitioned.PartitionedRegionFunctionStreamingMessage.operateOnPartitionedRegion(PartitionedRegionFunctionStreamingMessage.java:98)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.internal.cache.partitioned.PartitionMessage.process(PartitionMessage.java:332)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionMessage.scheduleAction(DistributionMessage.java:374)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionMessage$1.run(DistributionMessage.java:440)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionManager.runUntilShutdown(DistributionManager.java:662)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in org.apache.geode.distributed.internal.DistributionManager$9$1.run(DistributionManager.java:1108)
at Remote Member '192.168.2.6(57006)<v3>:32773(version:GEODE 1.2.0)' in java.lang.Thread.run(Thread.java:745) |
Analysis
TBD - More work here.
The test:
- creates a LuceneIndex in 3 servers on an older version
- rolls each server one at a time and:
- creates the region
- executes a query
- verifies the results
The LuceneQueryImpl.findKeys method causes LuceneQueryFunction to be invoked in an older member. That older member tries to create an IndexWriter and fails.