Table of Contents |
---|
Overview
Some Geode users have data models containing nested and complex objects. Lucene text search in Geode 1.2.0 supports indexing and querying only the top-level fields in the data object. The objective of this feature is to support indexing and querying an arbitrary depth of nested objects.
Goals
User can specified specify fields in nested object to be indexed.
Query User can query on these nested fields and collections of nested fields as well as top level fields.
Out of Scope
- Item in collection will not be index and searched.
API
...
Approach
Expose a LuceneSerializer interface and let the user write code to convert their objects to lucene docs.
Provide a default LuceneSerializer when creating the index that will flatten nested objects (eg, person.address.zip just gets stored as a list of zipcodes on the top level document).
- Provide a query syntax to search nested, flattened, fields.
Out of Scope
- Extending core Apache Lucene functionality to define a syntax for creating multiple documents and performing automatic joins in the StandardQueryParser for handling parent/child relationships with collections of nested fields.
API Change
- Add a new method to create a lucene index that takes a callback. The callback gives the user explicit control of how their value is converted to lucene documents and stored in the index.
Code Block |
---|
public LuceneIndexFactory {
/**
* Configure the way objects are converted to lucene documents for this lucene index
* @param luceneSerializer A callback which converts a region value to a
* Lucene document or documents to be stored in the index.
*/
public LuceneIndexFactory setLuceneSerializer(LuceneSerializer luceneSerializer);
}
/**
* An interface for writing the fields of an object into a lucene document
* The region key will be added as a field to the returned documents.
* @param index lucene index
* @param value user object to be serialized into index
*/
public interface LuceneSerializer {
Collection<Document> toDocuments(LuceneIndex index, Object value);
} |
XML Configuration
<cache xmlns:lucene="http://geode.apache.org/schema/lucene" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" version="1.0"> <region name="region" refid="PARTITION"> <lucene:index name="index"> <lucene:field name="a" analyzer="org.apache.lucene.analysis.core.KeywordAnalyzer"/> <lucene:field name="b" analyzer="org.apache.lucene.analysis.core.SimpleAnalyzer"/> <lucene:field name="c" analyzer="org.apache.lucene.analysis.standard.ClassicAnalyzer"/> <lucene:serializer> <class-name>org.apache.geode.cache.lucene.FlatFormatSerializer</class-name> </lucene:serializer> </lucene:index> </region> </cache> |
We will also provide a built-in implementation for LuceneSerializer called FlatFormatSerializer(). With this example serializer users can specify nested fields using the syntax fieldnameAtLevel1.fieldnameAtLevel2 for both indexing and querying.
For example, a in the following data model Customer object contains both a Person field. A object and a collection of Page objects. The Person object also contains a Page fieldobject.
Code Block |
---|
public class Customer implements Serializable { private String name; private StringCollection<String> symbol; // search integer in string formatphoneNumbers; private intCollection<Person> revenuecontacts; private int SSN; // search int private Person contact; // search nested object Page[] myHomePages; ...... } public class Person implements Serializable { private String name; private String email; private int revenue; private String address; private String[] phoneNumbers; private Page homepage; ....... } public class Page implements Serializable { private int id; // search integer in int format private String title; private String content; final String desc = "At client and server JVM, initializing cache will create the LuceneServiceImpl object," +" which is a singleton at each JVM."; ...... } |
Example
The example below demonstrates how to index on the nested fields: following example demos how to index on nested field such as contactcontacts.name, contacts.email, contacts.address, contacts.homepage.title, or contact.email, .
Note: each segment is a field name, not the type name. This will tell the system to find the parent and grandparent field, because it's possible that several fields will have the same type. For example, a field type, because Customer class could have 2 Person fields: Person contact more than one field of type Person; e.g. Person contacts and Person deliveryman. The email field to be indexed is from contact field, not deliveryman field.name is used to identify the parent field.
Code Block |
---|
// Get LuceneService LuceneService luceneService = LuceneServiceProvider.get(cache); // Create Index on fields, some are fields in nested objects: luceneService.createIndexFactory().addField("name").addField("symbol").addField("revenue").addField("SSN")setLuceneSerializer(new FlatFormatSerializer()) /* an out-of-box LuceneSerializer implementation */ .addField("name").addField("contactcontacts.name").addField("contactcontacts.email").addField("contactcontacts.address").addField("contactcontacts.homepage.title") .create("customerIndex", "Customer"); // Now to create region Region CustomerRegion = ((Cache)cache).createRegionFactory(shortcut).create("Customer"); |
gfsh command line:
Code Block |
---|
gfsh create lucene index --name=customerIndex --region=/Customer --field=name,contacts.name,contacts.email,contacts.address,contacts.homepage.title --serializer=org.apache.geode.cache.lucene.FlatFormatSerializer |
The syntax for querying the 2. When querying, the syntax of nested field is the same as toplevel field, i.e. without parent field's name. For example: "email:tzhou12*".
In order to find the parent document using the child object's field or grandchild object's field (such as use contact's email to find the customer), we need to use lucene's Join package. Geode provides
LuceneQueryProvider interface to implement such a query.
Code Block |
---|
package org.apache.geode.cache.lucene;
public interface LuceneQueryProvider extends Serializable {
/**
* @return A {@link Query} which will be executed against a Lucene index.
* @param index The {@link LuceneIndex} the query is being executed against.
* @throws LuceneQueryException if the provider fails to construct the query object. This will be
* propagated to callers of the {@link LuceneQuery} find methods.
*/
public Query getQuery(LuceneIndex index) throws LuceneQueryException;
} |
...
for a top level field, but with the additional qualifying parent field name, such as "contacts.name:tzhou11*". This distinguishes which "name" field when there can potentially be more than one 'name' field at different hierarchical levels in the object.
Code Block |
---|
LuceneQuery query = luceneService.createLuceneQueryFactory().create("customerIndex", "Customer", "contacts.name:tzhou11*", "name");
PageableLuceneQueryResults<K,Object> results = query.findPages(); |
Out-Of-Box implementation
We will provide an out-of-box implementation for the LuceneSerializer: FlatFormatSerializer.
It will still create one document for each parent object adding the nested object as embedded fields of the document. The field name will use the qualified name. Collections will be flattened and treated as tokens in the single field.
For example, the FlatFormatSerializer will convert a Customer object into a document as
(name:John11),(contacts.name:tzhou11), (contacts.email:tzhou11@gmail.com), (contacts.address:15220 Wall St), (contacts.homepage.id:11), (contacts.homepage.title: Mr. tzhou11), (contacts.homepage.content: xxx)
Risks and Mitigations
With this solution, collections (lists and maps) will be treated as a single flattened field, with the risk that queries into a collection may produce the wrong results. For example, a person entry with 2 address fields, one containing "main street" and the other containing zipcode "90210", a query such as:
person.address.zip=90210 and person.address.street=main
would incorrectly return this person entry. Because Apache Lucene does not define a standard approach for this, we are providing the LuceneSerializer interface to allow a user to write code to convert their objects to separate Lucene documents and to use Lucene ParentBlockJoinQuery to produce the desired results. For example, using the above query, the user could write the following code to produce the desired results:
Code Block |
---|
Such as , such as ToParentBlockJoinQuery, ToChildBlockJoinQuery, JoinUtil.
In this spec, we provided 2 examples of LuceneQueryProvider implementation using ToParentBlockJoinQuery, one for parent-child join, another for grandparent-parent-child join.
Code Block |
---|
// Example of parent-child join using ToParentBlockJoinQuery public class ToParentBlockJoinQueryProvider implements LuceneQueryProvider { String parentFilterField; String parentFilterString; String queryOnChild; String defaultFieldOnChild; private transient Query luceneQuery; public ToParentBlockJoinQueryProvider(String parentFilterField, String parentFilterString, String queryOnChild, String defaultFieldOnChild) { this.parentFilterField = parentFilterField; this.parentFilterString = parentFilterString; this.queryOnChild = queryOnChild; this.defaultFieldOnChild = defaultFieldOnChild; } @Override public Query getQuery(LuceneIndex index) throws LuceneQueryException { if (luceneQuery == null) { final StandardQueryParser queryParser = new StandardQueryParser(new KeywordAnalyzer()); Query childQueryaddressQuery = null; try { childQuery = queryParser.parse(queryOnChild"zip:90210 AND street:main", defaultFieldOnChild); } catch (QueryNodeException e) { } BitSetProducer parentFilter""); BitSetProducer parentDocFilter = new QueryBitSetProducer(new WildcardQueryTermQuery(new Term(parentFilterField"parent", parentFilterString"true"))); final ToParentBlockJoinQuery addressPart luceneQuery = new ToParentBlockJoinQuery(childQueryaddressQuery, parentFilterparentDocFilter, ScoreMode.Total); TopDocs } return luceneQuery; } } // Search using with nested object using the above LuceneQueryProvider // It will search child's field "email" for "tzhou11*" and return all the matched parent object, i.e. Customer object // The parent filter "*" will return all the matched parent object in this example. ToParentBlockJoinQueryProvider provider = new ToParentBlockJoinQueryProvider("symbol" /*parentField*/, "*" /*parentFilter*/, "email:tzhou11*" /*childQueryString*/, "email" /*childField*/); LuceneQuery query = luceneService.createLuceneQueryFactory().create("customerIndex", "Customer", provider); PageableLuceneQueryResults<K,Object> results = query.findPages();people = searcher.search(addressPart, 10); |