Overview

Many Geode users create data models using nested and complex objects. Lucene text search in Geode 1.2.0 release supports indexing and querying only the top-level fields in the data object.

The objective of this feature is to support indexing and querying an arbitrary depth of nested objects.

Goals

1. User can specify fields in nested object to be indexed. 

2. User can query on these nested fields and collections of nested fields as well as top level fields. 

Approach

1. Expose a LuceneSerializer interface and let the user write code to convert their objects to lucene docs.

2. Provide a default LuceneSerializer when creating the index that will flatten nested objects (eg, person.address.zip just gets stored as a list of zipcodes on the top level document).

3. Provide a query syntax to search nested, flattened, fields.
   

Out of Scope

1. Extending core Lucene functionality to define a syntax for creating multiple documents and performing automatic joins in the StandardQueryParser for handling parent/child relationships with collections of nested fields.

API Change

• Add a new method to create a lucene index that takes a callback. The callback gives the user explicit control of how their value is converted to lucene documents and stored in the index. 
  

public LuceneIndexFactory {
 /**
 * Configure the way objects are converted to lucene documents for this lucene index
 * @param luceneSerializer A callback which converts a region value to a 
 * Lucene document or documents to be stored in the index.
 */
 public void setLuceneSerializer(LuceneSerializer luceneSerializer);
}  
  
/**
 * An interface for writing the fields of an object into a lucene document
 * The region key will be added as a field to the returned documents.
 */
public interface LuceneSerializer {
  Collection<Document> toDocuments(Object value);
}

• We will provide a built-in implementation for LuceneSerializer. Use fieldnameAtLevel1.fieldnameAtLevel2 to specify a field in the nested object both for indexing and querying. 

For example, the following Customer object contains a Person field. The Person object contains a Page field.

public class Customer implements Serializable {
  private String name;
  private Person contact; // search nested object 
  ......
}
public class Person implements Serializable {
  private String name;
  private String email;
  private int revenue;
  private String address;
  private Page homepage;
  .......
}
public class Page implements Serializable {
  private int id; // search integer in int format
  private String title;
  private String content;
  ......
}

The following example demonstrates how to index nested fields: contact.name, contact.email, contact.address, contact.homepage.title.

Note: each segment is a field name, not a field type, because Customer class could have more than one field of type Person; e.g. Person contact and Person deliveryman. The field name is used to identify the parent field.

// Get LuceneService
LuceneService luceneService = LuceneServiceProvider.get(cache);

// Create Index on fields, some are fields in nested objects:
luceneService.createIndexFactory().setLuceneSerializer(new FlatFormatSeralizer()) /* an out-of-box LuceneSerializer implementation */
      .addField("name").addField("contact.name").addField("contact.email").addField("contact.address").addField("contact.homepage.title")
      .create("customerIndex", "Customer");

// Now to create region
Region CustomerRegion = ((Cache)cache).createRegionFactory(shortcut).create("Customer");

The syntax for querying the nested field is the same as for a top level field, but with the additional qualifying parent field name, such as "contact.name:tzhou11*". This distinguishes which "name" field when there can potentially be more than one 'name' field at different hierarchical levels in the object.

LuceneQuery query = luceneService.createLuceneQueryFactory().create("customerIndex", "Customer", "contact.name:tzhou11*", "name");
 
PageableLuceneQueryResults<K,Object> results = query.findPages();

Out-Of-Box implementation

We'll provide an out-of-box implementation for the LuceneSerializer: FlatFormatSerializer.

It will still create one document for each parent object adding the nested object as embedded fields of the document. The field name will use the qualified name. Collections will be flattened and treated as tokens in the single field.

For example, the FlatFormatSerializer will convert a Customer object into a document as

(name:John11),(contact.name:tzhou11), (contact.email:tzhou11@gmail.com), (contact.address:15220 Wall St), (contact.homepage.id:11), (contact.homepage.title: Mr. tzhou11), (contact.homepage.content: xxx)

Risks and Mitigations

With this solution, collections (lists and maps) will be treated as a single flattened field, with the risk that queries into a collection may produce the wrong results. For example, a person entry with 2 address fields, one containing Main Street and the other containing zipcode 90210, a query such as:

person.address.zip=90210 and person.address.street=main

would incorrectly return this person entry. Because Apache Lucene does not define a standard approach for this, we are providing the LuceneSerializer interface to allow a user to write code to convert their objects to separate Lucene docs and to use Lucene ParentBlockJoinQuery to produce the desired results. For example, using the above query, the user could write the following code to produce the desired results:

final StandardQueryParser queryParser = new StandardQueryParser();
Query addressQuery = queryParser.parse("zip:90210 AND street:main", "");
BitSetProducer parentDocFilter = new QueryBitSetProducer(new TermQuery(new Term("parent", "true")));
final ToParentBlockJoinQuery addressPart = new ToParentBlockJoinQuery(addressQuery, parentDocFilter, ScoreMode.Total);
TopDocs people = searcher.search(addressPart, 10);