Page History

...

...

1. Public Ignite Java API (IgniteCache, ClientCache) for direct querying secondary indexes:
   1. Query result is Iterator<Cache.Entry<K.V>>, result data is sorted by an index conditions;
   2. Decrease an overhead on SQL stuff like query parsing, query planning, etc;
   3. Guarantee to run queries on index (Ignite SQL does not support index hints to force IndexScan operations Actually SQL do support index hints https://ignite.apache.org/docs/latest/perf-and-troubleshooting/sql-tuning#index-hints)
   4. Clear Java syntax instead of sometimes non-intuitive Ignite SQL;
   5. Clear Java syntax instead of sometimes non-intuitive Ignite SQL;
   6. Flexibility in returned fields (cache key, cache val, both, Flexibility in returned fields (cache key, cache val, both, only indexed fields). It helps improve performance for some cases even more.
2. Improved public Ignite Java API for creating secondary indexes:
   1. Functional indexes with Java functional interfaces;
   2. New Java API for dynamic creation of secondary indexes;
   3. Remove SQL dependency (annotation QuerySqlField) if only index feature is required and SQL is not needed.
3. Transactional support for "select for update" index queries:
   1. User will able to update cache entries for result of an index query within same transaction.

...

1. Java API: CacheConfiguration.setQueryEntity.setIndexes(QueryIndex idxs... );
2. Java API: CacheConfiguration.setIndexedTypes + annotation QuerySqlField.index;
3. SQL API: Create index with SQL query.

For Java API (ways 1, 2) an index is described with (IgniteCache, CacheValue.class, fields, indexName).

For SQL API (way 3) an index is described with (Schema, Table, fields, indexName).Every index created with those API can be described with: CacheContext, cache value class / type, index name, fields 

Index Query API will support those different index descriptions. API provides:

1. Public class IndexQuery for building index query:
   1. it inherits public class Query<R> with R = Cache.Entry<K, V>;
   2. it accepts index descriptions at a constructor;
   3. it accepts index conditions (joint with AND) at object methods.
2. User provides index conditions that matches index fields and their order (ts desc, price asc) to leverage on an index structure.

...

// Creates an index query for an index, created with Java API.specifying value class or value type: 
// 1. Specify index description at constructor.
// 2. Index name is optional param, can try do best to find right index basing on specified Value.class in constructor and fields in conditionscriteria fields.
// 3. Index conditionscriteria (joint with AND operation) with methods. Order of fields in criteria doesn't matter

QueryCursor<Cache.Entry<Long, Good>> cursor = ignite.cache("GOOD").query(
	IndexQuery.forTypenew IndexQuery<Long, Good>(Good.class, idxName?)  // idxName is optional.
		.setCriteria(gt("ts", lastMidnightTs)
		., lt("price", 123.0))
);

// Create index with SQL query: "CREATE INDEX GOOD_TS_DESC_PRICE_ASC_IDX on GOOD_TBL (ts DESC, price ASC)"
// 1. Table name should be specified because it is possible to have the same index name for different tables (e.g., __key_PK).
// 2. Index name is optional too (do our best to find right index to run the query).
QueryCursor<Cache.Entry<Long, Good>> cursor = ignite.cache("QueryCursor<Cache.Entry<Long, Good>> cursor = ignite.cache("GOOD").query(
	IndexQuery.forTablenew IndexQuery<Long, Good>("GOOD_TBLTYPE", "GOOD_TS_DESC_PRICE_ASC_IDX"idxName?)  // idxName is optional.
		.setCriteria(gt("ts", lastMidnightTs)
		., lt("price", 123.0)
);

Classes to implement the API:

))
);

Classes to implement the API:

// Public packages.

// IndexQuery extends
public IndexQuery<K, V> extends Query<Cache.Entry<K, V>> {

	private List<IndexCriteria> criteria = new ArrayList<>();

	// Index description.
	private @Nullable String idxName;
	private String valType;

// Public packages.

// IndexQuery extends
public IndexQuery<K, V> extends Query<Cache.Entry<K, V>> {

	private List<IndexCondition> idxConds = new ArrayList<>();

	// Index description.
	private @Nullable String idxName;
	private @Nullable String valClass;
	private @Nullable String schemaName;

	public static IndexQuery forType(Class<?> valClass, String? idxName) {
		return new IndexQuery(valCls, idxName);		
	}

	public static IndexQuery forTable(Class<?> valClass, String? schema, String? idxName) {
		return new IndexQuery(valClass, schema, idxName);
	}

	public IndexQuery lt(String field, Object val) {
		IndexCondition cond = RangeIndexCondition();
		idxConds.addcriteria.add(IndexCriteriaBuilder.lt(field, val));

		return this;
	}

	// Other methods are:
	// eq, notEq, gt, gte, lt, lte, between, in, notIn, min, max, predicate
}

// Internal packages.

class IndexCriteriaBuilder {
	public static IndexCriteria lt(String field, Object val);
}

abstract class IndexConditionIndexCriteria extends Serializable {
	private final List<String> fields;
}

// min, max
class MinMaxIndexConditionMinMaxIndexCriteria extends IndexCondition {
	private final boolean max;
}

// gt, gte, lt, lte, between
class RangeIndexConditionRangeIndexCriteria extends IndexConditionIndexCriteria {
	private final @Nullable Object lower;
	private final @Nullable Object upper;

	private final boolean lowerInclusive;
	private final boolean upperInclusive;
}

// in, notIn, eq, notEq
class InIndexConditionInIndexCriteria extends IndexConditionIndexCriteria {
	private final Object[] vals;

	// Flag for not-in condition.
	private final boolean inverse;
}

// predicate
class PredicateIndexConditionPredicateIndexCriteria extends IndexConditionIndexCriteria {
	private IgnitePredicate<?> predicate;
}

...

1. Introduce new type of query - INDEX;
2. Final query processing is performed within IndexQueryProcessor;
3. Entrypoint for distributed index queries is the IndexQueryProcessor.queryDistributed method, it executes MapReduce query, that leverage on other cache queries (ScanQuery, TextQuery):
   1. the Map phase is node-local querying of index, it returns sorted data by definition;
   2. filtering with IndexQuery.setFilter is analogue of ScanQuery.setFilter;
   3. the Reduce phase is responsible for the Merge Sort of data from multiple nodes;for implementing it's suggested to extend the h2.twostep package - move basic functionality to the ignite-core module, then different query processors can extends them (h2, index query processor).from multiple nodes;
4. Entrypoint for local query is the IndexQueryProcessor.queryLocal method. It accepts users IndexQueryand IndexingQueryFilter for filtering result cache entries (primary partition, MVCC, etc);The method predicate accepts a function, that should be deployed on other nodes with GridCacheDeploymentManager, MVCC, etc).

Rules to write criteria:

1. Order of field in criteria doesn't matter.
2. Fields can be used any times, Ignite uses AND to build final index range.

Rules to choose index:

1. If index name specified - then get index with this name and check that criteria fields match it.
2. If index name isn't specified - then find index that matches all specified criteria fields. Fields have to be prefix of an index.

// Internal package.

public class IndexQueryProcessor extends GridProcessorAdapter {
	// Provides access to indexes.
	private final IndexProcessor processor;

	// Entrypoint for local query.
    public Iterator<Cache.Entry<?,?>> queryLocal(IndexQuery idxQuery, @Nullable IndexingQueryFilter filters) throws IgniteException {
		// 1. If user specified index name, then check a query:
		// - sort query fields in index key order;
		// - check that it's a valid index query: fields covers index keys (from left to right)
		// - fail otherwise.
		// 2. If user doesn't specified index name:
		// - get all indexes for specified cache and Value.class;
		// - find index by filtering by a query fields (index keys must be in the same order as specified query fields, try sort fields to match any index).
		// - validate index query (see validation steps from 1.)
		Index idx = index(idxQuery.desc());

		// 1. Parse index conditions.
		// 2. Validate index condition, index type.
		// 3. Maps index conditions to an index query methods.
		// 4. Perform index operations, get Cursor.
		GridCursor<IndexRow> cursor = query(idx, idxQry.conditions());

		// 1. Map IndexRow to Cache entry.
		// 2. Apply specified cache filters (primary partitions, MVCC versions, etc.)
		// 3. Wrap cursor to iterator and return.
		return map_and_filter(cursor, filters);
	}

	private GridCursor<IndexRow> query(Index idx, List<IndexCondition> conditions) {
		// eq(key) -> idx.findOne(key)
		// notEq(key) -> idx.find(null, null, current -> current != key)
		// gt(key) -> idx.find(key, null, current -> current != key)
		// gte(key) -> idx.find(key, null)
		// lt(key) -> idx.find(null, key, current -> current != key)
		// lte(key) -> idx.find(null, key)
		// between(left, right) -> idx.find(left, right), inclusive only
		// in(keys...) -> idx.findOne(keys[0]) && idx.findOne(keys[1]) ...
		// notIn(keys...) -> idx.find(null, null, current -> !keys.contains(current))
		// min() -> idx.findFirst()
		// max() -> idx.findLast()
		// predicate(p) -> idx.find(null, null, p)
	}
}

...

1. index conditions joint with OR;
2. different sort order: idx(ts desc, price asc) but query(ts desc, price desc);
3. Cache is wide and IndexScan on few fields is much cheaper than ScanQuery;
4. predicatesetFilter operation with a custom user condition.

IndexName is optional. If it's not specified than take PK index of specified table.

// The "predicate" operation can't find best index for a user query. So user must provide an index name for this operation. 
QueryCursor<Cache.Entry<Long, Good>> cursor = ignite.cache("GOOD").query(
	new IndexQuery<>("GOOD_TBL", "GOOD_TS_DESC_PRICE_ASC_IDX")
		.predicatesetFilter((good) -> good.ts > lastMidnightTs || price > 100)
);

...