Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.


IDIEP-112115
Author
Sponsor
Created

  

Status
Status
colourGrey
titleDRAFT


Table of Contents

Motivation

A criteria API represent an abstract paginated query to be executed on a cache. Supporting criteria API queries may result in several benefits:

...

Scan and Index Queries are types of queries supported in Apache Ignite 2.x

Description

Criteria Query is logically just a facade for SQL – same way as Criteria API is a facade for SQL in JPA. The easiest way to implement this feature then is to just generate SQL queries.

There are different gateways to a table called Table Views. Currently we have RecordView and KeyValueView. Each view should have the Criteria Query API. The only difference is the type of objects the query result maps to.

The flow:

  1. The user supplies a Criteria and a QueryOptions.
  2. The client (or the API facade in embedded mode) generates the SELECT query based on the Criteria and sets the options (for now – just pageSize) in the query properties.
  3. The client runs the SQL as usual, potentially via the public API.
  4. The client obtains the ResultSet and turns it into an Iterator; it uses the view-defined Mappers to turn the Tuple into the target types.

Suggested Predicates List for Criteria API

...

  • Comparisons: =, >, <, >=, <=, IN
  • Combination: AND, OR, NOT
  • Nulls: IS NULL, IS NOT NULL

Criteria query works with transactions similarly to SQL

...

  • Query methods accept a transaction instance
  • If transaction is null, an implicit read-only transaction is opened

...

Configuration options

  • Ability to enforce an index(indexHint) to be used

  • Ability to configure pageSize for query. This is useful for tuning

API


Code Block
languagejava
titleProposed API
collapsetrue
/**
 * Represents a predicate. Implementations of this interface are basic building blocks for performing criteria queries.
 */
public interface Criteria {
    /**
     * CreateCreates a predicate forthat tests testingwhether the column value is equal to athe given valuecondition.
       *
     * @param columnName Column name.
     * @param valuecondition ColumnTarget value.
     * @return the created <b>equal</b> predicate instance.condition. 
     */
    static <R>Expression Criteria equalcolumnValue(String columnName, Comparable<R>Condition valuecondition);
 {
        throw new UnsupportedOperationException("Not implemented yet.");/**
    }

    /**
 Creates a condition that *test Createthe aexamined predicateobject foris testingequal whetherto the column is greater thanspecified {@code value}.
       *
     * @param columnNamevalue ColumnTarget namevalue.
     */
 @param value Column value. static <R> Condition equalTo(Comparable<R> value);

     /**
 @return the created <b>greaterThan</b> predicate* instance.
Creates a condition that test */
the examined object is staticequal <R>to Criteriathe greaterThan(String columnName, Comparable<R> value) {
  specified {@code value}.
     *
     * throw@param new UnsupportedOperationException("Not implemented yet.");value Target value.
    }

     */
    static Condition equalTo(byte[] value);

    /**
     * CreateCreates a predicatecondition forthat testingtest whetherthe theexamined columnobject is greaternot equal thanto orthe equalspecified to{@code value}.
       *
     * @param columnNamevalue ColumnTarget namevalue.
     * @param value Column value./
    static <R> Condition greaterThan(Comparable<R> value);

     /**
 @return the created <b>greaterThanOrEqualTo</b> predicate* instance.
Creates a condition that test */
the examined object is staticgreater <R>than Criteriaor greaterThanOrEqualTo(String columnName, Comparable<R> value) {equal than the specified {@code value}.
     *
   throw  * new UnsupportedOperationException("Not implemented yet.");
    }@param value Target value.
     */
    static <R> Condition greaterThanOrEqualTo(Comparable<R> value);

    /**
     * CreateCreates a predicatecondition forthat testingtest whetherthe theexamined columnobject is less than the specified {@code value}.
     *
     * @param columnNamevalue Column namevalue.
     */
 @param value Column value. static <R> Condition lessThan(Comparable<R> value);

     /**
 @return the created <b>lessThan</b> predicate* instance.
Creates a condition that test */
the examined object is staticless <R>than Criteriaor lessThan(String columnName, Comparable<R> value) {equal than the specified {@code value}.
     *
   throw  new UnsupportedOperationException("Not implemented yet.");
    }* @param value Column value.
     */
    static <R> Condition lessThanOrEqualTo(Comparable<R> value);

    /**
     * CreateCreates a condition predicatethat fortest testingthe whetherexamined theobject columnis is lessfound thanwithin orthe equalspecified to{@code valuecollection}.
     *
     * @param columnNamevalues Column namevalues.
     * @param@return value Column value.
     * @return the created <b>lessThanOrEqualTo<the created <b>in</b> predicate instance.
     */
    static <R> CriteriaCondition lessThanOrEqualToin(String columnName, Comparable<R> value) {
        throw new UnsupportedOperationException("Not implemented yet.");
    }Comparable<R>... values);

    /**
     * CreateCreates a predicatecondition forthat testingtest whetherthe theexamined firstobject argumentis is anfound elementwithin ofthe aspecified certain{@code collection}.
     *
     * @param columnName Column name.
     * @param values Column values.
     * @return the created <b>in</b> predicate instance.
     */
    static <R> CriteriaCondition in(String columnName, Comparable<R>byte[]... values) {;

    /**
     * Creates the throw{@code new UnsupportedOperationException("Not implemented yet.");
and} of the expressions.
     }*

     /*** @param expressions Expressions.
     */
 Creates   astatic predicate that will perform the logical <b>and</b> operation on the given {@code predicates}Expression and(Expression... expressions);

    /**
     * Creates the {@code or} of the expressions.
     *
     * @param criterionsexpressions the child predicates to form the resulting <b>and</b> predicate from.Expressions.
     */
    static Expression or(Expression... expressions);

    /**
     * @returnCreates the created <b>and</b>negation of the predicate instance.
     */
    static Criteria and(Criteria... criterions) {* @param expression Expression.
     */
   throw newstatic Expression UnsupportedOperationExceptionnot("Not implemented yet."Expression expression);
    }

    /**
 * Provides methods to *iterate Createsover aoperation predicateresults thatand willrelease performunderlying the logical <b>or</b> operation on the given {@code predicates}.resources.
 */
public interface Cursor<T> extends Iterator<T>, AutoCloseable {
     /**
     * @paramCloses criterionsthis cursor theand childreleases predicatesany tounderlying formresources.
 the resulting  <b>or< */b>
 predicate from.
  @Override void close();
}

/**
 * @returnProvides themethods createdto <b>or</b>iterate predicateover instance.
operation results and release underlying */
resources in an asynchronous static Criteria or(Criteria... criterions) {
        throw new UnsupportedOperationException("Not implemented yet.");way.
 */
public interface AsyncCursor<T> {
    /**
     * Returns the current page content if the operation returns results.
     */
    }Iterable<T> currentPage();

    /**
     * CreatesReturns athe predicatecurrent thatpage willsize negateif the resultoperation of the given {@code predicate}return results.
     */
     * @param criteria the predicate to negate the value of.int currentPageSize();

    /**
     * @returnFetches the created <b>not</b> predicate instancenext page of results asynchronously.
     */
    static Criteria not(Criteria criteria) {CompletableFuture<? extends AsyncCursor<T>> fetchNextPage();

    /**
     * Indicates whether throwthere are new UnsupportedOperationException("Not implemented yet.");more pages of results.
     */
    }boolean hasMorePages();

    /**
     * CreatesCloses athis predicatecursor thatand willreleases addany SQL to where clauseunderlying resources.
     */
     * @param sql Regular SQL where clause.
     * @param arguments Arguments for the statement.CompletableFuture<Void> closeAsync();
}

/**
 * Criteria query options.
 */
public class CriteriaQueryOptions {
    /** Default options. */
    public *static @returnfinal theCriteriaQueryOptions createdDEFAULT <b>sql</b> predicate instance.
= builder().build();

     */
private final int pageSize;

    /**
 static Criteria sql(String sql, @Nullable* ObjectConstructor...
 arguments) {
   *
     throw* new UnsupportedOperationException("Not implemented yet.");@param pageSize Page size.
    }
}

/** */
 * Criteria query result cursor. Implements {@link Iterable} only for convenience, e.g. {@link #iterator()}
 * can be obtained only once. Also if iteration is started then {@link #getAll()} method calls are prohibited.
 */
public interface CriteriaQueryCursor<T> extends Iterator<T>, Iterable<T>, AutoCloseable {
    /**
     * Gets all query results and stores them in the collection.
     * Use this method when you know in advance that query result is
     * relatively small and will not cause memory utilization issuesprivate CriteriaQueryOptions(int pageSize) {
        this.pageSize = pageSize;
    }

    /**
     * Creates a new builder.
     *
     * @return Builder.
     */
    public static Builder builder() {
        return new Builder();
    }

    /**
     * Gets a page size - the maximum number of result rows that can be fetched at a time.
     *
 <p>Since all the results will* be@return fetched, all the resources will be closedBatch size.
     */
     * automatically after this call, e.g. there is no need to call {@link #close()} method in this case.public int pageSize() {
        return pageSize;
    }

     /**
     * @return List containing all query resultsBuilder.
     */
    defaultpublic static List<T>class getAll()Builder {
        return stream().collect(toList());
private int pageSize = 1000;

    }

    /**
 {@inheritDoc} */
    @Override
   * default Iterator<T> iterator() {
        return this;
    }
Sets the page size (the number of entries that will be sent to the cluster in one network call).
    /**
     *
 Returns   a sequential Stream over the elements* covered@param bypageSize thisPage cursorsize.
     *
     * @return SequentialThis Streambuilder overinstance.
 the elements covered by this cursor.
     */
        defaultpublic Stream<T>Builder streampageSize(int pageSize) {
        return StreamSupport.stream(spliterator(), false);
    }
    if (pageSize <= 0) {
    /**
     * Closes all resources related to this cursor.throw If the query execution is in progressnew IllegalArgumentException("Page size must be positive: " + pageSize);
     * (which is possible in case of invoking from another thread), a cancel will be attempted.       }

            this.pageSize = pageSize;

     * Sequential calls to this method have noreturn effect.this;
     * <p>Note: don't forget to close query cursors. Not doing so may lead to various resource leaks.
     */
 }

        /**
         * Builds the options.
       @Override void close();
}

/**
 *     Represents a criteria builder.* Implementations@return ofCriteria thisquery interfaceoptions.
 are basic building blocks for performing criteria queries.
 */
public interface CriteriaBuilder extends Criteria {
   public staticCriteriaQueryOptions ColumnObject columnName(String columnNamebuild() {
            throwreturn new UnsupportedOperationException("Not implemented yet."CriteriaQueryOptions(pageSize);
    }

    CriteriaBuilder and(Criteria criteria);
}
    CriteriaBuilder or(Criteria criteria);

    }
}

/**
 * Represents an object *which Representscan abe columnqueried with criteria.
 Implementations*
 of* this@param interface<T> are basic building blocks for performing criteria queriesEntry type.
     */
 public   interface ColumnObjectCriteriaQuerySource<T> {
    /**
     /**
 Criteria query over cache entries.
     *
 Create a predicate for testing* the@param columntx is equalTransaction to aexecute giventhe value.
query within or {@code null}.
     *
 @param criteria If {@code null} then all entries *will @param value Column valuebe returned.
         * @return@throws theSqlException created <b>equal</b> predicate instanceIf failed.
         */
    default Cursor<T> query(@Nullable Transaction <R>tx, CriteriaBuilder@Nullable equal(Comparable<R>Criteria valuecriteria);
 {
        /**
   return query(tx, criteria, CriteriaQueryOptions.DEFAULT);
    }

  * Create a/**
 predicate for testing whether the* columnCriteria isquery greaterover thancache valueentries.
         *
         * @param valuetx ColumnTransaction value.
to execute the query within or {@code null}.
  * @return the created* <b>greaterThan</b>@param predicatecriteria instance.
If {@code null} then all entries will be  */returned.
     * @param opts <R>Criteria CriteriaBuilder greaterThan(Comparable<R> value);

   query options.
     /**
 @throws SqlException If failed.
     */
 Create a predicate forCursor<T> testingquery(@Nullable whetherTransaction thetx, column@Nullable isCriteria greatercriteria, than@Nullable or equal to value.
  CriteriaQueryOptions opts);

    /**
     * Execute *
criteria query over cache entries in an asynchronous way.
 * @param value Column value.*
     * @param tx Transaction *to @returnexecute the createdquery <b>greaterThanOrEqualTo</b>within predicate instanceor {@code null}.
     * @param criteria If  */
        <R> Criteria greaterThanOrEqualTo(Comparable<R> value);

   {@code null} then all entries will be returned.
     * @return Operation future.
     /**
 @throws SqlException If failed.
     */
 Create a predicate fordefault testingCompletableFuture<AsyncCursor<T>> whetherqueryAsync(@Nullable theTransaction columntx, is@Nullable lessCriteria thancriteria) value.{
        return queryAsync(tx,  *criteria, CriteriaQueryOptions.DEFAULT);
    }

     * @param value Column value./**
     * Execute criteria query *over @returncache theentries createdin <b>lessThan</b>an predicateasynchronous instanceway.
     *
     */
 @param tx Transaction to execute the query <R>within Criteria lessThan(Comparable<R> value);

or {@code null}.
     * @param criteria If  /**
      {@code null} then all entries will be returned.
   * Create a* predicate@param foropts testingCriteria whetherquery theoptions.
 column is less than or* equal@return toOperation valuefuture.
     * @throws SqlException If *failed.
     */
    *CompletableFuture<AsyncCursor<T>> queryAsync(
 @param value Column value.
        @Nullable *Transaction @returntx,
 the created <b>lessThanOrEqualTo</b> predicate instance.
       @Nullable Criteria */criteria,
            @Nullable CriteriaQueryOptions opts
 <R> Criteria lessThanOrEqualTo(Comparable<R> value);
}

public interface KeyValueView<K, V> extends DataStreamerTarget<Entry<K, V>>, CriteriaQuerySource<Entry<K, V>> /**{
    // ...
}

public interface RecordView<R> extends *DataStreamerTarget<R>, CreateCriteriaQuerySource<R> a{
 predicate for testing whether the first argument is an element of a certain collection.// ...
}


Code Block
languagejava
titleUsage examples
// Criteria
try (Cursor<Entry<Tuple, Tuple>> cursor = kvView().query(
         *null,
         * @param values Column values.and(columnValue("intCol", equalTo(42)), columnValue("primitiveIntCol", greaterThan(9000)), columnValue("booleanCol", equalTo(true))),
         * @return the created <b>in</b> predicate instance.
         */
        <R> Criteria in(String columnName, Comparable<R>... values);
    }
}

/**
 * Criteria query options.
 */
public class CriteriaQueryOptions {
    /** Default options. */
    public static final CriteriaQueryOptions DEFAULT = builder().build();

    private final int pageSize;

    /**
     * Constructor.
     *
     * @param pageSize Page size.
     */
    private CriteriaQueryOptions(int pageSize) {
        this.pageSize = pageSize;
    }

    /**
     * Creates a new builder.
     *
     * @return Builder.
     */
    public static Builder builder() {
        return new Builder();
    }

    /**
     * Gets a page size - the maximum number of result rows that can be fetched at a time.
     *
     * @return Batch size.
     */
    public int pageSize() {
        return pageSize;
    }

    /**
     * Builder.
     */
    public static class Builder {
        private int pageSize = 1000;

        /**
         * Sets the page size (the number of entries that will be sent to the cluster in one network call).
         *
         * @param pageSize Page size.
         * @return This builder instance.
         */
        public Builder pageSize(int pageSize) {
            if (pageSize <= 0) {
                throw new IllegalArgumentException("Page size must be positive: " + pageSize);
            }

            this.pageSize = pageSize;

            return this;
        }

        /**
         * Builds the options.
         *
         * @return Criteria query options.
         */
        public CriteriaQueryOptions build() {
            return new CriteriaQueryOptions(pageSize);
        }
    }
}

/**
 * Represents an object which can be queried with criteria.
 *
 * @param <T> Entry type.
 */
public interface CriteriaQuerySource<T> {
    /**
     * Criteria query over cache entries.
     *
     * @param tx Transaction to execute the query within or {@code null}.
     * @param criteria If {@code null} then all entries will be returned.
     * @throws SqlException If failed.
     */
    default CriteriaQueryCursor<T> queryCriteria(@Nullable Transaction tx, @Nullable Criteria criteria) {
        return queryCriteria(tx, criteria, CriteriaQueryOptions.DEFAULT);
    }

    /**
     * Criteria query over cache entries.
     *
     * @param tx Transaction to execute the query within or {@code null}.
     * @param criteria If {@code null} then all entries will be returned.
     * @param opts Criteria query options.
     * @throws SqlException If failed.
     */
    CriteriaQueryCursor<T> queryCriteria(@Nullable Transaction tx, @Nullable Criteria criteria, CriteriaQueryOptions opts);

    /**
     * Execute criteria query over cache entries in an asynchronous way.
     *
     * @param tx Transaction to execute the query within or {@code null}.
     * @param criteria If {@code null} then all entries will be returned.
     * @return Operation future.
     * @throws SqlException If failed.
     */
    default CompletableFuture<CriteriaQueryCursor<T>> queryCriteriaAsync(@Nullable Transaction tx, @Nullable Criteria criteria) {
        return queryCriteriaAsync(tx, criteria, CriteriaQueryOptions.DEFAULT);
    }

    /**
     * Execute criteria query over cache entries in an asynchronous way.
     *
     * @param tx Transaction to execute the query within or {@code null}.
     * @param criteria If {@code null} then all entries will be returned.
     * @param opts Criteria query options.
     * @return Operation future.
     * @throws SqlException If failed.
     */
    CompletableFuture<CriteriaQueryCursor<T>> queryCriteriaAsync(
            @Nullable Transaction tx,
            @Nullable Criteria criteria,
            CriteriaQueryOptions opts
    );
}

public interface KeyValueView<K, V> extends DataStreamerTarget<Entry<K, V>>, CriteriaQuerySource<Entry<K, V>> {
    // ...
}

public interface RecordView<R> extends DataStreamerTarget<R>, CriteriaQuerySource<R> {
    // ...
}
Code Block
languagejava
titleUsage examples
collapsetrue
// Criteria
try (var cursor = kvView().queryCriteria(
        null,
        and(equal("intCol", 42), greaterThan("primitiveIntCol", 9000), equal("booleanCol", true)),
        CriteriaQueryOptions.builder().pageSize(10).build()
)) {
    // ...
}

// CriteriaBuilder
try (var cursor = kvView().queryCriteria(
        null,
        columnName("intCol").equal(42)
                .and(columnName("primitiveIntCol").greaterThan(9000))
                .and(columnName("booleanCol").equal(true))
)) {
     // ... 
}

//  SQL where clause as criteria.
try (var cursor = kvView().queryCriteria(null, sql("intCol = ? AND primitiveIntCol > ? AND booleanCol = true", 42, 9000)CriteriaQueryOptions.builder().pageSize(10).build()
)) {
     // ... 
}

Risks and Assumptions

Expected to have similar performance for single-key criteria query, single-key SQL, table.get(PK). At the same time, SQL parsing may take much longer than expected.

Reactive API should be added for both SQL and Criteria, so it's not a point of this IEP.

USE INDEX-style hint looks like is not supported at this moment, but we still can add API and SQL generation for it.

NOTE: Comparing with table.get(PK) should be done with disabled partition awareness

Future improvement for criteria query

  • Lookup affinity node if PK or colocation key is in the criteria.
  • Create cache of prepared statement used in criteria query

Reference Links

Scan Query in Apache Ignite 2.x: https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-scan-queries.

Index Query in Apache Ignite 2.x: https://ignite.apache.org/docs/latest/key-value-api/using-cache-queries#executing-index-queries.

Tickets

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyIGNITE-20865
// Links or report with relevant JIRA tickets.