Motivation

Currently, Ignite provide several interfaces to query data:

...

The goal of IEP is to provide consistency and interoperability between Key-Value, SQL, Scan query, Index Query APIs.

Description

Three query API has it's own advantages, use-cases and details, so description will be query API specific. In separate section different isolation levels will be described.

Description of transaction support for Key-Value API:

• Transaction state lives in JVM heap in IgniteTxState.
  • During, get or put invocation Ignite checks local state to get or store tx version of entry.
• Entries locks lives in JVM heap: GridCacheMvcc.
• On commit time Ignite writes changed entries to WAL and off-heap page-memory using ARIES-like protocol to preserve durability.
• Having only committed data in WAL and off-heap allows to exclude UNDO logic on failover.

SQL and other types of queries works on top of datastructures(BPlusTree) that stores data off-heap. This lead to the current behavior when transaction and queries not overlap.

Types of the solutions:

Described implementation and issue has two obvious solutions:

...

This following of description part provides high-level details of proposed implementation:

Scan

• updated and deleted entries must be modified, transformed, filtered before returning from iterator.
• new entries must be added to iterator. Looks like both can be done on the query runner side without observable performance issues.

SQL

New SQL Calcite engine gives Ignite the power to control execution flow. Proposed design of execution flow for different types of queries:

INSERT, UPDATE, DELETE, MERGE

Currently, executed on transaction coordinator node. But, separate thread used to permorm plan with ModifyNode. This means we must do the following:

1. Suspend transaction.
2. Store transaction inside BaseQueryContext.
3. Plan and start execute query.
4. Wrap ModifyNode in transaction resume/suspend.
5. Resume transaction after query invocation.

SELECT

To invoke select statements correctly execution engine must know IgniteTxState in time of table(index) scan. Ignite must send IgniteTxState (possibly filtering out some entries) to the node that invokes query step. This will allow to SQL engine to mixup transaction state and table(index) data to achieve correct results.

Transaction isolation levels

READ_COMMITTED

No additional locks required. Currently, queries works on this isolation level but without knowledge of transaction changes.

REPEATABLE_READ

Ignite must acquire shared-lock on each entry that conforms query criteria. This can be achieved in two ways: clause based locks, entry by entry locks. Decision must be made additionally in time of implementaion.

SERIALIZABLE

Ignite must acquire exclusive-lock on each entry that conforms query criteria. This can be achieved by clause based locks.

Lock performance

With SQL and Scan queries on strict isolation level it very simple for the end-user to acquire a lot of locks during "simple" query. Mechanism like CalciteQueryEngineConfiguration#getQueryMemoryQuota, CalciteQueryEngineConfiguration#getGlobalMemoryQuota must be created(or reused) to prevent user from harming server nodes.

Risks and Assumptions

Assumptions:

1. Primary Ignite use-case is OLTP with many short-lived small transactions.
2. SQL, Scan queries inside transaction doesn't touch many entries in single execution (mostly PK, index searches).
3. MVCC implementation is out of scope.

Risks:

1. Server node OOM failures due to the huge heap usage by transactions state.
2. Performance penalty of SELECT queries executed inside transaction.

Phases

PHASE 1

• Read-Committed isolation ONLY.
• Transaction aware SQL.
• Transaction aware Scan query.
• Transaction quotas.

PHASE 2

• Repeatable read, serializable isolation support.

PHASE 3

• Off heap locks implementation.
• Durability, failover protocol changes.

Discussion Links

Reference Links

Tickets