...

1. Batch writing to data pages
2. Batch updates in B+ Tree

Diagram overview

TBD

Batch write to data pages

Divide the input data rows into 2 lists:

1. Objects whose size is equal to or greater than the size of a single data page.
2. Other objects and remainders (heads) of large objects.

Sequentially write objects and fragments that occupy the whole page. The data page is taken from "reuse" bucket, if there is no page in reuse bucket - allocate a new one.

...

Overall changes to support batch updates in PageMemory can be divided into following phases.

Phase 1:

...

Batch insertion in FreeList to improve rebalancing

• Implement insertDataRows operation in FreeList - insert several data rows at once.
• Implement invokeAll operation in BPlusTree: support searching and inserting range of keys.
• Enable batch insertion in preloader (enabled by a special system propertyPreloader should insert a batch of data rows before initializing cache entries and should remove additional rows in case of fallback (if cache entry was not initialized).

Phase 2:

...

• InvokeAll operation should support batch update of existing keys.

...

 DataStreamer support

• Add support for batch updates in DataStreamer (inserts in FreeList in the isolated updater (similar to the preloader).

Phase

...

3: putAll support

• Add support for Implement batch updates in IgniteCache putAll.

...

• operations in B+ tree (findAll/putAll/removeALl/invokeAll).

• Examine the performance difference between the following approaches and select the best:
  A.  single updates (current approach)
  B.  sort + BPlusTree.invokeAll() + FreeList.insertDataRow
  C.  sort + BPlusTree.findAll + FreeList.insertDataRows + BPlusTree.putAll

Phase 4: MVCC support

• Add support for MVCC (TRANSACTIONAL_SNAPSHOT) cache mode.

Risks and Assumptions

1. Concurrent update of the cache entry is blocked until the batch operation with the same key is completed, therefore the batch size must be strictly limited.
2. Duplicate keys processing (WAL rebalancing) based on assumption that duplicate keys will be ordered according to cache entry version (a newer version is the latest)In-memory eviction policy can be configured in such a way that will lead to OOM when using batch writing to data pages, so in some degenerate cases the batch writing must be disabled.
3. Heap usage/GC pressure.

Prototype testing results

...