...

Motivation

One frequent Frequent usage pattern for Ignite is bulk data loading. Users need to be able to load data to Ignite from external sources as fast as possible. Ignite is not optimized for this use case at the moment, as bulk data loading process goes through the same code paths as normal cache updates. This IEP aims to improve bulk data loading performance.

Description

WAL optimization

When doing initial data load sometimes it is OK to relax crash-recovery guarantees. We can disable WAL for particular cache, cache group or data region, then load data, then enable it again. This mode could increase data loading time by a factor of 2x-4x.

Optimize CREATE INDEX

Secondary indexes negatively affects write performance. Common pattern is to drop indexes, load data and then create indexes again. This doesn't work for Ignite at the moment because index creation is slow. 

All proposed changes can be split in two groups - infrastructure improvements and index improvements. Note that some proposals are in conflict with each other so careful evaluation is a must.

...