Motivation

...

Possible partition file sending approaches

Hot swap cache data storage 

Preload entries from loaded partition file

Proposed Changes (hot swap approach)

The Apache Ignite can support cache rebalancing as transferring partition files using zero copy algorithm [1] based on an extension of communication SPI and Java NIO API.

Hot swap cache data storage 

Preload entries from loaded partition file

Proposed Changes (hot swap)

Process Overview

In the process of balancing data:

...

• Unlimited number of WAL-files to store temporary data records;
• Iterating over stored data records during an asynchronous writer thread insert new records;
• WAL-per-partiton approach is need to be used;
• Write operations to storage must have higher priority over read operations;

Expected problems to be solved

• We must stop updating indexes on demander when the data is ready to be transferred from the supplier node. All async cache updates on demander must not cause the index update;
• The previous partition metadata page and all stored meta information must be destroyed in memory and restored from the new partition file;

...