...

• In-memory rebalancing;
• Historical rebalancing (Presistence native presistence enabled);

Limitation of data balancing

Regardless of which rebalance mode is used `SYNC` or `ASYNC` `SYNC` or `ASYNC` (defined in `CacheRebalanceMode` enum), the Apache Ignite rebalance implementation has a number of limitations caused by a memory-centric desing architecture. Although all cache data is sent between peer's in batches (used `GridDhtPartitionSupplyMessage`) it still processes entries one by one. Such approach have the low impact with a pure in-memory Apache Ignite use case but it leads to additional fsync's and logging WAL records with the native persistence enabled. 

By default,  `setRebalanceThreadPoolSize` is set to `1` and `setRebalanceBatchSize` to `512K` which means that thousands of key-value pairs will be processed single-thread and individually. This lead to: 

• Extra chages of Ignite node datastructure. Adding each entry record into `CacheDataStore` will traverse and modify each index tree N-times. We will allocate space in `FreeList` N-times within `FreeList` and we will have to additionally store in WAL page delta records ~ O(N*log(N)) page delta records; ;
• For instance, batch with N-entries will produce N WAL reocrds which might end up with N fsyncs (fsync-wal configuration)Each entry will produce a separate WAL record (and probably fsync'ed). For batch wiht N-entries, we might end up doing N fsyncs;

Rebalancing procedure doesn't utilize network and storage device throughput to full extent.

...