Page History

...

By rebalancing the Apache Ignite cluster, it balances the distribution of primary and backup data copies would be balanced according to applied affinity function on the new set of peer nodes. Imbalanced data increases the likelihood of data loss and can affect peer utilization during data requests. On the other hand, a balanced set of data copies optimizes each Ignite peer 's requests load and each Ignite peer 's disk resources consumption.

...

• Although all cache data is sent between peer's peers in batches (GridDhtPartitionSupplyMessage used) it still processes entries one by one. Such an approach have the has low impact with a pure in-memory use case usage but it leads to additional fsync's and logging WAL records with the Apache Ignite native persistence enabled. 
  
  By default, setRebalanceThreadPoolSize is set to 1 and setRebalanceBatchSize to 512K which means that thousands of key-value pairs will be processed single-thread and individually. Such approach impacts on: 
  • The extra unnecessary changes to keep a node keep node data structures up to date. Adding each entry record into CacheDataStore will traverse and modify each index tree N-times. It will allocate the space N-times within FreeList structure and will have to additionally store WAL page delta records with approximate complexity ~ O(N*log(N));
  • Batch with N-entries will produce N-records in WAL which might end up with N fsyncs (assume fsync WAL mode configuration enabled);
  • Increased the chance of huge JVM pauses. The more attendant serving objects we produces produce by applying changes, the more often GC happens and the greater chance of JVM pauses we havearise;
    
    

• Rebalancing The rebalancing procedure doesn't utilize the network and storage device throughput throughout to its full extent even with enough meaningful values of setRebalanceThreadPoolSize. For instance, trying to use a common recommendation of N+1 threads (N  – the number of CPU cores available) to increase rebalance speed will drammatically slowdown computation performance on demander node. This can be easily shown on the graphs below.

  CPU utilization (supplier, demaner)
  setRebalanceThreadPoolSize – 9 setRebalanceBatchSize – 512K	setRebalanceThreadPoolSize – 1 setRebalanceBatchSize – 512K

  
  

...