ID | IEP-28 |
Author | |
Sponsor | |
Created | 31-Oct-2018 |
Status | DRAFT |
The Apache Ignite cluster balance procedure with enabled persitence currently doesn't utilize network and storage device throughout to its full extent. The balance procedure processes cache data entries one by one which is not efficient enough for the cluster with enabled persistence.
The Apache Ignite needs cache rebalacing with transfering partition files using zero-copy algorithm based on extension of communication SPI.
There are two participants in the process of balancing data:
The process of ordering cache groups for rebalancing remains the same. The whole process is described in terms of rebalance single cache group:
To achieve the zero-copy file tramsmission approach the CommunicationSpi needs to support direct (pipe) connections between two nodes:
The new abstraction layer over partition file needs to be created to support FileChannel#transferTo the zero copy feature of Java NIO and to support transferring partition file by chunks.
If operating system (JDK must be greater 1.6.0_18) does not support zero-copy file transfer, sending a file with FileChannel#transferTo might fail or yield worse performance. For example, sending a large file doesn't work well in Windows.
// Links to discussions on the devlist, if applicable.
// Links to various reference documents, if applicable.
// Links or report with relevant JIRA tickets.