Page History

...

CommunicationSpi

To benefit from zero file copy we must delegate the file transferring to FileChannel#transferTo(long, long, java.nio.channels.WritableByteChannel) [2] because the fast path of transferTo method is only executed if the destination buffer inherits from an internal JDK class.

• The CommunicationSpi needs to support pipe connections between two nodes;
  • The WritableByteChannel needs to be accesses on the supplier side;
  • The ReadableByteChannel needs to be read on the demander side;
• The CommunicationListener must be extended to respond on new incoming pipe connections;

Partition transmission

Preloader

A new implementation of cache entries preloader assume to be done. The new implementation must send and receive cache partition files over the CommunicationSpi channels by chunks of data with validation received itemsThe cache partition file transfer over the network must be done using chunks with validation of received piece of data on the demander side.

• The new layer over the cache partition file must support direct using of FileChannel#transferTo method over the CommunicationSpi pipe connection;
• The process manager must support transferring the cache partition file by chunks of predefined size (multiply to the page size) one by one;
• The connection bandwidth of the cache partition file transfer must have an ability to be limited at runtime;

...

Checkpointer

When the supplier node receives the cache partition file demand request it must prepare and provide the cache partition file to transfer over network. The will send the file over the CommunicationSpi. The cache partition file can be concurrently updated by checkpoint thread during its transmission. To guarantee the file consistency Сheckpointer must use Copy-on-Write [3] tehniques assume to be used to guarantee the data consistency during chunk transfer.  
The checkpointing tehnique and save a copy of updated chunk into the temporary file.

The checkpoint process description on the supplier node – items 4, 5, 6 of the Process Overview.

(new) Catch-up temporary WAL

During the cache partition file transmitting, the demander node must hold all corresponding data entries on the new temporary WAL storage to apply them laterWhile the demander node is in the partition file transmission state it must save all cache entries corresponding to the moving partition into a new temporary WAL storage. When the partition file completely received these entries will be applied on it one by one. Asynchronous operations are written to the end of temporary WAL storage until it becomes empty. The file-based FIFO technique assumes to be usedapproach assume to be used by this process.

• The new write-ahead-log manager for writing temporary records must support
  • Unlimited number of wal-files to store temporary data records;
  • Iterating over stored data records during an asynchronous writer thread inserts new records;
  • WAL-per-partiton approach need to be used;

...