Page History

...

When the supplier node receives the cache partition file demand request it will send the file over the CommunicationSpi. The cache partition file can be concurrently updated by checkpoint thread during its transmission. To guarantee the file consistency Сheckpointer must use Copy-on-Write [3] tehnique and save a copy of updated chunk into the temporary file.The checkpoint process description on the supplier node – items 4, 5, 6, 7, 8 of the Process Overview.

Catch-up temporary WAL

While the demander node is in the partition file transmission state it must save all cache entries corresponding to the moving partition into a new temporary WAL storage. These entries will be applied later one by one on the received cache partition file. All asynchronous operations will be enrolled to the end of temporary WAL storage during storage reads until it becomes fully read. The file-based FIFO approach assumes to be used by this process.

...

• Unlimited number of wal-files to store temporary data records;
• Iterating over stored data records during an asynchronous writer thread inserts new records;
• WAL-per-partiton approach need to be used;
• Write operations to temporary WAL storage must have higher priority over read operations;

...

Rebuild indexes

The cache group index groups indexs must be rebuilded before setting the partition state to the OWNING state. 

...

In case of crash recovery, there is no additional actions need to be applied to keep the cache partition file consistency. We are not recovering partition with the MOVING state, thus the single partition file will be lost and only it. The cache partition file will be fully loaded on the next rebalance procedure.

Topology change

Each topology change event JOIN/LEFT/FAILED may or may not change cache affinity assignments of currently rebalacning caches. If assignments is not changed and the node is still needs partitions being rebalanced we can continue the current rebalance process (see for details IGNITE-7165).

Activation\deactivation

Unstable connection

A new connection must be established and the download process of partition file must be continued from the last successfully send cache partition chunk.

Crash recovery

To provide default cluster recovery guarantee we must to: 

• Start the checkpoint process when the temporary WAL storage becomes empty;
• Wait for the first checkpoint ends and set owning status  status to the cache partition;

Topology change

Activation\deactivation

Unstable connection

A new connection must be established and the download process of partition file must be continued from the last successfully send part.

...

Phase-2

The SSL must be disabled to take an advantage of Java NIO zero-copy file transmission using FileChannel#transferTo method. If we need to use SSL the file must be splitted on chunks the same way to send them over the socket channel with ByteBuffer. As the SSL engine generally needs a direct ByteBuffer to do encryption we can't avoid copying buffer payload from the kernel level to the application level. 

...