THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Page History
...
File transfer between nodes
Partition preloader must support The node partition preloader machinery download cache partition file relocation files from one cluster node to another cluster nodes which owns desired partitions (the zero copy algorithm [1] assume to be used by default). To achieve this, the file transfer machinery transmission process must be implemented at Apache Ignite over Communication SPI.
CommunicationSpi
Ignite's IThe Comminication SPI must support to:
- establishing opening channel connections to the a remote node to an arbitrary topic (GridTopic is used) with predefined processing policyinitial meta information;
- listening incoming channel creation events and registering connection handlers on the particular nodeconnections and handling them by registered handlers;
- an arbitrary set of channel parameters on connection handshake (some initial Message assumed to be used);
API
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
public interface CommunicationListenerEx<T extends Serializable> extends EventListener {
/**
* @param nodeId Remote node id.
* @param initMsg Init channel message.
* @param channel Locally created channel endpoint.
*/
public void onChannelOpened(UUID nodeId, Message initMsg, Channel channel);
} |
GridIoManager
The file transmission IO manager must support to:
- using different approaches of incoming data handling – buffered and direct : CHUNK (read channel into ByteBuffer), FILE (zero-copy approach of FileChannel#transferTo);
- transferring send and receive data by chunks of predefined size with saving storing intermediate results;
- re-establishing reestablishing connection between nodes if an error occurs and continue file uploadsending\downloadreceiving;
- limiting connection bandwidth (upload and download) at runtime;
API
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
public interface TransmissionHandler {
/**
* @param err The err of fail handling process.
*/
public void onException(UUID nodeId, Throwable err);
/**
* @param nodeId Remote node id from which request has been received.
* @param fileMeta File meta info.
* @return Absolute pathname denoting a file.
*/
public String filePath(UUID nodeId, TransmissionMeta fileMeta);
/**
* <em>Chunk handler</em> represents by itself the way of input data stream processing.
* It accepts within each chunk a {@link ByteBuffer} with data from input for further processing.
*
* @param nodeId Remote node id from which request has been received.
* @param initMeta Initial handler meta info.
* @return Instance of chunk handler to process incoming data by chunks.
*/
public Consumer<ByteBuffer> chunkHandler(UUID nodeId, TransmissionMeta initMeta);
/**
* <em>File handler</em> represents by itself the way of input data stream processing. All the data will
* be processed under the hood using zero-copy transferring algorithm and only start file processing and
* the end of processing will be provided.
*
* @param nodeId Remote node id from which request has been received.
* @param initMeta Initial handler meta info.
* @return Intance of read handler to process incoming data like the {@link FileChannel} manner.
*/
public Consumer<File> fileHandler(UUID nodeId, TransmissionMeta initMeta);
} |
...
Overview
Content Tools
Apps