Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Apache Ignite needs to support peer-2-peer cache partition file transfer with using zero-copy algorithm based on extension of communication SPI. 

Streaming via CommunicationSpi

Handshake message

The handshake message patched to support new type of connection.

Code Block
languagejava
themeConfluence
titleHandshakeMessage2.java
/** */
private static final byte PIPE_DATA_TRANSFER_MASK = 0x01;

/**
 * @return If socket will be used to transfer raw files.
 */
public boolean usePipeTransfer() {
    return (flags & PIPE_DATA_TRANSFER_MASK) != 0;
}

/**
 * @param usePipeTransfer {@code True} if socket should be used to transfer raw files.
 */
public final void usePipeTransfer(boolean usePipeTransfer) {
    flags = usePipeTransfer ? (byte)(flags | PIPE_DATA_TRANSFER_MASK) : (byte)(flags & ~PIPE_DATA_TRANSFER_MASK);
}


Extension communication SPI

Communication SPI support new connection type to communicate with peers via sockets.

Code Block
languagejava
titleCommunicationSpi.java
    /**
     * @return {@code True} if new type of direct connections supported.
     */
    public default boolean pipeConnectionSupported() {
        return false;
    }

    /**
     * @param src Source cluster node to initiate connection with.
     * @return Channel to listen.
     * @throws IgniteSpiException If fails.
     */
    public default ReadableByteChannel getRemotePipe(ClusterNode src) throws IgniteSpiException {
        throw new UnsupportedOperationException();
    }

    /**
     * @param dest Destination cluster node to communicate with.
     * @param out Channel to write data.
     * @throws IgniteSpiException If fails.
     */
    public default void sendOnPipe(ClusterNode dest, WritableByteChannel out) throws IgniteSpiException {
        throw new UnsupportedOperationException();
    }

...

  1. How many streaming connetions will be supported by CommunicationSpi?
    Single connection between pair of nodes.
  2. Will bandwidth of CommunicationSpi connection be controlled at runtime?
    Yes.
  3. Create Standalone
  4. How to choose the best rebalance type on the new node joined topology?
  5. ASYNC and SYNC cache rebalacing via p2p?


References

  1. Zero Copy I: User-Mode Perspective – https://www.linuxjournal.com/article/6345