Page History

QPID-6262

Shortcomings of the current transport system.

...

1. It uses blocking IO. Messages are pushed onto the wire either synchronously (by another IO thread performing an enqueue) or asynchronously (the QueueRunner).  If the socket is unable to accept messages, then the socket write will block.  This in turn, once the IoSender's buffer becomes full will cause IoSender's send operation to block.   If the socket does not begin to accept data within a fixed time frame, send() throws a SenderTimeoutException.  The handling of this exception is awkward and has been related to a number of defects.  We currently have no better mechanism to deal with this timeout apart from closing the connection.     It would be better if non blocking IO were used and we only attempted to send messages if we knew the socket was able to take some data.
2. Since IO uses blocking IO, we must use a dedicated thread per connection as we have no way to predict when a socket will block.   The current IO model actually uses two threads per connection.    Users with larger Broker instances will have hundreds/thousands of IO threads.  Each thread requires its own stack-space so such a Broker will have a very heavy memory footprint.     If we were to use non-blocking IO, a thread pool could be utilised, reducing the thread count, thus reducing the memory footprint.
3. It has two SSL implementations.  It uses SslServerSocket based implementation for Ports configured to use SSL only and uses an SSLEngine based approach when the Port supports both plain and SSL connections (where a sniff is required).
4. The current IO transport model is seen as impeding progress in a wider queue/message threading refactorrefactoring

Desirable characteristics of a new IO transport system.

1. Non-blocking IO based
2. Exposes a mechanism so that a caller may determine if the underlying socket can accept data.  This will used by messaging layer to determine if it can send a message. This will ultimately allow TCP/IP back pressure to influence the distribution of messages amongst consumers.
3. The IO transport's send operation accepts one or more ByteBuffers.
   1. Once a byte buffer is passed to the send method, the transport takes ownership of the ByteBuffer and has responsibility for transferring its entire contents.
   2. The caller may not further alter a byte buffer that has been passed to the transport.
   An IO thread will be used to write the byte buffer(s) contents down the wire
   1. .
4. There will only be a single IO thread writewriting/reading for a connection at any one time
   1. Each iteration should:
      1. write as many byte buffers as possible until all pending byte buffers are sent, or the socket signals it is unable to take any more data.  In the latter case, the algorithm must add the IO_WRITE flag to the selector in order that it is notified when the socket can accept more data
      2. read as much as possible until the socket yields no more bytes and pass resulting byte buffers to the receivers
      3. write any new byte buffers resulting from the read data
5. The IO thread needs to wake up when:
   1. new byte buffers are available to send
   2. the socket indicates that it is ready to accept more data
6. IO threads will be drawn from an IO thread pool.