Page History

TCP Network Performance

(Abstracted and extended from a discussion from the NuttX Google group)

Question

For some unknown reason, I am seeing poor TCP network performance.

Answer

First let's talk about TCP send performance.

Source of Performance Bottlenecks

General TCP send performance is not determined by the TCP stack as much as it is by the network device driver. Bad network performance is due to time lost BETWEEN packet transfers. The packet transfers themselves go at the wire speed*. So if you want to improve performance on a given network, you have to reduce time lost between transfers. There is no other way.

...

It would be good to have a real in-depth analysis of the network stack performance to identify bottlenecks and generate ideas for performance improvement. No one has ever done that. If I were aware of any stack related performance issue, I would certainly address it.

RFC 1122

There is one important feature missing the NuttX TCP that can help when there is no write buffering: Without write buffering send() will not return until the transfer has been ACKed by the recipient. But under RFC 1122, the host need not ACK each packet immediately; the host may wait for 500 MS before ACKing. This combination can cause very slow performance when small, non-buffered transfers are made to an RFC 1122 client. However, the RFC 1122 must ACK at least every second (odd) packet so sequences of packets with write buffering enabled do not suffer from this problem.

Update:  RFC 1122 support was added to the NuttX TCP stack with commit 66ef6d143a627738ad7f3ce1c065f9b1f3f303b0 in December of 2019.  That, however, that affects only received packet ACK behavior and has no impact on transmitted packet performance; write buffering is still recommended.

TCPBlaster

I created a new test application at apps/examples/tcpblaster to measure TCP performance and collected some data for the configuration that happens to be on my desk. The tcpblaster test gives you the read and write transfer rates in Kb/sec (I won't mention the numbers because I don't believe they would translate any other setup and, hence, would be misleading).

...

If you do discover any significant performance issues with the stack I will of course gladly help you resolve them. Or if you have ideas for improved performance, I would also be happy to hear those.

What about Receive Performance?

All of the above discussion concerns transmit performance, i.e., "How fast can we send data over the network?" The other side is receive performance. Receive performance is very different thing. In this case it is the remote peer who is in complete control of the rate at which packets appear on the network and, hence, responsible for all of the raw bit transfer rates.

...