...

Concurrent file transfer

Modifying OODT thread number

The performance of OODT PushPull was evaluated for directory transfer using a directory structure of 16x1024MB binary files. This would result in an averaged outcome compared to single file transfer and also eliminating some anomalies which arose in some cases.

PushPull has the option of specifying the number of threads for download of multiple files in parallel e.g (/cas­pushpull­0.6/etc/push_pull_framework.properties):

org.apache.oodt.cas.pushpull.file.retrieval.system.recommended.thread.count=4

which will be used when the optimised variable is set to false:

org.apache.oodt.cas.pushpull.crawler.use.tracker=false

By manually setting the number of threads for concurrent file transfer, the optimised thread number was found to be 4 which was therefore used for further benchmarking of concurrent file transfer on a modified network.

Image Added

Concurrent file transfer with packet delay

Using the recommended/optimised thread number for concurrent file transfer, the performance of PushPull is evaluated with induced packet delay into the network. While the bandwidth decreases for increasing induced packet delay, as expected, the tool is still capable of transferring files concurrently even with packet delays of ~1s (1000ms).

Image Added

Concurrent file transfer with packet loss

Little or no performance variation was found for packet loss of < 4%. When a large percentage of packet loss was induced, > 16%, the transfer time was exponentially increased up to a point ( > 24%) where the tool was not capable of transferring.

Image Added

Concurrent file transfer with packet corruption 

The performance of PushPull was found to drop to ~0.5 of the “clean network link” for a packet corruption = 24%. Beyond this point PushPull was not able to transfer files as the tool became unresponsive.

Image Added

Concurrent file transfer with packet reordering

A peculiar, non-linear, behaviour was found when inducing packet reordering < 4% for concurrent file transfer. These tests were once again repeated but still confirmed this behaviour. In particular from 1 ­-2% an increase is noted in the transfer rate. For packet reordering > 8% a general decreasing trend is noted for the transfer rate. While the transfer rate decreases, it still remains > 30 MB/s indicating that the tool manages packet reordering well with very little variation (or decrease) in performance observed, even for reordering > 80%.

Image Added

Concurrent file transfer with packet duplication

Packet duplication shows a strong linear trend. Although a decrease in the transfer rate is seen, it is only ~3 MB/s going from 1-80%. . Therefore, PushPull is more than capable to deal with packet duplication.

Image Added