Compliance
To validate the reliability of our dsperf tool, we performed a comparative analysis against results obtained using established and widely adopted network benchmarking utilities.
In particular, we used the following tools:
Tool | Brief Description | Main Available Metrics |
iperf (https://iperf.fr/) | Widely used tool to measure bandwidth between two hosts. Supports TCP, UDP (and SCTP in some versions). | Throughput Transfer time Initial RTT Jitter, packet loss (only UDP) MSS, TCP Window Size |
netperf (https://hewlettpackard.github.io/netperf/) | Tool for measuring network performance. Supports tests like TCP_STREAM, TCP_RR and UDP_STREAM. Provides detailed metrics on latency, throughput, and transaction rate. | Throughput RTT (TCP_RR) Latency Message Size Transactions per second |
nuttcp (https://www.nuttcp.net/Welcome%20Page.html) | Network performance measurement tool for TCP/UDP throughput. Also provides CPU usage, user/system/wall-clock time and supports parallel streams and configurable segment size. | Throughput Average RTT CPU Usage (TX/RX) User/system time Packet loss % (only UDP) Retransmissions |
The network transmission capacity is measured using the traffic generation functions of third-party tools.
The obtained measurements are then compared with those obtained using dsperf, with the same network configuration and data block size.
We used a simple network composed of commercial hardware to perform the measurements. We opted for embedded devices as they are simpler and are produced with integrated boards that include transceivers for three different network technologies.
This ensures that all nodes have the same characteristics and configuration.
To improve the statistical accuracy of the results, each test was repeated 400 times for different transferred block-size on IEEE 802.11b network.
The following table show the measures taken and aggregated:
We have analyzed the dispersion factory for each 400 samples:
The variability of the data, measured for a block-size, is acceptable for all the tools.
Iperf highlights peaks attributable to characteristic periodic processing.
Dsperf does not present peaks because it performs a loop to calculate the total time instead of estimating it at intervals.
So we compared the total transfer time and throughput measurements for the tools:
Considering the dispersion factor the measures can be considered compatible in terms of accuracy and repeatability. Therefore, we can assume that dsperf is a valid tool for measuring the performance of both the underlay and the overlay layers.