Want a quick benchmark of the network link between two servers? Compare the private networking feature offered by your favorite cloud providers? Wondering how much overhead a VPN adds?

Try qperf. It’s a quick, easy way to check TCP and UDP bandwidth and latencies between two servers.

Installing qperf

On CentOS/RHEL 7, qperf is available from the EPEL repository. Install it like this:

# enable the EPEL repo
$ sudo yum install epel-release

# install the package itself
$ sudo yum install qperf

The Ubuntu/Debian repositories do not have qperf, so you’ll need to build it from source. This is thankfully quite easy:

# get the tools needed to build qperf
$ sudo apt-get install -y make gcc libc-dev

# download the qperf source tarball
$ wget https://www.openfabrics.org/downloads/qperf/qperf-0.4.9.tar.gz

# unpack it
$ tar xvf qperf-0.4.9.tar.gz

# build it
$ cd qperf-0.4.9
$ ./configure
$ make

# the qperf binary is now at qperf-0.4.9/src/qperf

Using qperf

You need to run qperf on two machines. On the first one, qperf is run in a “server” mode and on the second one, qperf can be invoked to run various tests.

Running qperf without any arguments starts it in server mode. On the second node, invoke it with the IP of the first server, and the test(s) to run. The tests we are interested in are:

  • tcp_bw and udp_bw – the rate of transfer over TCP and UDP, reported at byte-level (MB/s) and packet-level (messages/sec)
  • tcp_lat and udp_lat – the average per-hop time taken for a TCP or UDP packet

By default qperf runs each specified test for 2 seconds. Add a -v flag to get a verbose output. Here it is in action:

qperf in action

The above screencast shows the execution of the tcp_bw and tcp_lat tests. The first node (node1), with the IP 10.99.0.1, is running qperf in the server mode.

Here is a sample output of the UDP bandwidth and latency tests (you can list any or all tests in the same command line):

root@node2:~# qperf -v 10.99.0.1 udp_bw udp_lat
udp_bw:
    send_bw         =   248 MB/sec
    recv_bw         =   238 MB/sec
    msg_rate        =  7.25 K/sec
    send_cost       =   302 ms/GB
    recv_cost       =   547 ms/GB
    send_cpus_used  =   7.5 % cpus
    recv_cpus_used  =    13 % cpus
udp_lat:
    latency        =  46.7 us
    msg_rate       =  21.4 K/sec
    loc_cpus_used  =  11.5 % cpus
    rem_cpus_used  =    10 % cpus

You can also see the CPU that qperf had to consume to send and receive the data that it did. You can also see an even more detailed output with the -vv flag.

Instructing qperf to run the tests for a longer time, say 60s, reduces the influence of outliers (qperf only reports averages, and no other stats). You can do this by using the -t NN flag, like so:

root@node2:~# qperf -t 60 -v 10.99.0.1 tcp_bw tcp_lat
tcp_bw:
    bw              =   239 MB/sec
    msg_rate        =  3.65 K/sec
    time            =    60 sec
    send_cost       =   254 ms/GB
    recv_cost       =  1.15 sec/GB
    send_cpus_used  =  6.07 % cpus
    recv_cpus_used  =  27.6 % cpus
tcp_lat:
    latency        =  47.4 us
    msg_rate       =  21.1 K/sec
    time           =    60 sec
    loc_cpus_used  =    11 % cpus
    rem_cpus_used  =    10 % cpus

And finally, you can also choose the size of the message to be used for the tests. (You can also make it repeat the test with increasing sizes using the -oo option.) Here is qperf being run with a 4 kB packet size:

root@node2:~# qperf -v -m 4k 10.99.0.1 tcp_bw tcp_lat
tcp_bw:
    bw              =   220 MB/sec
    msg_rate        =    55 K/sec
    msg_size        =     4 KB
    send_cost       =   429 ms/GB
    recv_cost       =  1.16 sec/GB
    send_cpus_used  =   9.5 % cpus
    recv_cpus_used  =  25.5 % cpus
tcp_lat:
    latency        =   145 us
    msg_rate       =  6.91 K/sec
    msg_size       =     4 KB
    loc_cpus_used  =     7 % cpus
    rem_cpus_used  =     5 % cpus

Applying qperf

Here are some examples of where qperf can be useful:

  • How does the private networking feature offered by Digital Ocean compare to, say, Linode’s in terms of network throughput and latency?
  • Your containers are talking to each other over a Weave overlay network – is this slowing things down?
  • How much will adding encryption to a Weave network slow things down?

Interested in the answers to these questions? Let us know in the comments and we’ll try to do a blog post about them in the future.

Going from here

qperf is a handly little tool to have in your toolbox. Another similar tool is iperf.

While qperf and iperf actively measure your network link, there is another class of tools that passively monitor network transfer rates, errors and the like. Our own OpsDash is one of them.