TCP LabsWACREN Network Monitoring and Measurement Workshop

Antoine Delvaux — a.delvaux@man.poznan.pl perfSONAR developer

30.09.2015

Hands-on session• We’ll explore practical aspects of TCP • Checking the effect of loss and latency on TCP

performance • Looking the TCP window size • Checking and setting buffers values

• Looking at the effects of tuning on TCP performance

2

VM setup• On your local setup: 3 VM – Debian + perfsonar-testpoint – CentOS + perfSONAR-TestPoint – perfSONAR toolkit full install (CentOS)

• Need to check/change network card before boot – Should be on Bridged adapter – If not possible, then in Host-only adapter or NAT

• Local network on your machine (IP if host-only adapter config): – 192.168.56.2 : your host – 192.168.56.3 : Debian host — wacren00 – 192.168.56.4 : CentOS host — wacrencos – 192.168.56.5 : perfSONAR — wacrenps

3

VM reconfig• Login: pst/abuja • Beware of keyboard: AZERTY – Change after login with: • On Debian: sudo dpkg-reconfigure keyboard-configuration

• On CentOS (as root) vi /etc/sysconfig/keyboard

• Timezone • On Debian:

sudo dpkg-reconfigure tzdata

4

Mininet• What is mininet? – “Mininet creates a realistic virtual network, running

real kernel, switch and application code, on a single machine (VM, cloud or native), in seconds, with a single command”

– http://mininet.org/

• Used by many networking courses: – https://github.com/mininet/mininet/wiki/Teaching-

and-Learning-with-Mininet

5

Mininet• Simulates network using OpenFlow (Open

vSwitch), all on your local machine • Simulate network topology and link characteristics – bandwidth – losses – delays – queuing – etc…

• You define the network topology in a Python script

6

Launch and use mininet• Default topology: h1 — s1 — h2

– $ sudo mn– *** Creating network– (…)– mininet> help

• Ping hosts in mininet shell: – pingall– h1 ping h2

• Show interfaces – h1 ifconfig– h2 ifconfig

7

What is iperf?• Throughput: – Rate of successful message delivery over a communication channel – How much data can I shovel into the network for some given amount of

time?

• What does this tell us? – Opposite of utilization (e.g. its how much we can get at a given point in

time, minus what is utilised) – Utilisation and throughput added together are capacity

• Tools that measure throughput are a simulation of a real work use case (e.g. how well could bulk data movement perform)

• Ways to game the system – Parallel streams – Manual window size adjustments – ‘memory to memory’ testing – no spinning disk

8

How does iperf work?• Data Transfer between memory on local and

remote host through end systems TCP/IP-Stack and network

• Two modes: TCP, UDP • Best practise: – Interactive session on both sides – Take result from Server (Receiver)

9

iperf server• One program, two roles : – Server (Receiver) – Client (Sender)

• Run the server as daemon or interactively – On wacrenps (CentOS host) – iperf3 -s

10

iperf client outputpst@wacren00:~$ iperf3 -c 192.168.56.5Connecting to host 192.168.56.5, port 5201[ 4] local 192.168.56.3 port 49833 connected to 192.168.56.5 port 5201[ ID] Interval Transfer Bandwidth Retr Cwnd[ 4] 0.00-1.00 sec 78.1 MBytes 655 Mbits/sec 135 140 KBytes [ 4] 1.00-2.00 sec 72.0 MBytes 604 Mbits/sec 90 134 KBytes [ 4] 2.00-3.00 sec 72.8 MBytes 611 Mbits/sec 90 130 KBytes [ 4] 3.00-4.00 sec 75.7 MBytes 635 Mbits/sec 90 140 KBytes [ 4] 4.00-5.00 sec 69.8 MBytes 585 Mbits/sec 90 109 KBytes [ 4] 5.00-6.00 sec 50.7 MBytes 425 Mbits/sec 46 31.1 KBytes [ 4] 6.00-7.00 sec 64.8 MBytes 544 Mbits/sec 45 214 KBytes [ 4] 7.00-8.00 sec 75.6 MBytes 634 Mbits/sec 90 221 KBytes [ 4] 8.00-9.00 sec 73.8 MBytes 619 Mbits/sec 90 222 KBytes [ 4] 9.00-10.00 sec 75.1 MBytes 630 Mbits/sec 90 226 KBytes - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth Retr[ 4] 0.00-10.00 sec 709 MBytes 594 Mbits/sec 856 sender[ 4] 0.00-10.00 sec 708 MBytes 594 Mbits/sec receiver

iperf Done.

11

iperf server output

12

[nmmadmin@localhost ~]$ iperf3 -s-----------------------------------------------------------Server listening on 5201-----------------------------------------------------------Accepted connection from 192.168.56.3, port 49832[ 5] local 192.168.56.5 port 5201 connected to 192.168.56.3 port 49833[ ID] Interval Transfer Bandwidth[ 5] 0.00-1.00 sec 75.6 MBytes 634 Mbits/sec [ 5] 1.00-2.00 sec 72.0 MBytes 604 Mbits/sec [ 5] 2.00-3.00 sec 72.9 MBytes 612 Mbits/sec [ 5] 3.00-4.00 sec 75.8 MBytes 636 Mbits/sec [ 5] 4.00-5.00 sec 69.8 MBytes 586 Mbits/sec [ 5] 5.00-6.01 sec 51.9 MBytes 430 Mbits/sec [ 5] 6.01-7.00 sec 62.6 MBytes 531 Mbits/sec [ 5] 7.00-8.00 sec 75.8 MBytes 636 Mbits/sec [ 5] 8.00-9.00 sec 73.2 MBytes 614 Mbits/sec [ 5] 9.00-10.00 sec 75.6 MBytes 634 Mbits/sec [ 5] 10.00-10.03 sec 2.67 MBytes 661 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth[ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec sender[ 5] 0.00-10.03 sec 708 MBytes 592 Mbits/sec receiver

iperf udp measurementpst@wacren00:~$ iperf3 -c 192.168.56.5 -u -b 50MConnecting to host 192.168.56.5, port 5201[ 4] local 192.168.56.3 port 59084 connected to 192.168.56.5 port 5201[ ID] Interval Transfer Bandwidth Total Datagrams[ 4] 0.00-1.00 sec 5.40 MBytes 45.2 Mbits/sec 691 [ 4] 1.00-2.00 sec 5.96 MBytes 50.0 Mbits/sec 763 [ 4] 2.00-3.00 sec 5.95 MBytes 49.8 Mbits/sec 761 [ 4] 3.00-4.00 sec 5.97 MBytes 50.1 Mbits/sec 764 [ 4] 4.00-5.00 sec 5.95 MBytes 49.9 Mbits/sec 762 [ 4] 5.00-6.00 sec 5.96 MBytes 50.0 Mbits/sec 763 [ 4] 6.00-7.00 sec 5.96 MBytes 50.0 Mbits/sec 763 [ 4] 7.00-8.00 sec 5.95 MBytes 49.9 Mbits/sec 762 [ 4] 8.00-9.00 sec 5.98 MBytes 50.2 Mbits/sec 765 [ 4] 9.00-10.00 sec 5.95 MBytes 49.9 Mbits/sec 762 - - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams[ 4] 0.00-10.00 sec 59.0 MBytes 49.5 Mbits/sec 0.000 ms 0/0 (-nan%) [ 4] Sent 0 datagrams

iperf Done.

13

iperf in mininet

• Default topology: h1—s1—h2 (2 hosts, 1 switch) – $ sudo mn– *** Creating network– (…)– mininet> help

• Ping hosts in mininet shell: – pingall– h1 ping h2– iperf– h2 iperf3 -s -D– h1 iperf3 -c h2

• We need to have iperf3 running in server mode on h2 (cannot run interactively in mininet)

• Notes: the mininet `iperf` command uses iperf version 2

14

Simulating latency• Mininet topology hsssh • h1—s1—s2—s3—h2 – sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh– …– mininet>

• h1 ping h2• h2 iperf3 -s -D• h1 iperf3 -c h2• h1 iperf3 -c h2 -u -b 100M

• What’s different? • latency • throughput • UDP tests: no retransmits, no cwnd

15

Simulating losses• Mininet topology hsssh-loss • h1—s1—s2—s3—h2 and losses between s1 and s2 – sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh-loss– …– mininet>

• h1 ping h2• h2 iperf3 -s -D• h1 iperf3 -c h2

• What’s different? • do ping see losses? • latency • retransmits • congestion window (Cwnd)

16

Increased latency

• Mininet topology hsssh-loss-lat • h1—s1—s2—s3—h2 with losses between s1 and s2

and increased delay between s2 and s3 – sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh-loss-lat– …– mininet>

• h1 ping h2• h2 iperf3 -s -D• h1 iperf3 -c h2

• What’s different? • throughput • retransmits • congestion window (Cwnd)

17

iperf

18

Throuput Expectations• Q: What iperf through should you expect to

see on a uncongested 10Gbps network? • A: 3 - 9.9 Gbps, depending on – RTT – TCP tuning – CPU core speed, and ratio of sender speed to receiver

speed

• It is good to have tools to measure!

19

OWAMP• OWAMP = One Way Active Measurement Protocol (RFC 4656)

– E.g. ‘one way ping’

• Some differences from traditional ping: – Measure each direction independently (recall that we often see things like congestion

occur in one direction and not the other) – Uses small evenly spaced groupings of UDP (not ICMP) packets – Ability to ramp up the interval of the stream, size of the packets, number of packets

• OWAMP is most useful for detecting packet train abnormalities on an end to end basis – Loss – Duplication – Out of order packets – Latency on the forward vs. reverse path – Number of Layer 3 hops

• Does require some accurate time via NTP – the perfSONAR toolkit does take care of this for you.

20

About time…• One Way Measurement means… • … time measure happens on different host,

need for accurate time synchronisation – sync to good, close (network wise) time servers – avoid pool.ntp.org servers that are too overloaded or

not close enough to you – perfSONAR configuration provides a different set – even better if you have a Stratum 1 time server on

your network (and good service to your users!) • OWAMP refuses to start if NTP sync is not good enough

21

OWAMP usefulness• OWAMP is very useful in regular testing – Congestion or queuing often occurs in a single direction – Packet loss information (and how often/how much occurs

over time) is more valuable than throughput • This gives you a ‘why’ to go with an observation.

– If your router is going to drop a 50B UDP packet, it is most certainly going to drop a 1500B/9000B TCP packet

• Overlaying data – Compare your throughput results against your OWAMP – do

you see patterns? – Alarm on each, if you are alarming (and we hope you are

alarming …)

22

OWAMP lab

• Mininet topology hsssh – sudo service owampd stop– sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh– …– mininet>

• h2 service owampd start• h1 owping h2• h1 owping -a 1 h2

• Output in 2 directions

23

OWAMP with loss

• Mininet topology hsssh-loss – sudo mn --custom topology/topo-ps-tutorial.py --topo hsssh-loss– …– mininet>

• h2 service owampd restart• h1 owping -a 1 h2

• What’s different? • same losses on each direction?

24

Some Quick Words on BWCTL

• BWCTL is a wrapper around a few tools: – iperf, iperf3, nuttcp, ping, owping, traceroute and tracepath

• Some general notes: – Runs as a daemon and as a CLI client – Use ‘-c’ to specify a ‘catcher’ (receiver) – Use ‘-s’ to specify a ‘sender’ – The bwctl client can be a 3rd machine! – Will default to IPv6 if available (use -4 to force IPv4 as needed, or

specify things in terms of an address if your host names are dual homed)

• Policy specification can do things like prevent tests to subnets, or allow longer tests to others. See the man pages for more details (bwctl.limits)

25

BWCTL tools• Couple of varieties of testers that BWCTL knows how to talk with (use

-T switch): – Iperf2

• Default for the command line (e.g. bwctl –c HOST will invoke this) • Some known problems (CPU hog, hard to get UDP testing to be correct)

– Iperf3 • Default for the perfSONAR regular testing framework, can invoke via command

line switch (bwctl –T iperf3 –c HOST) • New brew, has features iperf2 is missing (retransmissions, JSON output,

daemon mode, etc.) – Nuttcp

• Different code base, more control over how the tool behaves on the host (bind to CPU/core, etc.)

• Similar feature set to iperf3 – Traceroute: useful to do reverse traceroutes – Tracepath: very useful to see MTU and asymmetric routing issues

26

BWCTL lab: preparation• Don’t need mininet for these, can run directly on

the VM (we’ll use only the client) • We need to install it on VM

– cd /etc/apt/sources.list.d– sudo wget http://downloads.perfsonar.net/debian/perfsonar-wheezy-release.list

– wget -qO - http://downloads.perfsonar.net/debian/perfsonar-wheezy-release.gpg.key | sudo apt-key add -

– sudo apt-get update– sudo apt-get install bwctl-client

• BWCTL doesn’t work behind a firewall, it needs a public IP, but we’ll trigger BWCTL tests remotely

27

BWCTL lab

• On the VM: – bwtraceroute -T tracepath -c newy-pt1.es.net -s nmm0.wacren.net– bwtraceroute -T traceroute -c newy-pt1.es.net -s nmm0.wacren.net

– bwtraceroute -T tracepath -c amst-pt1.es.net -s nmm1.wacren.net– bwtraceroute -T traceroute -c amst-pt1.es.net -s nmm1.wacren.net

– bwtraceroute -T tracepath -c nmm1.wacren.net \ -s jnb-wits-1g.perfsonar.ac.za

– bwtraceroute -T traceroute -c nmm1.wacren.net \ -s jnb-wits-1g.perfsonar.ac.za

– bwtraceroute -T tracepath -c nmm0.wacren.net -s pfsnr-pe.kenet.or.ke– bwtraceroute -T traceroute -c nmm0.wacren.net -s pfsnr-pe.kenet.or.ke

28

BWCTL lab• Test reverse direction, swap -s and -c • Try other tools, try other hosts – psmp-gn-bw-01-ath-gr.geant.net – perfsonar2.mi.infn.it – perfsonar02.hephy.oeaw.ac.at – perfmon.dur.scotgrid.ac.uk – psmp-gn-bw-01-tal-ee.geant.net – ps0001.m45.ihep.su – repos.indiacms.res.in – dur-dut-1g.perfsonar.ac.za – ps00-bsas.innova-red.net

29