Technical Rehearsals Matter!

Bob Riddle, Internet2

14 January 2004

Network Trouble-Shooting

Rehearse, rehearse, & rehearse

Questions to answer before opening night•Can I “stretch” the stage from here to there?

– do you need a 2 lane road or an expressway?

– will you need the highway during rush hour? –do you need to be able to “talk”, “listen”, or both?

•Do I live in the right kind of “neighborhood” ?– time to think about FDX, multicast, & switches– time to make some new friends & acquire new toys

•What sort of guarantees can I offer my boss?

Stretching the “Stage”

•Speed: do you need a freeway, city street, or a dirt road? (mbps)

•Latency: is it a “telephone” call or a “movie”? (rtt)

•Packet Loss: how much of the content do you care about? (100% … 90% … 50% … 10% …)

Issues to consider when

“stretching” the stage

Stretching the “Stage” -cont-

Plan for a meaningful rehearsal

What kind of highway do you need?• H.323 ~= 384 kbps• Mpeg1 ~= 1.5 mpbs• Mpeg2 ~= 4 – 16 mbps• DV ~= 30 mbps• HDTV ~= 19.2 – 270 mbps

What kind street do you really need?• One way? (streaming, broadcast)• Two way? (interactive voice & video)• Many way? (more than 2 end points)

Do the math! (and budget for the asphalt!)

Rush hour traffic or 3 a.m. traffic?

Some lessons learned from the Bing Theater• Good idea - Test early & test often

– Sunday evening exercising the network

– Monday morning “exercise”– Monday afternoon “exercise”

• Result – Hayward suffered from congestion• Lessons learned:

– Sun night traffic != to weeknight network traffic– Mon a.m. traffic != to Mon p.m. network traffic

it’s important to rehearse the network during show time

You have to live in the Right Neighborhood

Take a look at your local neighborhood

1. Run internal tests across your LAN

2. Make sure to test what you hope to use!

3. Find a friend “next door” to test with

4. Learn about ping, traceroute, iperf, Internet2 Detective, VRVS & what they can tell you

5. Start thinking about what “roads” you’ll need to travel to the next neighborhood(s)

My list or “normal” E2E tests

Bandwidth tests• iperf (http://dast.nlanr.net/Projects/Iperf/)

– allows for tcp, udp, multicast, bi-directional– allows testing for FDX needs– allows testing at various rates (KkMm)

Multicast Tests• iperf (http://dast.nlanr.net/Projects/Iperf/)• Vic, rat (http://www.vrvs.org)• inSORS (http://internet2.insors.net)• rcBridge (http://if.anu.edu.au/SW/rcbridge.html)

Use the tools!

Results of such tests – NWS example!• bandwidth tests:

– discovered DHCP server gateway problem at NWS

– discovered 30 mbps with 0% packet loss was the best

• FDX testing:– discovered misconfiguration between PC & switch in Ann

Arbor Internet2 office– PC diagnostics said everything was fine, switch display

indicated HDX connection to the PC

• Human tests:– involve participants in “successful” network testing– need someone with the correct event perspective

Can I buy these tools at Wal-Mart?

What did we learn from the Rialto event?• 2 days to learn/fix/configure the network (help!)• What if we could have shipped a no-brainer PC to “almost” endpoint that would all you to remotely exercise all these tools?

Introducing the “cakebox”• No operator, no monitor, keyboard, or mouse required• Small, inexpensive, and easy to ship PC• Just plug in a network cable & a power cable• Provide user interface for non-network geeks

http://envoy.internet2.edu/pioneer/

Tools - cont -

http://detective.internet2.edu

Runs on Windows XP

Runs on Mac OSX

Runs E2E on detective machines

Its purpose is to tell you if there is any “hope”!

Other tools you can use

http://e2epi.internet2.edu//

http://www.pingplotter.com/

http://www.insecure.org/nmap/

http://www.traceroute.org/

E2EP (end-to-end performance group)

Questions?

www.internet2.edu