Experiences with theQWEST/SCinet Bandwidth

Challenge2000-2002

John Shalf, Wes BethelLBNL

Shaky City Data Caches HPC Resources

Sensor Nets

Simulations

STM

HandheldDevices

Dr Jane

Collaborators

Visapult Architecture

SourceVolume

Backend

Viewer Component

Visapult Architecture

SourceVolume

3D 2D

Visapult Architecture

SourceVolume

3DGigabits

2DMegabits

Visapult Architecture

SourceVolume

3DGigabits

2DMegabits

SC2000 Demo Configuration

NTON

8 node StorageCluster (DPSS)

Network Throughput: 5 sec peak 1.48 Gbits/sec (72 streams: 20.5Mbits/stream); 60 minute sustained average: 582 Mbits/sec

Compute Cluster

(8 nodes)

Berkeley Lab:.75 TB, 4 server DPSS

ANL Booth Linux Cluster

OC-48OC-48

2 x 1000 BT

HSCC

SGI Origin (8 CPU)

1.5 Gb/s4 x 1000BT

QwestASCI Booth:

SGI Origin (8 CPU)

4 x 1000BT

Visapult VisualizationApplication

File Transfer Application

SC2000 Network Throughput

Refactoring the Design• Congestion avoidance

- Good for internet- Bad bad baaaad for PVCs and other dedicated networks.

(switched lambdas?)

• Multistream TCP- Erratic performance- Requires a lot of tuning- Unfriendly to other users- Unfriendly to visualization applications

• We want full control of the “throttle”- Very much like network video

Refactoring the Design• TCP is the wrong thing for interactive vis!

- Layer 3 latency/jitter (all buffering effects)- Poor response to bursty traffic- Vis needs interactivity and minimal latency!

• Network Video / UDP streams- Present packets to app. immediately (low latency) - Full control of data rate- Lossy, but effects of loss can be managed

• SOCK_RDM

Effect of Loss on Visapult

Evolving Binary Black Hole Merger Simulation (100 timesteps)

SC2001 BWC Testbed

SC2001 BWC Results

•3.3 Gigabits Peak (MRTG of ESNet/QWEST OC-48)•Lots of time spent tuning (packet pacing is nontrivial)

SC2002 Planning• Better UDP packet pacing with poor clock

granularity- Error diffusion algorithm from dithering

• Omnidirectional viewing for Visapult

• Nascent Grid Infrastructure- Global Grid Testbed Collaboration (GGTC)- GGTC won 3 out of 4 HPC challenge awards

Global Testbed

Bandwidth Scavenging Service

Czech Republic

90% utilization on path from Brno

To GEANT (to SC) via Praha

Steady @ 16+ Gigabits!Visapult/Cactus SC02 Bandwidth Challenge Results

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

0 100 200 300 400 500 600 700 800

Time (seconds)

Megabits

10G-Link-1

10G-Link-2

10G-Link-3

Cumulative

Why do we do this??• Opportunity to test the next generation network capacity

(2 years before production)• Opportunity to test software ideas that will be required to

make use of the next generation network (2 year lead)• Opportunity to test future network engineers (give them a

2 year lead on the problems with future networks)• Prescient

- I-Way 1995 (presaged vBNS experiences and “The Grid”)- 1997 multi-stream tests (presaged multi-stream TCP protocols)- 2001 10GigE and UDP-based protocols

• 10GigE backbone for SC2002• RUDP, SABUL, TSUNAMI user-space UDP-based protocols

What should you know for 2003• Manual throttle (UDP-based protocols) are here to stay.

- Hopefully SOCK_RDM will cover most needs- Whaaa? Those idiots are going to burn down the network! Next big

thing: resource management

• RSVP & DiffServ were developed to manage this very situation with regard to network video

• RSVP & DiffServ are never going to happen- Gregory Bell, “Failure to Thrive: QoS and the Culture of Operational

Networking,” Proceedings of the ACM SIGCOMM 2003 Workshops, RIPQoS Workshop.

• Next Big Thing? : Pluggable Congestion Management- AIMD for internet (can even mimic multistream TCP behavior)- Fixed rate for PVCs and switched lambdas- Get hints from monitoring services (NWS) to manage flow end-to-end

Concern about the future of BWC• The future and eternal reigning champion

of the BWC?

Concern about the future of BWC• The future and eternal reigning champion

of the BWC?

iperf --udp -b <infinity>

Concern about the future of BWC• The future and eternal reigning champion of the BWC?

- How do we prevent this?

Concern about the future of BWC• The future and eternal reigning champion of the BWC?

- How do we prevent this?

• There is not technical solution to this problem- It is a social problem, not a technical one

Concern about the future of BWC• The future and eternal reigning champion of the BWC?

- How do we prevent this?

• There is not technical solution to this problem- Can’t ban UDP- Can’t eliminate high bandwidth category

Concern about the future of BWC• The future and eternal reigning champion of the BWC?

- How do we prevent this?

• There is not technical solution to this problem- Can’t ban UDP- Can’t eliminate high bandwidth category

• Must answer- Does this advance science or art?- Is this a glimpse of the future?

Concern about the future of BWC• The future and eternal reigning champion of the BWC?

- How do we prevent this?

• There is not technical solution to this problem- Can’t ban UDP- Can’t eliminate high bandwidth category

• Must answer- Does this advance science or art?- Is this a glimpse of the future?- What the heck are judges for anyways?

Acknowledgements• Wes Bethel• Brian Tierney• Dan Gunter• Jason Lee• Eli Dart• Mike Bennett• Chip Smith• John Christman• Raju Shah• Ed Seidel• Gabrielle Allen• Tom Goodale• Thomas Radke• Mirek Ruda• Ludek Matskya

• NERSC• LBNL• NCSA• PSC• ANL• AEI/Potsdam• University of Amsterdam• ESNet• QWEST• Abilene• StarLight• Teragrid • Force10 Networks• Scinet

More Information• Visapult:

- V1.0: http://vis.lbl.gov/projects/visapult- V2.0: http://vis.lbl.gov/projects/visapult2

• Cactus- http://www.cactuscode.org

• VisPortal (Grid Portal for managing Visapult)- http://frost.lbl.gov/visportal/

• SC2002 Global Grid Testbed- http://scb.ics.muni.cz/static/SC2002/Testbed.html

• Related Publications- Special Issue on Visualization on the Grid, IEEE Computer

Graphics and Applications, Volume 23, Number 2, March/April 2003.