Network Measurements Session Introduction

Joe MetzgerNetwork Engineering GroupESnet

Eric BoydDeputy Technology OfficerInternet2

July 16 2007Joint Techs at FERMI

Why is Network Measurement Important?• Users dependence on the network is increasing

– Distributed Applications– Moving Larger Data Sets– The network is becoming a critical part of large science experiments

• The network is growing much more complex– ESnet had 6 core devices in 05’, 25+ in 08’– ESnet had 6 core links in 05’, 40+ in 08’, 80+ by 2010?– Dynamic Circuits– Network Security Issues

• The community needs to better understand the network– Users must know what performance levels to expect.– Network Operators need to be able to demonstrate that the network

meets or exceeds those expectations.– Application Developers must understand the ‘wizards gap’ and have

access to tools that differentiate between network problems and application problems.

Data Transfer times over R&E Networks10PB 300,240.0 Gbps 25,020.0 Gbps

3,127.5 Gbps 1,042.5 Gbps 148.9 Gbps 34.7 Gbps

1PB 30,024.0 Gbps 2,502.0 Gbps 312.7 Gbps 104.2 Gbps 14.9 Gbps 3.5 Gbps

100TB 2,932.0 Gbps 244.3 Gbps 30.5 Gbps 10.2 Gbps 1.5 Gbps 339.4 Mbps

10TB 293.2 Gbps 24.4 Gbps 3.1 Gbps 1.0 Gbps 145.4 Mbps 33.9 Mbps

1TB 29.3 Gbps 2.4 Gbps 305.4 Mbps 101.8 Mbps 14.5 Mbps 3.4 Mbps

100GB 2.9 Gbps 238.6 Mbps 29.8 Mbps 9.9 Mbps 1.4 Mbps 331.4 Kbps

10GB 286.3 Mbps 23.9 Mbps 3.0 Mbps 994.2 Kbps 142.0 Kbps 33.1 Kbps

1GB 28.6 Mbps 2.4 Mbps 298.3 Kbps 99.4 Kbps 14.2 Kbps 3.3 Kbps

100MB 2.8 Mbps 233.0 Kbps 29.1 Kbps 9.7 Kbps 1.4 Kbps 0.3 Kbps

5 Minutes 1 Hour 8 Hours 24 Hours 7 Days 30 Days

RED: Something is broken! Usually TCP tuning or HW problems within 100 feet of end points.GREEN: Supported by R&E Backbones today (may have local campus challenges)

WHITE: Requires special engineering.

TCP Throughput LimitsThroughput Limits by RTT and Window Size

1,000,000

10,000,000

100,000,000

1,000,000,000

10,000,000,000

0 20 40 60 80 100 120

RTT in Milliseconds

Throughput in Bits Per Second

16K

32K

64K

128k

512K

1M

2M

8M

Window Size in Bytes

TCP Throughput LimitsThroughput Limits by RTT and Window Size

1,000,000

10,000,000

100,000,000

1,000,000,000

10,000,000,000

0 20 40 60 80 100 120

RTT in Milliseconds

Throughput in Bits Per Second

16K

32K

64K

128k

512K

1M

2M

8M

Window Size in Bytes

Gbps on Campus withany windowsize

Need 1 MB Windows toGet 100 Mbps Cross country

Default OS Window sizes.Is this enoughFor you?

Scale of the Integration Challenge• Measurement infrastructure needs to:

– Obey agreed-upon protocols (schema and semantics)

– Be interoperable across administrative boundaries

– Integrate with middleware (federated trust) infrastructure

– Integrate with circuit provisioning software

Scale of the Deployment Challenge• Universities, national labs, regionals, and

national backbones are all autonomous

• Measurement infrastructure needs to:– Be deployed widely (Metcalf’s Law)– Be locally controlled– Work well with existing local infrastructure– Integrate easily into local processes

Internet2 Connectors

8

MAGPI

3ROXCalREN-2 South

Great Plains Network

Indiana GigaPoP

MREN

Merit

LONI

Internet2

ESnet

NoX

NYSERNet

OARnet

OmniPoPSoX

Oregon GigaPoP Pacific Northwest

GigaPoP

ESnet Connects

9

Lawrence Livermore National Lab (T3)

Brookhaven National Lab (T1)SLAC (T2)

Fermi National Accelerator Lab (T1)

Lawrence Berkeley National Lab (T3)

ESnetArgonne National Lab (T3)

Nine Universities Connect through CalREN-2 South

10

University of Arizona (T3)

UC Irvine (T3) UC Santa Cruz (T3)UC Davis (T2)

UCLA (T3)

UC Riverside (T3)

UC San Diego(T3)

UC Santa Barbara (T3)

California Institute of Technology (T2)

CENIC

Universities Connecting through Oregon GigaPoP and Pacific NW

GigaPoP

11

University of Oregon (T3) University of Washington (T3)

Oregon GigaPOPPacific Northwest GigaPOP

Four Universities Connect through LONI

12

University of Texas, Arlington (T2)

LONI

University of Texas, Dallas (T3)

Southern Methodist University (T3)

University of Mississippi (T3)

Seven Universities Connect through Great Plains Network

13

Oklahoma State University (T3)

Great Plains Network

University of Nebraska-Lincoln (T2)

University of Kansas (T3)University of Oklahoma (T2)

Kansas State University (T3)

University of Iowa (T3)

Iowa State University (T3)

Two Universities Connect through OmniPoP

14

University of Wisconsin, Madison (T3)

OmniPoP

University of Wisconsin, Milwaukee (T2)

Five Universities Connect through MREN

15

University of Notre Dame (T3)

University of Illinois at Chicago (T3) University of Chicago (T2)

Northwestern University (T3)

Univ of Illinois, Urbana-Champaign (T3)

MREN

Universities that Connect through Indiana GigaPoP and OARnet

16

Indiana University (T2)

Indiana GigaPoP

Purdue University (T2)

OARnet

Ohio State University (T3)

Two Universities Connect through Merit

17

Michigan State University (T2)

Merit

University of Michigan (T2)

Eight Universities Connect through SoX

18

Florida International University (T3)

Duke University (T3)University of Florida (T2)

Vanderbilt University (T3)

University of Puerto Rico (T3)

Florida State University (T3)

University of Tennessee (T3)

University of South Carolina (T3)

SoX

Two Universities Connect through 3ROX

19

Carnegie Mellon University (T3)

3ROX

University of Pittsburgh (T3)

Three Universities Connect through MAGPI

20

Princeton University (T3)

MAGPI

University of Pennsylvania (T3)

Rutgers University (T3)

Seven Universities Connect through NYSERNet

21

SUNY Albany (T3)

Columbia University (T3)New York University (T3)

University of Rochester (T3)

SUNY Stony Brook (T3)

Cornell University (T3)

SUNY Buffalo (T3)

NYSERNet

Nine Universities Connect through NoX

22

MIT (T2 and T3)

Brandeis University (T3)Harvard University (T2)

Boston University (T2 and T3)

Brown University (T3)

Yale University (T3)

Northeastern University (T3)Tufts University (T3)

U Mass, Amherst (T3)

NoX

LHC Measurement Requirements 11. Monitor up/down status of cross domain circuits

A. Publish status via a web services interfaceB. Provide tools to visualize stateC. Generate NOC alarms when circuits change states

2. Monitor Link/Circuit Capacity, Errors & Utilization

A. Publish statistics via a web services interfaceB. Provide tools to visualize the dataC. Generate NOC alarms when thresholds are crossed

LHC Measurement Requirements 23. Continuously measure delay between participants

A. Manage multiple sparse meshs of continuous tests and store results in an MA

B. Publish results via a standardized web service interfaceC. Provide a tool to visualize the dataD. Provide tools to automatically analyze data and generate NOC

alarms

4. Make scheduled bandwidth measurements across paths of interest

A. Manage multiple regularly scheduled sparse meshes of tests and store results in an MA

B. Publish results via a standardized web service interfaceC. Provide a tool to visualize the dataD. Provide tools to automatically analyze data and generate NOC

alarms

LHC Measurement Requirements 35. Measure & Publish Topology of both primary

and backup pathsA. Publish statistics via a web services interface

B. Provide tools to visualize the data over time

Directions Forward• Deploy measurement tools

– To quantify the service your receiving/delivering

• Set User Expectations– 100 to 300 Mbps per stream

• Educate your user base– So they know what is possible

Questions?• Joe Metzger (metzger@es.net)

• Eric Boyd (eboyd@internet2.edu)