1
ESnet UpdateSummer 2007
Joint Techs Workshop
Joe BurresciaESnet General Manager
July 16,2007
Energy Sciences NetworkLawrence Berkeley National Laboratory
Networking for the Future of Science
2
TWC
SNLL
YUCCA MT
BECHTEL-NV
PNNLLIGO
INEEL
LANL
SNLAAlliedSignal
PANTEX
ARM
KCP
NOAA
OSTI ORAU
SRS
JLAB
PPPL
Lab DCOffices
MIT
ANL
BNL
FNALAMES
NR
EL
LLNL
GA
DOE-ALB
OSC GTNNNSA
International (high speed)10 Gb/s SDN core10G/s IP core2.5 Gb/s IP coreMAN rings (≥ 10 G/s)Lab supplied linksOC12 ATM (622 Mb/s)OC12 / GigEthernetOC3 (155 Mb/s)45 Mb/s and less
NNSA Sponsored (12)Joint Sponsored (3)
Other Sponsored (NSF LIGO, NOAA)Laboratory Sponsored (6)
42 end user sites
SINet (Japan)Russia (BINP)CA*net4
FranceGLORIAD (Russia, China)Korea (Kreonet2
Japan (SINet)Australia (AARNet)Canada (CA*net4Taiwan (TANet2)Singaren
ESnet IP core: Packet over
SONET Optical Ring and Hubs
ELP
DC
commercial peering points
MAE-E
PAIX-PAEquinix, etc.
PN
WG
Po
P/
PA
cifi
cWav
e
ESnet 3 with Sites and Peers (Early 2007)
ESnet core hubs IP
Abilene high-speed peering points with Internet2/Abilene
Abilene
Ab
ilen
e
CERN(USLHCnet
DOE+CERN funded)
GÉANT - France, Germany, Italy, UK, etc
NYC
Starlight
SNV
Ab
ilene
JGI
LBNL
SLACNERSC
SNV SDN
SDSC
Equinix
SNV
ALB
ORNL
CHI
MRENNetherlandsStarTapTaiwan (TANet2, ASCC)
NA
SA
Am
es
AU
AU
SEA
CH
I-SL M
AN
LA
NA
bile
ne
Specific R&E network peers
Other R&E peering points
UNM
MAXGPoP
AMPATH(S. America)
AMPATH(S. America)
ES
net
Scie
nce D
ata
Netw
ork
(S
DN
) core
R&Enetworks
Office Of Science Sponsored (22)
ATL
NSF/IRNCfunded
Equinix
3
ESnet 3 Backbone as of January 1, 2007
Sunnyvale
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)MAN rings (≥ 10 G/s)Lab supplied links
Seattle
San Diego Albuquerque
El Paso
Chicago New York
City
Washington DC
Atlanta
Future ESnet Hub
ESnet Hub
4
ESnet 4 Backbone as of April 15, 2007
Clev.
Boston
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Future ESnet Hub
ESnet Hub
5
ESnet 4 Backbone as of May 15, 2007
SNV
Clev.
Boston
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Future ESnet Hub
ESnet Hub
6
ESnet 4 Backbone as of June 20, 2007
Clev.
Boston
Houston
Kansas City
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Future ESnet Hub
ESnet Hub
7
ESnet 4 Backbone Target August 1, 2007
Clev.
Boston
Houston
Los Angeles
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Future ESnet Hub
ESnet Hub
8
ESnet 4 Backbone Target August 30, 2007
Clev.
Boston
Houston
Boise
Los Angeles
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Future ESnet Hub
ESnet Hub
9
ESnet 4 Backbone Target September 30, 2007
Clev.
Boston
Houston
Boise
Los Angeles
10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Nashville
Future ESnet Hub
ESnet Hub
10
DC
ESnet 4 Backbone Target September 15, 2008
Houston 10 Gb/s SDN core (NLR)10/2.5 Gb/s IP core (QWEST)10 Gb/s IP core (Level3)10 Gb/s SDN core (Level3)MAN rings (≥ 10 G/s)Lab supplied links
Clev.
Houston
Kansas City
Boston
Sunnyvale
Seattle
San Diego Albuquerque
El Paso
Chicago
New York City
Washington DC
Atlanta
Denver
Los Angeles
Nashville
Future ESnet Hub
ESnet Hub
11ESnet4Core networks 50-60 Gbps by 2009-2010 (10Gb/s circuits),
500-600 Gbps by 2011-2012 (100 Gb/s circuits)
Cle
vela
nd
Europe(GEANT)
Asia-Pacific
New York
Chicago
Washington DC
Atl
anta
CERN (30+ Gbps)
Seattle
Albuquerque
Au
str
ali
a
San Diego
LA
Denver
South America(AMPATH)
South America(AMPATH)
Canada(CANARIE)
CERN (30+ Gbps)Canada(CANARIE)
Asi
a-Pac
ific
Asia Pacific
GLORIAD (Russia and
China)
Boise
HoustonJacksonville
Tulsa
Boston
Science Data Network Core
IP Core
Kansa
s
City
Au
str
ali
a
Core network fiber path is~ 14,000 miles / 24,000 km
162
5 m
iles
/ 2
545
km
2700 miles / 4300 km
Sunnyvale
Production IP core (10Gbps)
SDN core (20-30-40-50 Gbps)
MANs (20-60 Gbps) or backbone loops for site access
International connections
IP core hubs
Primary DOE LabsSDN hubs
High speed cross-connectswith Ineternet2/AbilenePossible hubs
USLHCNet
12Typical ESnet 4 Hub
M320 Router7609 Switch
10G Performance Tester
Peering Router
Secure Term ServerPower Controllers
OWAMP Measurement
Device
13ESnet 4 Factiods as of July 16, 2007
• Installation to date:o 10 new 10Gb/s circuits
o ~10,000 Route Miles
o 6 new hubs
o 5 new routers 4 new switches- Total of 70 individual pieces of equipment shipped
– Over two and a half tons of electronics
o 15 round trip airline tickets for our install team- About 60,000 miles traveled so far….
- 6 cities– 5 Brazilian Bar-B-Qs/Grills sampled
14ESnet Traffic Now Exceeding 2 PetaBytes/Month
TBytes Accepted
0.00
500.00
1000.00
1500.00
2000.00
2500.00
3000.00
Jan, 00 Jan, 01 Jan, 02 Jan, 03 Jan, 04 Jan, 05 Jan, 06 Jan, 07
2.4PBytes in May 2007
Te
raB
ytes
ESnet traffic historically has increased 10x every 47 months
1 PBytes in April 2006
15
ESnet Availability 8/2006 through 7/2007
0
500
1000
1500
2000
2500
ORNL 100.000SLAC 100.000NERSC 99.998SNLL 99.998
LBL 99.997PNNL 99.997LLNL 99.996FNAL 99.995IARC 99.995SRS 99.995
Yucca 99.993BJC 99.991ANL 99.990JGI 99.990
LANL-DC 99.986LLNL-DC 99.986
MIT 99.985JLab 99.982PPPL 99.982
GA 99.973BNL 99.970
DOE-ALB 99.955
LANL 99.955SNLA 99.955Allied 99.951Pantex 99.949Bechtel 99.948
NREL 99.941
DOE-NNSA 99.918DOE-GTN 99.889
INL 99.883
ORAU 99.878OSTI 99.868
Ames-Lab 99.851
NOAA 99.782Lamont 99.624
Outage Minutes
Aug
SepOct
Nov
Dec
Jan
Feb
MarApr
May
Jun
Jul
ESnet Availability
“5 nines” (>99.995%) “3 nines” (>99.5%)“4 nines” (>99.95%)
Dually connected sites Note: These availability measures are only for ESnet infrastructure, they do not include site-related problems. Some sites, e.g. PNNL and LANL, provide circuits from the site to an ESnet hub, and therefore the ESnet-site demarc is at the ESnet hub (there is no ESnet equipment at the site. In this case, circuit outages between the ESnet equipment and the site are considered site issues and are not included in the ESnet availability metric.
16OSCARS Overview
Path Computation• Topology
• Reachability• Contraints
Scheduling• AAA
• Availability
Provisioning• Signalling• Security
• Resiliency/Redundancy
OSCARSGuaranteedBandwidth
Virtual Circuit Services
On-demand Secure Circuits and Advance Reservation System
17OSCARS Status Update
• ESnet Centric Deploymento Prototype layer 3 (IP) guaranteed bandwidth virtual circuit service deployed in ESnet
(1Q05)o Layer 2 (Ethernet VLAN) virtual circuit service under development
• Inter-Domain Collaborative Effortso Terapaths
- Inter-domain interoperability for layer 3 virtual circuits demonstrated (3Q06)- Inter-domain interoperability for layer 2 virtual circuits under development
o HOPI/DRAGON- Inter-domain exchange of control messages demonstrated (1Q07)- Initial integration of OSCARS and DRAGON has been successful (1Q07)
o DICE- First draft of topology exchange schema has been formalized (in collaboration with NMWG)
(2Q07), interoperability test scheduled for 3Q07- Drafts on reservation and signaling messages under discussion
o UVA- Integration of Token based authorization in OSCARS under discussion
18ESnet Measurement Update
• perfSONAR Services Deployedo Measurement Archives serving Utilization information (Demo
later today)o E2EMP serving end-to-end circuit status informationo Visualization tools
• Bandwidth & Latency Measurement Pointso Deployed at ESnet4 hubs as they are configured
• Working with the LHC communityo To define a set of perfSONAR services that can meet the
network measurement & monitoring needs of large distributed science application communities.