Supporting Advanced Scientific Computing Research • Basic Energy Sciences • Biological and Environmental Research • Fusion Energy
Sciences • High Energy Physics • Nuclear Physics
ESnetUpdate
Steve Cotter, Dept Head Joint Techs Winter 2009
College Station, TX
• Network Update • OSCARS • Futures
Agenda
ESnet4 – Jan 2009
StarLight
MAN LAN (32 A of A)
PNNL
FNL
ORNL
LLNL
GA
BNL
LANL
IP router
Lab
Optical node
SDN router Lab Link
MAN NLR 10G 20G SDN
SDN IP
Peering Link
SINet (Japan) Russia (BINP)
CERN/LHCOPN (USLHCnet:
DOE+CERN funded) GÉANT - France, Germany, Italy, UK, etc
CA*net4 France GLORIAD (Russia, China) Korea (Kreonet2
MREN StarTap Taiwan (TANet2, ASCGNet)
AMPATH CLARA
(S. America)
CUDI (S. America)
Japan (SINet) Australia (AARNet) Canada (CA*net4 Taiwan (TANet2) Singaren Transpac2 CUDI
KAREN/REANNZ ODN Japan Telecom America NLR-Packetnet Internet2 Korea (Kreonet2)
KAREN / REANNZ Transpac2 Internet2 Korea (kreonet2) SINGAREN Japan (SINet) ODN Japan Telecom America
CA*net4
GÉANT in Vienna (via USLHCNet circuit)
ESnet Confidential
2008 Hub & Wave Install Timeline
JUL AUG SEP OCT NOV DEC
CHIC STAR
NASH ATLA
DENV
PNWG
WASH
NEWY AOFA
CLEV BOST
HOUS KANS
ALBU ELPA
LOSA SDSC
DENV BOIS
Created by Mike O’Connor Mod by JimG
1st set of Juniper MX’s arrived at
LBNL in Mid June
2nd set of Juniper MX’s arrived at
LBNL in Mid Sept
MX480 IP MX960 SDN MX960 SDN
MX960 IP
MX960 SDN
MX960
MX960
MX960
MX960
MX960
MX480
MX480&MX960
MX960 x2
MX480 MX960
MX960 x2
MX960 x2
MX480
MX480
MX960
MX960
2008 HUB Installs 6 MX480’s
19 MX960’s SITE Installs 1 M120 PPPL
1 M10i LASV-HUB
PPPL M120
LASV M10i
NEW HUB INSTALL
EXISTING HUB UPGRADE
1 OC12 LANV-SUNN 1 10GE Internet2 STAR-CHICHIC
14 10GE Internet2 wave installed/split & accepted
19 10GE Internet2 waves installed/split & accepted
6 10GE Internet2 waves installed, split & accepted
1 10G Framenet XC in WASH
2 10GE Internet2 waves installed, split & accepted
1 10GE NLR AOFA-WASH 1 ORNL-NASH 10G IP
1 10GE MAN-LAN #2 1 10GE NRL Temp WASH-STAR 1 10GE CIC-OMNIPop at STAR
Current Hub Count: • 21 Completed: 32 AofA, NEWY*, WASH, ATLA, NASH, CLEV*, BOST*,
CHIC, STAR, KANS*, HOUS*, ELPA*, DENV, ALBU, BOIS*, PNWG, SUNN, SNV(Qwest), LOSA*, SDSC, LASV(SwitchNap)* *9 New Hubs since July 2008
Current Backbone Wave Count: • Internet2 / Level3 Waves:
– IP Waves: 17 new/split for a total of 25 – SDN Waves: 25 new/split for a total of 30
• NLR Waves: – 1 new wave for a total of 5 – 1 temp wave (STAR-WASH) for used during NLR northern path upgrade
Hub & Wave Count
MAN Upgrades Timeline
ESnet Confidential
JAN FEB MAR
MX960
SNV
Created by Mike O’Connor Mod by JimG
LBL-MR2, SNV-MR1, SNLL-MR2. LL NL-MR2 (complete)
MX960
LBNL
MX480
SNLL MX480
LLNL MX480
SLAC
MX960
NERSC
MX960
FNAL MX480
FNAL
MX960
LBNL
MX480
JGI SLAC-MR2 (complete)
FNAL’s MX Shipping Feb 3rd
MX960
BNL
MX960
ANL
MX480
BNL Final BAMAN 6509 replacements mid Feb.
1 10GE FRPG (upgrade from 1GE) DENV
2 new 10GE LIMAN (waves #3 & #4) & DF circuit between AofA-NEWY to BNL
ESnet4 Metro Area Rings
Newport News - Elite
JLab ELITE
ODU
MATP
Wash., DC
Atlanta MAN
180 Peachtree
56 Marietta (SOX)
ORNL (backup)
Wash., DC
Houston
Nashville
San Francisco Bay Area MAN
LBNL
SLAC
JGI
LLNL
SNLL
NERSC
SU
NN
USLHCNet
FNAL
600 W. Chicago
Starlight
ANL
West Chicago MAN Long Island MAN
BNL
111- 8th (NEWY)
32 AoA, NYC
USLHCNet
• LI MAN expansion, BNL diverse entry • FNAL and BNL dual ESnet connection • Upgraded Bay Area MAN switches
• 1 OC-12 between SUNN-LASV (supporting Las Vegas sites and future OC-12 to GA)
• 1 10GE NLR AofA-WASH (backup for SDN & USLHCnet)
• 1 ORNL-NASH 10G IP • 1 10G FrameNet connection at WASH hub • 1 10GE MAN-LAN connection (second10G peering link) • 1 10GE NRL temp WASH-STAR • 1 10GE CIC-OMNIPop at STAR • 1 10GE FRPG (upgrade from 1GE) DENV
New ESnet4 Connections
• New LIMAN 10G waves #3 & #4 to BNL (Early Feb) • OC12 between LASV hub and General Atomic (Feb) • 10GE between BOST hub to MIT (Feb) • OC12 between DENV hub and Pantex (TBD) • 1GE wave in BOIS to INL via IRON (TBD) • 10GE SDN wave between PNWG hub to PNNL (TBD) • 10GE SDN wave between NASH hun to ORNL (TBD)
Future Installs
0
500
1000
1500
2000
LIG
O
100.
000
MS
RI
100.
000
NG
A 10
0.00
0%
NS
TEC
10
0.00
0%
OR
NL
100.
000
GA
99.9
99
JGI
99.9
99
LBL
99.9
99
PN
NL
99.9
99
SLA
C
99.9
99
SN
LL
99.9
99
NE
RS
C
99.9
98
BN
L 99
.996
B
JC
99.9
95
FNA
L 99
.995
LL
NL
99.9
95
DO
E-
NN
SA
SR
S
99.9
94
Y12
99
.994
P
PP
L 99
.992
Yu
cca
99.9
92
MIT
99
.990
A
NL
99.9
89
IAR
C
99.9
89
DO
E-A
LB
99.9
78
LAN
L 99
.978
S
NLA
99
.978
P
ante
x 99
.977
LA
NL-
DC
99
.976
LL
NL-
DC
99
.976
K
CP
99.9
73
JLab
99
.964
N
RE
L 99
.961
IN
L 99
.892
D
OE
-GTN
99
.885
N
OA
A 99
.855
O
STI
99
.852
B
echt
el
99.8
08
Am
es-L
ab
99.7
67
Lam
ont
99.7
18
OR
AU
99
.660
Out
age
Min
utes
Site Availability 2/2006 to 1/2007 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
0
500
1000
1500
2000
AN
L 10
0.00
FN
AL
100.
00
Lam
ont
100.
00
LIG
O
100.
00
OR
NL
100.
00
PN
NL
100.
00
SN
LL
100.
00
DO
E-A
LB
99.9
99
JGI
99.9
99
LBL
99.9
99
MS
RI
99.9
99
NG
A 99
.999
S
LAC
99
.999
Yu
cca
99.9
99
NN
SA
99.9
98
SN
LA
99.9
98
LAN
L 99
.997
B
NL
99.9
94
MIT
99
.993
S
RS
99
.993
A
mes
-Lab
99
.991
P
PP
L 99
.991
IN
L 99
.989
N
ER
SC
99
.989
N
STE
C
99.9
89
NR
EL
99.9
85
LLN
L 99
.984
K
CP
99.9
81
IAR
C
99.9
73
GA
99.9
72
LAN
L-D
C
99.9
70
LLN
L-D
C
99.9
70
Pan
tex
99.9
55
DO
E-G
TN
99.9
20
JLab
99
.913
B
echt
el
99.8
96
Y12
99
.865
B
JC
99.8
64
NO
AA
99.8
64
OR
AU
99
.854
O
STI
99
.852
Out
age
Min
utes
Site Availability 2/2008 to 1/2009 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan
5 9s 41%
4 9s 22%
3 9s 37%
Improved Site Availability
5 9s 39%
4 9s 19%
3 9s 32%
2 9s 10%
• Network Update • OSCARS • Futures
Agenda
OSCARS Current (v0.5) Implementation
User User App
AAAS Authentication Authorization Auditing Subsystem
Notification Broker API Resv API
WBUI Web Based User Interface
OSCARS Core - Reservation Management - Path Computation - Scheduling - Inter-Domain Communications
PSS Path Setup Subsystem - Network Element Interface
NS NotificationSubsystem
HTTPS HTTPS (SOAP) RMI SSHv2
ESnet Public WebServer (Proxy)
WS Interface
IDC InterDomain Controller
Source
Sink
SDN
SDN
IP
IP
IP
IP Link
ESnet WAN
SDN
Notification Call-back Event API
• Well defined inter-module interfaces • Exchange of static topology information • PCE integrated into OSCARS Core
ESnet IDC (OSCARS)
OSCARS Future Implementation
User User App
AAAS Authentication Authorization Auditing Subsystem
Notification Broker API Resv API
WBUI Web Based User Interface
OSCARS Core - Reservation Management - Scheduling - Inter-Domain Communications
PSS Path Setup Subsystem - Network Element Interface
NS NotificationSubsystem
HTTPS HTTPS (SOAP) RMI SSHv2
ESnet Public WebServer (Proxy)
WS Interface
IDC InterDomain Controller
PCE Path Computation Engine
ESnet IDC (OSCARS)
Notification Call-back Event API
Source
Sink
SDN
SDN
IP
IP
IP
IP Link
ESnet WAN
SDN
• Exchange of dynamic topology information • includes time dimension
• PCE separated from OSCARS Core • PCEs can be daisy changed • allows PCE to be pluggable • facilitates a research framework for collaboration
• Operational VC support – As of 12/2/08, there are 16 long-term production VCs
instantiated, all of which support HEP • 4 VCs terminate at BNL • 2 VCs support LHC T0-T1 (primary and backup) • 12 VCs terminate at FNAL • 2 VCs support LHC T0-T1 (primary and backup) • For BNL and FNAL LHC T0-T1 VCs, except for the ESnet PE router
at BNL (bnl-mr1.es.net) and FNAL (fnal-mr1-es.net), there are no other common nodes (router), ports (interfaces), or links between the primary and backup VC.
• Short-term dynamic VCs – Between 1/1/08 and 12/2/08, there were roughly 2650
successful HEP centric VCs reservations • 1950 reservations initiated by BNL using Terapaths • 1700 reservations initiated by FNAL using LambdaStation
OSCARS Use within ESnet
OSCARS is a Production Service
Site VLANS ESnet PE
ESnet Core
USLHCnet (LHC OPN)
VLAN
USLHCnet VLANS
USLHCnet VLANS
USLHCnet VLANS
USLHCnet VLANS
Tier2 LHC VLANS
T2 LHC VLAN
Tier2 LHC VLANS
OSCARS setup all VLANs
OSCARS generated and managed virtual circuits at FNAL – one of the US LHC Tier 1 data centers. This circuit map (minus the yellow callouts that explain the diagram) is automatically generated by an OSCARS tool and assists the connected sites with keeping track of what circuits exist and where they terminate.
Spectrum Now Monitors OSCARS Circuits
• Network Update • OSCARS • Futures
Agenda
Historical ESnet Traffic Patterns
Log Plot of ESnet Monthly Accepted Traffic, January 1990 – December 2008
Tera
byte
s / m
onth
Oct 1993 1 TBy/mo.
Aug 1990 100 MBy/mo.
Jul 1998 10 TBy/mo.
38 months
57 months
40 months
Nov 2001 100 TBy/mo.
Apr 2006 1 PBy/mo.
53 months
Traffic Increases 10X Every 47 Months
July 2010 10 PBy/mo.
Network Traffic, Science Data, and Network Capacity
Historical Projection
All
Four
Dat
a S
erie
s ar
e N
orm
aliz
ed to
“1” a
t Jan
. 199
0
(HEP data courtesy of Harvey Newman, Caltech, and Richard Mount, SLAC. Climate data courtesy Dean Williams, LLNL, and the Earth Systems Grid Development Team.)
4 Pb/y 2010
40 Pb/y 2010
ESnet4 – 2010
StarLight
MAN LAN (32 A of A)
PNNL
FNL
ORNL
LLNL
GA
BNL
LANL
IP router
Lab
Optical node
SDN router Lab Link
MAN NLR 10G
30/40/50G SDN IP
50
50
30
30
50
40
40
40
40
40
40