tab, 03. march 2006 bruno hoeft german lhc – wan present state – future concept...
TRANSCRIPT
TAB, 03. March 2006Bruno Hoeft
German LHC – WANpresent state – future concept
Forschungszentrum Karlsruhe GmbHInstitute for Scientific Computing
P.O. Box 3640D-76021 Karlsruhe, Germany
http://www.gridka.de
Bruno Hoeft
TAB, 03. March 2006Bruno Hoeft
GridKaFZK-LAN
WAN (March 03rd. 06)
DWDM10GE
FZKR-Inet-IWR
X-WiNDWDMAR
HEP
10GE
T0 DWDM lightpath
T0‘ DWDM lightpathBackup (end 2006)T1 DWDM lightpathBandwidth ntd
T2 DWDM lightpathBandwidth ntd
10GE physical link
1GE physical link
TAB, 03. March 2006Bruno Hoeft
X-WiNAR
GridKa - backboneFZK-LAN
HEP
DWDM
10GE10GE
10GE
T0 DWDM lightpath
T0 DWDM lightpathBackup (end 2006)T1 DWDM lightpathBandwidth ntd
T2 DWDM lightpathBandwidth ntd
10GE physical link
1GE physical link
T0CERN T1- SARA (CBF – DFN / Surfnet)
WAN (March/April 06)
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
STATUS PHONE
WS-X6548-GE-TX
4748
373835
3625
262324
131411
121
2
4 8 P O R T
12
11
9
10
7
8
5
6
3
4
1
2
24
23
21
22
19
20
17
18
15
16
13
14
36
35
33
34
31
32
29
30
27
28
25
26
48
47
45
46
43
44
41
42
39
40
37
38
10/100/1000 BASE-TETHERNET
SWITCHING MODULE
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
WS-X6748-SFP
48 PORT GIGABIT ETHERNET SFP
STATUS
12
34
56
78
910
1112
1314
1516
1718
1920
2122
2324
2526
2728
2930
3132
3334
3536
3738
3940
4142
4344
4546
4748
WS-X6704-10GE
4 PORT 10 GIGABIT ETHERNET
LINK 1
LINK 2
LINK3
LINK 4
PORT1TX RX
PORT2TX RX
PORT3TX RX
PORT4TX RX
FANSTATUS
1
2
3
4
5
6
7
8
9
Power Supply 1 Power Supply 2
Catalyst 6500SERIES
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SWITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLE PORT 2
PORT 1
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SWITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLE PORT 2
PORT 1
-48 V TO -60 V80 A
Cisco Systems, Inc.Switch must be in off "O" position to Install/Remove power supply.
Fastener must be fully enaged prior to operating power supply.
INPUTOK
FANOK
OUTPUTFAIL
-48 V TO -60 V80 A
Cisco Systems, Inc.Switch must be in off "O" position to Install/Remove power supply.
Fastener must be fully enaged prior to operating power supply.
INPUTOK
FANOK
OUTPUTFAIL
R-Inet-GISCisco C6509
FZKR-Inet-IWR
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
STATUS PHONE
WS-X6548-GE-TX
47
48
3738
35
36
25
26
2324
1314
11
12
1
2
4 8 P O R T
12
11
9
10
7
8
5
6
3
4
1
2
24
23
21
22
19
20
17
18
15
16
13
14
36
35
33
34
31
32
29
30
27
28
25
26
48
47
45
46
43
44
41
42
39
40
37
38
10/100/1000 BASE-TETHERNET
SWITCHING MODULE
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
WS-X6748-SFP
48 PORT GIGABIT ETHERNET SFP
STATUS
12
34
56
78
910
1112
1314
1516
1718
1920
2122
2324
2526
2728
2930
3132
3334
3536
3738
3940
4142
4344
4546
4748
WS-X6704-10GE
4 PORT 10 GIGABIT ETHERNET
LINK 1
LINK 2
LINK3
LINK 4
POR
T1
TX RX
POR
T2
TX RX
POR
T3
TX RX
POR
T4
TX RX
- CNAF (CBF – DFN / Switch / Garr)(Q3/4)
Atlas T1-T1 2010
Hi-Lumi 2*500 MB/s
TAB, 03. March 2006Bruno Hoeft
https://uimon.cern.ch/twiki/bin/view/LHCOPN/LHCopnTables
New location:https://twiki.cern.ch/twiki/bin/view/LHCOPN/LHCopnTables
TAB, 03. March 2006Bruno Hoeft
X-WiNAR
GridKa - backboneFZK-LAN
HEP
DWDM
10GE10GE
10GE
T0 DWDM lightpath
T0 DWDM lightpathBackup (end 2006)T1 DWDM lightpathBandwidth ntd
T2 DWDM lightpathBandwidth ntd
10GE physical link
1GE physical link
T0CERN T1-SARA (CBF – DFN / Surfnet)-CNAF (CBF – DFN / Switch / Garr)(Q3/4)
WAN (2006/7)
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
STATUS PHONE
WS-X6548-GE-TX
4748
373835
3625
262324
131411
121
2
4 8 P O R T
12
11
9
10
7
8
5
6
3
4
1
2
24
23
21
22
19
20
17
18
15
16
13
14
36
35
33
34
31
32
29
30
27
28
25
26
48
47
45
46
43
44
41
42
39
40
37
38
10/100/1000 BASE-TETHERNET
SWITCHING MODULE
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
WS-X6748-SFP
48 PORT GIGABIT ETHERNET SFP
STATUS
12
34
56
78
910
1112
1314
1516
1718
1920
2122
2324
2526
2728
2930
3132
3334
3536
3738
3940
4142
4344
4546
4748
WS-X6704-10GE
4 PORT 10 GIGABIT ETHERNET
LINK 1
LINK 2
LINK3
LINK 4
PORT1TX RX
PORT2TX RX
PORT3TX RX
PORT4TX RX
FANSTATUS
1
2
3
4
5
6
7
8
9
Power Supply 1 Power Supply 2
Catalyst 6500SERIES
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SWITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLE PORT 2
PORT 1
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SWITCH FABRIC
EJECT
DISK 0
EJECT
DISK 1
CONSOLE PORT 2
PORT 1
-48 V TO -60 V80 A
Cisco Systems, Inc.Switch must be in off "O" position to Install/Remove power supply.
Fastener must be fully enaged prior to operating power supply.
INPUTOK
FANOK
OUTPUTFAIL
-48 V TO -60 V80 A
Cisco Systems, Inc.Switch must be in off "O" position to Install/Remove power supply.
Fastener must be fully enaged prior to operating power supply.
INPUTOK
FANOK
OUTPUTFAIL
R-Inet-GISCisco C6509
FZKR-Inet-IWR
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
STATUS PHONE
WS-X6548-GE-TX
47
48
3738
35
36
25
26
2324
1314
11
12
1
2
4 8 P O R T
12
11
9
10
7
8
5
6
3
4
1
2
24
23
21
22
19
20
17
18
15
16
13
14
36
35
33
34
31
32
29
30
27
28
25
26
48
47
45
46
43
44
41
42
39
40
37
38
10/100/1000 BASE-TETHERNET
SWITCHING MODULE
WS-SUP720
SUPERVISOR 720 WITH INTEGRATED SW
ITCH FABRIC EJECT
DISK 0
EJECT
DISK 1
CONSOLEPORT 2
PORT 1
WS-X6748-SFP
48 PORT GIGABIT ETHERNET SFP
STATUS
12
34
56
78
910
1112
1314
1516
1718
1920
2122
2324
2526
2728
2930
3132
3334
3536
3738
3940
4142
4344
4546
4748
WS-X6704-10GE
4 PORT 10 GIGABIT ETHERNET
LINK 1
LINK 2
LINK3
LINK 4
POR
T1
TX RX
POR
T2
TX RX
POR
T3
TX RX
POR
T4
TX RX
T0CERN T0‘
LHC-OPN 2007:T0-T1: Atlas 100 MB/s + CERN - GridKa Alice 25 MB/s +
LHCb 30 MB/s +CMS 280 MB/s 10 Gb capacity
TAB, 03. March 2006Bruno Hoeft
Experiments
•Atlas2007 2008 2009 2010 2010+
•T0 T1 : 100 MB/s ???•T1 T0 : 2,51 MB/s ???•T1 T1: 50MB/s 250-300 MB/s 300-350 MB/s 500 MB/s Hi-Lumi 2x500 MB/s•T1 t2 : 30MB/s 60 MB/s 70 MB/s ~100 MB/s (3 tier-2 centres)•T2 t1 : ignore
•Alice – no roadmap•T0 T1 300 Mb/s•T1 T2 20 MB/s (for 3.5 tier-2s)•T1 T2 3 MB/s
•CMS -- no roadmap•T0 t1: 280 MB/s•T1 t2: 60 MB/s – 600 Mbit•T2 t1: 12 MB/s – 120 Mbit
•LHCb• data taking
•T0 T1: 6,7 MB/s 9 MB/s•T1 T1: 11 MB/s 22 MB/s
• reprocessing •T1 T1: 22 MB/s 60 MB/s•T0 T1: 21 MB/s 29 MB/s
• 1 month stripping•T1 T1: 76MByte/sec•T0 T1: 15MByte
T0 T1 in 2007 ?? 240MByte ??
TAB, 03. March 2006Bruno Hoeft
PCSS Poznan- Alice (5-7MB/s) Warsaw University
-CMS (60 MB/s)-Alice (5-7MB/s)-LHCb ??
Cyfronet Cracow-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)-LHCb ??
FZU Prague-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)
LMU + MPI Munich-Atlas (10/20/25/33 MB/s)UNI Freiburg
-Atlas (10/20/25/33 MB/s)
GSI Darmstadt-Alice (5-7MB/s)
DESY Hamburg-CMS (60 MB/s)-Atlas (10/20/25/33 MB/s)
UNI Wuppertal-Atlas (10/20/25/33 MB/s)
RWTH Aachen-CMS (60 MB/s)
GridKa
Russia ??
TAB, 03. March 2006Bruno Hoeft
German LHC net
• p2p star net– T2 Centres exchanging lhc data
– all Tier-2 sites linked to GridKa
– Exception: t2 – t2 link• Desy and RWTH Aachen (redundance and strong connection
between federated CMS Tier-2 site)
– Routing between German T2 centres at GridKa• one Router excl. for WAN LHC connectivities
– Interlinks to Tier-2 to Tier-3 centres (100Mbit/s cms + atlas) general research network special volumen agreement for ger. LHC / interface rates
TAB, 03. March 2006Bruno Hoeft
LHC Network
PCSS Poznan- Alice (5-7MB/s) Warsaw University
-CMS (60 MB/s)-Alice (5-7MB/s)-LHCb ??
Cyfronet Cracow-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)-LHCb ??
FZU Prague-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)
LMU + MPI Munich-Atlas (10/20/25/33 MB/s)UNI Freiburg
-Atlas (10/20/25/33 MB/s)
GSI Darmstadt-Alice (5-7MB/s)
DESY Hamburg-CMS (60 MB/s)-Atlas (10/20/25/33 MB/s)
UNI Wuppertal-Atlas (10/20/25/33 MB/s)
RWTH Aachen-CMS (60 MB/s)
GridKa
Russia ??CERN
SARA/Nikhef
CNAF/Bologna
TAB, 03. March 2006Bruno Hoeft
LHC Network
PCSS Poznan- Alice (5-7MB/s) Warsaw University
-CMS (60 MB/s)-Alice (5-7MB/s)-LHCb ??
Cyfronet Cracow-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)-LHCb ??
FZU Prague-Atlas (10/20/25/33 MB/s)-Alice (5-7MB/s)
LMU + MPI Munich-Atlas (10/20/25/33 MB/s)UNI Freiburg
-Atlas (10/20/25/33 MB/s)
GSI Darmstadt-Alice (5-7MB/s)
DESY Hamburg-CMS (60 MB/s)-Atlas (10/20/25/33 MB/s)
UNI Wuppertal-Atlas (10/20/25/33 MB/s)
RWTH Aachen-CMS (60 MB/s)
GridKa
Russia ??CERN
SARA/Nikhef
CNAF/Bologna
German – LHC Network 2007:T1-T2: German GSI ~10 MB/s +
UNI Freiburg 10 MB/s +Munich 10 MB/s +RWTH Aachen 60 MB/s +UNI Wuppertal 10 MB/s +Desy 70 MB/s ??
European Belgium ???Switzerland ???FZU Prague 20 MB/sPoland 100 MB/s
German – LHC Network 2008:T1-T2: German GSI ~10 MB/s +
UNI Freiburg 20 MB/s +Munich 20 MB/s +RWTH Aachen 60 MB/s +UNI Wuppertal 10 MB/s +Desy 80 MB/s ??
European Belgian ???Switzerland ???FZU Prague 30 MB/sPoland 110 MB/s
TAB, 03. March 2006Bruno Hoeft