lhc-o ptical p rivate n etwork network at gridka 10gbit lan/wan evaluations
DESCRIPTION
Firewall. FS NAS. FS NAS. FS NAS. German / European. Tier-0/Tier-1 (T0/T1) LHC-OPN Lightpath. IP via MPLS Tunnel. Tier-2 1) LHC Lightpath. International IP. RWTH-Aachen. CNAF (2007 10Gbit). Uni-Dresden. GSI / FZU Prague / PSNC Poland (2006/7/8 1Gbit). - PowerPoint PPT PresentationTRANSCRIPT
Forschungszentrum Karlsruhei n d e r H e l m h o l t z - G e m e i n s c h a f tForschungszentrum Karlsruhe GmbH, Institut für Wissenschaftliches Rechnen,Postfach 36 40, 76021 KarlsruheBruno Hoeft, Marc Garcia Marti
LHC-OpticalPrivateNetwork NETWORK AT GRIDKA10Gbit LAN/WAN evaluations
Total Cost of Ownership (TCO):- Dramatically reduced costs (per NIC/high end server)- the troughput increased from 2.8Gbit to 7.0Gbit with TOE (standard MTU)- eight 1GE standard server = one 10GE Server- higher administration needs to be taken into account- PCI-Express will allow for the full rate of 10GE- 250/6000 Euro per 1GE/10GE router port
Routing:- External Routing via BGP
-eBGP with the Forschungszentrums AS-No.: 34878 exterior Boarder Gateway Protocol-iBGP synchronising the eBGP router interior Boarder Gateway Protocol
- Internal redundance secured via OSPF- Layer-2 tunnel for distributed Storage Server via Policy Based Routing - Private network space separated into switched VLans
DFN
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
R-inet-IWRR-inet-GridKa
VLAN 192.108.45.0 /26
VLAN 33_x 10.33.x.x /24LCG-se/ce/ui/bdii/...
AR DWDM
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
Power Supply 1 Power Supply 2
9 8 7 6 5 4 3 2 1
FANSTATUS
WS-C6509-NEB
OSPF
eBG
P
eBG
P
iBGP
VLAN / Switched
... ...
Pol
icyB
ased
Rou
ting
+
Layer-2 Tunnel
0
5000
10000
15000
20000
25000
30000
35000
40000
45000
2003 2004 2005 2006
Year
Eu
ro
Euro / ServerEuro / NIC
Mmrbc set to 512 bytesMmrbc set to 2048 bytesMmrbc set to 4096 bytes
Intel NIC – tuned for throughput Chelsio
Standard mtu
8160 Bytes mtu
8180 Bytes mtu
9216 Bytes mtu
Standard mtu8180 Bytes mtu9216 Bytes mtu
Standard mtu8180 Bytes mtu9216 Bytes mtuStandard mtu (server side)8180 Bytes mtu (server side)9216 Bytes mtu (server side)Standard mtu
8160 Bytes mtu8180 Bytes mtu9216 Bytes mtuStandard mtu (server side)8160 Bytes mtu (server side)8180 Bytes mtu (server side)9216 Bytes mtu (server side)
NIC – tuned for Latency- Intel 0.016ms- The throughput of the Intel NICs is neglectable after tuning for latency
- Chelsio 0.020ms-The throughput of the Chelsio NIC is still normal after tuning for latency
- PCI-X Bus: theoretical limit 8.5 Gbps approx. - tested max. rate 7.8 Gbps (linux kernel packet generator)
Window size 14.0 MByte TxIntDelay=500 RxIntDelay=500Window size 2.0 MByte TxIntDelay=500 RxIntDelay=500Window size 8.0 MByte TxIntDelay=250 RxIntDelay=250Window size 8.0 MByte TxIntDelay=74 RxIntDelay=74Window size 8.0 MByte TxIntDelay=15 RxIntDelay=15
Cisco 7609
Juniper M160
Juniper M160 Juniper M160
Cisco 12416
Cisco 12410
Cisco 6509
10gtk113oplapro96
Gridka (Karlsruhe)
DFN
GEANT
CERN
„Connecting WLCG Tier-2 Centres to GridKa“of Dr. Andreas Heiss presents more details about the at GridKa assigned Tier-2 sites
Dr. Doris Ressmann will present in her talk more detailed information at the dCache setup of GridKa.„using TSM to create a high performance Tape connection“Track: Computing Facilities and Networking, Monday 14:20-14:40
CNAF (2007 10Gbit)
International IP IP viaMPLS Tunnel
Tier-21)
LHC Lightpath
Tier-0/Tier-1 (T0/T1) LHC-OPN Lightpath
German / European
T0 DWDM lightpathT0 DWDM lightpathBackup (end 2006)T1 DWDM lightpath Bandwidth 10GET2 DWDM lightpathBandwidth 1/10GET2 L2-IP-Tunnel (MPLS)
Ethernet 1 GbitFiberChannel 2GbitEthernet 10 Gbit
FZK-LAN
routerprivate network
…
…
FS
…
…
FS NASFS NAS
FS NAS
FSFS
SAN
…
…
switch
Firewall
LCG(RGMA/MON/BDII)
…
LCG(RB/LFC/DC-SRM)
GridKa-InternetRouter
FZK-InternetRouter
DFN X-WiN
AR
T2HEP
DWDM
T0 T0‘T1T2T2
10GE10GE
T2
Desy(20071/10Gbit)
GSI /FZU Prague /PSNC Poland(2006/7/81Gbit)
SARA (2007 10Gbit)
}
CERN2007 10Gbit240Mbyte netto
Fnal
SLAC
Uni-Freiburg
Uni-Dresden
LCG(se/ce/ui)
LoginServerALICE/Atlas/CMS/LHCb
LoginServerBabar/CDF/DZero/Compass
dcache2)
pools
dC
ach
e Hea
dn
od
e
MPI u. Uni-München
10GE
10GE
ComputeNode
ComputeNode
ComputeNode
ComputeNode
…
…
dcache2)
pools…
internextern
RWTH-Aachen
Uni-Wuppertal
WAN setup -- GridKa - CERN
WAN throughput / TCP cloning
1GE versus 10GE Server Nodes
1) 2)Email: [email protected]; [email protected]: +49 7247 82 8172Web: www.gridka.de www.fzk.de/iwr