http:// ca*net 4 international grid testbed [email protected] tel: +1.613.785.0426
TRANSCRIPT
Problem
1. TCP throughput over long fat pipes very susceptible to packet loss, MTU, TCP kernel, Buffer memory, trail drop, AQM optimized for commodity Internet, etc
2. Packet loss can result from congestion, but also underlying BER– achieve a gigabit per second with TCP on a coast-to-coast path (rtt = 40 msec),
with 1500 byte packets, the loss rate can not exceed 8.5x10^-8 packets– “End to end” BER for optical networks 10^-12 to 10^-15 which means packet loss
rate of approximately 10^-8 to 10^-11– The bigger the packet the greater the loss rate!!!
3. Cost of routers significantly greater than switches for 10 Gbps and higher (particularly for large number of lambdas)
4. Lots of challenges maintaining consistent router performance across multiple independent managed networks
– MTU, auto-negotiating Ethernet, insufficient buffer memory5. Require consistent and similar throughput for multiple sites to maintain coherency for
grids and SANs and new “space” storage networks using erasure codes e.g. Oceanstore6. For maximum throughput OS and kernel bypass may be required7. Many commercial SAN/Grid products will only work with QoS network
Possible Solutions
1. For point to point large file transfer a number of possible techniques such as FAST, XCP, parallel TCP, UDP, etc– Very scalable and allows same process to be used for all sorts of file
transfer from large to small– But will it address other link problems?
2. Datagram QoS is a possibility to guarantee bandwidth– But requires costly routers and no proven approach across
independent managed networks (or campus)– Does not solve problem of MTU,link problems, etc
3. E2E lightpaths - all solutions are possible– Allows new TCP and non TCP file transfers– Allows parallel TCP with consistent skew on data striping– Allows protocols that support OS bypass, etc– Guarantees consistent throughput for distributed coherence and
enables news concepts of storing large data sets in “space”– Uses much lower cost switches and bypasses routers
What are E2E lightpaths?
> Customer controlled E2E lightpaths are not about optical networking– E2E lightpaths do not use GMPLS or ASON
> The power of the Internet was that an overlay packet network controlled by end user and ISPs could be built on top of telco switched network
– CA*net 4 is an optical overlay network on top of telco optical network where switching will be controlled by end users
> More akin to MAE-E “peermaker” but at a finer granularity– “Do you have an e2e lightpath for file transfer terminating at a given IX?
Are you interested in peering with my e2e lightpath to enable big file transfer?”
– Lightpath may be only from border router to border router> With OBGP can establish new BGP path that bypasses most (if not all) routers
– Allows lower cost remote peering and transit– Allows e2e lightpaths for big file transfer
e2e Lightpaths Of elephants and mice
Normal IP/BGP path
x.x.x.1y.y.y.1OBGP path
Only y.y.y.1 advertised to x.x.x.1 via OBGP path
Only x.x.x.1 advertised to y.y.y.1 via OBGP path
Optical “Peermaker”
Application or end user controls peering of BGP optical paths to set up dedicated route for transfer of elephants
Small mice traffic is routed over normal IP path
CA*net 4
Halifax
Edmonton
Seattle
VancouverWinnipeg
MontrealOttawa
Chicago
Halifax
New York
Regina
Fredericton
CharlottetownVictoria
Saskatoon
Minneapolis
St. John's
Calgary
TorontoCA*net 4 Node
Existing CA*net 4 OC192Boston
TransLight OC192
Canada sets land speed recordVancouver <-> Geneva
CANARIE2xGbE
circuits StarLight SURFnet 2xGbE
circuits
NetherLight
www.iGrid2002.org
for more info on iGrid2002
www.iGrid2002.org
for more info on iGrid2002
VANCOUVEROTTAWA
CHICAGO
1 x GE loop-back on OC-24
Sustained Throughput ~11.1 Gbps Ave. Utilization = 93%
8 x GE @ OC-12 (622Mb/s)
SAN land speed record
ATLAS/CMS: Data Grid Hierarchy
Tier 1
Tier2 Center
Online System
CERN 700k SI95 ~1 PB Disk; Tape Robot
FNAL: 200k SI95; 600 TBIN2P3 Center INFN Center RAL Center
InstituteInstituteInstituteInstitute ~0.25TIPS
Workstations
~100-1500 MBytes/sec
2.5 Gbps
0.1–10 GbpsPhysics data cache
Low level Trigger data ~PByte/sec
~2.5 Gbps
Tier2 CenterTier2 CenterTier2 Center
~2.5 Gbps
Tier 0 +1
Tier 3
Tier 4
Tier2 Center Tier 2
Experiment
International Grid Testbed
> Joint CERN, SURFnet, STAR LIGHT, TransLight project> Objectives:
1. To validate and test software for customer control and routing of lightpaths2. Test remote processing of of low level trigger data from the ATLAS test
beam.3. Develop and adapt grid applications which are designed to interact with a
LightPath Grid Service which treats networks and network elements as grid resources which can be reserved, concatenated, consumed and released.
4. Characterize the performance of bulk data transfer over an end-to end lightpath.
5. To investigate and test emerging technologies and its impact on high speed long distance optical networks. These technologies include 10 Gbit Ethernet, RDMA/IP, Fibre Channel/IP, serial SCSI, HyperSCSI over long distance ethernet, etc.
6. Collaborate with the EU ESTA project which is developing 10 GbE equipment with CERN, industrial and other academic partners.