1NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
10 Gigabit Networking at TelstraClearJörg Micheel
2NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
Outline
• Some Metro Ethernet equipment in use at TelstraClear• Test equipment, testing process• Project: University of Auckland 10 Gigabit Campus Ring• Measurement results and performance figures• Outlook: REANNZ Advanced Network implementation
3NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
Extreme Networks Black Diamond – metro core switch
• Black Diamond 10808 (BD10K)• 22 rack mount units• 1280 Gbps capacity• Up to 48 10-Gigabit ports• Up to 480 10/100/1000 ports• Powerful VLAN, Virtual router Layer2
and Layer3 capabilities• Proprietary EAPS link-protection
protocol provides continuity in case of fiber cut
• L2/L3 Quality-of-Service• L2/L3 hardware filtering and priority• Jumbo frames at 9212• High availability, hardware
redundancy
4NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
Extreme Networks Summit X450 – edge switch
• X450-24t with 24 ports 10/100/1000 copper, four combined SFP GigE ports
• X450-24x with 24 ports 1-GigE SFP, four combined 10/100/1000 ports
• Optional dual 10-Gigabit Ethernet uplinks• 1 RU form factor• 160 Gigabits-per-second capacity• 65 million packets-per-second forwarding performance• Coming: X450a with stacking capability• Other features similar to Black Diamond series
5NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
Network test equipment – EXFO PacketBlazer
• 8510 – wire speed full duplex 10/100/1000
• 8510G – wire speed full duplex 10G LAN/WAN PHY
• BERT testing at Layer 1• Frame testing at Layer 2• RFC 2544 Performance
test suite for network devices
• Throughput, back-to-back packets, frame loss and latency, jitter
• Detailed QoS analysis• Can extend concept of
device testing to (parts of) a network
6NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland 10 Gigabit campus ring (1)
Requirements:• Resilient infrastructure to interconnect eight University of
Auckland campus sites• Provide multiple 1-Gigabit services at Layer 2 between campuses
Solution:• Dual-implementation with Summit X450 at six sites within
Auckland metropolitan domain• Dark fiber between the sites• Supports multiple unconstraint 1-Gigabit VLAN services with QoS
up to the maximum 10 Gigabit backbone capacity• Resiliency via Ethernet Advanced Protection Switching (EAPS)• SLA: 99.99% end-to-end throughput• SLA: 500 milliseconds maximum round-trip-time• SLA: 250 milliseconds maximum packet delay variation
7NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland 10 Gigabit campus ring (2)
ITSS
Symonds St
Newton Data
Medicine
Epsom
Tamaki
8NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland – test setup
ITSS
Symonds St
Newton Data
Medicine
Epsom
Tamaki
12 VLANS
12 VLANS12
VLANS
12 VLANS12
VLANS
12 VLANS
9NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland – throughput results
• 10 Gigabit backbone performance: full line rate for any packet sizes between 128 and 9212 bytes
• Any single 1-Gigabit loop VLAN: loss free at any packet size
• 10 loop 1-Gigabit VLAN performance (saturated core, with all services active):
100% 99% 98% 97% 96%64 8.12 6.10 4.07 2.02 0.00
128 4.71 3.54 1.16 0.00 0.00256 2.54 0.60 0.00 0.00 0.00512 1.31 0.00 0.00 0.00 0.00
1024 0.66 0.00 0.00 0.00 0.001280 0.51 0.00 0.00 0.00 0.001518 0.42 0.00 0.00 0.00 0.00
Packet size in bytes
Link utilization
Percent loss
10NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland – delay and jitter results
• Fiber run length 44.7 km – 218.7 microseconds (RI=1.467)
• OTD(min) = 258.4 microseconds• Switching, buffering accounts for 39.6 microseconds (15%
of total)• Per switch latency 3.3 microseconds, inclusive
serialization
• RTTD(64) 517.0 microseconds at 100% of 1-Gigabit load• PDV(64) 1.2 microseconds
Meets SLA requirements comfortably, after adjustment for additional 10 km fiber spur
11NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland – latency as a factor of packet size
A -> B B -> A A -> B B -> A A -> B B-> A64 258,436 258,385 258,549 258,483 259,465 258,591
128 261,985 261,934 262,047 262,001 262,089 262,088256 265,226 265,175 265,296 265,239 265,381 265,278512 273,662 273,611 273,717 273,699 273,817 273,766
1024 289,866 289,814 289,920 289,882 289,969 289,9691280 299,537 299,485 299,581 299,565 299,691 299,6401518 306,687 396,687 306,777 306,739 306,893 306,7919212 579,630 579,629 579,727 579,689 579,784 579,835
RTTD 64PDV 64
Latency (in nanoseconds)Packet size (in bytes)
Summary results (in nanoseconds)
516,821 517,032 518,0561,235
Minimum Average Maximum
A -> B -> A A -> B -> A A -> B -> A
12NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland – EAPS protection switching
• Measured the impact of a ring break upon site-to-site traffic
• Worst case scenario – re-routing once around the entire ring
• Measurements conducted at single site• Metric – number of packets lost• Worst case timing was measured at 217 milliseconds• Typical timing is around 100-150 milliseconds
Exceeds SLA expectations for sub-second failover
13NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
University of Auckland – test summary
• Installation meets / exceeds original design parameters• Network is capable of delivering unconstraint full 1-
Gigabit services across the metro domain• Fiber spur by far dominates latency (85%)• Switching latency is negligible, a few microseconds per
packet at most• Serialization delay can become the dominating factor with
large packet sizes and many hops
Consider the QoS impact when designing applications for either bulk data transfer or sensitive audio/video conferencing applications
14NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
REANNZ Advanced Network
• 10 Gigabit backbone overlay on TelstraClear DWDM
• Gigabit access in all major cities
• Layer 2 network with QoS
• Routers in Auckland, Wellington, Christchurch
• International access: 155 Mbits/sec to Sydney, 622 Mbits/sec to California
• Rollout in 2006 and 2007
15NZNOG 2006 – Victoria University Wellington, March 23rd, 2006
Acknowledgements and references
• University of Auckland ITSS department – Robert Beattie, Manager
• REANNZ – Charles Jarvie, CEO• A cast of dozens of hands at TelstraClear and JazzTech
Thank you!