AOL TW 0 12/11/2002 5:05 PM

AOL Transit Data NetworkAOL Transit Data NetworkAS 1668AS 1668

www.atdn.netwww.atdn.netDecember 5, 2002December 5, 2002

Vijay GillVijay Gill

AOL TW 2 12/11/2002 5:05 PM

AOL Transit Data Network (ATDN)

• What is ATDN• Current Topology (December 2002)• Design Philosophy behind ATDN• Backbone Performance and Load• Next Steps

AOL TW 3 12/11/2002 5:05 PM

• International IP Network that connects all AOL/TW facilities to the internet- OC192 Backbone in US- OC48 Backbone in Europe

• Provides:

- AOL/TW Facility Interconnections- Internet Connectivity- AOL Broadband/Dial Access Connectivity- Inter-Data Center Connectivity

What Is ATDN?

AOL TW 4 12/11/2002 5:05 PM

ATDN and Supported Infrastructure

Daily Facts:• 100 Gigabit/sec of edge traffic• 14.8B URLs/day, 250k URLs/sec• 391M emails• 245M stock quotes• 1.38M IM’s• 125,000+ concurrent streams• 34+ million AOL subscribers• 3+ million CompuServe subscribers• 125+ million AIM registrations

worldwide• 110+ million ICQ registered users• Peak simultaneous usage:

- 2.9M AOL users online- 3.9M active AIM users- 1.8M active ICQ users

Infrastructure:• 25,000+ host servers• ~ 500,000 sq ft raised floor space• ~ 800 optical backbone routers

- ATDN – 70+ 10-gig capable routers in the backbone

- ATDN – 113+ edge routers• ~ 3000 L2/L4-7 Switches• 66,000 interfaces polled every 5

mins• Over 1 million network variables

captured

AOL TW 5 12/11/2002 5:05 PM

• 36 PoPs on 4 Continents

• Interconnections:

- 200 unique ASes

- 125 ISPs

• 170 Gbps of Internet Capacity

• 500 Gbps Total Edge Capacity

• 100 Gbps Traffic Through ATDN (Nov 2002)

ATDN Statistics

AOL TW 6 12/11/2002 5:05 PM

Tokyo

Honolulu

New YorkSunnyvale

Los AngelesAtlanta

Vienna

Denver Columbus

Houston

KansasCity

TampaBay

Chicago

Charlotte

PhoenixDallas

Seattle

NewarkAshburn

San Jose

Albany

SaoPaulo

Brazil

London

ParisFrankfurt

Circuit Size

622 Mbps

155 Mbps

2.5 Gbps

10 Gbps

AOL TW 7 12/11/2002 5:05 PM

ATDN Logical Design

AOL TW 8 12/11/2002 5:05 PM

AOL Metro Connection (AOLWAVE)

NTC

Building 1

SJE

Building 2

Denver 10G

Seattle 2.5G

LAS 10GHON STM4

HON STM4

Seattle 2.5GSJG

! Provider 1 Ring will be used as an OC48 “Collector” ring and for the transitional phase-out of the SJE PoP.

AOL TW 9 12/11/2002 5:05 PM

ATDN Hub Design

bb2bb1

pop1 pop2 pop3

OC192 backbone linkOC192 backbone link

OC3, OC12, OC48, OC192Channelized OC12/DS3

T3, T1, E1, E3Channelized T3/T1

AOL TW 10 12/11/2002 5:05 PM

Rack Elevation

AOL TW 11 12/11/2002 5:05 PM

Data Center Building Blocks

Aggregation Routers

L2/L4-7 Switches

AOL TW 12 12/11/2002 5:05 PM

Complete life cycle in NetOps- Architect, design, implement, and operate

Design Criteria- Diversity of component, paths, logical units- No Single Point of Failure (SPOF)

!Redundant capacity to support peak load - Routed topology- System review to ensure performance & survivability goals

ATDN Design PhilosophyATDN Design Philosophy

AOL TW 13 12/11/2002 5:05 PM

• Standards based (as much as possible)• Consistency

- Replicate design throughout all POPs ! Boxes might be different sizes but nodal architecture is

the same• “Choose the path more traveled”

- Problem with being on the cutting edge is that someone’s got to bleed

• Empower new employees with the ability to easily understand what has been built- Repetitive tasks get pushed into things like NOCs, which

have a very high turnover rate

Design Goals – Operational Simplicity

AOL TW 14 12/11/2002 5:05 PM

• When to touch the network- Routing policy based on simple performance metrics and

cost (price/Mb)- No fancy tricks – Traffic engineering based on using

common policy across similar peers (e.g., free or transit) rather than exception policy

• Achieved through engineering simplicity- Focus on reducing OPEX- It’s not about building a network that’s cool or is a challenge

to engineer! “Make it simple, but no simpler”

Design Goals – Operational Simplicity

AOL TW 15 12/11/2002 5:05 PM

Cost Per Bit – Major Components

Source: KPCB

Historical and forecast market price and unit cost of Transatlantic STM-1 circuit (on 25 year IRU lease)

2,000

4,000

6,000

8,000

10,000

12,000

14,000

16,000

18,000

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Pric

e pe

r STM

-1 ($

m)

PRICE

COST

SG&

A pe

r STM

-1(%

)

SG&A

AOL TW 16 12/11/2002 5:05 PM

• Two Level Hierarchy- Edge and Core

• Optimized for minimal cost of operation (OPEX and SG&A)• Overprovision bandwidth (for now)• Scaleable

ATDN Architecture

AOL TW 17 12/11/2002 5:05 PM

• Separation of routing functions in the core and the edge • Well-defined interfaces between the edge and core

- Easy to isolate problem equipment, circuits, or traffic sources/sinks

- Reduces risk and complexity of substituting substantially different hardware or software in the network as it evolves

• Network subsystems can be evolved separately as long as the interface presented remains unchanged

Forced Hierarchy

AOL TW 18 12/11/2002 5:05 PM

• Core consists of large transit hubs- Yields good aggregation- Having fewer transit hubs reduces the number of transit

paths in the network! Prevents combinatorial explosion of possible paths! Traffic engineering is relatively simple

ATDN Design Highlights

AOL TW 19 12/11/2002 5:05 PM

• BGP/OSPF- Migration to IS-IS

• Run IGP lean, carry everything in iBGP• IGP cost structure:

- Local- Regional- Long-haul- International

Routing Architecture

AOL TW 20 12/11/2002 5:05 PM

• No protected rings at the SONET level• Take unprotected bandwidth• Rely on IP for protection• No single failure of any component should isolate a

major hub• Primary, secondary and tertiary backup paths

Routing Architecture Robustness

AOL TW 21 12/11/2002 5:05 PM

Network Protection

AOL TW 22 12/11/2002 5:05 PM

• Matrix of city-pair flows- Macro flows are tractable- Watch bandwidth usage

• Capacity Planning- Add capacity where and when required- Increase primary and backup capacity in sync

• Egress capacity is the bottleneck

Traffic Engineering

AOL TW 23 12/11/2002 5:05 PM

• Standard Filtering (external)- Nothing longer than /24, no RFC1918, no private

ASNs, etc., no martians- Full edge packet filtering on AOL space as well as

RFC1918 source/dest- All routes tagged with communities to record

origination- Customers are filtered by prefix

• Peers- No prefix/as-path filters applied to peers other than

above- Maximum prefix used as a coarse sanity-check

Routing Policy

AOL TW 24 12/11/2002 5:05 PM

Traffic Topology

AOL TW 25 12/11/2002 5:05 PM

ATDN Frankfurt to Honolulu

AOL TW 26 12/11/2002 5:05 PM

ATDN Traffic Growth Oct 01-Current

AOL TW 27 12/11/2002 5:05 PM

Traffic

AOL TW 28 12/11/2002 5:05 PM

• Discontinuity of demand curve requires architecture/design to support sudden growth spurts- Acquisitions, etc.- New applications- Relief of bottlenecks elsewhere in network (backlog)- Sudden acts of major players (RR, Datacenters, etc.)

Capacity and Topology Growth Thoughts

AOL TW 29 12/11/2002 5:05 PM

Base + Peak Capacity/Robustness Requirements

• In-service capacity can be divided into base capacity and peak capacity- Base capacity must be provided with high assurance in the

presence of any single location failure of any scale- Peak capacity is provided in "all-up" condition and may be

impacted by failures• The base demand is defined as the 95th percentile 5-minute

load seen under unconstrained conditions

AOL TW 30 12/11/2002 5:05 PM

• The peak demand is defined as the 95th percentile 2-second load seen under unconstrained conditions (if we could observe it)

• Experience suggests that the peak load is about twice the base load

Base + Peak Capacity/Robustness Requirements

AOL TW 31 12/11/2002 5:05 PM

Restoration Time Requirements

• Probably driven by VOIP-type applications• Frequency/duration tradeoff

- 50 ms does not appear to be a hard requirement- One sub-second transient in a voice call is probably

acceptable to customer (?)- Many >150 ms transients in a voice call might affect

customer perception of quality of service (?)

AOL TW 32 12/11/2002 5:05 PM

• Probably a mix of short-reach and long-reach- SR to interface to exotic transmission (soliton) or local OEO- LR (WDM-compatible) to interface to lower-cost transmission

such as metro fiber between hubs- SONET framing for OC-192c initially- Longer-term, 10GB Ethernet or other technology may be

desirable, especially for higher speeds

Interface Requirements

AOL TW 33 12/11/2002 5:05 PM

Q&A

• Questions