Evolution 0.9:The Evolution of the U.S.

Peering Ecosystem

Gigabit Peering Forum VIIHerndon, VA September 9, 2003

William B. Norton

Co-Founder & Chief Technical Liaison

Equinix, Inc.

Internet Researcher

• Four Years of Research• Document Internet Operations Practices –

Interconnect/Peering/IXes/etc.• Research Process – Focus with Community and

– Write White Paper version M.m– Walk throughs & Feedback– Fix/Update Paper

• 200 Walkthroughs later – a document reflecting community Internet Operations knowledge

So far 6 White Papers

Internet Operations White Papers

1) “Interconnection Strategies for ISPs”2) “Internet Service Providers and Peering”3) “A Business Case for Peering”4) “The Art of Peering: The Peering

Playbook”5) “The Peering Simulation Game”6) “Do ATM-based Internet Exchanges

Make Sense Anymore?” Freely available. Send e-mail to wbn@equinix.com

New white Paper: The Evolution of the U.S. Peering Ecosystem, Agenda…

Agenda

The Evolution of the Internet Peering Ecosystem– The Internet Peering Ecosystem Environment– Each Player, their Motivations, their Behavior

• Four Major Events, led to …

• Three Major Evolutions in the Peering Ecosystem

The Evolution of the U.S. Peering Ecosystem

Act I: 1998-2000

The Internet Peering Ecosystem

• The Environment

• The Players– Their role in the Peering Ecosystem– Their behavior in the Peering Ecosystem– Their relationship to others in the Peering

Ecosystem

• The Evolution of the Peering Ecosystem

The Internet Peering Ecosystem• Many “Internet Regions” (usually defined

by geopolitical boundaries)• Each with its own Peering Ecosystem

consisting of– Tier 1 ISPs– Tier 2 ISPs– Content Playersand Enterprises

Content / Enterprise Companies

Tier 1 ISPs

Tier 2 ISPs

Internet Region

Interconnect Regions• Large Internet Regions have multiple

geographical areas where ISPs peerInternet Region

Interconnect Region

Interconnect Region

Interconnect Region

Interconnect Region

Interconnect Region

Player: Tier 1 ISPs

Transit (Downstream)Connections

Backbone Backhaul

Peering Interconnections

Tier 1 ISPTier 1 ISP

Transit Customers

Def: A Tier 1 ISP is an ISP that has access to the entire RegionalInternet routing table solely through Peering Relationships (I.e. doesn’t buy transit from anyone).

Tier 1 Peering• Full Mesh Peering among Tier 1 ISPs

within each Interconnect Region

Tier 1 PeeringTier 1 PeeringTier 1 ISPTier 1 ISP

Tier 1 ISPTier 1 ISP

Tier 1 ISPTier 1 ISP

Tier 1 ISPTier 1 ISP

Tier 1 ISPTier 1 ISP

Tier 1 ISPTier 1 ISP

Some Tier 1 ISPs in the U.S.

Sprint

AT&T

UUNet

C&W

Level3

Qwest

Tier 1 Peering Across the U.S.

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

Sprint

AT&T

UUNet

C&W

Level3

Qwest

P P

P P

P

P PPP

P

PP

P

8 Interconnect Regions

Tier 1 Peering Motivations

• Generally not interested to peer with anyone else

1) They already get the traffic for free

2) They may be foregoing revenue (they could sell transit)

3) They don’t want to give away that which differentiates them

Player: Tier 2 ISPDef: A Tier 2 ISP is an ISP that purchases

(and therefore resells) transit within an Internet Region.

Transit (Downstream)Connections

Backbone Backhaul(optional)

Peering Interconnections(optional)

Tier 2 ISPTier 2 ISP

Transit Customers

UpstreamTransitConnection(s)

Tier 2 ISP Peering Motivations

1) Reduce transit costs

2) Improve Performance

3) Greater control over routing

Tier 2 Peering• Sparse Mesh Peering in each Interconnect

Region

Tier 2 PeeringTier 2 PeeringTier 2 ISPTier 2 ISP

Tier 2 ISPTier 2 ISP

Tier 2 ISPTier 2 ISP

Tier 2 ISPTier 2 ISP

Tier 2 ISPTier 2 ISP

Tier 2 ISPTier 2 ISP

Tier 2 ISPs

• Widely varying characteristics and behaviors

• Scope: Global, National, Regional, Local• Underfunded to well funded• Food fights to solid relationships within the

community• Big Peoples Table and Little Peoples Table

perhaps a bad generalization

The Content Provider and Enterprise Player

• Always buy transit

• Focus on Content

• Focus on Service

• Few Network Engineers

• SLAs valuable

Content Provider (and Enterprise)

Transit Provider Interconnections (Upstream(s))

The Internet Peering Ecosystem

Content / Enterprise Companies

Tier 1 ISPs

Tier 2 ISPs

Internet Region

Peering Inclinations and Peering Policies

• Def: A Peering Inclination is a predilection towards or against peering as demonstrated by Peering behavior in a Peering Ecosystem

• Def: A Peering Policy documents and defines the prerequisites to peering.

Four types of Peering Inclination

• Open Peering – Anyone, Anywhere

• Selective Peering – Some Requirements

• Restrictive Peering – No

• No Peering – No, prefer buying transit

The Evolution of the U.S. Peering Ecosystem

Act II: 2000-2003

Evolution of the U.S. Peering Ecosystem

Four key events leading to drastic disruption in the Peering Ecosystem:

1) The 1999/2000 Telecom Collapse2) The growth of the Used Equipment Market3) The Upstream Provider for the Cable Companies

(@Home) went bankrupt4) Peer-to-Peer file sharing systems (like Kazaa)

grow in popularity, traffic grows exponentially between Access Providers (1.5MB MP3 700MB AVI files)

Leads to 3 Major Evolutions

Evolution #1 – Cable Companies are Peering

• 40% of traffic Kazaa• 3-5 Gbps of transit

traffic-> Gbps of peering potential !

Significant because1) Volume2) Open Peering3) Kazaa Effect

(Optional)Backbone Backhaul

Cable Company

Transit Provider Interconnections (Upstream(s))

Peering

U.S. Cable Companies Eyeballs

MSO Country YE02 Change in 02Comcast USA 3,620,300 1,199,100Time Warner USA 2,613,000 1,027,000Cox USA 1,407,900 524,400Charter USA 1,180,000 572,300CableVision USA 770,100 263,500Shaw (Big Pipe) Canada 750,000 ?Rogers Canada 650,000 ?Adelphia USA 610,000 305,900Bright House USA 490,000 159,000Mediacom USA 191,000 76,000RCN USA 160,400 49,800Insight USA 144,800 56,700Cable One USA 78,100 45,200Total 12,665,600 4,278,900

Cable Modem Subscribers

Source: The Bridge & Leichtman ResearchSource: I$P HO$TING ReportMay 2003, Volume VII, Number 5, Page 8Article "Dialup Bakeoff - Is it really so bad at EarthLink? Or so good at United?"Source: Bill Norton conversations with Rogers and Shaw 2002 (numbers may be understated)

Cable Company

Cable Company

Cable Company

Cable CompanyCable Company

Cable Company

Cable Company

AOL

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

Content

2000-2003 Evolution #1 – Cable Companies Peering

Tier 2 ISPs

CableCos

Tier 1 ISPs

Peering

Evolution #2 – Network Savvy Large Scale Content Companies

get into Peering• To reduce transit costs • To improve the end user experience• They need to move out of bankrupt colo anyway

Significant because

1) Volume of traffic is huge

2) Content Providers have Open Peering

3) Leading players pave the way

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

Content

2000-2003 Evolution #2 – Large Scale Network Savvy Content

Peers

Tier 2 ISPs

LSNSC

Tier 1 ISPs

Other Broadband Players

• Significant, but…

• Not Open Peers so disincentive to peer

• So less volume and less disruptive

Evolution #3 – Cable Companies Freely Peer with Content

Companies• Content literally on the Cable Company

Network

• Lowest possible latency from content to eyeballs

• Internet Gaming, Broadband Streaming, etc.

Content

1998-2000

Tier 2 ISPs

Tier 1 ISPs

Transit$

Thickness representsIn-Group Peering

Content

2000-2003 Evolution #1 – Cable Companies Peering

Tier 2 ISPs

CableCos

Tier 1 ISPs

Content

2000-2003 Evolution #2 – Large Scale Network Savvy Content

Peers

Tier 2 ISPs

LSNSC

Tier 1 ISPs

Content

2000-2003 Evolution #3 – Cable Peers with Large Scale Network

Savvy Content

Tier 2 ISPs

LSNSC

Tier 1 ISPs

CableCos

International Dynamics – Separate White Paper

Content

Tier 2 ISPs

LSNSC

Tier 1 ISPs

CableCos

Content

Tier 2 ISPs

Tier 1 ISPs

NTT

KDDI

IIJ

JTCableCos

DSL

JP U.S.

U.S. Tier 1 ISPs are Tier 2 ISPs in Japan Internet RegionJapan Tier 1 ISPs are Tier 2 ISPs in the U.S. Internet Region

Summary 2003 Peering Ecosystem Evolving

NoPeering Open Selective Restrictive

Lg.Content

Tier2 ISPs

CableCos

RBOCs

Tier 1 ISPs

Foreign ISPs

Excite@Home