1

Internet Topology

COS 461: Computer Networks

Spring 2007 (MW 1:30-2:50 in Friend 004)

Jennifer Rexford

Teaching Assistant: Ioannis Avramopoulos http://www.cs.princeton.edu/courses/archive/spring07/cos461/

2

Returning the Midterm Exam

• Exam scoring break down–Average: 90–High: 99–Median: low 90s

• See the course Web site–Exam–Answer key

3

Goals of Today’s Lecture

• IP routers– Interface cards– Switching fabric– Route processor

• Router-level topology with a single network– Points of Presence (PoPs)– Backbone and enterprise network topologies

• AS-level topology of the Internet– Autonomous System (AS) numbers– Tier-1 ISPs, regional providers, and stub ASes– Business relationships between ASes

4

IP Routers

5

Inside a High-End Router

SwitchingFabric

Processor

Line card

Line card

Line card

Line card

Line card

Line card

6

Router Physical Layout

Juniper T series

Cisco 12000

Crossbar

Linecards

7

Line Cards (Interface Cards, Adaptors)

• Interfacing – Physical link– Switching fabric

• Packet handling– Packet forwarding– Decrement time-to-live– Buffer management– Link scheduling– Packet filtering– Rate limiting– Packet marking– Measurement

to/from link

to/from switch

lookup

Rec

eive

Transm

it

8

Switching Fabric

• Deliver packet inside the router– From incoming interface to outgoing interface– A small network in and of itself

• Must operate very quickly– Multiple packets going to same outgoing interface– Switch scheduling to match inputs to outputs

• Implementation techniques– Bus, crossbar, interconnection network, …– Running at a faster speed (e.g., 2X) than links– Dividing variable-length packets into cells

9

Packet Switching

R1Link 1

Link 2

Link 3

Link 4

Link 1, ingress Link 1, egress

Link 2, ingress Link 2, egress

Link 3, ingress Link 3, egress

Link 4, ingress Link 4, egress

ChooseEgress

ChooseEgress

ChooseEgress

ChooseEgress

“4”

“4”

10

Router Processor

• So-called “Loopback” interface–IP address of the CPU on the router

• Control-plane software–Implementation of the routing protocols–Creation of forwarding table for the line cards

• Interface to network administrators–Command-line interface for configuration–Transmission of measurement statistics

• Handling of special data packets–Packets with IP options enabled–Packets with expired Time-To-Live field

11

Error Reporting

• Examples of errors a router may see– Router doesn’t know where to forward a packet– Packet’s time-to-live field expires

• Router doesn’t really need to respond– Best effort means never having to say you’re sorry– So, IP could conceivably just silently drop packets

• But, silent failures are really hard to diagnose– IP includes basic feedback about network problems– Internet Control Message Protocol (ICMP)

12

Internet Control Message Protocol

• ICMP runs on top of IP– In parallel to TCP and UDP– Though still viewed as an integral part of IP

• Diagnostics– Triggered when an IP packet encounters a problem

E.g., time exceeded or destination unreachable

– ICMP packet sent back to the source IP address Includes the error information (e.g., type and code) … and an excerpt of the original data packet for identification

– Source host receives the ICMP packet And inspects the except of the packet (e.g., protocol and ports) … to identify which socket should receive the error

13

Example: Time Exceeded

host DNS... host host DNS...

router routerrouter

host

1.2.3.7

8.9.10.11

5.6.7.156

• Host sends an IP packet– Each router decrements the time-to-live field

• If time-to-live field reaches 0– Router generates an ICMP message– Sends a “time exceeded” message back to the source

Time exceeded

14

Ping: Echo and Reply

• ICMP includes a simple “echo” function– Sending node sends an ICMP “echo” message– Receiving node sends an ICMP “echo reply”

• Ping tool– Tests the connectivity with a remote host– … by sending regularly spaced echo commands– … and measuring the delay until receiving the reply

• Pinging a host– “ping www.cs.princeton.edu” or “ping 12.157.34.212”– Used to test if a machine is reachable and alive– (However, some nodes have ICMP disabled… )

15

Router-level topology of a network

16

Intra-AS Topology

• Node: router

• Edge: link

17

Hub-and-Spoke Topology

• Single hub node–Common in enterprise networks–Main location and satellite sites–Simple design and trivial routing

• Problems–Single point of failure–Bandwidth limitations–High delay between sites–Costs to backhaul to hub

18

Princeton Example

• Hub-and-spoke–Four hub routers and many spokes

• Hub routers–Outside world (e.g., AT&T, USLEC, …)–Dorms–Academic and administrative buildings–Servers

19

Simple Alternatives to Hub-and-Spoke

• Dual hub-and-spoke– Higher reliability– Higher cost– Good building block

• Levels of hierarchy– Reduce backhaul cost– Aggregate the bandwidth– Shorter site-to-site delay …

20

Backbone Networks

• Backbone networks–Multiple Points-of-Presence (PoPs)–Lots of communication between PoPs–Accommodate traffic demands and limit delay

21

Abilene Internet2 Backbone

22

Points-of-Presence (PoPs)

• Inter-PoP links–Long distances–High bandwidth

• Intra-PoP links–Short cables between

racks or floors–Aggregated bandwidth

• Links to other networks–Wide range of media and

bandwidth

Intra-PoP

Other networks

Inter-PoP

23

Where to Locate Nodes and Links

• Placing Points-of-Presence (PoPs)–Large population of potential customers–Other providers or exchange points–Cost and availability of real-estate–Mostly in major metropolitan areas (“NFL cities”)

• Placing links between PoPs–Already fiber in the ground–Needed to limit propagation delay–Needed to handle the traffic load

24

AS-level topology of the Internet

25

Internet Routing Architecture

• Divided into Autonomous Systems– Distinct regions of administrative control– Routers/links managed by a single “institution”– Service provider, company, university, …

• Hierarchy of Autonomous Systems– Large, tier-1 provider with a nationwide backbone– Medium-sized regional provider with smaller backbone– Small network run by a single company or university

• Interaction between Autonomous Systems– Internal topology is not shared between ASes – … but, neighboring ASes interact to coordinate routing

26

Autonomous System NumbersAS Numbers are 16 bit values.

• Level 3: 1 • MIT: 3• Harvard: 11• Yale: 29• Princeton: 88• AT&T: 7018, 6341, 5074, … • UUNET: 701, 702, 284, 12199, …• Sprint: 1239, 1240, 6211, 6242, …• …

Currently just over 20,000 in use.

27

AS Topology

• Node: Autonomous System

• Edge: Two ASes that connect to each other

1

2

3

4

5

67

28

What is an Edge, Really?

• Edge in the AS graph– At least one connection between two ASes– Some destinations reached from one AS via the other

AS 1

AS 2

d

Exchange Point

AS 1

AS 2

d

AS 3

29

Interdomain Paths

1

2

3

4

5

67

ClientWeb server

Path: 6, 5, 4, 3, 2, 1

30

Business Relationships

• Neighboring ASes have business contracts–How much traffic to carry–Which destinations to reach–How much money to pay

• Common business relationships–Customer-provider

E.g., Princeton is a customer of AT&T E.g., MIT is a customer of Level 3

–Peer-peer E.g., Princeton is a peer of Patriot Media E.g., AT&T is a peer of Sprint

31

Customer-Provider Relationship

• Customer needs to be reachable from everyone– Provider tells all neighbors how to reach the customer

• Customer does not want to provide transit service– Customer does not let its providers route through it

d

d

provider

customer

customer

provider

Traffic to the customer Traffic from the customer

advertisements

traffic

32

Customer Connecting to a Provider

Provider Provider

1 access link 2 access links

Provider

2 access routers

Provider

2 access PoPs

33

Multi-Homing: Two or More Providers

• Motivations for multi-homing–Extra reliability, survive single ISP failure–Financial leverage through competition–Better performance by selecting better path–Gaming the 95th-percentile billing model

Provider 1 Provider 2

34

Princeton Example

• Internet: customer of AT&T and USLEC

• Research universities/labs: customer of Internet2

• Local non-profits: provider for several non-profits

AT&T USLEC Internet2

35

Peer-Peer Relationship

• Peers exchange traffic between customers – AS exports only customer routes to a peer– AS exports a peer’s routes only to its customers– Often the relationship is settlement-free (i.e., no $$$)

peerpeer

Traffic to/from the peer and its customers

d

advertisements

traffic

36

AS Structure: Tier-1 Providers

• Tier-1 provider– Has no upstream provider of its own– Typically has a national or international backbone– UUNET, Sprint, AT&T, Level 3, …

• Top of the Internet hierarchy of 12-20 ASes– Full peer-peer connections between tier-1 providers

37

Efficient Early-Exit Routing

• Diverse peering locations– Both costs, and middle

• Comparable capacity at all peering points– Can handle even load

• Consistent routes– Same destinations advertised

at all points– Same AS path length for a

destination at all points

Customer A

Customer B

multiplepeeringpoints

Provider A

Provider B

Early-exit routing

38

AS Structure: Other ASes

• Tier-2 providers– Provide transit service to downstream customers– … but, need at least one provider of their own– Typically have national or regional scope– E.g., Minnesota Regional Network– Includes a few thousand of the ASes

• Stub ASes– Do not provide transit service to others– Connect to one or more upstream providers– Includes vast majority (e.g., 85-90%) of the ASes

39

Characteristics of the AS Graph

• AS graph structure– High variability in node degree (“power law”)– A few very highly-connected ASes– Many ASes have only a few connections

1 10 100 1000

CC

DF

1

0.1

0.01

0.001

AS degree

All ASes have 1 or more neighbors

Very few have degree >= 100

40

Characteristics of AS Paths

• AS path may be longer than shortest AS path

• Router path may be longer than shortest path

s d

3 AS hops, 7 router hops

2 AS hops, 8 router hops

41

Shared Risks

• Co-location facilities (“co-lo hotels”)– Places ISPs meet to connect to each other– … and co-locate their routers, and share space & power– E.g., 32 Avenue of the Americas in NYC

• Shared links– Fiber is sometimes leased by one institution to another– Multiple fibers run through the same conduits– … and run through the same tunnels, bridges, etc.

• Difficult to identify and accounts for these risks– Not visible in network-layer measurements– E.g., traceroute does not tell you links in the same ditch

42

Conclusions

• Internet topology–Network inside a router–Network inside an AS–Network connecting ASes

• Coming up: two-tiered routing system–Intradomain routing–Interdomain routing

• Reading– 4.2, 4.3.3, 4.3.4