mapping the internet topology traceroute a basic tool for toppgy

1

Mapping the Internet Topology

Yuval ShavittSchool of Electrical EngineeringSchool of Electrical Engineering

http://www.eng.tau.ac.il/~shavitthttp://www.netDimes.org

TracerouteA Basic Tool for Topology Discoveryp gy y

Simple delay/loss probing with ping

C:\>ping www.fer.hr

Pinging www.fer.hr [161.53.72.111] with 32 bytes of data:

Reply from 161.53.72.111: bytes=32 time=113ms TTL=49Reply from 161.53.72.111: bytes 32 time 113ms TTL 49Reply from 161.53.72.111: bytes=32 time=111ms TTL=49Reply from 161.53.72.111: bytes=32 time=113ms TTL=49Reply from 161.53.72.111: bytes=32 time=118ms TTL=49

Ping statistics for 161.53.72.111:Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:Minimum = 111ms, Maximum = 118ms, Average = 113ms

ICMP

ICMP is the IP error diagnosis protocol.

IP header

CodeType

Checksum

Sequence number

Any ICMP data

ICMP Message Types

MeaningType No.

Echo reply0

Destination unreachable3

Source quench4

Redirect5

Echo8

R t d ti t9

PING

Router advertisement9

Router solicitation10

Time exceeded11

Parameter problem12

Timestamp13

Timestamp reply14

Information requeste15

Information reply16

ping

• Can be used to– Check host is alive– Measure RTD to a serverMeasure RTD to a server

• Application layer “ping”– One can generate application layer messages to

test application reaction time– Most common:

• TCP SYN message to port 80

2

traceroute

• Useful to learn the route characteristics between two hosts.

• Sends a series of probes to successive nodesSends a series of probes to successive nodes along a route to an intended destination and records the source address and time delay of the message returned by each.

• Based on ICMP “TTL expired” message

IP datagram format

ver length

32 bits

16-bit identifierInternetchecksum

time tolive

32 bi IP dd

IP protocol versionnumber

header length(bytes)

max numberremaining hops

(decremented at

forfragmentation/reassembly

total datagramlength (bytes)head.

lentype ofservice

“type” of data flgs fragmentoffset

upperlayer

data (variable length,typically a TCP

or UDP segment)

32 bit source IP addresseach router)

upper layer protocolto deliver payload to

32 bit destination IP address

Options (if any) E.g. timestamp,record routetaken, pecifylist of routers to visit.

ICMP Message Types

MeaningType No.

Echo reply0

Destination unreachable3

Source quench4

Redirect5

Echo8

R t d ti t9

Type Code description3 0 dest. network unreachable3 1 dest host unreachable3 2 dest protocol unreachable3 3 dest port unreachable3 6 dest network unknown3 7 dest host unknown

Router advertisement9

Router solicitation10

Time exceeded11

Parameter problem12

Timestamp13

Timestamp reply14

Information requeste15

Information reply16

traceroute

traceroute

Regular UDP packets• successive TTLs

timeA B C D E

ICMP “TTL expired” message

ICMP “port unreachable” message

traceroute versions

• UNIX: – default send UDP packets

• Start at port 33435, and increment port per packet!• Why this is not advisable?W y s s o dv s b e?

– traceroute –l sends ICMP “ECHO request”– tcptraceroute uses TCP SYN messages

• If port is close gets RST reply• If port is open gets SYN ACK and reply with RST• Best to overcome firewalls

• Windows– ICMP “ECHO request”

C:₩>tracert www.fer.hr

Tracing route to www.fer.hr [161.53.72.111]over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.200.2542 19 ms 20 ms 19 ms vxr.tau.ac.il [132.66.8.10]3 17 ms 22 ms 20 ms c6509.tau.ac.il [132.66.8.20]4 21 ms 19 ms 19 ms tel-aviv.tau.ac.il [132.66.4.1]5 19 ms 23 ms 18 ms gp1-tau-fe.ilan.net.il [128.139.191.70]6 20 ms 20 ms 20 ms iucc.il1.il.geant.net [62.40.103.69]

i i i [ ]7 69 ms 69 ms 69 ms il.it1.it.geant.net [62.40.96.154]8 82 ms 82 ms 82 ms it.ch1.ch.geant.net [62.40.96.33]9 101 ms 98 ms 98 ms ch.at1.at.geant.net [62.40.96.1]

10 105 ms 105 ms 105 ms at.hu1.hu.geant.net [62.40.96.178]11 117 ms 112 ms 113 ms hu.hr1.hr.geant.net [62.40.96.145]12 113 ms 115 ms 115 ms carnet-gw.hr1.hr.geant.net [62.40.103.218]13 120 ms 122 ms 123 ms 193.198.228.614 114 ms 112 ms 119 ms 193.198.229.1015 120 ms 119 ms 119 ms 161.53.16.1416 114 ms 114 ms 113 ms duality.cc.fer.hr [161.53.72.111]

Trace complete.

3

From Traceroute to IP Maps

• Problem 1:– TR gives interface IP not router ID

• Problem 2:• Problem 2:– TR can result in false links

• Problem 3:– Non responding routers

Alias ResolutionPing (echo request) X

or UDP X (high port number

M

echo reply Y

⇒ IP X = IP Y

ICMP (port unreachable) Y

Aliasing Based on IP Identifier

• x<y<z, z-x small likely aliases

• If |x-y|>200Aliases are

disqualified, third packet is not sent

Traceroute and Load-Balancing

LSrc

B

A C

D

TTL = 2 DstE

Actual topology:

16

Src L

A

D

C

TTL = 3

False link

B

Missing nodes and links

E Dst

Inferred topology:

Anomalies: false loops and cycles

L DSrc

B

A

C

TTL = 3

Dst

Actual topology:

17

Src L D

B

Dst

TTL = 2

TTL = 4

Inferred topology:

Load Balancing can lead to ‘Loops’

TTL = 2

real topology traceroute possible outcome

TTL 2

TTL = 3TTL = 4

4

Anomalies: false diamonds

LSrc

B

A C

D

DstE

Actual topology:

19

Src L

A

D

E Dst

B

C

Inferred topology:

Reducing The Load Balancing Problem

• Load balancing can be done in two ways:1. Packet based2. Flow based (S IP, D IP, S port, D port, prot)2. Flow based (S IP, D IP, S port, D port, prot)

• Sending all probes as a single flow will force the same route selection in flow based Load balancer

• How to do this?

Paris traceroute• Solves the problem with per-flow load balancing

Probes to a destination belong to same flow

• How to identify probes?Use the UDP checksum

• Does not address per-packet load balancing

21

LSrc

B

A C

D

TTL = 2 TTL = 3 DstE

Port 1 Port 1

Port 1Checksum 1

Checksum 3Checksum 2

TTL = 1

Does not address per packet load balancing

• Diamonds appear in 30% of the destinationsParis traceroute removes 10,662 from 19,159 (56%)

• Loops appear in 4.5% of the measured routesParis traceroute removes 5 047 from 5 795 (87%)

Measurement artifacts are common

From LIP6 vantage point:

22

Paris traceroute removes 5,047 from 5,795 (87%)• Cycles appear in 0.25% of the measured routes

Paris traceroute removes 3,886 from 5,674 (68%)• Other causes

Routing changesNAT boxesBuggy routersPer-packet load balancing

Forwards the probe

Anomalies:Loops caused by buggy routers

Src A B Dst

XRejects the probe

-bash$ traceroute Dsttraceroute to Dst1 B 0.289 ms2 B 0.278 ms

23

Forwards the probe with TTL equal to 0

Src B

TTL = 1 TTL = 2X TTL = 1

Dst

with a TTL of 0 and sends it back to the source

3 Dst 0.578 ms

-bash$ traceroute-paris Dsttraceroute to Dst1 B 0.289 ms !T02 B 0.278 ms3 Dst 0.578 ms

Rejects the probe with a TTL of 0 and sends it back to the source

Forwards the probe with TTL equal to 0

Anomalies:Loops caused by NAT boxes

Src Dst (NAT) B CA

D t

Response TTL = 252IP Identifier = 9356

Response TTL = 254IP Identifier = 12375

24

Src

TTL = 2

(NAT)

TTL = 3

DstA

BDst

TTL = 3

2Response TTL = 253IP Identifier = 5286

5

C:₩>tracert www.colbud.hu

Tracing route to www.colbud.hu [81.182.250.153]over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.200.2542 19 ms 21 ms 18 ms vxr.tau.ac.il [132.66.8.10]3 20 ms 21 ms 21 ms c6509.tau.ac.il [132.66.8.20]4 21 ms 20 ms 19 ms tel-aviv.tau.ac.il [132.66.4.1]5 20 ms 22 ms 19 ms gp1-tau-fe.ilan.net.il [128.139.191.70]6 26 ms 22 ms 21 ms iucc il1 il geant net [62 40 103 69]

Non-Responding Routers

6 26 ms 22 ms 21 ms iucc.il1.il.geant.net [62.40.103.69]7 91 ms 92 ms 92 ms il.nl1.nl.geant.net [62.40.96.117]8 97 ms 97 ms 97 ms nl.de1.de.geant.net [62.40.96.101]9 95 ms 96 ms 93 ms ffm-b2-pos2-3.telia.net [213.248.77.89]

10 96 ms 96 ms 150 ms ffm-bb2-pos2-3-0.telia.net [213.248.64.177]11 110 ms 112 ms 114 ms bpt-b1-pos2-0.telia.net [213.248.64.26]12 * * * Request timed out.13 112 ms 110 ms 111 ms 10ge-0-0.core0-ip2.net.telekom.hu [145.236.85.2]14 112 ms 114 ms 110 ms tenge1-2.core0.adatpark.hu [145.236.89.10]15 114 ms 112 ms 114 ms fixip-lns2.adatpark.hu [195.228.253.58]16 120 ms 122 ms 124 ms 153-250-182-81.adsl-fixip.axelero.hu [81.182.250.153]

Trace complete.

1

2

3

4

3

4

11

4

True network

10

5

11

9

6

8

77

6

8

77

8

10

26

3

4

1

7

4

Measured network

1

6

8

7

7

8

10

27

1

2

3

7

4

1

77

4

revisitedTrue networkSuppose one router is unresponsive:

3

4

1

7

10

5

11

9

6

?

7

6

?

77

?

10

28

7

3

4

1

7

7

4

10 ? 7

Measured network

1

6

6-?-7

7

10-?-7

10

29

Unresponsive Routers - Example

True network with 3 unresponsive routersThe non-blue nodes are unresponsive.

Measured network with 51 unknown nodesThe non-blue nodes are unknowns (gaps).

30

6

Unifying UnknownsIs it possible to identify a group of unknownnodes that represent the same unresponsiverouter?

31

AS Topology Discovery

The Internet Structure

routers


The AS graph




i i i [ ]

private network

Tel Aviv Uni.

ILANAS378

MACHBA

from IP to AS routes

7 69 ms 69 ms 69 ms il.it1.it.geant.net [62.40.96.154]8 82 ms 82 ms 82 ms it.ch1.ch.geant.net [62.40.96.33]9 101 ms 98 ms 98 ms ch.at1.at.geant.net [62.40.96.1]


Trace complete.

DANTE

HR-ZZ

CARnet

AS20965GEANT

CARnetAS2108

378 20965 2108

IP to AS

7

How map IP to AS?

• Routing address registry– Voluntary public registry such as whois.radb.net– Used by prtraceroute and “NANOG traceroute”– Incomplete and quite out-of-date

• Mergers, acquisitions, delegation to customers

• Origin AS in BGP paths– Public BGP routing tables such as RouteViews– Incomplete and inaccurate… but usually right

• Multiple Origin ASes (MOAS), no mapping, wrong mapping

Rifining IP-to-AS Mapping

• Start with initial IP-to-AS mapping– Mapping from BGP tables is usually correct– Good starting point for computing the mapping

C ll t BGP d t t th• Collect many BGP and traceroute paths– Signaling and forwarding AS path usually match– Good way to identify mistakes in IP-to-AS map

• Successively refine the IP-to-AS mapping– Find add/change/delete that makes big difference– Base these “edits” on operational realities

Extra AS due to Internet eXchange Points

• IXP: shared place where providers meet– E.g., MAE-East, MAE-West, LINX, AMS-IX– Large number of fan-in and fan-out ASesLarge number of fan in and fan out ASes

A

B

C

D

E

F

G

Traceroute AS path BGP AS path

Physical topology and BGP session graph do not always match.

B

C

F

G

A E

Extra AS due to Sibling ASes

• Sibling: organizations with multiple ASes:– E.g., Sprint AS 1239 and AS 1791– AS numbers equipment with addresses ofAS numbers equipment with addresses of

another

Traceroute AS path BGP AS path

A

B

C

D

E

F

G

H

A

B

C

D

E

F

G

Sibling ASes “belong together” as if they were one AS.

Weird Paths Due to Unannounced Addresses

A B12.0.0.0/8

C

A C

A C A C

B A C B C

C does not announce part ofits address space in BGP

(e.g., 12.1.2.0/24)

Fix the IP-to-AS map to associate 12.1.2.0/24 with C

A

Interfaces in multiple ASes

CBA-B-C

A CBC-B-A

Are A and C neighbor ASes?

What AS does the middle router belong to, B or C?

8

Traceroute Inherent Problems

Same problems as with router level maps:• Load balancing• Routing change during traceroute• Routing change during traceroute• Effects:

– Wrong links– Loops– Missing links

Making AS Maps Larger

Revealing the Internet StructureRevealing the Internet Structure

Revealing the Internet StructureRevealing the Internet StructureDiminishing return!Diminishing return!

⇓

Deploying more boxes does not

30 new links

7 new links

NO new links

boxes does not pay-off

9

Revealing the Internet StructureTo obtain the ‘horizontal’links we need strong presence in the edge

Number of VPs

• Large instrumentation boxes (Ark)– Cannot reach more than 10s of VPs

• Community base software agents (DIMES)– Reached 100s VP– Can grow much more

• Very light agents (Ono)– Reached 1000s VP– Hard to clean noise

• Small hardware devices (RIPE ATLAS)– Reached 100s VP

DIMES

CAIDA’s Archipelago (Ark)

• 54 boxes

May 2011

TAU’s DIMES• ~1000 active agents daily DIMES

May 2011

RIPE’s ATLASMay 2011

• 400 active (600 distributed)So how many is enough?

• [Chen et al. 02], [Bradford et al. 01]: when you combine more and more points of view the return diminishes very fasty

• A few VPs are enough– The mass of the tail is significant

No. of views

contribution

10

How to Measure Contribution?

• Graph size is not the only factor.• We want our Internet view to share the

same properties as the real Internet?same properties as the real Internet?– Remove measurement bias

• How to quantify this?

Data Set

• Data is obtained from DIMES– Community-based infrastructure, using almost

1000 active measuring software agentsg g– Agents follow a script and perform ~2 probes

per minute (ICMP/UDP traceroute, ping)– Most agents measure from a single AS (vp)

• But some (appear to) measure from more…• Data need to be filtered to remove artifacts

– Traceroute data collected during March 2008

Filtering the data

• For each agent and each week, classify how many networks it measured the Internet from Typical cases: yp– ASi:15300, ASj:8 – ASi:10000, ASj:3178– ASi:10000, ASj:412 , ASk:201– 18000, 12, 11, 9, 9, 3, 3, 2, 2, 1, 1, 1, 1, 1, ….

Measurements Per Agent

Week 4,2008

Measurements per Network

500

Agents per Network

11

Filtering Results• 96% of the agents have less than 4

different vps• High degree ASs tend to have more agents• High number of measurements for all vps• High number of measurements for all vps

degrees

Diminishing Returns?

• Barford et. al. – the utility of adding many vps quickly diminishes – In terms of ASes and AS-linksIn terms of ASes and AS links

• Shavitt and Shir – utility indeed diminishes but the tail is long and significant– Tail is biased towards horizontal links

• We wish to quantify how different aspects of AS-level topology are affected by adding more vps

Creating topologies per VP

sort by

Topology Size

• The return (especially for AS links) does not diminishes fast!

VP with small local topology can contribute many new links!

Direction of Detected Links

• For each link: Plot max adjacent AS degree and max adjacent ASes degree difference

Low degree gdifference – indicates tangential links and links between small-size ASesHigh degree

difference –indicates radial links towards the core

Convergence of Properties

• Taking several common AS-level graph properties, and analyze their convergence as local topologies are addedp g– Keeping the sort order by number of links

• Slow convergence indicates the need to have broad and diverse set of vps

12

Density and Average DegreeSlow convergence of density and average degree – easy to detect ASes but difficult to find all links

Power-law and Max Degree

Fair convergence of power-law exponent

Fast convergence of maximal degree – core links are easily detects

Betweenness and Clustering

Radial links decrease cc

Fast convergence of max bc – Level3 (AS3356), a tier-1 AS is immediately detected as having max bc

Tangential links increase cc

Revisitng Sampling Bias

• Lakhina et al. – AS degrees inferred from traceroute sampling are biased– ASes in vicinity to vps have higher degreesASes in vicinity to vps have higher degrees– Power-law might be an artifact of this!

• Dall’asta et al. – no…it is quite possible to have unbiased degrees with traceroutes

• Cohen et al. – when exponent is larger than 2, resulting bias is neglible

Evaluating Sampling Bias

• For each AS find:– All the vps that have it in their local topology– The Valley-Free distance in hopsThe Valley Free distance in hops

Up-hill to the core (c2p), side-ways in the core (p2p) and down-hill from the core (p2c)

Dataset VPs and Distances

Low degree ASes are seen from less vps than high-degree Ases…this makes sense!makes sense!

In our dataset, most ASes have a vp that is only 1-2 hops away!

13

Average Distance per Degree

Low degree ASes are seen from farther vps sampling bias?vps…sampling bias?

No real bias! •More VPs are located in high-degree ASes

•There are high-degree ASes that are seen from “far” vps•Broad distribution – all ASes are pretty close-by to a vp!

Revisiting Diversity Bias

• What is the effect of diversity in vps geo-location and network type?– Some infrastructures rely on academicSome infrastructures rely on academic

networks for vp distribution – does it have an effect on the resulting topology?

• We compare iPlane and DIMES– Classify AS into types: t1,t2, edu, comp, ix, nic

using Dimitropoulos et al.

Diversity Bias Evaluation

iPlane uses many PlanetLab nodes (edu), while DIMES resides mostly at homes (tier-2)mostly at homes (tier 2)

Indeed DIMES have higher t2 and comp degrees and iPlane have higher edu degrees – results are slightly biased to vps’ types!

In Search of Ground Truth• One week is not sufficient for active

measurements

• Both iPlane and DIMES have lower average degrees than RouteViews– Except iPlane’s edu and ix!– Diversity bias exists – need diverse vp types!

Measuring Within a Network

• Comparing vp average degrees to quantify the effect of measuring within a network

Indeed, the average degree when measuring within a network is mostly higher (hmm…tier-1 doesn’t count cause most vps are the same!)

Conclusion

• VP distribution is important– Number, AS type, geo-location

• AS-level graph properties are affected• AS-level graph properties are affected– Some converge very fast– Other converge slowly

• Community based projects have practically unlimited growth potential!

14

BGP Listeners

• Passive listening to BGP messages can easily give AS level topology information– U. of Oregon: RouteViewsU. of Oregon: RouteViews– RIPE

• Problems:– Aggregation– limited

PoP Level Maps

What the PoP is ?• PoP – Point of Presence of the ISPThe Internet Structure

routers


The AS graph


The AS graph The PoP level graph

15

PoP Discovery (RocketFuel)

• Group interfaces to routers• use DNS names

– S1-bb11-nyc-3-0.sprintlink.net is a Sprint router in New York CityNew York City

• use connectivity information– if a router connects only to routers in Seattle, it is in

Seattle

• Use the router role in the topology?– only backbone routers connect to other cities– use DNS names

• s1-gw2-sea-3-1.sprintlink.net is a Sprint gateway router[Spring et al. SIGCOMM 2002]

iPlane PoP Discovery Approach

monitormonitor

][Madhyastha et al. IMC 2006

monitorDelay measurements

Problems with this Approch

• Delay measurements are noisy– Change of time– Routing anomalies: permanent stretchRouting anomalies: permanent stretch

Bad Routing Example

Tel Aviv University

Azriely Center

< 6 km

BEZEK-NET, AS6810

Azriely to TAUTAU to Azriely

16

PoP Discovery (DIMES)

• Most delay measurements are OK• But some have noise

• Use structure to clean ‘bad’ measurements




i i i [ ]

Can we measure link delay?

Linkdelay19-22-1249

Min.0191719182069

Negative delays

7 69 ms 69 ms 69 ms il.it1.it.geant.net [62.40.96.154]8 82 ms 82 ms 82 ms it.ch1.ch.geant.net [62.40.96.33]9 101 ms 98 ms 98 ms ch.at1.at.geant.net [62.40.96.1]


Trace complete.

1316771727-6

698298105112113120112119113

3000

3500

4000

4500

5000Link Delay Measurements Histogram

elay

s

A delay of a link inside TAU

-150 -100 -50 0 50 100 150 200 2500

500

1000

1500

2000

2500

3000

Link delay [ms]

Dis

tribu

tion

of th

e d

a

mon

g 1

ms

bin s

negative delay

What is the graph obtained from running traceroutes thru a POP?

Network Motifs and Significance

• “Network Motifs: Simple Building Blocks of Complex Networks” R. Milo, S. Shen-Orr, S. Itzkovitz, N. Kashtan, D. Chklovskii, and U. Alon. Network motifs: simple building blocks of complex networks. Science, 2002.

• Qualitative measure of statistical significance• Qualitative measure of statistical significance, the Z score:

σμ−

=XZ

Motifs in IP interface Graphs• Z-Score in IP level Graphs

• We should look for motifs with small weights of the edges – representing small physical delays.

17

POP Extraction FlowRemove all edges with:•delay higher than 5 ms•small number of measurements.

Build an induced sub-graph of G for each connected

component.

Divide the sub-graph toRun LocalizationUnify adjacent parent/child Divide the sub graph to parent/child groups (on

sub-graphs)

Run Localization Algorithm (on sub-

graphs)

Unify adjacent parent/child groups to POPs cores, then unify

adjacent POPs

Connect unassociated nodes to the nearest POP

cores.

[Feldman & Shavitt, Globecom ‘08]

Example

Parent-Child ClassificationC1

C2

Parent pair - both of them points to two or more nodes.Child pair - both of them are pointed by two or more nodes.Groups of parent/child nodes are union of parent/child

P1

P2

blue nodes - parent , red nodes - child,blue and red nodes - both parent and child, dashed nodes - neither of two.

are union of parent/child couples.

Localization Algorithm• Checks whether all nodes that belong to the same parent/child group are

located at the same physical location (belongs to the same POP).• Are nodes A and B collocated? (multi-dimensions)

AB

A BI II

• If we assume Gaussian noise – in case I nodes are closer• If we assume Impulse noise – most likely we should select case II

Localization example

• Is 5 co-located with 1,2,3?– No: by node 10– Yes: by nodes 9 & 11– Maybe: by node 6

• Clearly {6,7,8} are notcollocated with {9,10,11}

Localization Algorithm (cont.)• For bipartite graph G(V,E) with weight function

• For each pair of parent nodes

• Define common children group CC with members {cc1,cc2,…}

• Error Ratio Vector

18

Localization Algorithm (cont.)• Compare to a threshold

– Log function brings small errors towards zero.– Using log function on x and 1/x has equal impact

|)),((log(| vuERmedian

– Median is used to disregard small number of outliers– Note: It is also possible to take an absolute difference into

account too. However no significant benefit was observed.

POP Extractionconnect nodes and core elements.

• Repeat Localization algorithm for child nodes, too• Unify adjacent parent/child groups to formulate

POP cores• Unify adjacent POP cores• Optional: Connect “lonely nodes” (singletons) to

POPs.

PoP Discovery• Using Traceroute measurements

– 30M-40M measurements per week.– 5.5M-6.5M distinct edges discovered.– ~1000 agents in over 200 ASes are used for the1000 agents in over 200 ASes are used for the

measurements.– 2.5M IP addresses in over 26,000 ASes are being targeted.– Using Median algorithm to estimate distance between

nodes.

PoP Discovery• Running on bi-weekly basis

– Increased number of discovered PoPs compared to 1 week period.

– Less sensitive to topology changes than 4 weeks period.p gy g p

Distinct EdgesIPs in PoPsNo. of PoPsTime Frames

±20%<1%<1%1 week to 1 week+43%+79%+58%1 week to 2 weeks+59%+15%+10%2 weeks to 4 weeks

PoP Discovery• Discovered PoPs

– ~4400 discovered PoPs.– Over 50,000 IPs within discovered PoPs.

• Over 100,000 IPs with singletonsg

• Discovered mostly large PoPs and not access PoPs.• Enhancements

– Targeting iPlanes’s PoP’s IP addresses – increased the number of discovered PoPs by less than 20%.

– Targeted measurements to specific AS doubled the number of discovered PoPs in small ASes.

• Had some effect in large ISPs but not to that extent.

PoP Discovery• The number of discovered PoPs directly relates to the number

of discovered IP edges.

19

PoP Discovery

• Sensitivity to delay threshold:Number of PoP IPs Number of PoPs

• Sensitivity to number of measurements threshold:Number of PoP IPs Number of PoPs


The AS graph

The PoP level graph

Embedding PoPs in Geography

GeoLocation: Possible Approaches

• Use available GeoIP databases– Better first validate their accuracy

• Use triangulation from known locations• Use triangulation from known locations– We know where PlanetLab nodes are– Can we clean the noise?

Evaluating GeoLocation databases

Verizon/MCI/UUNET (ASN 703)10-nodes PoP (w/Singletons)

How to Evaluate GeoIP DBs

• Main problem: NO ground truth!• Idea: compare several databases

Problem: some databases may get their data– Problem: some databases may get their data from the same source

• Better idea:– Use aggregation into PoPs to check DB

coherency

Evaluation of Geolocation Databases

• Seven databases were used for the evaluation.– NetAcuity (Digital Element) – High end– GeoBytes– GeoIP (MaxMind)– IPligence Max– IPligence Max– IP2Location– HostIP.info – Free service– Spotter – Research tool

• Dataset: DIMES measurements, March 2010– 52K IP addresses (+ 52K singletons IP addresses)– 3800 PoPs

20

Vendor Reported Accuracy

115

†US state accuracy

Evaluation methods

• Null Replies • Agreement within a database - coherency• “Ground Truth” location• Comparison Between databases

– Similarity– By majority Vote

• Database anomalies

Null Replies

117

• For each IP in the PoP (N IPs), each database (M) get a vote on the geo-location– Number of votes N•M

PoP Location

• Using the votes we define the PoP locationand convergence radius

Stage 1118

PoP Location and ‘Convergence’

Stage 1119

Alternative Method

• CAIDA use GeoIP DB voting for IP location:– Select a threshold. They use 80kmSelect a threshold. They use 80km– Cluster locations within the threshold– Node can belong to different clusters– The largest cluster is selected as winner– Use the centroid of the winning cluster

21

PoP spread radius

CDF of Range of ConvergenceConvergence within Databases

PoP spread radius

CDF of Location Votes Percentage Within 500km from PoP Center

Ground Truth evaluation

• Using CAIDA’s 25K “Ground Truth” IP addresses– January-2010 database, based on DNS & ISP collaboration– In the results, city range considered at 100km range

Database IP hits Country Match City MatchG b t 67 3% 80 1% 26 5%

10.1K wrongly Geobytes 67.3% 80.1% 26.5%

HostIP.Info 28.1% 89.0% 17.9%IP2Location 100% 76.0% 13.3%

IPligence 100% 76% 0.7%Netacuity 67.9% 96.9% 79.1%

Spotter 54.1% --- 27.8%

20.5K wrongly located in

Washington DC

located in Washington DC

Correlation among Databases

Heatmap – Median distance between databases CDF- distances between databases

Database Correlation

IPligence- MaxMind (CAIDA)

Comparison with CAIDA

21685

85134

134216

22


Database Anomalies - Disagreement Between DatabasesDIMES

Verizon/MCI/UUNET (ASN 703)10-nodes PoP (w/Singletons)


Database Anomalies - Disagreement Between DatabasesDIMES

Global Crossing (ASN 3549)160-nodes PoP (w/Singletons)

Database Anomalies –False Location Replies

Qwest as an example• 70 PoPs were discovered by the algorithm• MaxMind assigned the PoPs to 55 different locations• HostIP.Info assigned the PoPs to 46 different locations• IP2Location assigned the PoPs to 35 different locations• IP2Location assigned the PoPs to 35 different locations• IPligence located the PoPs in only one distinct location;

– All the PoPs were placed in Denver, where Qwest HQ are located.– Out of 20291 Qwest entries in IPligence, 20252 are located in Denver.

• MaxMind had the same problem as IPligence in their May-2009 DB, but it was fixed in July-2009 DB.

Improving Active Measurements

Spotter Geolocation tool, active measurements basedOur findings (Tech report):◦ Failed to locate a third of the IP addresses◦ Its range of convergence was considerably higher than other databases

Idea: Leverage DIMES infrastructure to improve Spotter ◦ More vantage points, in locations possibly closer to the target◦ Larger dataset◦ Conclusions from PoP study help analyze incorrect location assignments

Goal: study the limits of active measurement based geolocation

Agreement Between Databases – By Majority Vote

CDF of Database Location Deviation From PoP Median.

Long tail.

Evaluating GeoLocation databasesAgreement Between Databases – By Majority Vote

23

Many bad news:• Ground truth has bias• Coherency ≠ Accuracy

SummaryDIMES

• Coherency ≠ Accuracy – BUT: incoherency ⇒ inaccuracy

• Database correlation– Majority vote is tricky

133Most results appear in arXiv:1005.5674, May 2010 & JSAC Dec 2011

What Topology Can Tell US

Background

• Internet economics is changing rapidly• [Gill, et al. PAM 2008]

Large content providers (CPs) bypass many– Large content providers (CPs) bypass many tier-1 ISPs by pushing their networks closer to the users

– The Internet becomes flatter.• [Labovitz et al., SIGCOMM 2010]

– Traffic from CPs is increasing in time– Most of it is routed outside the core

Background (cont.)

• [Dhamdhere and Dovrolis , CoNext 2010]– a new Internet model that captures the Internet

transition from a hierarchy of transit providers y pto a flatter interconnection of peers.

Our Work

• A longitudinal study over 5 years• Observe the change in the Internet

ecosystem using topology instead of trafficecosystem using topology instead of traffic• Focus on content providers and tier-1 ISPs

Methodology

• Collect one month of traceroutes every 3 months

• IP to AS mapping. AS route to topologypp g p gy– When crossing IXP ignore it A-X-B ⇒ A-B– Mapping error do not change the trends

• DIMES has a large vantage point churn– Use iPlane data instead– DIMES shows, in most cases, the same trend

but with larger topologies and more noise

24

ASes Measured

Contend Provides

• Google

• Yahoo!

Tier-1 ISPs

• AT&T

• Level 3

• MSN

• Facebook

• Amazon

• Sprint

• Qwest

• Global Crossing

DegreeMSNGoogleYahoo!AmazonFacebook

Content Providers:A clear increase in the degree

Tier-1 ISPs:Mostly decreasing

CPs Neighbor DegreeAmazon and Facebook:Move from reliance on tier-1 to more diverse neighborsneighbors

Tier-1 Neighbors

Degree increase:•Lose low degree customers•Other customers O e cus o e sincrease degree (?)

CP Neighbor Country

2007

2008

2009

2010

April

New CPs grow in the USVeteran CPs already exploited their US growth potential

25

CP Neighbor Type

New CPs diversify their neighbor typeDecrease the percentage of LTP

Clustering Coefficient

CPs:CC mostly decreasing

Tier-1:Mostly increasing (losing small stub customers)

k-Pruning Analysis

k-Pruning Analysis

1-shell

k-Pruning Analysis

1-shell

26

k-Pruning Analysis

1-shell

k-Pruning Analysis

1-shell

k-Pruning Analysis

1-shell

2-shell

k-Pruning Analysis

1-shell

2-shell

3-shell = core

k-Pruning Analysis

1-shell

2-shell

3-shell = core

k-Pruning Analysis

1-shell

2-shell

3-shell

4-shell = core

27

Shell IndexIn DIMES: MSN is below Yahoo! and Google

All tier-1 ISPs are in the core

• The veteran CP are in the core (or close)•The new CPs advance very fast towards the core

Core Make-up

Content/Access/Hosting Provider

Large Transit Provider

Not only LTPs,Not noticed by [Carmi et al., PNAS 2007]

ProviderSmall Transit

Provider

Enterprise Customer

[Dhamdhere and Dovrolis, IMC’08]

How Do CPs Connect?

CPs tend to use IXPs

Plotted with DIMES data

Centrality

• Betweenness centrality1. Use shortest paths on the undirected AS graph2. Examine all traceroute containing the AS2. Examine all traceroute containing the AS

• Calculate the portion of traceroute where the AS is in the middle of the AS level route

• More accurate than using ToR + valley free SPR

– The two methods gave similar results• PageRank

Tier-1 Betweenness Centrality

• Sprint & Level3 are slowly decreasing• The rest are stable

CPs Betweenness CentralityNote the scale

Google and Yahoo!main AS serve as a AS10310transit to sibling ASes:In 14% and 22% of the traces, respectively. AS15169

28

PageRankNote the scale

Summary

• CPs increase their connectivity• Diversify their connections:

From mostly tier 1 to tier 2 and others– From mostly tier-1 to tier-2 and others– From US centric to global

• Tier-1 centrality is decreasing

• Indeed we see a flatter Internet

mapping the internet topology traceroute a basic tool for toppgy

Documents