Finding Nearby Objects in Peer-to-Peer Networks

Kirsten Hildrum

Object Location

The Big Picture

Behind the Cloud

?

Two solutions

• One directory– One participant stores

a lot, but average is low

– Stretch is high

• Everyone has a directory– Low stretch!– Everyone has lots of

data– Changes are very

expensive

Distribute directoryEach peer is responsible for part of directory

Q: Who is responsible for my object?A: Hash peers into same namespace as

objects.

Q: How do I get there?A: Peer-to-peer networks route to names.

Must efficiently self-organize into network

Efficiency is…

With respect to network• Few neighbors• Short routes (few hops

& low stretch)• Easy routing (No

routing tables!)• Dynamic

With respect to directory• Not too many entries

(4,4)

hard

easy

• Low stretch dynamic peer-to-peer network

• Tolerate failures in network

• Adapting to network variation

• Related results

• Future work

Outline

System Neighbors

Motivating Structure

Hops

CAN, 2001 O(r) grid O(rn1/r)

Chord, 2001 O(log n) hypercube O(log n)

Pastry, 2001 O(log n) hypercube O(log n)

Tapestry, 2001 O(log n) hypercube O(log n)

Distributed Hash Tables

• These systems give– Guaranteed location – Join and leave algorithms– Load-balanced storage

• No stretch guarantees

Low Stretch Approaches

System Stretch Space Balanced Metric

Awerbuch Peleg, 1991

polylog polylog no General

PRR, 1997 O(1) O(log n) yes Special

Thorup-Zwick O(k2) O(kn1/k) yes General

RRVV, 2001 polylog polylog yes General

• Not dynamic

Result #1: Make a dynamic low-stretch scheme

PRR/Tapestry

City

State

Country

PRR/Tapestry

Two object types: red and blue, so two trees

Level 1

Level 2

Level 3

Neighbor TableFor “5471” (Octal)

Routing Levels1234

1xxx

2xxx

0xxx

3xxx

4xxx

5xxx

6xxx

7xxx

50xx

51xx

52xx

53xx

54xx

55xx

56xx

57xx

540x

541x

542x

543x

544x

545x

546x

547x

5470

5471

5472

5473

Ø

5475

Ø

5477

Balancing Load

1

NodeID5123

3

3

2

22

4

3

NodeID5471

NodeID5416

NodeID5061

NodeID5432

NodeID5455

NodeID5470

Building It• Every node is a root!• Can extend to more than two colors• Pefix routing in base b promoting at rate 1/b• b=16 match one hex digit or four bits at a time

• Space: O(b logb n) – Every node at every level stores b parents

We want the links to be of geometrically increasing length

Node Joins

• Correctness O(log n) insert & delete – Concurrent inserts in a lock-free fashion

• Keeping low stretch– Required a nearest-neighbor search algorithm

as subroutine– Keep O(1) stretch in special metrics

• Algorithms implemented in version of Tapestry used in OceanStore

Concurrent Insert

Solving the Problem

Theorem 1 When peer A is finished inserting, it knows about all relevant peers that have finished insertion.

Hildrum, Kubiatowicz, Rao & Zhao, “Distributed Object Location in a Dynamic Network” SPAA2002

Sequential Consistency(without locks)

BA

0

20

40

60

80

100

120

0 5 10 15 20 25 30 35

Time (minutes)

Co

nsi

stan

cyNode Join in Practice

largegroup join

~210

killnodes

RouteToID operations on 40 node clusterChurn: nodes join/leave every 10 seconds, average lifetime = 2mins

150 128 203

Constant churn

Dynamic challenge: find parent

• Need to find closest at level

• Idea Use routing structure backwards

• Other approaches• Clarkson ‘97

• Karger & Ruhl ‘02

• Krauthgamer Lee ‘03

Growth Restriction

• Growth Restriction:– Ball of radius 3r

contains only a factor of c more nodes than ball of radius r.

– Stronger than “polynomial growth”

• Base b at least c– Promote less than 1/c

r3r

Algorithm Idea, in pictures

new

The Trimming Lemma

• Circle of radius r around new with at least one level i node.

• To find level (i-1) must find its parent.

• Level (i-1) node in little circle must point to a level i node in 3r of new

<2r

r

3r

How Many?

• Keep fixed k= O(log n) • If k red nodes in little

circle, then about ck in big circle

• Expect only about ck/b blue nodes

• If c/b < 1, expect less than k

r

3r

log n per level & log n levels, so total is O(log2 n)

Summary

In growth-restricted networks (with no additional space!):

Theorem 2 [HKRZ02] Can find nearest neighbor with high probability with O(log2 n) messages

Theorem 3 [HKMR04] Can find nearest neighbor, and messages is O(log n) with high probability

+ more dynamic, low-stretch network

• Low stretch dynamic peer-to-peer network

• Tolerate failures in network

• Adapting to network variation

• Related results

• Future work

Outline

Behind the Cloud Again

Dealing with faults

• Multiple paths– Castro et. al– One failure along path,

path breaks

• Wide path– Paths faulty at the same

place to break

• Exponential difference in width effect

• “retrofit” Tapestry to do latter in slightly malicious networks

Failed!

Still good…

Effective even for small overhead

Theorem 4 In growth restricted spaces, can make probability of failed route less than 1/nc for width O(clog n)Hildrum & Kubiatowicz, DISC02

0

10

20

30

40

50

60

70

80

90

100

0.1 0.2 0.3 0.4 0.5

Fraction of Bad Nodes

% f

aile

d r

ou

tes 1

2

3

4

5

6

Wide path vs. multiple paths

0

10

20

30

40

50

60

70

80

90

0 0.1 0.2 0.3 0.4 0.5 0.6

Fraction of Bad nodes

Fa

ile

d P

ath

s

4

4 Single

• Low stretch dynamic peer-to-peer network

• Tolerate failures in network

• Adapting to Network Variation

• Related results

• Future work

Outline

Network not homogeneous

Previous schemes assumed global growth rate. But what if there isn’t one?

San Francisco

Nebraska

Paris

Using the local c

• Idea: Determine local c and use it

• Challenge: Paths still meet up, stretch still low, and space reasonable

• Technique: “Fix” or match bits– In variable size chunks– PRR97, LAND, Tapestry,

Pastry matched bits in fixed size chunks

r3r

Summary

• Previous approaches assumed a global c• Had to know c in advance of construction Hildrum, Krauthgamer & Kubiatowicz

[SPAA04]:

Dynamic, locally optimal low-stretch network

• Low stretch dynamic peer-to-peer network

• Tolerate failures in network

• Adapting to network variation

• Related results

• Future work

Outline

Congestion

• Latency might not be right measure

• Reduce bandwidth usage

• When demands are known, can use linear program to find a set of paths

• But what if demands are not known…

Summary

• Räcke02 showed scheme exists O(log3 n)• Azar, Cohen, Fiat, Kaplan, & Räcke in STOC03

find optimal routing, but not hierarchical

Theorem 6 Hierarchical scheme can be constructed in polynomial time. (Within O(log2 n log log n) of optimal)– Harrelson, Hildrum, & Rao, “A Polynomial-time Tree

Decomposition to Minimize Congestion” SPAA03

Online Auctions for Digital Goods

Online Answer bidder i before seeing the next bid– Bob offers you 50¢, yes or no?– Don’t get to see Alice’s bid before answering

Digital Each additional copy is free– Want to sell to everyone at their bid– Can’t sell to everyone at their bid, or else all bids will be low

Bar-Yossef, Hildrum, Wu, “Online Auctions for Digital Goods” SODA 2002:

Theorem 7 Constant competitive ratio when opt makes O(maxBid log log maxBid)

Theorem 8 When best fixed price sells few, can get:

)maxBid(loglogexp O

• Low stretch dynamic peer-to-peer network

• Tolerate failures in network

• Adapting to network variation

• Related results

• Future work

Outline

Future Work• Object location questions

– Tradeoff between degree, stretch, load balance as function of growth rate

– Smooth transition between general-metric systems and growth-restricted systems

• Distributed low-congestion network

• Peer-to-peer networks with selfish players

• Sensor networks

Thank You!