Capturing Complexity in Networked

Systems Design:

The Case for Improved Metrics

Sylvia Ratnasamy

Intel Research

Motivation

Our community values “simple”, “clean” system designs

“The advantage of Chord is that it is less complicated…” – Chord, Sigcomm’01

“The complexity of the architectural design…” – Sprint, IEEE Network 2000

“A design for multicast that is simple…” – Perlman. Simple Multicast, IETF Draft.

“Complex routing algorithms may have difficulty scaling… simple floods are

sufficient” -- TinyOS, NSDI’04

“This paper proposed a simple and effective approach …” – SOSR, OSDI’04

“Thus, simplicity is our primary design requirement …” – BVR, NSDI’05

Motivation

Our community values “simple”, “clean” system designs

• But lacks metrics that rigorously capture this aesthetic

• Best practice: metrics borrowed from the theory community• e.g., total_messages, total_state

• Problem: at times incongruent with our notion of “simplicity”• e.g., flooding: inefficient but simple• e.g., dijkstra’s: O(n) state, but simple• e.g., leaderID vs. neighborID

This Paper

• Can we quantify design “simplicity”?– more rigorously compare-and-contrast design options– align design goals of algorithms, systems communities

• First stab: complexity metrics for the algorithmic component of a networked system

• Note– just one aspect to system complexity– intent: complement, not replace, existing metrics

Outline

• Strawman

• Simple evaluation scenarios

• Limitations and Future Directions

Strawman

Two observations:– much of system design centers around state

• what state is required?• how is it constructed/maintained?• how is it used?

– what distinguishes wide-area state:• derived from remote nodes dependencies are distributed • relayed via intermediate nodes more dependencies

Current metrics mostly treat all state as equal

Proposal

For a given piece of state s, measure the ensemble of distributed dependencies that yield s

• First cut: measure counting – akin to existing metrics: #msgs, #state– amenable to evaluation by simple examination, simulation

Value vs. Transport Dependencies

s is value dependent on x, s is transport dependent on next(B) state at A, R1, R2, R3.

vs = #pieces of state on which s is value dependent

tsx = #pieces of state relied on to transport x to s

A R1 R2 R3 B

s=x

next(B)

x=30°

next(B) next(B)next(B)

Counting Dependencies

A R1 R2 R3 B

s=x

next(B)

x=30°

next(B) next(B)

vx = txx = 0 vs = 1 ; tsx = 4

next(B)

Counting Dependencies

A R1 R2 R3 B

s=x x=30+y°

C R4

y=20°

vy = tyy = 0 vs = 2 tsx = 4

vx = 1 txy = 2

Note: value dependencies accumulate

Dependencies Complexity

single input: state s, derived from x

– complexity of s, cs = (vx + 1) tsx + cx

A R1 R2 R3 B

x=y+30°

C R4

y=20°

vy = tyy = 0 vs = 2 , tsx = 4vx = 1 txy = 2

s=x

cy = 0 cx = 1.2+0 = 2 cs = 2.4+2 = 10

single input: state s, derived from x

– complexity of s, cs = (vx + 1) tsx + cx

BA

x=0vs = d , tss-1 = 1vx = i tii-1 = 1

c = 0 cs = O(d2)

s=1 s=2 s=ds=i

…

cs = 1 cs = 3

Dependencies Complexity

multiple inputs: state s, derived from x1, x2, …, xm

two basic variants:

1. s = ALL (xi) cs = Σ ((vxi + 1) tsxi + cxi)

– if vxi = cxi = 0, tsxi = 1, then cs = m

2. s = ANY (xi) cs = 1/m Σ ((vxi + 1/m) tsxi + cxi)

– if vxi = cxi = 0, tsxi = 1, then cs = 1/m

Dependencies Complexity

multiple paths: state s derived from x, multiple paths from x to s

simple case: m disjoint paths, each incurring d transport dependencies

tsx = d/m, cs = vs tsx + cx

• if vx = cx = 0, then cs = d/m

Dependencies Complexity

Complexity: toy scenarios

scenario vs cs

s=x; s,x are 1 hop apart 1 O(1)

s=x; s,x are k hops apart 1 O(k)

s=distance(x); s,x k hops apart k O(k2)

s=ALL(x1, x2, …, xm) s, xi are 1 hop apart

m O(m)

s=ANY(x1, x2, …, xm)

s, xi are 1 hop apart1/m O(1/m)

s=x m distinct 1-hop paths from x to s

1 O(1/m)

Preliminary Evaluation (summary)

system vs cs croute

distance-vector O(d) O(d2) O(d3)

link-state O(1) O(d) O(d3)

compact [SPAA04]?? O(n) > O(nd) > O(nd2 )

tree-based O(n1/2) O(n) O(n3/2)

geo-routing O(1) O() O(n)virtual-geo [mobicom03] O(n) O(n3/2) O(n5/2)

Limitations, Future Directions

• scope: no validation of assumptions, correctness, quality, performance, etc.

• correlated inputs• discriminating transport dependencies• capturing absent dependencies• capturing robustness

– count “reverse” dependencies?

• role of time• evaluating the metric: future studies

Summary

• design simplicity: valued, but poorly measured

• question: is design complexity measurable?

• conjecture: capturing dependencies in state is key (w.r.t. algorithmic component of a system)

Thanks! Questions?

complexity of operations

• an operation acts on different pieces of state

• each piece of state has a complexity

• compute complexity of an operation by (appropriately) combining complexity of state it acts on

cs = 0 1 ... O((d-1)2) O(d2)

BA

x=0 s=1 s=2 s=ds=i

…

complexity of forwarding operation: cfwd = O(d3)

complexity of operations

• an operation acts on different pieces of state

• each piece of state has a complexity

• compute complexity of an operation by (appropriately) combining complexity of state it acts on

cs = k

cs = k

cs = k

cs = km

12

3

…fetch(obj)

complexity of fetch: cfetch = O(k/m)

Future Studies

• centralized solutions simpler?– e.g., RCP vs. DV

• designing for low dependency– e.g., soft state-based approaches

• layered vs. customized– e.g., PHTs [sigcomm’05] vs. Mercury [sigcomm’04]

• different routing options– mesh (RON, ESM), DHT-inspired (VRR/ROFL),

centralized (RCP), compact routing-based, landmark routing, coordinate-based, …