The Internet as a computing surface

Avogadro-scale Computing MIT, April 17, 2008

Scott KirkpatrickHebrew University of Jerusalem and CBA, MIT

Questions …• Based on 4 yrs of Internet measurement activity

– DIMES, ETOMIC, EVERGROW and Internet Archive

• Is the Internet Avogadro-scale?• Routing IS asynchronous conformal computing – analyze it. • But the Internet is multi-layered

– Data is centrally managed and closely-held

• The current direction of growth is towards greater local independence• Independence and asynchrony have their costs

– The details (data model, computational model, geographic distribution) are critical

Is the Internet really Avogadro-scale?

• Avogadro’s number is really 2^78.99 ~ 80 bits

• Internet addresses:– IPv4 2^32, IPv6 2^128, IPv4 with NAT ~ 2^64– China takes NAT-ing the furthest, with only a very few entry points, huge internal address

space.

• Web content: modern search engines crawl 10^6 web pages/sec, ~2.5x10^12 pages each month before discarding. “Deep web” maybe 100x larger.

• IPv4 operates in layers, divided by subnetworks (ASes).• AS-AS uses BGP routing, inside AS uses shortest-path with link-state, may

conceal all under MPLS…

Next discuss how the Internet is connected

• K-shell analysis shows an interesting kind of hierarchical structure on the largest scale, gives unambiguous identification of the “nucleus.”

• It’s fractal, so the structure – if it has actually evolved to be a solution to more general problems – can be applied on many scales.

• Percolation properties show that both local and long-ranged connectivity coexist

K-shell for network visualization

DIMES monitoring project:www.netdimes.org

K-shell picture gives unique nucleus + fractal

Carmi et al., PNAS 2007

Meduza (מדוזה) model

This picture has been stable from January ‘04 (kmax = 30) to present day, with little change in the nucleus composition. The precise definition of the tendrils: sites isolated from the largest cluster in all the crusts – connect only to the core.

“Disruptive” Alternatives to today’s Protocols

• The Medusa structure has consequences: routing, viewed as a computation, is changing…

• Monitoring becomes a function in which every router participates.

• Information shared beyond today’s “customer-provider privilege” – even one step helps a lot

• Regional networking reduces dependence on long-haul carriers in the nucleus.

• Each trend is towards more local interchange in a flatter “surface”

Synchronization and in-band Control

• Roughening is an issue with distributed data-intensive CA-style computation. Can graded longer-range communications eliminate it? Are occasional small-world links enough? Which is more effective for the headaches required?

SK, Science Perspective, 2004

Compute-intensive CA’s are different beasts

• Single data source, serial execution algorithms (like sort) suffer no synchronization overheads.

• Load and unload are critical

• Ultimate “wallpaper computing” will develop its own models

• Hierarchy ain’t going away anytime soon!