bandwidth on the internet mat ford isoc standards & technology isoc advisory council meeting,...

Bandwidth on the Internet

Mat Ford

ISOC Standards & Technology

ISOC Advisory Council Meeting, Hiroshima, Japan

Sunday November 8th 2009

Takeaways, upfront

• Gross bandwidth growth more than catered for by new capacity

• No Internet ‘brownouts’ anytime soon

• Broadband creating new realities for access nets• More bandwidth provisioning isn’t the answer

• Additional capacity is quickly consumed (by design)

• Interests of content providers, ISPs and users not always well-aligned

• We need more data

• IETF has several streams of relevant work

• ISOC has a panel on this topic during IETF76

2009-11-08 Bandwidth on the Internet, ISOC AC Meeting 2

Bandwidth, defined

• A term that describes the amount of information that can be passed through a communications channel in a given amount of time; that is, the capacity of the channel. The bandwidth is usually expressed in 'bits per second’…

• Or bytes per second, leading to lots of confusion

• Used for expressing• Capacity of a channel or interface (e.g. 100Mbps ethernet i/f)

• Rate of information transfer (e.g. downloading at 3.2KB/s)

2009-11-08 3Bandwidth on the Internet, ISOC AC Meeting

Internet is sharing

• Early objections to feasibility of packet-switching centred on the need for very large packet buffers to handle uncontrolled loads from end hosts

• Leonard Kleinrock credited with demonstrating the theoretical feasibility of packet-switched communications


• Internet protocols have their origin in the desire to share computers in diverse locations

• Extension of the OS-paradigm of shared resources

Rules of the road

• Resource sharing• Central to notions of Internet’s value and

success

• Enables interconnection of diverse applications

• Over heterogeneous networking media with diverse speeds

• Sharing creates potential for demand to outstrip supply

• Necessitates a sharing mechanism

• This is congestion control


How much bandwidth?

• Cisco Forecasts 44 Exabytes per Month of IP Traffic in 2012


Some rough consensus?


Kenjiro Cho et al. see ~40% growth per annum since 2005, for IX peaks

What about capacity?


• Telegeography: Capacity growing by 50% per annum

Congestion


Definitions of congestion

• Arrival rate > service rate (λ > μ)• Congestion = arrival rate/service rate (p=λ/μ)

• In the presence of queues, congestion only present when packets are dropped?

• This is TCP’s definition of congestion

• For network operators, congestion is usually measured over longer timescales, e.g. 15mins

• For economists, congestion occurs when increased use would impose a cost on existing users

• In this case congestion can occur before queues start to build or packets are dropped


Evolution of congestion control

• Congestion control originally handled by ICMP

• Widespread incidents of congestion collapse in mid-1980s• Early TCP implementations had very bad retransmission behavior that

pushed the entire network into a state where most packets were lost and the resultant throughput was negligible.

• Addition of slow start, congestion avoidance, fast retransmit and fast recovery algorithms to TCP


Image source: Internet Protocol Journal, Vol 9, No. 2

Standards evolution

• RFC 793 ‘Transmission Control Protocol’, STD 7 (Sept. 1981)

• Doesn’t specify any congestion control mechanism

• Congestion control handled by ICMP Source Quench

• RFC 1122 ‘Requirements for Internet Hosts – Communication Layers’ (October 1989)

• Mandates implementation of a congestion control algorithm

• RFC 2581 ‘TCP Congestion Control’ (April 1999)• Defines the accepted mechanism

• Required for the avoidance of congestion collapse

• Flow-rate fairness


Ref: RFC 4614 ‘A Roadmap for TCP Specification Documents’

TCP

• Implemented on end hosts

• Regulates sending rate for a single flow

• Doesn’t observe sending rate over time, or whether there are multiple flows from a single end host

• Doesn’t regulate aggregate demand

• Doesn’t know where congestion occurred, when it occurs

• TCP will keep getting faster unless• All data is sent

• Congestion experienced

• End system maxed out


TCP in action


Animation credit: Guido Appenzeller

TCP isn’t enough

• In his original paper, Van Jacobson noted some limitations

• While algorithms at the transport endpoints can insure the network capacity isn’t exceeded, they cannot insure fair sharing of that capacity. Only in gateways, at the convergence of flows, is there enough information to control sharing and fair allocation. Thus, we view the gateway ‘congestion detection’ algorithm as the next big step.’

• Whether or not multiple flows share common bottleneck links isn’t clear from the edge – only net ops have this view


Cable vs. DSL architectures


• voluntarily polite algorithms in endpoints• pushes until congested• equalises rates of data flows

a game of chicken – taking all and holding your ground pays

or start more ‘TCP-fair’ flows than anyone else (Web: x2, p2p: x5-100)

or for much more data than others (video streaming or p2p file-sharing x200)• net effect of both (p2p: x1,000-20,000 higher traffic intensity)

24

TCP-fairness

bandwidthbandwidth22

bandwidth1capa

cit

y

time

(VoIP, VoD)unresponsive

flow3

Slide credit: Bob Briscoe

2009-11-08 Bandwidth on the Internet, ISOC AC Meeting

In defence of flow-rate fairness

• Floyd and Allman (2008)

• ‘We do not, however, claim that flow-rate fairness is necessarily an optimal fairness goal or resource allocation mechanism for simple best-effort traffic. Simple best-effort traffic and flow-rate fairness are in general not about optimality, but instead are about a low-overhead service (best-effort traffic) along with a rough, simple fairness model (flow-rate fairness).’


What’s different this time?

• Congestion collapse isn’t new• Happened in the 80s

• Predicted again in the 90s (Metcalfe’s gigalapse)

• More predictions swirling around now

• Growing diversity of uses/users

• Increasing traffic volumes

• Increasing economic value

• Emergence of public policy debates


Impact of all-you-can-eat broadband


5%

45%

15%

27%

20%16%

20%

8%

40%

4%

BroadbandUsage Distribution

% of subscribers % traffic

sour

ce:

Ella

coya

200

7(n

ow A

rbor

Net

wor

ks)

Source: Cho et al., Proceedings of ACM CoNEXT2008

A changed landscape

• Services that are now dominant consumers of bandwidth were simply not viable at pre-broadband access data rates

• Natural suppression of user demand

• Congestion is at the edge

• Huge growth of mass-market broadband since 2000• ‘a seismic shift in the nature of the congestion problem’

• Bandwidth hungry apps and multiple users per connection

• Raised expectations

• Access nets now the dominant constraint on achievable throughput

• Changing diurnal patterns of consumption

• Growing symmetry of traffic patterns


29

ISPs forced to subvert TCP

1. equal bottleneck flow rates(TCP)?

2. access rate shared between active users, but weighted by fee (weighed fair queuing, WFQ)?

3. volume capstiered by fee?

4. heaviest applications of heaviest usersthrottled at peak times by deep packet inspection (DPI)?

29

bit-rate

time

bit-rate

timebit-rate

timebit-rate

time



Congestion isn’t a bug, it’s a feature

• Want transmissions to occur as fast as possible

• Lack of congestion implies network could be more efficient

• Techniques are available to enable the use of congestion signals to control flow rates without lost packets/retransmissions


time

bit-rate

time

bit-rate

Graphic credit: Bob Briscoe

31

• light usage can go much faster• hardly affects completion time of

heavy usage• doesn’t have to shift into night

• BitTorrent & Microsoft have protocols to do this

but... punished by #2, #3 & #4

NOTE: weighted sharing doesn't imply differentiated network service

• just weighted aggressiveness of end-system's rate response to congestion

bit-rate

time

bit-rate

time

bit-rate

time

1. TCP

4. deeppacketinspection(DPI)

weightedTCPsharing

bit-rate

time

2. (weighted) fairqueuing

bit-rate

time

3. volume caps

Better alternativesSlide credit: Bob Briscoe


32

• Becoming impossible to deploy a new use of the Internet• Must negotiate arbitrary blocks and throttles en route

• Two confusable motives• Fairer cost sharing

• Competitive advantage to own services

• How to deconfuse? how to encourage fairer cost sharing?• Make cost of usage transparent

• Fixing Internet technology should avoid need for legislation

32


Limiting the horizon


BBC iPlayer example

• Original version used P2P to minimise BBC wholesale bandwidth costs

• Latest update, P2P technology disabled• Disliked by end-users

• Disliked by ISPs

• BBC claim wholesale bandwidth costs have fallen

• iPlayer Desktop warns users to beware ISP bandwidth caps and charges


Wrapping up


• Recent studies conclude• P2P declining

• Being replaced with video and other web2.0 content

• Network capacity growing faster than traffic volumes

• Interests of content providers, ISPs and users not always well-aligned

• Fair management of congestion is a whole network issue

We need more data

• http://mitas.csail.mit.edu/

• MINTS

• CAIDA

• M-Lab

• Transparency wrt congestion management policies of net ops is a good start

• Mismanagement of congestion is stifling application innovation

• Want to avoid imposition of regulation that enshrines TCP flow-rate fairness in law – there is ample scope for innovation in congestion management mechanisms at both sub-second and sub-month timescales.


http://mitas.csail.mit.edu/

Network operators play a key role

• Determine supply of network resources

• Limit demand (access link capacity)

• Is there a role for network operators at shorter time scales?


An inflection point?

• Can the Internet community move to a new notion of what constitutes ‘fair’ resource allocation on the Internet?

• TCP has ‘incumbent’s advantage’

• If it ain’t broke, don’t fix it!

• Has the environment changed so significantly that it is broke?

• Who should have responsibility for prioritising traffic during overload conditions?

• How can the answer to that question be made to work in practice?


Relevant IETF work

• Conex BoF• Exposing expected congestion along the forwarding path of the

Internet

• Ledbat WG• Congestion control algorithm for scavenger service

• Alto WG• Protocols for better-than-random peer selection

• MultipathTCP WG• Simultaneously use multiple paths in a single TCP session

• Homegate BoF• Getting uniform set of requirements to aid deployment


ISOC Panel

• Internet Bandwidth Growth: Dealing with Reality• 11.45am – 12.45pm, Tuesday 10th November

• Light lunch provided

• Orchid West Room, ANA Crowne Plaza Hotel

• Panellists• Leslie Daigle (Moderator)

• Kenjiro Cho, IIJ

• Richard Woundy, Comcast

• Danny McPherson, Arbor Networks

• Lars Eggert, Nokia Research Center, IETF Transport AD

• Audiocast & report will be available


References

• The Evolution of Internet Congestion• Steven Bauer, David Clark, William Lehr (MIT)

• Internet Cost Transparency: Mending Value Chain Incentives• Bob Briscoe (BT)

• ATLAS Internet Observatory 2009 Annual Report• C. Labovitz et al. (Arbor Networks, Inc.)

• Observing Slow Crustal Movement in Residential User Traffic• Cho et al. (IIJ)


bandwidth on the internet mat ford isoc standards & technology isoc advisory council meeting,...

Documents

isoc ac meeting slide

isoc ac meeting16 slide

isoc ac meeting17 slide

isoc ac meeting6 slide

isoc ac meeting20 slide

isoc ac meeting15 slide

isoc ac meeting2 slide

isoc ac meeting5 slide