algorithmic nuggets in content delivery - · pdf filealgorithmic nuggets in content delivery...

Algorithmic Nuggets in Content Delivery

Bruce M. Maggs Ramesh K. SitaramanDuke and Akamai UMass, Amherst and Akamai

[email protected] [email protected]

This article is an editorial note submitted to CCR. It has NOT been peer reviewed.The authors take full responsibility for this article’s technical content. Comments can be posted through CCR Online.

ABSTRACTThis paper “peeks under the covers” at the subsystems thatprovide the basic functionality of a leading content deliv-ery network. Based on our experiences in building one ofthe largest distributed systems in the world, we illustratehow sophisticated algorithmic research has been adapted tobalance the load between and within server clusters, man-age the caches on servers, select paths through an overlayrouting network, and elect leaders in various contexts. Ineach instance, we first explain the theory underlying thealgorithms, then introduce practical considerations not cap-tured by the theoretical models, and finally describe what isimplemented in practice. Through these examples, we high-light the role of algorithmic research in the design of com-plex networked systems. The paper also illustrates the closesynergy that exists between research and industry whereresearch ideas cross over into products and product require-ments drive future research.

1. INTRODUCTIONThe top-three objectives for the designers and operators

of a content delivery network (CDN) are high reliability,fast and consistent performance, and low operating cost.While many techniques must be employed to achieve theseobjectives, this paper focuses on technically interesting al-gorithms that are invoked at crucial junctures to provideprovable guarantees on solution quality, computation time,and robustness to failures. In particular, the paper walksthrough the steps that take place from the instant that abrowser or other application makes a request for contentuntil that content is delivered, stopping along the way toexamine some of the most important algorithms that areemployed by a leading CDN.

One of our aims, as we survey the various algorithms, isto demonstrate that algorithm design does not end whenthe last theorem is proved. Indeed, in order to develop fast,scalable, and cost-effective implementations, significant in-tellectual creativity is often required to address practicalconcerns and messy details that are not easily captured bythe theoretical models or that were not anticipated by theoriginal algorithm designers. Hence, much of this paper fo-cuses on the translation of algorithms that are the fruits ofresearch into industrial practice. In several instances, wedemonstrate the benefits that these algorithms provide bydescribing experiments conducted on the CDN.

A typical request for content begins with a DNS queryissued by a client to its resolving name server (cf. Figure 1).The resolving name server then forwards the request to the

Edge Server

Client

Origin

Authorita4ve Name Server (Global and Local Load

Balancing)

Overlay Rou4ng

Content

DNS

Figure 1: A CDN serves content in response to aclient’s request.

CDN’s authoritative name server. The authoritative nameserver examines the network address of the resolving nameserver, or, in some cases, the edns-client-subnet provided bythe resolving name server [9], and, based primarily on thisaddress, makes a decision about which of the CDN’s clustersto serve the content from. A variant of the stable marriagealgorithm makes this decision, with the aim of providinggood performance to clients while balancing load across allclusters and keeping costs low. This algorithm is describedin Section 2.

But DNS resolution does not end here. The task of indi-cating which particular web server or servers within the clus-ter will serve the content is delegated to a second set of nameservers. Within the cluster, load is managed using a consis-tent hashing algorithm, as described in Section 3. The webserver address or addresses are returned through the resolv-ing name server to the client so that the client’s application,such as a browser, can issue the request to the web server.The web servers that serve content to clients are called edgeservers as they are located proximal to clients at the “edges”of the Internet. As such, Akamai’s CDN currently has over170,000 edge servers located in over 1300 networks in 102countries and serves 15-30% of all Web traffic.

When an edge server receives an HTTP request, it checksto see if the requested object is already present in the server’scache. If not, the server begins to query other servers in

ACM SIGCOMM Computer Communication Review 52 Volume 45, Number 3, July 2015

the following order. First it asks the other servers in thesame cluster, who share a LAN. If none of these servershave the object, it may ask a “parent” cluster. If all elsefails, the server may issue a request for the object to anorigin server operated by the content provider, who is acustomer of the CDN. As new objects are brought into theCDN’s edge server, it may be necessary to evict older itemsfrom the cache. Section 4 explains how algorithms based onBloom filters are used to decide which objects to cache andwhich not to.

Often, CDNs deliver content that is not cacheable. Exam-ples include dynamic web sites, web applications, and livevideo streaming. In this scenario, the CDN’s edge serveracts as a “front end” for a web site, but must forward re-quests for uncacheable content to the content provider’s ori-gin. Using the CDN servers as relays in an “overlay routingnetwork”, however, can actually reduce download time andincrease availability. The idea is that exploring multipleoverlay paths between the origin and the edge servers andpicking the best ones can provide paths with lower latencyand higher throughput. Further, the multiple paths provideredundancy in case of path failures, enhancing availability.In Section 5, we explain how multi-commodity flow pro-vides an algorithmic framework for constructing good over-lay routing networks.

A CDN is a large distributed system where individualservers and networks can fail at any time. The system has tobe built to be resilient to such failures and should continue tofunction with only minimal degradation in performance. ACDN has dozens of software components that collect data,analyze it, and make key decisions based on it. Two ex-amples described above are global and local load-balancingdecisions, which are computed using real-time data feeds intime scales of minutes or seconds. A common strategy em-ployed by the CDN for fault tolerance is to compute thesolution independently on several machines, and then to ap-ply a leader-election algorithm to determine which machineshould distribute its solution to other CDN components thatdepend on it. Because different machines typically computesolutions at different times and with slightly different inputs,there may be slight differences in the solutions. Section 6 de-scribes several variants of leader election algorithms. Thereare multiple algorithms because different responses to net-work partitions are desired in different contexts.

In each section, an effort is made to describe an abstractversion of the problem to be solved along with one or morealgorithms for solving the problem. Typically, however,there are practical considerations not captured in the ab-stract model. Hence in each section we also present the mostimportant practical considerations and explain how they im-pact the actual implementations. Our focus is primarily onthe algorithms employed by a CDN. For a broader look atthe systems architecture of Akamai’s CDN, the reader is re-ferred to [10] and [24]. For deeper discussion of Akamai’soverlay networks, the reader is referred to [30].

2. STABLE ALLOCATIONSGlobal load balancing is the process of mapping clients to

the server clusters of the CDN. Rather than making load bal-ancing decisions individually for the billions of clients aroundthe world, cluster assignments are made at the granularityof map units. Each map unit can be viewed as the tuple〈IP address prefix, traffic class〉, where the first element of

the tuple is a set of client IP addresses and the second ele-ment describes the type of traffic accessed by those clients.The IP addresses of a specific map unit are intended to be“proximal” to each other in the Internet and can be viewedas a single unit from the perspective of load balancing, i.e.,they are likely to experience similar performance when as-signed to the same CDN cluster. The second tuple of a mapunit is the traffic class that the clients of the map unit access.A traffic class indicates the type of content service accessedby the users. There are tens of traffic classes such as video,web content, applications, software downloads, etc., each ofwhich has distinctive properties. As a simple example, themap unit 〈1.2.3.4/24, video〉 corresponds to the set of clientsin a specific /24 prefix who are accessing videos. Note thatthe same set of users accessing web content form a differ-ent map unit 〈1.2.3.4/24,web〉. If the block of IP addressesis too coarse, then assigning the map unit to a single clus-ter may produce suboptimal performance for some of thoseaddresses. But if the granularity is too fine, there will bemany map units, leading to scalability issues. Dividing theclients of the global Internet into prefixes at the right levelof granularity is a complex problem in its own right, but be-yond the scope of this paper. In all, it is customary to havetens of millions of map units to capture all of the clients andtraffic classes of the trillions of requests per day served byAkamai.

Global Load Balancing. The goal of global load bal-ancing is to assign each map unit Mi, 1 ≤ i ≤ M , to aserver cluster Cj , 1 ≤ j ≤ N , such that the clients repre-sented by each map unit can download the requested con-tent from their assigned cluster with high availability andperformance. For each map unit, the candidate clusters areordered in descending order of preference where a higherpreference indicates better predicted performance for theclients of that map unit. For example, a candidate clusterwith a higher preference might be one with lower latency,lower packet loss, and higher throughput to the clients inthe map unit. Likewise, each server cluster Cj has pref-erences regarding which map units it would like to serve.A number of factors dictate a cluster’s preferences for mapunits, including the contract terms under which the clusterwas deployed, whether the IP addresses are local to the au-tonomous system (AS) in which the cluster is deployed, etc.For example, a cluster deployed in an upstream provider ISPmay prefer to serve clients in the ISP’s downstream customerISPs.

Map units originate demand while the clusters have ca-pacity to serve that demand. With each map unit Mi ademand di is associated, which represents the amount ofcontent accesses that clients belonging to that map unit arelikely to generate. Demands can vary rapidly in real-time asclients access more (or less) content.“Flash crowds,” wherethe demands from certain map units sharply increase are notuncommon. Each cluster Cj also has a notion of capacitycj that represents the maximum demand that the clustercan serve. Capacity is dictated by how much processingpower, memory and disk capacity, and network bandwidththe servers in the cluster possess. The goal of global loadbalancing is to assign map units to clusters such that pref-erences are accounted for and capacity constraints are met.

Stable Allocations. Stable allocations [14] is a classi-cal algorithmic paradigm that forms the basis for assigningmap units to clusters in global load balancing. The simplest


M1: (C2, C1,C3)

M2: (C2, C1,C3)

M3: (C2,C3, C1)

C1: (M2, M1, M3)

C2: (M1, M2,M3)

C3: (M1, M3,M2)

Figure 2: Each map unit has a preference order ofclusters from where to download content, while eachcluster has a preference order of which map units toserve content. A stable marriage (marked in bold) isa matching of map units to clusters such that no un-matched pair prefer each other over their matchedpartners.

example of stable allocations is the stable marriage prob-lem, which was first studied by Gale and Shapley in 1962 inthe provocatively titled paper “College admissions and thestability of marriage” [12]. Since the original work, stablemarriage and related stable allocation problems have beenused to match students to schools in New York city [2], med-ical interns to hospitals [27], and organ donors to patients[28]. Notably, Roth and Shapley won the Nobel Prize inEconomics in 2012 for their work on stable allocation.

In our exposition, we start with the simplest form of stableallocation (Section 2.1), extend these results to more com-plex forms that appear in the CDN context (Section 2.2),and finally describe the real-world complexities of an oper-ational system that go beyond the algorithmic models (Sec-tion 2.3).

2.1 The Stable Marriage ProblemThe stable marriage problem is often described as the task

of matching men and women for marriage. There is an equalnumber of men and women, each seeking a partner of theopposite sex. The stable marriage problem can model asimple case of global load balancing where each map unitis a man with unit demand for content, each cluster is awoman with unit capacity to server content, and the numberof map units, M , equals the number of clusters, N . Anillustrative example is shown in Figure 2. Each map unitorders the clusters by preference, e.g., M1 prefers clusterC2, followed by C1, and then C3. Likewise, each cluster hasa preferential list of map units, e.g., cluster C1 prefers mapunit M2, followed by M1, and then M3. As in real-worldmatch-making, not everyone can have their top choice! Itis easy to see in our example that not every map unit canobtain the top-rated cluster C2. Nor can every cluster getits top choice, since two clusters rate M1 highest.

A stable marriage is a matching of men to women wherethere is no pair of participants that both prefer each other tothe partners that they are matched to. Given an assignment

of map units to clusters, an unmatched pair Mi and Cj issaid to be blocking if

1. Mi prefers Cj over its current partner Cj′ , and

2. Cj prefers Mi over its current partner Mi′ .

Note that the elements of a blocking pair have an incentiveto switch away from their current partners to marry eachother. An assignment that matches all map units with clus-ters such that there are no blocking pairs is said to be stable.As can be seen from the example, the assignment of M1 toC2, M2 to C1, and M3 to C3 is stable.

The notion of stability is a natural way to account for thepreferences of all participants. Finding a stable marriageis achieved by a simple and distributed algorithm called theGale-Shapley algorithm [12]. The algorithm works in roundswhere men propose and women (provisionally) accept, form-ing an “engagement.” Initially, all men and women are“free.” As the rounds progress they become engaged, al-though engagements are sometimes broken. In each round,each free man proposes to the most preferred woman that hehas not already proposed to, whether or not she is alreadyengaged. Among the proposals received in a round, a freewoman accepts the one from the man she most prefers. If awoman is already engaged, she accepts a new proposal, andbreaks her previous engagement, only if she receives a pro-posal from a man who she prefers over her current fiancee. Ifthere are multiple such proposals, she accepts the one fromthe most preferred man. The algorithm terminates wheneveryone is engaged. The fundamental theorem of stablemarriage follows.

Theorem 2.1 ([14]). The Gale-Shapley algorithm ter-minates with a matching that is stable after no more than N2

proposals, where N is number of men (and women). All pos-sible executions of the algorithm with men as the proposersyield the same matching where each man has the most pre-ferred partner that he can have in any stable marriage.

The stable marriage produced by Gale-Shapley is often called“man-optimal” since each man is married to the most pre-ferred woman that he could be married to in any stablemarriage. It may also be termed “woman-pessimal” be-cause each woman is married to the least preferred man thatshe could be married to in any stable marriage. Our loadbalancing algorithm produces a “map-unit-optimal” stablemarriage as map units play the role of the proposers. Thus,the solution provides each map unit with the most preferredcluster possible in any stable marriage, a property that fitswell with the CDN’s mission of maximizing performance forclients.

2.2 Algorithmic ExtensionsGlobal load balancing uses a generalized form of the Gale-

Shapley algorithm described in Section 2.1. A list of of gen-eralizations needed to model our load balancing problemmore closely is provided below.

1) Unequal number of map units and clusters. In our prob-lem, there are vastly more map units than clusters, i.e., tensof millions of map units versus thousands of clusters.

2) Partial preference lists. Given that there are tens ofmillions of map units and thousands of clusters, it is un-necessarily expensive to measure and rank every cluster foreach map unit. It suffices to rank only those clusters that


are expected to provide a minimum level of performance.In fact, it may suffice to measure and rank for each mapunit the top dozen or so clusters that are likely to providethe best performance. Likewise, the clusters need only ex-press preferences for the subset of the map units that arelikely candidates for assignment. For instance, a map unitin Boston may never need to rank a cluster in Tokyo andvice versa, since they are unlikely to be paired up.

3) Modeling integral demands and capacities. While thecanonical stable marriage problem considers unit value de-mands and capacity, it can be extended to the case in whichcapacity and demand are expressed as arbitrary integers.For each map unit, we estimate the content access trafficgenerated by its clients and represent that as a demandvalue. Likewise, for each cluster, we can estimate its ca-pacity, which is the amount of demand it can serve.

The above variants can be solved by generalizing the Gale-Shapley algorithm in straightforward ways. A survey ofthese common variants with further references can be foundin [15]. The extension of stable marriage to a many-to-manyproblem where a man can marry many wives (polygamous)and a woman can marry many husbands (polyandrous) isalso relevant [6].

2.2.1 Resource TreesWith the above extensions, our model for the global load

balancing problem is better, but it still inadequate in an im-portant regard. A server cluster cannot be accurately mod-eled as a single resource whose capacity can be representedas a single number. Instead, a server cluster has a numberof different resources and interrelated resource constraints.These resources that include processing power, memory anddisk capacity, and network bandwidth. Further, the capa-bilities of the servers may restrict what traffic classes theymay serve. For example, the amount of video traffic thata cluster can serve may be separately constrained from theamount of web traffic that it might serve.

50 Bps

30 Fps

Apps Web Video

30 Fps 40 Fps 30 Fps (25 Fps)

(25 Fps)

(30 Fps)

E D C

B

A

Figure 3: An example of a resource tree with capac-ities shown in black. The residual capacities afterassigning a video map unit with 20 units of demandrequiring 20 Bps and 5 Fps is shown in blue.

We illustrate the resource constraints of a cluster using asimple example. Suppose a server cluster has two resources,each with a separate capacity. First is a network resource

that constrains the rate at which data can be sent out of thecluster modeled by a capacity expressed in the units of bytesper second (Bps). The second models all other resources ofthe server using a single number expressed in the units of“flytes” per second (Fps). The flyte capacity captures allnon-network server resources such as the processor, memory,and disk. Besides raw resource constraints, the cluster mayhave limits on how much of each traffic class it can serve.As an example, assume that the cluster can serve three traf-fic classes: video, web, and applications, each with its owncapacity limitations. The different resource constraints canbe expressed as a tree as shown in Figure 3. The leaves ofthe tree bound the maximum number of Fps that can beused for each traffic class. The parent node B bounds themaximum number of Fps that can be served from the clusteracross all traffic classes. The root node A bounds the totalnetwork traffic (in Bps) that can be sent from the cluster.

The resource tree allows us to express a hierarchy of con-straints on the mix of traffic that a cluster is able to serve.A map unit is assigned to a leaf of the tree and it consumesresources on each node in its leaf-to-root path. For instance,a unit of demand from a map unit accessing video exhaustsf Fps and b Bps at each node in the path from the leaf Cto the root A, where f and b are scale factors. Each traf-fic class has its own scale factor. For instance, a unit ofdemand from a video map unit will consume a lot of Bpsbut not a lot of Fps, since video traffic is network intensive.In contrast, a banking application served by the cluster willexhaust a lot of Fps but not many Bps, since such traffic isprocessor intensive but the traffic sent out is small.

To continue our example of Figure 3, suppose 20 units ofdemand from a video map unit are assigned to the cluster,where the scale factor dictates that each unit of video de-mand requires 0.25 Fps and 1 Bps. The residual capacity ofthe cluster can now be calculated by following the leaf-to-root path and decrementing the consumed values, as shownin the figure. More map units can be assigned to the clusteras long as there is sufficient residual capacity left in all nodesin the corresponding leaf-to-root path.

The Gale-Shapley algorithm can be generalized to themore general situation where each cluster has a resource treewith capacity associated with each node of the tree [13]. Toillustrate how the generalized algorithm works using the ex-ample in Figure 3, suppose that a second map unit with 26units of application traffic is assigned to the cluster, whereeach unit of demand requires 1 Fps and 0.2 Bps. Adding thenew map unit to the cluster requires an additional 26 Fpsand 26× 0.25 = 6.5Bps in its leaf-to-root path. This addi-tion will cause a capacity violation at node B of the tree, butno capacity violation at node A or node E. Suppose that thecluster has a higher preference for map units with applica-tion traffic, rather than video traffic. The algorithm looks atthe first node in the leaf-to-root path where there is a capac-ity violation and evicts a lower preference map unit in favorof the higher preference one. In our case, 4 units of videodemand are evicted to create 1 Fps of capacity at node B toaccommodate the additional 26 units of application trafficdemand. By iteratively performing such reassignments thealgorithm converges to a stable allocation of map units.

2.3 Implementation ChallengesWhile algorithms provide a conceptual framework for a

problem solution, numerous architectural and system chal-


lenges also need to be addressed, some of which we list be-low.

1) Complexity and scale. A key challenge is the complex-ity and scale of global load balancing across tens of millionsof map units and thousands of clusters for over a dozen traf-fic classes with more complex resource trees than the simpleexample we provide in Figure 3.

2) Time to solve. The map unit assignment must be re-computed every 10 to 30 seconds, as network performanceand client demand change on short time scales. Networklinks can become congested or fail in seconds, changing thepreference order of map units. Server capacities can varyat a similar granularity as demand rises and falls. Further,global and local events can cause flash crowds where demandfor some map units shoots up, requiring quick reassignmentto other clusters. The generalized Gale-Shapley algorithmhas a strong advantage in that its run time is nearly-linear inthe product of the number of map units and the maximumlength of the map unit preference lists. It is also amenable toa distributed implementation where map unit assignmentscan proceed in parallel.

3) Demand and capacity estimation. The demand of amap unit is an estimate of the downloading activity thatwe are likely to see from its clients. Further, the actualserver resources consumed by a unit of demand, i.e., thescale factors, are often difficult to estimate and only becomemeasurable after the demand is assigned to a cluster, thusrequiring a tight feedback loop to estimate demand and ca-pacity based on past history.

4) Incremental and persistent allocation. While our algo-rithmic model assumes that a stable allocation is performedfrom scratch in each cycle, it is undesirable to do so for tworeasons. First, typically only a small fraction of map unitsneed to be reassigned, e.g., map units that have experiencedsignificant changes in demand or preference since the lastcycle. Second, recomputing the entire stable allocation canresult in a very different solution than the previous one, evenif the demands and preferences have not changed in majorways. A map unit reassigned from (say) cluster A to clus-ter B for which it has similar preference causes unnecessaryhavoc, because cluster A has already cached the content re-quested by the map unit clients, while cluster B may nothave cached different content, causing cache misses whenclients are switched to cluster B. This phenomenon calls fora “sticky” solution that changes assignments only when thepreference differences are sufficiently large. A particularlyharmful form of impersistence is oscillation, in which theglobal load balancer moves cyclically between different so-lutions as a result of small changes in input values.

The above challenges require additional innovations, push-ing us into realms where the clean, provable algorithmicmodels no longer hold. Nevertheless, global load balancingis a good example of a problem in which algorithmic and ar-chitectural innovations reinforce each other and neither canexist without the other.

3. CONSISTENT HASHINGConsistent hashing [17, 22] is used by the CDN to balance

the load within a single cluster of servers. Consistent hash-ing was the first algorithmic innovation in Akamai’s CDNand the patents on consistent hashing (e.g., [18]) formedan important component of the start-up company’s initialportfolio of licensed intellectual property.

In a traditional hash table, objects from a universe U aremapped to a set of buckets B. For example, if U is the setof positive integers, and B is the set of integers [0, n − 1],then the mapping U → B might be achieved using a simplehash function h such as h(x) = ((ax + b) mod P ) mod n,where P is a prime number such that P ≥ n, a is chosenat random from [1, P − 1], and b is chosen at random from[0, P −1]. This hash function maps elements of U uniformly(over the random choices of a and b) to the buckets B andalso provides pairwise independence, i.e., for any x1, x2 ∈ U ,

Pr[h(x1) = y1&h(x2) = y2] = Pr[h(x1) = y1] Pr[h(x2) = y2].

When a hash table is used in a sequential computer pro-gram, an arbitrary subset S ⊂ U of O(n) objects of interestare inserted into the buckets of B, i.e., each element x ∈ Sis inserted into bucket h(x). Using an array of linked lists torepresent the buckets, operations such as inserting an objectinto a bucket, removing an object from a bucket, or testingwhether an object is already in a bucket can be implementedin a straightforward manner so that the expected time peroperation is constant. Alternative data structures that sup-port these operations, such as balanced search trees, typi-cally require Θ(log n) time per operation (but may supportadditional operations).

In the CDN setting, an object is a file such as a JPEGimage or HTML page, and a bucket is the cache of a distinctweb server. A new complication arises in this context: aserver may fail, in which case it can no longer fulfill its roleas a bucket. Unfortunately, the class of hash functions likeh(x) above, which map elements of U directly to bucketsB, provides no efficient mechanism for dealing with the lossof a bucket. Simply remapping all of the objects in thelost bucket to another bucket is not ideal because then onebucket stores double the expected load. On the other hand,balancing the load by renumbering the existing buckets andrehashing the elements using a new hash function has thedisadvantage that many objects will have to be transferredbetween buckets.

SERVER

OBJECT

Figure 4: Consistent hashing first maps both objectsand buckets (servers) to the unit circle. An objectis then mapped to the next server that appears onthe circle in clockwise order.

Consistent hashing solves this problem in a clever way. In-


stead of mapping objects directly to buckets, both objectsand buckets are first mapped to the unit circle. As shownin Figure 4, each object is then mapped to the next bucketthat appears in clockwise order on the unit circle. This ap-proach provides a more graceful mechanism for dealing withthe loss of a bucket: if a bucket fails then the objects thatwere formerly mapped to the bucket are instead mapped tothe next bucket that appears in clockwise order on the unitcircle. Similarly, if a new bucket is added, e.g., the failedserver comes back on-line, the only objects that have to bemoved are those that are mapped to the new bucket.

A simple enhancement to consistent hashing that improvesthe balance of objects among the buckets is to map eachbucket to multiple locations on the unit circle. For example,each of the n buckets might be mapped to logn instances.The expected load for each instance is 1/ logn the load fora bucket, but the expected load for a bucket is the sameas if there was only one instance. The distribution of load,however, is much more tightly centered around the expecta-tion. As before, an object is mapped to the next instance ofa bucket that appears in clockwise order on the unit circle.If a server fails, all of the corresponding bucket’s instancesare removed from the unit circle. The objects that were inthe bucket are now remapped across a set of buckets ratherthan to a single bucket.

3.1 Popular ObjectsAn important practical consideration not addressed in the

original consistent hashing work, but described in [16], isthat some web objects may be far more popular than oth-ers. In some circumstances, such as a flash-crowd event, asingle object may be so popular that it is not possible fora single server within a cluster to satisfy all of the requestsfor the object. Instead, the object must be served by morethan one server. A straightforward extension to consistenthashing would be to map a popular object to the next kservers that appear in clockwise order on the unit circle,where k is a function of the popularity of the object. Onedisadvantage of this approach, however, is that if two verypopular objects happen to hash to nearby positions on theunit circle, the buckets that they map to will overlap, andmay have difficulty handling the requests for both objectssimultaneously.

The approach taken by the CDN is to use a separate map-ping of the buckets to the unit circle for each object. Thismodification to consistent hashing preserves the propertythat when a bucket is removed, the only objects that mustbe moved to different buckets are the ones that were for-merly in the removed bucket, and when a bucket is added,the only objects that must be moved are the ones that arenow mapped to the new bucket.

Suppose that there are two popular objects, each of whichis to be mapped to k buckets. Regardless of whether thebuckets are mapped to the unit circle using the same hashfunction for both objects or different hash functions, the ex-pected number of buckets containing both objects is Θ(k2/n).But the distributions are very different. For example, if thesame hash function is used, the probability that at leastk/2 buckets receive both popular objects is Θ(k/n). Onthe other hand, if different hash functions are used, theneach of the k instances of the second object is independentlymapped to a bucket containing the first object with proba-bility at most k/n. Thus, the probability that at least k/2

of these instances map to buckets containing the first ob-ject is at most

(k

k/2

)(k/n)k/2, which is at most (2ek/n)k/2

(sometimes a loose upper bound), which is typically lessthan Θ(k/n), as long as k < n/2e.

3.2 Consistent Hashing in PracticeAlthough an object is a single file to be served by a web

server, the CDN does not hash each object independently.Instead, when a content provider signs up as a customerof the CDN, they are granted one or more serial numbers.The content provider’s objects are grouped together by se-rial number, and all objects with the same serial number arehashed to the same bucket (or same set of buckets, if popu-lar). The advantage of grouping a content provider’s objectstogether arises when multiple objects must be fetched by thebrowser in order to render a single web page. If these ob-jects were all independently hashed to different servers, thebrowser would have to open a separate HTTP connectionto each server. By mapping the objects to the same set ofservers, the browser can make a persistent HTTP connec-tion to just one randomly-chosen server in the set, and thenfetch all of the objects from that server. The overhead forestablishing and ramping up a new TCP connection is thuspaid only once, rather than once per object.

We are now in a position to provide a concrete exam-ple of how the CDN employs consistent hashing in prac-tice. Suppose that the serial numbers assigned to contentproviders range from 1 to 2048 and that the CDN has as-signed the serial number 212 to a customer that operatesthe web site www.example.com. In order to use the CDNto deliver the images that appear on its web pages, the cus-tomer creates a new domain name images.example.com, andthen uses the DNS CNAME mechanism to indicate that im-ages.example.com is an alias for the canonical domain namea212.g.akamai.net. (The “g” in this domain name stands for“general,” a domain name used by the CDN for deliveringstatic objects.) In this example, the customer embeds theURL http://images.example.com/logo.jpg in the HTML forthe web page http://www.example.com so that the browserwill fetch the web site’s logo from the CDN.

When an authoritative name server operated by the CDNreceives a request to resolve the name a212.g.akamai.netfrom a resolving name server, it first determines which clus-ter should serve the ensuing HTTP requests from browserssharing this resolving name server. The cluster is chosenusing the algorithm described in Section 2. Once the clusterhas been selected, consistent hashing is used to determinewhich server or servers within the cluster should serve ob-jects under serial number 212. In this example the fourservers in the chosen cluster have been assigned the IP ad-dresses 192.168.1.1 through 192.168.1.4. (Although theseaddresses are private, in practice the CDN’s servers almostalways use public addresses.) For each serial number thereis a random map of the buckets to the unit circle. Themap can be represented as an ordered list of servers, e.g.,for serial number 212 the map might be 212: 192.168.1.3192.168.1.1 192.168.1.4 192.168.1.2. (Since each serial num-ber has its own random mapping of buckets to the unit cir-cle, we can assume that the serial number itself is mappedto position 0.) Depending on the current popularity of thecontent being served under serial number 212, the authori-tative name server for the CDN returns the first k addresseson this list, skipping any servers that are not operational.


The k servers in the DNS response are listed in random orderto help spread the load among them.

4. BLOOM FILTERSBloom filters are used to approximately represent dynam-

ically evolving sets in a space efficient manner. Bloom filterswere first introduced by Burton Bloom [7] and have foundnumerous elegant applications in networking [8]. Bloom fil-ters are useful in content delivery in two different contexts:content summarization and content filtering. We illustrateboth uses and focus on a simple example of content filter-ing that offers significant benefits in the form of better diskresource usage and performance.

4.1 The BasicsSuppose that we want to store a set S = {e1, e2, · · · , en}

in a manner that allows us to efficiently insert new elementsinto the set and answer membership queries of the form “Iselement e in set S?”. A simple data structure is a hash func-tion h that maps elements to a hash table T [1 · · ·m], whereeach table entry stores a binary value that is initialized to0. When an element e is inserted into set S, we simply setthe entry with index h(e) to 1, i.e., we set T [h(e)] to 1. Tocheck if e ∈ S, we simply need to check if T [h(e)] is 1. Notethat “false positives” are possible with this solution whenan element e′ 6∈ S “collides” with an element e ∈ S, i.e.,when h(e) = h(e′). In this case, the membership query fore′ will return the incorrect answer, i.e., we will conclude thate′ ∈ S when it is not. Note that a“false negative” cannotoccur since when we conclude that an element is not in Swe are always correct.

Bloom filters are a generalization of the simple hash tablesolution described above. Rather than use one hash func-tion, a Bloom filter uses multiple hash functions h1, h2, · · · , hk

to reduce the probability of a false positive. As before, wehave a table T [1 · · ·m] where each T [i] can store a binaryvalue and all entries are initialized to 0. When an elemente is inserted into set S the bits T [hi(e)], 1 ≤ i ≤ k, are setto 1. To check if e ∈ S, we need to check if T [hi(e)] = 1,for all 1 ≤ i ≤ k. False positives are still possible withthis solution, but the likelihood is smaller since an elemente′ 6∈ S will be mistakenly believed to belong to S only ifother elements have set the bits T [hi(e

′)] to 1 for all k hashfunctions hi.

It is instructive to quantify the tradeoff between the falsepositive probability p, the number of elements n, the numberof bits m, and the number of hash functions k. For simplic-ity, assume that for any element e and hash function hi,hi(e) is distributed independently and uniformly over therange of values 1 to m (although in practice weaker hashfunctions might be used). For any 1 ≤ l ≤ m, we evaluatethe probability that T [l] = 0, after inserting all n elementsof S into the table. For each e ∈ S and 1 ≤ i ≤ k, theprobability that hi(e) 6= l is (1 − 1/m). The probabilitythat T [l] = 0 after inserting all n elements with k probeseach is simply (1 − 1/m)kn. Thus, the probability that anentry T [l] = 1 after n insertions is simply 1− (1− 1/m)kn.A false positive occurs when e′ 6∈ S but T [hi(e

′)] = 1, for all1 ≤ i ≤ k. Although not strictly true, we assume that theprobabilities that the different entries T [l] are set to 1 areindependent. Thus, the false positive probability p can be

approximated as follows.

p =(

1− (1− 1/m)kn)k

≈ (1− e−kn/m)k. (1)

(The reader is referred to a standard reference such as [23]for a more rigorous derivation of the relationship between p,m, n, and k.) For a given n and m, one can find the k thatminimizes the probability p by differentiating the right-handside of Equation 1 with respect to k and setting it to zero.Solving for k provides the following.

k ≈ (ln 2) · (m/n). (2)

That is, k should be chosen to be roughly proportional tothe ratio of the size of the hash table and the number ofelements that you expect to be placed in the Bloom filter.Plugging in the value of k from Equation 2 into Equation 1,we get the following relation.

p ≈ 2−k ≈ 2−m ln 2

n . (3)

As we will see, the above equation is key for sizing the Bloomfilter for specific applications.

4.2 Content Summarization and FilteringThere are two key applications of Bloom filters to CDNs.

The first application is cache summarization where a Bloomfilter is used to succinctly store the set of objects storedin a CDN server’s cache. Note that using a Bloom filterfor this purpose is much more space-efficient than storing alist of URLS associated with the objects in cache. Cachesummarization can be used to locate which server cache haswhich objects. For instance, if the CDN’s servers periodi-cally exchange cache summaries, a server that does not havea requested object in its cache can find it on other servers us-ing their cache summaries. Cache summarization has beenwell-studied in the academic literature [11]. It is also im-plemented in popular web caching proxies like Squid, whichcreate cache summaries called “digests” [29]. For reasons ofefficiency, Squid uses a Bloom filter with k = 4 where a sin-gle 128-bit MD5 hash of the object’s identifier is partitionedinto four 32-bit chunks and treated as four separate hashes.

Note that cache summarization requires the Bloom filterto support deletion of elements in addition to insertions. Inparticular, when an object is evicted from cache, it has toremoved from the Bloom filter as well. This requires a vari-ant called counting Bloom filters where the table values areno longer binary but are counters instead [11]. To insertan element e into the set the entries T [hi(e)], 1 ≤ i ≤ kare incremented. To delete an element, the k locations aredecremented. Testing for membership of a particular ele-ment checks if the k locations are non-zero.

The second application for Bloom filters is what we callcache filtering, in which a Bloom filter is used to determinewhat objects to cache in the first place. To motivate theneed for cache filtering, we collected access logs from a sin-gle cluster of Akamai’s CDN consisting of 45 servers thatcache and serve Web traffic. Using the logs, we studied therelative popularity of the over 400 million distinct web ob-jects that clients accessed from the servers over a period oftwo days. As shown in Figure 5, nearly three-quarters ofthe objects were accessed only once and 90% of the objectswere accessed fewer than four times during the two days.

In normal operation without cache filtering, a CDN’s servercaches each object that it serves by storing a copy of the ob-


0

20

40

60

80

100

1 4 16 64 256 1024

Percen

t of o

bjects

Number of accesses

Figure 5: On a typical CDN server cluster servingweb traffic over two days, 74% of the roughly 400million objects in cache were accessed only once and90% were accessed less than four times.

ject in its disk cache. When the disk cache fills up, theobjects are evicted using a cache replacement policy. Butthere is no reason to cache objects that are likely to be ac-cessed only once and never accessed again. Such objects arejocularly referred to as “one-hit-wonders”. Our popular-ity analysis suggests that over three-quarters of the objectsare one-hit wonders and a significant amount of disk spacewould be saved if they were not cached. In addition, thenumber writes to disk would also be reduced significantly ifthese objects were not cached.

Cache-on-second-hit rule. A simple cache filtering rulethat avoids caching one-hit-wonders is to cache an objectonly when it is accessed for a second time within a specifictime period. The rule can be implemented by storing the setof objects that have been accessed in a Bloom filter. Whenan object is requested by an client, the server first checks tosee if the object has been accessed before by examining theBloom filter. If not, the object is fetched and served to theclient, but it is not cached. If the object has been accessedbefore, it is fetched, served, and also stored in the server’sdisk cache.

The astute reader may have observed that as more ob-jects are added to a Bloom filter and it starts to fill up, theprobability of false positives increases. A simple approach tocircumventing this issue is to have two Bloom filters, a pri-mary and a secondary. All new objects are inserted into theprimary filter, until it is reaches a threshold for maximumnumber of objects. At that point, the primary becomes thesecondary filter and a new filter with all entries initializedto zero becomes the primary. As the old secondary filter isdiscarded, objects that have not been accessed recently areforgotten. When checking if an object has been accessed inthe recent past, both primary and secondary Bloom filtersneed to be queried.

4.3 Empirical BenefitsTo illustrate the benefits of the cache-on-second-hit rule

using Bloom filters, we describe a simple experiment con-ducted by Ming Dong Feng using a cluster of about 47 pro-duction servers serving live traffic on the field. Each serverhas eight hard disks. The workload served by these servershave a “cold footprint” consisting of web and videos from

social networking sites. Bloom filters that implement thecache-on-second-hit rule were turned on on March 14th andturned off on April 24th to create before, during, and afterscenarios for analysis. Measurements were made every fewtens of seconds and averaged across all machines every sixhours. Thus, our figures show multiple measurements perday, enabling us to see both intra- and inter-day variations.

Figure 6 shows the average byte hit rate of the servers,where byte hit rate is simply the percent of bytes served toclients that were found in the server cache. For instance, abyte hit rate of 75% would mean that for every 100 bytesserved to the client, 75 bytes were found in cache and 25bytes had to be fetched from a peer, parent, or origin server.Note that during the period when Bloom filtering was turned

70.00 72.00 74.00 76.00 78.00 80.00 82.00 84.00 86.00 88.00 90.00

17-‐Feb 27-‐Feb 9-‐Mar 19-‐Mar 29-‐Mar 8-‐Apr 18-‐Apr 28-‐Apr 8-‐May 18-‐May 28-‐May

Byte Hit Ra

te (%

)

Date

Figure 6: Byte hit rates increased when cache filter-ing was turned on between March 14th and April24th because not caching objects that are accessedonly once leaves more disk space to store more pop-ular objects.

on, the byte hit rate increases from around 74% to 83%.Most CDNs implement cache replacement algorithms suchas LRU, which evict less popular objects such as one-hit-wonders when the cache is full. Filtering out the less pop-ular objects and not placing them in cache at all providesadditional disk space for more popular objects and increasesthe byte hit rate.

In Figure 7, we show the impact of Bloom filters on thenumber of disk writes performed by the servers. The diskrelated metrics we report are measured by a utility similarto Linux’s iostat [1] which runs continually on productionservers and reports metrics relevant to disk performance.Not having to store the one-hit-wonders in cache reducesthe aggregate rate of disk writes by nearly one-half. To bemore precise, the rate of disk writes drops from an average of10209 writes per second to 5738 writes per second, a decreaseof 44%. One consequence of fewer disk writes is that the

0

2000

4000

6000

8000

10000

12000

14000


Disk writes per se

cond

Date

Figure 7: Turning on cache filtering decreases therate of disk writes by nearly one half because objectsaccessed only once are not written to disk.

latency of accessing an object from disk decreases. As shownin Figure 8, the average disk latency drops from an average


of 15.6 milliseconds to 11.9 milliseconds, a decrease of about24%.

0.00

5.00

10.00

15.00

20.00

25.00


Disk Laten

cy (m

sec)

Date

Figure 8: The latency of reading content from diskdecreases when cache filtering is turned on, sincethere are fewer competing disk writes.

4.4 Implementation ChallengesThus far we have described cache filtering in its simplest

form as an illustrative example. Now we consider some real-world issues in implementing cache filtering in a productionCDN.

1. Speeding up the Bloom filter. The Bloom filter imple-mentation needs to be extremely fast because Bloom filteroperations are on the critical path for client-perceived CDNperformance, i.e., the time taken to perform these opera-tions adds to the download time perceived by the client.Rather than evaluating k different hashes, the approach ofusing a single suitably-sized MD5 hash and partitioning thebits into k hashes is often used in practice.

2. Sizing the Bloom filter. The Bloom filter implementa-tion should be space efficient. In fact, space efficiency is themain reason why Bloom filters are used in place of simplerhash table based solutions. Note that server memory is avaluable resource and is used to hold a “hot-object” cacheof the most popular objects. The Bloom filter must be sizedcorrectly since a filter that is too large wastes memory and afilter that is too small yield a large false positive probability.

In practice, the sizing of the Bloom filter is done as fol-lows. First, we decide how many objects n we are likelyto add to the Bloom filter while it is in use. Next, we de-cide what false positive probability p we can tolerate, i.e.,what is our tolerance to mis-classifying objects due to falsepositives. Knowing n and p, the size of the Bloom filter mcan be evaluated using Equation 3. Further, knowing n andm, the number of hash functions k can be evaluated usingEquation 2. To take a typical example, suppose that a sin-gle server is likely to see n = 40 million objects and we arewilling to tolerate a false positive probability of 0.1%, thensolving Equation 3 we get a Bloom filter table with 575.1million bits, or about 68.6 MBytes in size. This is a reason-ably small size and would fit easily in server memory. Then,using Equation 2, we would use k = 10 hash functions.

3. Complex cache filtering. Thus far, we have consideredonly the simplest form of cache filtering, i.e., the cache-on-second-hit rule. In addition to this rule, more complex rulesmay need to be implemented in practice that take into con-sideration other object popularity metrics and other charac-teristics such as size. Further, cache filtering applies not justto the disk cache, but also to the hot object cache in memory.For instance, an object needs to have a much higher level ofpopularity and also be small enough for it to be eligible tobe placed in memory. In general, there is content filtering

occurring at every level of the hierarchy of caches within aCDN server, including memory, solid-state disks, and harddisks. Implementing a hierarchy of complex cache filteringrules with variants of Bloom filters is a key challenge.

5. OVERLAY ROUTING

ORIGINEDGE

SERVERSOVERLAY CLIENTS

Figure 9: An overlay routing network enables fastand reliable communication between origins andedge servers. Different content (i.e., commodities)represented by different colors are routed simulta-neously from their respective origins to edge serverswho requested them. On receiving the content, theedge servers serve it to their proximal clients.

Most web sites have some content that is either not cacheableor cacheable for only short periods of time. Take, for exam-ple, a banking portal. A significant part of the portal is per-sonalized for individual users and cannot be cached, thoughsome page elements such as CSS, JavaScript, and imagesmay be common across users and can be cached. Besides dy-namic web content, CDNs also host applications that userscan interact with in real-time where much of the interac-tions cannot be cached. Further, in the world of media, livestreaming and teleconferencing are delivered by CDNs inreal-time and are uncacheable. Finally, even when contentcan be cached, there are often time-to-live (TTL) parame-ters that require the CDN to periodically refresh the con-tent. An example is a stock chart that changes continuallyand must be updated every few minutes.

A common framework that can capture all of the abovesituations is shown in Figure 9. There are (i) origins thatcreate the content, (ii) edge servers that clients access toconsume the content, and (iii) an overlay network that isresponsible for transporting the content from the origins tothe edges. Clients request the content from a proximal edgeserver, and the edge server in turn downloads the requestedcontent from the origin via the overlay network.

In the case of a web site, the origin is a collection of appli-cation servers, databases, and web servers deployed by thecontent provider in one or more data centers on the Internet.In the case of a live stream, the origin denotes the serversthat receive the stream in real-time from the encoders cap-turing the event. While the origin is usually operated by thecontent provider, CDNs often deploy a middlebox called a


reverse proxy near the origin to provide more efficient end-to-end transport between the origin and the edges. Theoverlay network is a large collection1 of servers operatedby CDN that can be used as intermediate nodes for rout-ing content from the origin to the edge servers. The edgeservers are operated by the CDN and are deployed in morethan a thousand data centers around the world, so as to beproximal to clients.

5.1 The Overlay Construction ProblemThe key problem in overlay routing is how to construct

an overlay to provide efficient communication between ori-gins and edge servers. An overlay construction algorithmtakes as input (i) client demands, which dictate which ori-gins need to send their content to which edge servers, and (ii)real-time network measurements of latency, loss, and avail-able bandwidth for the paths through the Internet betweenorigins, overlay servers, and edge servers. For each originthat needs to communicate with a specific edge server, theoverlay construction algorithm returns a set of one or moreoverlay paths that have high availability and performance.An overlay path starts at an origin, passes through one2

or more intermediate servers in the overlay network, andends at an edge server as shown in Figure 9. Content issent from the origin to the edge server along one of more ofthe constructed overlay paths. Note that overlay construc-tion must be performed frequently as both client demandand network characteristics change rapidly. Therefore, al-gorithms for overlay construction must be very efficient.

5.2 Multicommodity FlowA natural algorithmic framework for performing overlay

routing is to formulate it as a multicommodity flow prob-lem. As shown in Figure 9, the origins, the overlay servers,and the edge servers form the nodes of a flow network. Thelinks of the flow network are all possible pairs of nodes thatcan communicate with each other, with each link having acost and a capacity. Each origin is the source of a distinctcommodity that needs to be routed to a set of edge servers.Note that a commodity represents content hosted at an ori-gin, i.e., a commodity could be the contents of a web siteor a stream of packets from a live video. The goal of over-lay routing is to create routing paths for each commodityfrom its origin to the edge servers, while obeying capacityconstraints and minimizing cost. Note that performance re-quirements are often codified as cost such that minimizingcost maximizes performance.

We now list key aspects of overlay construction that mustbe modeled in our multicommodity flow formulation. Manyof these aspects make our problem different from the classi-cal multicommodity flow studied in the literature, requiringnew algorithmic innovations.

1. Multipath transport. Overlays use multiple paths be-tween each origin and edge to enhance reliability and perfor-mance. There are two different modes in which these pathscan be used, depending on the application context. In one

1While overlay servers and edge servers are conceptuallydistinct as shown here, a typical physical server may simul-taneously play both roles in a CDN.2A path from origin to the edge server that does not passthrough any intermediate overlay servers is called the directpath. Overlays always use the direct path when no overlaypath is superior to it.

mode that is common for live video, the paths are used si-multaneously with content sent across all the paths. Contentpackets are encoded across these paths so that packets loston one path can be recovered at the edge server using pack-ets from other paths. A second mode that is more commonin web delivery is to consider the multiple paths as candi-dates and have the edge server choose the best performingpath among the candidates in real-time. To select the bestpath, the edge server conducts periodic “races” between allthe candidate paths and chooses the one that provides thefastest download time.

2. Role of overlay servers. Overlay servers may receivecontent and send it in a replicated fashion to multiple re-ceivers. They may also perform coding and decoding oper-ations if network coding is used in the overlay. As a resultthere is no conservation of flow that classical multicommod-ity flow problems possess.

3. Cost function. Each link has an associated cost andcapacity. The cost function primarily encodes link perfor-mance, though one could also incorporate the bandwidthcosts incurred in using that link. Link performance is char-acterized differently for different application contexts. Fordynamic web content, the primary consideration is link la-tency, which impacts the download time perceived by theuser. In this case, lower latency links have lower cost. In thecase of video streaming, link throughput is a more importantmetric. A link with higher available bandwidth and lowerpacket loss that is capable of supporting higher throughputsis given a lower cost.

4. Capacity. Both node and link capacities are used toreflect the amount of traffic demand that can be routedthrough the node and link respectively. Capacity incorpo-rates both server and network resources as well as contrac-tual requirements that bound the amount of traffic that canbe sent.

5. Optimized transport protocols. CDNs often use opti-mized transport protocols between nodes that they own andoperate. Thus, the communication between overlay serversand edge servers are optimized. Often a reverse proxy ispresent at the origin or proximal to it, allowing for origintraffic to be optimized as well. The link performance modelthat is incorporated as a cost reflects the functioning of theseprotocols.

5.3 Algorithmic SolutionsThe considerations listed above lead to different variants

of the overlay construction problem for different applicationcontexts. As a result, multiple algorithms are used in prac-tice to solve these variants. We list two examples below.

1. Dynamic web content. Overlays that are more latencysensitive such as those for routing dynamic web content usecustomized algorithms for the well-studied all-pairs shortest-path (APSP) problem. Note that overlay construction in-volves both cost and capacity constraints. A first step toconstructing an APSP instance is to perform Lagrangianrelaxation (i.e., penalizing violations of constraints ratherthan forbidding them) [3] to encode the capacity constraintsas a cost that can be added to existing link-performance-dependent cost terms. Customized algorithms for APSPcan then be applied to compute the least cost paths be-tween each origin-edge pair, i.e., the algorithm yields pathsthat have the best performance without capacity violations.

2. Live videos. For throughput sensitive traffic such as


live videos, a different approach may be employed. In par-ticular, given the large amounts of data transported acrossthe overlay, particular attention must be paid to the vari-ous capacity constraints in the overlay and the bandwidthbottlenecks in the Internet. One approach is to formulatea mixed integer program (MIP) that captures all the per-formance, capacity, and bandwidth constraints. The MIPoften cannot be solved efficiently as it is NP-hard to so. Toovercome this problem, we can “relax” the integral variablesin the MIP so that they can take on real values, resulting ina linear program (LP). The LP can then be solved efficientlyto yield a fractional solution that can then be “rounded” toget an integral solution. The solution produced in this fash-ion is of course not optimal. But, in some cases, the solutioncan be shown to be near-optimal.

For a more in-depth treatment of an algorithmic solutionfor constructing overlays for live streaming, the reader isreferred to [5, 4], while the system architecture of a livestreaming network is described in [19]. The algorithm pro-duces an overlay using LP relaxation that is rounded to anintegral solution using GAP rounding. The overlay pro-duced is guaranteed to be within a logarithmic factor ofoptimal.

5.4 Empirical BenefitsThe empirical benefits of an overlay network as described

above are two-fold. First, on a consistent basis, overlayrouting provides better user-perceived performance, such asfaster download times or fewer video stream rebuffers. Theimprovement in performance is due to many factors that fa-vor overlay paths from origins to edge servers to the directpaths provided by the Internet. The overlay paths may havelower latency and/or higher throughput depending on theapplication context. The CDN can save the overhead of es-tablishing TCP connections between different nodes withinthe overlay by holding them persistently. The CDN may alsouse optimized transport protocols between their nodes. Ourempirical results, however, do not separate out the benefitsof each effect.

A second benefit is what one could term as a catastro-phe insurance. Catastrophic events such as a cable cut arenot rare on the Internet. During such an event, overlaysprovide alternate paths when the direct path from origin toedge is impacted, leading to outsized improvements in bothavailability and performance.

There has been much work in quantifying the benefits ofoverlays. A large-scale empirical study of Akamai’s routingoverlay is presented in [26]. The empirical observations re-ported here are based on [30], which describes many differentoverlay technologies at Akamai and their relative benefits.

5.4.1 Consistent benefitsThe routing overlay provides significant and consistent

performance benefits by discovering and using better per-forming overlay paths for communication, even when thereare no significant catastrophic network events. Figure 10,which was reported in [30], shows the performance benefitof an overlay for dynamic web content. During the operationof the overlay measured in this experiment, each edge serverwanting to download content from an origin chooses the bestoverlay path from a set of candidate overlay paths providedby the overlay construction algorithm. In this experiment,a web site hosted in an origin in Dallas was accessed by

Figure 10: A routing overlay provides significantspeedups by choosing better performing paths fromthe origin to the client. Key: North America (NA),Europe (EU), Asia.

“agents” deployed around the world that simulate clients bydownloading web content periodically using an embeddedbrowser3. For the experiment, the agents repeatedly down-loaded a single dynamic (i.e., uncacheable) 38KB web page.The agents downloaded the web content through Akamai’sCDN, which uses the routing overlay, and also downloadedthe content directly from origin. Speedup is the ratio of thedownload time of the file when downloaded directly fromthe origin to the download time of the same file when down-loaded using the routing overlay. Thus, speedup measuresthe amount of benefit that the routing overlay is providing.The results were then aggregated by continent and the av-erage speedup is shown in the figure. Note that the speedupis significant in all geographies. The speedup, however, in-creases as the client (i.e., agent) moves farther away fromthe origin. This is a natural consequence of the fact thatwhen the content needs to travel a longer distance from ori-gin to edge there are more opportunities for optimizing thatcommunication using better overlay paths.

5.4.2 Catastrophe InsuranceOverlay routing provides substantially more benefit dur-

ing catastrophic events. In [30], performance measurementscollected during the occurrence of one such event are re-ported. In April 2010, a large-scale Internet outage occurredwhen a submarine communications cable system (called SEA-ME-WE 4), which links Europe with the Middle East andSouth Asia, was cut. The cable underwent repairs from 25April to 29 April during which time several cable systemswere affected, severely impacting Internet connectivity inmany regions across the Middle East, Africa, and Asia.

Figure 11 shows the download time experienced by clientsin Asia using a routing overlay during the outage period incomparison with the download time experienced by the sameclients without the overlay. Specifically, times were mea-sured for agents distributed across India, Malaysia, and Sin-gapore to download a dynamic (i.e., uncacheable) web page

3Agents are often used for repeated “apples-to-apples” per-formance measurements of the same content over long timeperiods, something real users would find too boring to do!


Figure 11: Performance of the routing overlay dur-ing a cable cut.

approximately 70KB in size hosted at an origin in Boston.The agents downloaded two versions of the web page - onedirectly from the origin in Boston, and another through therouting overlay. It can be seen that during the outage, theperformance of the download directly from the Boston originto Asia was very slow. Downloads using the routing overlay,however, did not experience much degradation on accountof alternate overlay paths being used.

6. LEADER ELECTION AND CONSENSUSLeader election algorithms are employed ubiquitously by

the CDN in dozens of different distributed system compo-nents. Because these algorithms are tricky to design andimplement correctly, the CDN has developed a leader elec-tion software library. The algorithms in this library canproduce different leadership outcomes. For instance, somealgorithms guarantee that at least one server will be electedas a leader, while others guarantee at most one. Leader elec-tion is an instance of the more general problem of consensus.There are numerous scenarios where general consensus pro-tocols are key for the correct functioning of a CDN. For thispurpose, more complex algorithms such Paxos [20] are usedby multiple CDN components.

6.1 Leader ElectionWe start with the problem of leader election. Every decision-

making software component of the CDN is executed in areplicated fashion on many servers that often reside in mul-tiple data centers. Examples of components replicated inthis fashion include the global and local load balancers de-scribed in the earlier sections. The key reason for replicatingthe decision-making components is to ensure that the CDNwill continue to function even if some servers or even en-tire data centers fail. In fact, such failures are common inInternet-scale systems.

A server executing a decision-making component takesreal-time inputs from several sources and produces outputsthat are consumed by other components running on otherservers. When such a component is replicated across multi-ple servers, each server independently collects the requireddata and computes the outputs. Even if the software run-ning on these servers is identical, the outputs produced by

each server could be different, since each server could beusing slightly different or even partial sets of inputs. There-fore, it is necessary to select a server called the leader whoseoutput is used by other system components that require it.The leader is determined in a distributed fashion using aleader election algorithm. We now describe some of the keyassumptions and concepts that underlie leader election.

1) Failure model. The server failure model assumed by thealgorithm designers is that a server may cease to functionat any instant in time, and may likewise resume at any in-stant, but will not operate in a malicious or Byzantine [21]manner. (The consequences of server compromise go farbeyond any impact on leader election algorithms!) The net-work may also fail at arbitrary times, but, by assumption,only by preventing pairs of servers from communicating orby introducing arbitrarily large packet delays. In particu-lar, network failures may partition the servers into disjointsubsets, where the servers in each subset can communicatewith each other, but servers in different subsets cannot.

2) Candidate set. The servers participating in a leaderelection form the candidate set. The servers are always con-figured to know the candidate set in advance, although thereis never a guarantee that every candidate server can com-municate with every other candidate server. Ideally, serversparticipating in a leader election algorithm should be con-figured with the same candidate set or problems may arise.For example, ties in voting for a leader are often broken bysimply choosing the candidate with the smallest numberedIP address. If different participants have different candi-date sets, there may not be an agreed-upon smallest address.However, periods where different servers have different can-didate sets is unavoidable. When a new configuration is re-leased on the network, it is inevitable that some servers havethe latest candidate set while others have the older version.The leader election algorithm must be designed such that itdoes not produce adverse outcomes during such periods.

3) Health. It is important that the elected leader is “healthy”in terms of being able to perform its duties. A candidatemay be disqualified from participating in the election dueto “poor health.” Health is a numerical score that is com-puted differently for different components. Generally, a can-didate server is considered healthy if it has recent versionsof all the data feeds needed for its computation, has theresources to execute the computation, and is able to com-municate with other servers that require its outputs. Forinstance, a server performing global load balancing cannotbe considered healthy if it does not have sufficiently recentvalues for the map unit demands and cluster capacities thatare required to execute the stable marriage algorithm. Aleader is often chosen to be healthiest candidate, breakingties using their IP addresses. The aim is to prevent a serverfrom becoming the leader if it has an invalid, outdated, orincomplete view of the data needed to make decisions.

4) Electoral process. Elections are triggered when thereis no current leader. All candidates broadcast health val-ues with a fixed (but configurable) periodicity to every oneelse. Note that each candidate not only broadcasts their ownhealth but also the latest health values for others it has heardfrom. Each candidate chooses a leader using pre-determinedrules based on its own health value and the health values itreceives from others. The periodic broadcasts of health val-ues serves a similar role as the heartbeats in the leader elec-tion protocol of Raft [25]. Specifically, if the heart beat from


the current leader is not heard for a specified timeout period,it is presumed “dead” and will trigger the election of a newleader. The timeout value is chosen to be larger so as not totrigger unnecessary leadership changes, e.g., heartbeat witha 5-second periodicity could have a 60-second timeout.

5. Outcome requirements. There are two outcome require-ments that are common in leader election within the CDN.In an “at-most-one” election, at most one leader is desiredacross the entire CDN. In this case, leadership requires anabsolute majority within the entire candidate set. In thecase of a network partition, it is possible that no leader ischosen. Whereas in an “at-least-one” election, an absolutemajority is not required, we allow multiple leaders, one ineach partition. We now provide examples where these dif-ferent outcomes are appropriate.

6.2 At-Least-One Leader ElectionWhile the goal of a leader election algorithm is to select a

single global leader, in many circumstances it may be safe,or even desirable, to allow each subset in a network partitionto elect its own leader. One example comes from the systemthat uses consistent hashing to balance load within a cluster,as described in Section 3. Each server in a cluster is con-figured so that it has a list of the addresses of all servers inthe cluster, whether or not it can communicate with them.Each server also has an identical copy of the mapping ofservers in the cluster to the unit circle for each serial num-ber. Periodically (on a time scale of seconds) the serverswithin a cluster try to contact each other to exchange infor-mation about how much traffic (e.g., megabits per second)they are serving for each serial number. If there is a parti-tion within the cluster, then this information is exchangedwithin subsets.

The assignment of servers to serial numbers is computedas follows. Each server independently adds up the trafficfor each serial number. Then, based on the total traffic, theserver determines how many servers, k, to assign to the serialnumber, and creates a list of the first k servers within thesubset that appear on the unit circle. The lists computedby different servers in a subset should be nearly identical,although there may be small differences based on the tim-ings at which traffic measurements are taken or reported,or based on different views of which servers are currentlyon-line. Rather than use different solutions simultaneouslywithin a subset, however, a leader election algorithm is ap-plied, and the leader distributes its solution to the othermembers of the subset. The servers in the subset thenuse the leader’s solution to answer DNS queries for domainnames such as a212.g.akamai.net.

In the event of a partition within a cluster, therefore, eachsubset has its own leader and operates independently with-out interfering with other subsets. This circumstance is notideal, however, because each subset may not have enoughstorage capacity to cache all of the popular content that for-merly was distributed across the entire cluster, so cache hitrates may suffer. On the other hand, the servers in a subsetcannot direct clients to servers in other subsets, because theother subsets may be completely inoperable. Fortunately,partitions within a cluster are very rare, because the serverswithin a cluster are almost always housed in the same room.

6.3 At-Most-One Leader ElectionPerhaps surprisingly, there are circumstances in which it

is better for a leader election algorithm to fail to select anyleader rather than select different leaders for different sub-sets in a network partition. (At-most-one-leader semanticsis achieved by requiring a majority of the candidates to votefor the same leader.) Leader elections that produce at mostone leader is suitable for software components that obeytwo criteria: (i) the periodicity with which the componentexecutes and produces a new output is flexible, i.e., it is rel-atively safe for the other components of the CDN to operatebased on a slightly older output; (ii) the use of two differentversions of the output can lead to unpredictable or adverseconsequences.

As an example, consider the software component (called“topology discovery” [9]), which explores the current struc-ture of the Internet by performing network measurementsand then outputs a map unit definition table, which lists allthe map units, and assigns an identification number to eachmap unit. Over time, as the structure of the Internet and itsusers change, the map unit definitions also change. Severalcomponents may share a common map unit definition ta-ble to perform their functions. For instance, one componentuses the map unit definitions to compute the demand fromeach map unit, while the global load balancer discussed inSection 2 uses the map unit definitions and their demandsto route traffic to clusters. If there are two or more dif-ferent map unit definition tables in the system, numerousproblems may arise, such as data and decisions meant forone map unit accidentally being associated with or appliedto another. As with any other component, topology discov-ery is performed in a replicated fashion on several servers.However, for this component, if leader election produces noleader and no output for a short period of time, it is saferfor other components to operate with a last-known-good ver-sion of the map unit definition table than to operate withpotentially conflicting versions of it.

6.4 ConsensusThere are many contexts where full-blown consensus al-

gorithms are required for the proper functioning of a CDN.Consider the case of a CDN delivering an e-commerce Website. For better performance the user’s session state (such asthe contents of his/her shopping cart) can be stored in theproximal edge server that is serving the user, rather thanat a centralized origin site. But, what would happen if thatserver were to fail? It is not acceptable for the user to losehis or her session state! Thus, there is a need for the sessionstate to be replicated across different servers and the copiesto be kept consistent, so that server failures do not disruptuser transactions. There are several other such applicationswithin a CDN that require a distributed, replicated, fault-tolerant store that supports atomic reads and writes. ThePaxos algorithm [20] has been implemented to provide suchfunctionality in the CDN, and the Raft [25] algorithm ap-pears to be an attractive alternative.

7. CONCLUDING REMARKSThis paper explores the interaction between algorithmic

research and commercial distributed systems based on ourpersonal experiences in building one of the largest distributedsystems in the world, Akamai’s content delivery network.While our examples are by no means comprehensive, wehope that they illustrate how research influenced the designof the CDN and how the system-building challenges inspired


more research.Pasteur’s quadrants [31] provide an intuitive classification

of research based on whether its goal is a fundamental un-derstanding (i.e., basic research), a particular use for society(i.e., applied research), or both (i.e., use-inspired basic re-search)4. Our examples illustrate the impact of research ofall three types.

Basic research in stable allocations began decades beforethe first CDNs were built. Applying stable allocations tonetworked systems, however, was never the intention of theearly researchers. Yet the framework provides an elegantway to formulate and solve the global load-balancing prob-lem. Moreover, the impact is not a one-way street. Thereal-world implementation of the global load-balancing sys-tem required modeling multiple capacity constraints, whichinspired further research, eventually leading to algorithmsin which constraints are expressed as resource trees.

In contrast to stable allocations, the algorithmic work onconsistent hashing is a classic example of use-inspired basicresearch. A primary motivator of the research was in factload balancing within a CDN, although it has subsequentlyfound applications in numerous other contexts.

Finally, in our examples, developing the algorithms is notthe end, but the beginning. Much applied research is re-quired to translate algorithmic ideas into practice. In somecases, the simple tractable models used for analysis do notadequately model the complexities of the real world. There-fore, theorems of optimality may not strictly apply in prac-tice. In most cases, the algorithm that is actually imple-mented differs from the algorithms analyzed in the litera-ture. In all examples, however, provably effective algorithmsprovided the conceptual basis for the system that was built.From this standpoint, an algorithm that proposes a con-ceptually new way of thinking about a problem may be asimportant as being the most efficient in its class.

8. ACKNOWLEDGEMENTSThe systems presented in this paper are the product of

sixteen years of hard work by numerous Akamai engineersand researchers. While it is not feasible to list all of theirnames here, we have cited publications describing these sys-tems where possible. The research of the broader commu-nity laid the foundations for building these systems, and wehave cited the relevant work where appropriate. The au-thors would also like to thank specific colleagues for helpin preparing this paper. Ming Dong Feng conducted theBloom filter experiments described in Section 4. The em-pirical evaluation of overlay routing in Section 5 was firstreported in [30]. We also thank Mangesh Kasbekar, MingDong Feng, Marcelo Torres, Brian Dean, and Michel Goe-mans for insightful discussions about the material. Thiswork was supported in part by NSF grant CNS-1413998.

9. REFERENCES[1] iostat - linux man page.

http://linux.die.net/man/1/iostat.

[2] Atila Abdulkadiroglu, Parag A. Pathak, and Alvin E.Roth. The New York City high school match.American Economic Review, pages 364–367, 2005.

4The scientists often cited as canonical examples in eachquadrant are Niels Bohr (basic), Thomas Edison (applied),and Louis Pasteur (use-inspired basic).

[3] Ravindra K. Ahuja, Thomas L. Magnanti, andJames B. Orlin. Network Flows: Theory, Algorithms,and Applications. Prentice Hall, 1993.

[4] Konstantin Andreev, Bruce M. Maggs, AdamMeyerson, Jevan Saks, and Ramesh K. Sitaraman.Algorithms for constructing overlay networks for livestreaming. arXiv preprint arXiv:1109.4114, 2011.

[5] Konstantin Andreev, Bruce M. Maggs, AdamMeyerson, and Ramesh K. Sitaraman. Designingoverlay multicast networks for streaming. InProceedings of the Fifteenth Annual ACM Symposiumon Parallel Algorithms and Architectures, pages149–158. ACM, 2003.

[6] Mourad Baıou and Michel Balinski. Many-to-manymatching: stable polyandrous polygamy (orpolygamous polyandry). Discrete AppliedMathematics, 101(1):1–12, 2000.

[7] Burton H. Bloom. Space/time trade-offs in hashcoding with allowable errors. Communications of theACM, 13(7):422–426, 1970.

[8] Andrei Broder and Michael Mitzenmacher. Networkapplications of Bloom filters: A survey. InternetMathematics, 1(4):485–509, 2004.

[9] Fangfei Cheng, Ramesh K. Sitaraman, and MarceloTorres. End-user mapping: Next generation requestrouting for content delivery. In Proceedings of the 2015ACM Conference on SIGCOMM, SIGCOMM ’15,2015.

[10] John Dilley, Bruce Maggs, Jay Parikh, Harald Prokop,Ramesh Sitaraman, and Bill Weihl. Globallydistributed content delivery. Internet Computing,IEEE, 6(5):50–58, 2002.

[11] Li Fan, Pei Cao, Jussara Almeida, and Andrei Z.Broder. Summary cache: a scalable wide-area Webcache sharing protocol. IEEE/ACM Transactions onNetworking (TON), 8(3):281–293, 2000.

[12] David Gale and Lloyd S. Shapley. College admissionsand the stability of marriage. American MathematicalMonthly, pages 9–15, 1962.

[13] Michel Goemans. Load balancing in content deliverynetworks. IMA Annual Program Year Workshop:Network Management and Design, April 2003.

[14] Dan Gusfield and Robert W. Irving. The StableMarriage Problem: Structure and Algorithms. MITPress, Cambridge, MA, 1989.

[15] Kazuo Iwama and Shuichi Miyazaki. A survey of thestable marriage problem and its variants. InInformatics Education and Research forKnowledge-Circulating Society, 2008. ICKS 2008.International Conference on, pages 131–136. IEEE,2008.

[16] D. Karger, A. Sherman, B. Bogstad, R. Dhanidina,K. Iwamoto, B. Kim, L. Matkins, and Y. Yerushalmi.Web caching with consistent hashing. ComputerNetworks, 31(11):1203–1213, 1999.

[17] David Karger, Eric Lehman, Tom Leighton, MatthewLevine, Daniel Lewin, and Rina Panigrahy. Consistenthashing and random trees: Distributed cachingprotocols for relieving hot spots on the World WideWeb. In Proceedings of the 29th Annual ACMSymposium on Theory of Computing, pages 654–663,May 1997.


[18] David Karger, Eric Lehman, Tom Leighton, MatthewLevine, Daniel Lewin, and Rina Panigrahy. U.S.patent number 8,458,259: Method and apparatus fordistributing requests among a plurality of resources,August 2002.

[19] Leonidas Kontothanassis, Ramesh Sitaraman, JoelWein, Duke Hong, Robert Kleinberg, Brian Mancuso,David Shaw, and Daniel Stodolsky. A transport layerfor live streaming in a content delivery network.Proceedings of the IEEE, 92(9):1408–1419, 2004.

[20] Leslie Lamport. Paxos made simple. ACM SIGACTNews, 32(4):18–25, 2001.

[21] Leslie Lamport, Robert Shostak, and Marshall Pease.The Byzantine generals problem. ACM Transactionson Programming Languages and Systems (TOPLAS),4(3):382–401, 1982.

[22] Daniel M. Lewin. Consistent hashing and randomtrees: Algorithms for caching in distributed networks.Master’s thesis, Massachusetts Institute ofTechnology, 1998.

[23] Michael Mitzenmacher and Eli Upfal. Probability andcomputing: Randomized algorithms and probabilisticanalysis. Cambridge University Press, 2005.

[24] Erik Nygren, Ramesh K. Sitaraman, and Jennifer Sun.The Akamai network: a platform for high-performanceinternet applications. ACM SIGOPS OperatingSystems Review, 44(3):2–19, 2010.

[25] Diego Ongaro and John Ousterhout. In search of anunderstandable consensus algorithm. In Proceedings ofthe USENIX Annual Technical Conference, pages305–320, 2014.

[26] H. Rahul, M. Kasbekar, R. Sitaraman, and A. Berger.Towards realizing the performance and availabilitybenefits of a global overlay network. In Proceedings ofthe Passive and Active Measurement Conference, 2006.

[27] Alvin E. Roth and Elliott Peranson. The redesign ofthe matching market for American physicians: Someengineering aspects of economic design. Technicalreport, National Bureau of Economic Research, 1999.

[28] Alvin E. Roth, Tayfun Sonmez, and M. Utku Unver.Kidney exchange. The Quarterly Journal ofEconomics, pages 457–488, 2004.

[29] Alex Rousskov and Duane Wessels. Cache digests.Computer Networks and ISDN Systems,30(22):2155–2168, 1998.

[30] Ramesh K. Sitaraman, Mangesh Kasbekar, WoodyLichtenstein, and Manish Jain. Overlay networks: AnAkamai perspective. In Pathan, Sitaraman, andRobinson, editors, Advanced Content Delivery,Streaming, and Cloud Services. John Wiley & Sons,2014.

[31] Donald E. Stokes. Pasteur’s Quadrant: Basic Scienceand Technological Innovation. Brookings InstitutionPress, 1997.


algorithmic nuggets in content delivery - · pdf filealgorithmic nuggets in content delivery...

Documents