HyperDex: A Distributed, Searchable Key-Value Store

Robert EscrivaComputer Science

DepartmentCornell University

escriva@cs.cornell.edu

Bernard WongCheriton School of Computer

ScienceUniversity of Waterloo

bernard@uwaterloo.ca

Emin Gün SirerComputer Science

DepartmentCornell University

egs@systems.cs.cornell.edu

ABSTRACT

Distributed key-value stores are now a standard componentof high-performance web services and cloud computing ap-plications. While key-value stores offer significant perfor-mance and scalability advantages compared to traditionaldatabases, they achieve these properties through a restrictedAPI that limits object retrieval—an object can only be re-trieved by the (primary and only) key under which it wasinserted. This paper presents HyperDex, a novel distributedkey-value store that provides a unique search primitive thatenables queries on secondary attributes. The key insightbehind HyperDex is the concept of hyperspace hashing inwhich objects with multiple attributes are mapped into amultidimensional hyperspace. This mapping leads to effi-cient implementations not only for retrieval by primary key,but also for partially-specified secondary attribute searchesand range queries. A novel chaining protocol enables thesystem to achieve strong consistency, maintain availabilityand guarantee fault tolerance. An evaluation of the full sys-tem shows that HyperDex is 12-13× faster than Cassandraand MongoDB for finding partially specified objects. Ad-ditionally, HyperDex achieves 2-4× higher throughput forget/put operations.

Categories and Subject Descriptors

D.4.7 [Operating Systems]: Organization and Design

Keywords

Key-Value Store, NoSQL, Fault-Tolerance, Strong Consis-tency, Performance

1. INTRODUCTIONModern distributed applications are reshaping the land-

scape of storage systems. Recently emerging distributedkey-value stores such as BigTable [11], Cassandra [32] andDynamo [19] form the backbone of large commercial appli-cations because they offer scalability and availability prop-erties that traditional database systems simply cannot pro-vide. Yet these properties come at a substantial cost: the

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.SIGCOMM’12, August 13–17, 2012, Helsinki, Finland.Copyright 2012 ACM 978-1-4503-1419-0/12/08 ...$15.00.

data retrieval API is narrow and restrictive, permitting anobject to be retrieved using only the key under which itwas stored, and the consistency guarantees are often quiteweak. Queries based on secondary attributes are either notsupported, utilize costly secondary indexing schemes or enu-merate all objects of a given type.

This paper introduces HyperDex, a high-performance, scal-able, consistent and distributed key-value store that providesa new search primitive for retrieving objects by secondaryattributes. HyperDex achieves this extended functionalityby organizing its data using a novel technique called hyper-

space hashing. Similar to other hashing techniques [23, 29,36,47], hyperspace hashing deterministically maps objects toservers to enable efficient object insertion and retrieval. Butit differs from these techniques because it takes into accountthe secondary attributes of an object when determining themapping for an object. Specifically, it maps objects to co-ordinates in a multi-dimensional Euclidean space – a hyper-

space – which has axes defined by the objects’ attributes.Each server in the system is mapped onto a region of thesame hyperspace, and owns the objects that fall within itsregion. Clients use this mapping to deterministically insert,remove, and search for objects.

Hyperspace hashing facilitates efficient search by signifi-cantly reducing the number of servers to contact for eachpartially-specified search. The construction of the hyper-space mapping guarantees that objects with matching at-tribute values will reside on the same server. Through geo-metric reasoning, clients can restrict the search space for apartially-specified query to a subset of servers in the system,thereby improving search efficiency. Specificity in searchesworks to the clients’ advantage: a fully-specified search con-tacts exactly one server.

A naive hyperspace construction, however, may suffer froma well-known problem with multi-attribute data known as“curse of dimensionality [6].” With each additional secondaryattribute, the hyperspace increases in volume exponentially.If constructed in this fashion, each server would be respon-sible for a large volume of the resulting hyperspace, whichwould in turn force search operations to contact a large num-ber of servers, counteracting the benefits of hyperspace hash-ing. HyperDex addresses this problem by partitioning thedata into smaller, limited size subspaces of fewer dimensions.

Failures are inevitable in all large-scale deployments. Thestandard approaches for providing fault tolerance store ob-jects on a fixed set of replicas determined by a primarykey. These techniques, whether they employ a consensusalgorithm among the replicas and provide strong consis-

25

tency [18,46], or spray the updates to the replicas and onlyachieve eventual consistency [19,32,45,49], assume that thereplica sets remain fixed even as the objects are updated.Such techniques are not immediately suitable in our settingbecause, in hyperspace hashing, object attributes determinethe set of servers on which an object resides, and conse-quently, each update may implicitly alter the replica set.Providing strong consistency guarantees with low overheadis difficult, and more so when replica sets change dynami-cally and frequently. HyperDex utilizes a novel replicationprotocol called value-dependent chaining to simultaneouslyachieve fault tolerance, high performance and strong con-sistency. Value-dependent chaining replicates an object towithstand f faults (which may span server crashes and net-work partitions) and ensures linearizability, even as replicasets are updated. Thus, HyperDex’s replication protocolguarantees that all get operations will immediately see theresult of the last completed put operation – a stronger con-sistency guarantee than those offered by the current gener-ation of NoSQL data stores.

Overall, this paper describes the architecture of a new key-value store whose API is one step closer to that of traditionalRDBMSs while offering strong consistency guarantees, fault-tolerance for failures affecting a threshold of servers andhigh performance, and makes three contributions. First,it describes a new hashing technique for mapping struc-tured data to servers. This hashing technique enables ef-ficient retrieval of multiple objects even when the searchquery specifies both equality and range predicates. Second,it describes a fault-tolerant, strongly-consistent replicationscheme that accommodates object relocation. Finally, it re-ports from a full implementation of the system and deploy-ment in a data center setting consisting of 64 servers, anddemonstrates that HyperDex provides performance that iscomparable to or better than Cassandra and MongoDB, twocurrent state-of-the-art cloud storage systems, as measuredusing the industry-standard YCSB [15] benchmark. Morespecifically, HyperDex achieves 12-13× higher throughputfor search workloads than the other systems, and consis-tently achieves 2-4× higher throughput for traditional key-value workloads.

The rest of this paper is structured as follows: Section2 describes hyperspace hashing. Section 3 specifies how todeal with spaces of high dimensionality through data parti-tioning. Section 4 specifies the value-dependent replicationprotocol used in HyperDex. Section 5 outlines our full im-plementation of HyperDex. Section 6 evaluates HyperDexunder a variety of workloads. Section 7 discusses relatedwork for hyperspace hashing and HyperDex. We discusshow our system relates to the CAP Theorem in Section 8and conclude with a summary of our contributions.

2. APPROACHIn this section, we describe the data model used in Hyper-

Dex, outline hyperspace hashing, and sketch the high-levelorganization and operation of the system.

2.1 Data Model and APIHyperDex stores objects that consist of a key and zero or

more secondary attributes. As in a relational database, Hy-perDex objects match an application-provided schema thatdefines the typed attributes of the object and are persisted in

tables. This organization permits straightforward migrationfrom existing key-value stores and database systems.

HyperDex provides a rich API that supports a variety ofdatastructures and a wide range of operations. The systemnatively supports primitive types, such as strings, integersand floats, as well as composite types, such as lists, setsor maps constructed from primitive types. The dozens of op-erations that HyperDex provides on these datatypes fall intothree categories. First, basic operations, consisting of get,put, and delete, enable a user to retrieve, update, and de-stroy an object identified by its key. Second, the search op-eration enables a user to specify zero or more ranges for sec-ondary attributes and retrieve the objects whose attributesfall within the specified, potentially singleton, ranges. Fi-nally, a large set of atomic operations, such as cas andatomic-inc, enable applications to safely perform concur-rent updates on objects identified by their keys. Since com-posite types and atomic operations are beyond the scope ofthis paper, we focus our discussion on the get, put, delete,and search operations that form the core of HyperDex.

2.2 Hyperspace HashingHyperDex represents each table as an independent multi-

dimensional space, where the dimensional axes corresponddirectly to the attributes of the table. HyperDex assigns ev-ery object a corresponding coordinate based on the object’sattribute values. An object is mapped to a deterministiccoordinate in space by hashing each of its attribute valuesto a location along the corresponding axis.

Consider, for the following discussion, a table containinguser information that has the attributes first-name, last-name, and telephone-number. For this schema, Hyper-Dex would create a three dimensional space where the first-name attribute comprises the x-axis, the last-name attributecomprises the y-axis, and the telephone-number attributecomprises the z-axis. Figure 1 illustrates this mapping.

Hyperspace hashing determines the object to server map-ping by tessellating the hyperspace into a grid of non-over-lapping regions. Each server is assigned, and is responsiblefor, a specific region. Objects whose coordinates fall withina region are stored on the corresponding server. Thus, thehyperspace tessalation serves like a multi-dimensional hashbucket, mapping each object to a unique server.

The tessalation of the hyperspace into regions (called thehyperspace mapping), as well as the assignment of the re-gions to servers, is performed by a fault-tolerant coordinator.The primary function of the coordinator is to maintain thehyperspace mapping and to disseminate it to both serversand clients. The hyperspace mapping is initially created bydividing the hyperspace into hyperrectangular regions andassigning each region to a virtual server. The coordinatoris then responsible for maintaining this mapping as serversfail and new servers are introduced into the system.

The geometric properties of the hyperspace make objectinsertion and deletion simple. To insert an object, a clientcomputes the coordinate for the object by hashing each ofthe object’s attributes, uses the hyperspace mapping to de-termine the region in which the object lies, and contactsthat server to store the object. The hyperspace mappingobviates the need for server-to-server routing.

2.3 Search QueriesThe hyperspace mapping described in the preceding sec-

tions facilitates a geometric approach to resolving search

26

First Name

Phone Number

Last Name

John

Smith

Figure 1: Simple hyperspace hashing in three di-mensions. Each plane represents a query on a sin-gle attribute. The plane orthogonal to the axis for“Last Name” passes through all points for last_name

= ‘Smith’, while the other plane passes through allpoints for first_name = ‘John’. Together they rep-resent a line formed by the intersection of the twosearch conditions; that is, all phone numbers forpeople named “John Smith”. The two cubes showregions of the space assigned to two different servers.The query for “John Smith” needs to contact onlythese servers.

operations. In HyperDex, a search specifies a set of at-tributes and the values that they must match (or, in the caseof numeric values, a range they must fall between). Hyper-Dex returns objects which match the search. Each search

operation uniquely maps to a hyperplane in the hyperspacemapping. A search with one attribute specified maps toa hyperplane that intersects that attribute’s axis in exactlyone location and intersects all other axes at every point. Al-ternatively, a search that specifies all attributes maps toexactly one point in hyperspace. The hyperspace mappingensures that each additional search term potentially reducesthe number of servers to contact and guarantees that addi-tional search terms will not increase search complexity.

Clients maintain a copy of the hyperspace mapping, anduse it to deterministically execute search operations. Aclient first maps the search into the hyperspace using themapping. It then determines which servers’ regions intersectthe resulting hyperplane, and issues the search request toonly those servers. The client may then collect matchingresults from the servers. Because the hyperspace mappingmaps objects and servers into the same hyperspace, it isnever necessary to contact any server whose region does notintersect the search hyperplane.

Range queries correspond to extruded hyperplanes. Whenan attribute of a search specifies a range of values, the cor-responding hyperplane will intersect the attribute’s axis atevery point that falls between the lower and upper bounds ofthe range. Note that for such a scheme to work, objects’ rela-tive orders for the attribute must be preserved when mappedonto the hyperspace axis.

Figure 1 illustrates a query for first_name = ‘John’ andlast_name = ‘Smith’. The query for first_name = ‘John’

k v1 v2 v3 v4 v5 . . . vD-1vD-1 vD

hyperspace

k v1 v2 v3 v4 v5 . . . vD-1vD-1 vD

key subspace

subspace 0 subspace 1 subspace S

Figure 2: HyperDex partitions a high-dimensionalhyperspace into multiple low-dimension subspaces.

corresponds to a two-dimensional plane which intercepts thefirst_name axis at the hash of ‘John’. Similarly, the queryfor last_name = ‘Smith’ creates another plane which inter-sects the last_name axis. The intersection of the two planesis the line along which all phone numbers for John Smith re-side. Since a search for John Smith in a particular area codedefines a line segment, a HyperDex search needs to contactonly those nodes whose regions intersect that segment.

3. DATA PARTITIONINGHyperDex’s Euclidean space construction significantly re-

duces the set of servers that must be contacted to findmatch-ing objects.

However, the drawback of coupling the dimensionality ofhyperspace with the number of searchable attributes is that,for tables with many searchable attributes, the hyperspacecan be very large since its volume grows exponentially withthe number of dimensions. Covering large spaces with a gridof servers may not be feasible even for large data-center de-ployments. For example, a table with 9 secondary attributesmay require 29 regions or more to support efficient searches.In general, a D dimensional hyperspace will need O(2D)servers.

HyperDex avoids the problems associated with high-di-mensionality by partitioning tables with many attributesinto multiple lower-dimensional hyperspaces called subspaces.Each of these subspaces uses a subset of object attributes asthe dimensional axes for an independent hyperspace. Fig-ure 2 shows how HyperDex can partition a table with Dattributes into multiple independent subspaces. When per-forming a search on a table, clients select the subspace thatcontacts the fewest servers, and will issue the search toservers in exactly one subspace.

Data partitioning increases the efficiency of a search byreducing the dimensionality of the underlying hyperspace.In a 9-dimensional hyperspace, a search over 3 attributeswould need to contact 64 regions of the hyperspace (andthus, 64 servers). If, instead, the same table were parti-tioned into 3 subspaces of 3 attributes each, the search willnever contact more than 8 servers in the worst case, andexactly one server in the best case. By partitioning the ta-ble, HyperDex reduces the worst case behavior, decreasesthe number of servers necessary to maintain a table, andincreases the likelihood that a search is run on exactly oneserver.

Data partitioning forces a trade-off between search gener-ality and efficiency. On the one hand, a single hyperspacecan accommodate arbitrary searches over its associated at-tributes. On the other hand, a hyperspace which is too large

27

will always require that partially-specified queries contactmany servers. Since applications often exhibit search local-ity, HyperDex applications can tune search efficiency by cre-ating corresponding subspaces. As the number of subspacesgrows, so, too, do the costs associated with maintaining dataconsistency across subspaces. Section 4 details how Hyper-Dex efficiently maintains consistency across subspaces whilemaintaining a predictably low overhead.

3.1 Key SubspaceThe basic hyperspace mapping, as described so far, does

not distinguish the key of an object from its secondary at-tributes. This leads to two significant problems when im-plementing a practical key-value store. First, key lookupswould be equivalent to single attribute searches. AlthoughHyperDex provides efficient search, a single attribute searchin a multi-dimensional space would likely involve contactingmore than one server. In this hypothetical scenario, key op-erations would be strictly more costly than key operationsin traditional key-value stores.

HyperDex provides efficient key-based operations by cre-ating a one-dimensional subspace dedicated to the key. Thissubspace, called the key subspace, ensures that each objectwill map to exactly one region in the resulting hyperspace.Further, this region will not change as the object changesbecause keys are immutable. To maintain the uniquenessinvariant, put operations are applied to the key subspacebefore the remaining subspaces.

3.2 Object Distribution Over SubspacesSubspace partitioning exposes a design choice in how ob-

jects are distributed and stored on servers. One possibledesign choice is to keep data in normalized form, whereevery subspace retains, for each object, only those objectattributes that serve as the subspace’s dimensional axes.While this approach would minimize storage requirementsper server, as attributes are not duplicated across subspaces,it would lead to more expensive search and object retrievaloperations since reconstituting the object requires cross-serv-er cooperation. In contrast, an alternative design choice isto store a full copy of each object in each subspace, whichleads to faster search and retrieval operations at the expenseof additional storage requirements per server.

Hyperspace hashing supports both of these object distri-bution techniques. The HyperDex implementation, how-ever, relies upon the latter approach to implement the repli-cation scheme described in Section 4.

3.3 Heterogeneous ObjectsIn a real deployment, the key-value store will likely be

used to hold disparate objects with different schema. Hyper-Dex supports this through the table abstraction. Each tablehas a separate set of attributes which make up the objectswithin, and these attributes are partitioned into subspacesindependent of all other tables. As a result, HyperDex man-ages multiple independent hyperspaces.

4. CONSISTENCY AND REPLICATIONBecause hyperspace hashing maps each object to multiple

servers, maintaining a consistent view of objects poses achallenge. HyperDex employs a novel technique called value-

dependent chaining to provide strong consistency and faulttolerance in the presence of concurrent updates.

For clarity, we first describe value-dependent chaining with-out concern for fault tolerance. Under this scheme, a singlefailure leaves portions of the hyperspace unavailable for up-dates and searches. We then describe how value-dependentchaining can be extended such that the system can tolerateup to f failures in any one region.

4.1 Value Dependent ChainingBecause hyperspace hashing determines the location of

an object by its contents, and subspace partitioning createsmany object replicas, objects will be mapped to multipleservers and these servers will change as the objects are up-dated. Change in an object’s location would cause problemsif implemented naively. For example, if object updates wereto be implemented by simply sending the object to all af-fected servers, there would be no guarantees associated withsubsequent operations on that object. Such a scheme wouldat best provide eventual consistency because servers mayreceive updates out-of-order, with no sensible means of re-solving concurrent updates.

HyperDex orders updates by arranging an object’s repli-cas into a value-dependent chain whose members are deter-ministically chosen based upon an object’s hyperspace coor-dinate. The head of the chain is called the point leader, andis determined by hashing the object in the key subspace.Subsequent servers in the chain are determined by hashingattribute values for each of the remaining subspaces.

This construction of value-dependent chains enables ef-ficient, deterministic propagation of updates. The pointleader for an object is in a position to dictate the total orderon all updates to that object. Each update flows from thepoint leader through the chain, and remains pending un-til an acknowledgement of that update is received from thenext server in the chain. When an update reaches the tail,the tail sends an acknowledgement back through the chainin reverse so that all other servers may commit the pendingupdate and clean up transient state. When the acknowl-edgement reaches the point leader, the client is notified thatthe operation is complete. In Figure 3, the update u1 illus-trates an object insertion which passes through h1, h2, h3,where h1 is the point leader.

Updates to preexisting objects are more complicated be-cause a change in an attribute value might require relocatingan object to a different region of the hyperspace. Value-dependent chains address this by incorporating the serversassigned to regions for both the old and new versions ofthe object. Chains are constructed such that servers are or-dered by subspace and the servers corresponding to the oldversion of the object immediately precede the servers corre-sponding to the new version. This guarantees that there isno instant during an update where the object may disappearfrom the data store. For example, in Figure 3, the updateu2 modifies the object in a way that changes its mappingin Subspace 0 such that the object no longer maps to h2

and instead maps to h5. The value-dependent chain for up-date u2 is h1, h2, h5, h3. The update will result in the objectbeing stored at h5, and subsequently removed from h2, asacknowledgments propagate in reverse.

Successive updates to an object will construct chains whichoverlap in each subspace. Consequently, concurrent updatesmay arrive out of order at each of these points of overlap.For example, consider the update u3 in Figure 3. The value-dependent chain for this update is h1, h5, h3. Notice that it

28

h1 h2 h3

h4 h5 h6

keysubspace subspace 0 subspace 1

update u1

update u2

update u3

Figure 3: HyperDex’s replication protocol propa-gates along value-dependent chains. Each updatehas a value-dependent chain that is determinedsolely by objects’ current and previous values andthe hyperspace mapping.

is possible for u3 to arrive at h5 before u2. If handled im-properly, such races could lead to inconsistent, out-of-orderupdates. Value-dependent chains efficiently handle this caseby dictating that the point leader embed, in each update,dependency information which specifies the order in whichupdates are applied. Specifically, the point leader embedsa version number for the update, and the version number,hash and type of the previous update. For instance, u3 willhave a version number of 3 and depend upon update u2 withversion number 2 and type put. Servers which receive u3 be-fore u2 will know to delay processing of u3 until u2 is alsoprocessed.

By design, HyperDex supports destructive operations thatremove all state pertaining to deleted objects. Examples ofdestructive operations include delete and the cleanup as-sociated with object relocation. Such operations must becarefully managed to ensure that subsequent operations getapplied correctly. For instance, consider a del followed by aput. Since we would like a del to remove all state, the put

must be applied on servers with no state. Yet, if anotherdel/put pair were processed concurrently, servers which hadprocessed either del would not be able to properly orderthe put operations. Value-dependent chains ensure thatconcurrently issued destructive operations are correctly or-dered on all servers. Each server independently delays op-erations which depend upon a destructive operation untilthe destructive operation, and all that came before it, areacknowledged. This ensures that at most one destructiveoperation may be in-flight at any one time and guaranteesthat they will be ordered correctly. The delay for each mes-sage is bounded by the length of chains, and the number ofconcurrent operations.

4.2 Fault ToleranceTo guard against server failures, HyperDex provides ad-

ditional replication within each region. The replicas acquireand maintain their state by being incorporated into value-dependent chains. In particular, each region has f +1 repli-cas which appear as a block in the value-dependent chain.For example, we can extend the layout of Figure 3 to toler-ate one failure by introducing additional hosts h′

1 through

h′

6. As with regular chain replication [57], new replicas areintroduced at the tail of the region’s chain, and servers arebumped forward in the chain as other servers fail. For ex-ample, the first update in Figure 3 has the value-dependentchain h1, h

′

1, h2, h′

2, h3, h′

3. If h2 were to fail, the resultingchain would be h1, h

′

1, h′

2, h′′

2 , h3, h′

3. This transition will beperformed without compromising strong consistency.

Point leader failures do not allow clients to observe an in-consistency. For instance, if h1, the point leader, were to failin our previous example, h′

1 will take over the role of pointleader. When a client detects a point leader failure, it willnotify the application and preserve at-most-once semantics.Further, because all client acknowledgments are generatedby the point leader, the client will only see a response afteran object is fully fault-tolerant.

In our implementation, HyperDex uses TCP to transmitdata which ensures that messages need only be retrans-mitted when servers fail and are removed from the value-dependent chain. In response to failures, HyperDex serversretransmit messages along the chains to make progress. Uponchain reconfiguration, servers will no longer accept messagesfrom failed servers, ensuring that all future messages traversethe new, valid chain.

4.3 Server and Configuration ManagementHyperDex utilizes a logically centralized coordinator to

manage system-wide global state. The coordinator encap-sulates the global state in the form of a configuration whichconsists of the hyperspace mapping between servers and re-gions and information about server failures. The coordinatorassigns to each configuration an epoch identifier, a strictlyincreasing number that identifies the configuration. The co-ordinator distributes configurations to servers in order byepoch, and updates the configuration as necessary. Hyper-Dex’s coordinator holds no state pertaining to the objectsthemselves; only the mapping and servers.

HyperDex ensures that no server processes late-arrivingmessages from failed or out-of-date servers. Each HyperDexserver process (an instance) is uniquely identified by its IPaddress, port and instance id. Instance ids are assigned bythe coordinator and are globally unique, such that serverscan distinguish between two instances (e.g. a failed processand a new one) that reuse the same IP and port number.HyperDex embeds into messages the instance ids, the regionsof the hyperspace and indices into the chain for both thesender and recipient. A recipient acting upon a messagecan validate the sender and recipient against its most recentconfiguration. If the message contains a stale mapping, itwill not be acted upon. If, instead, the mapping is valid, thehost processes the message accordingly.

Each host changes its configuration in an all-or-nothingfashion which appears instantaneous to threads handlingnetwork communication. This is accomplished on each hostby creating state relevant to the new configuration, pausingnetwork traffic, swapping pointers to make the new statevisible, and unpausing network traffic. This operation com-pletes in sub-millisecond time on each host.

The logically centralized coordinator does not impose abottleneck. Configurations are small in practice, and pro-portional to the size of the cluster and number of tables ac-tive on the cluster. Furthermore, in cases where bandwidthfrom the coordinator becomes a bottleneck, the coordina-tor need only distribute deltas to the configuration. Clients

29

maintain a complete copy of the configuration in memoryand perform local computation on the hyperspace mapping.

4.4 Consistency GuaranteesOverall, the preceding protocol ensures that HyperDex

provides strong guarantees for applications. The specificguarantees made by HyperDex are:

Key Consistency All actions which operate on a specifickey (e.g., get and put) are linearizable [26] with all opera-tions on all keys. This guarantees that all clients of Hyper-Dex will observe updates in the same order.

Search Consistency HyperDex guarantees that a searchwill return all objects that were committed at the time ofsearch. An application whose put succeeds is guaranteed tosee the object in a future search. In the presence of con-current updates, a search may return both the committedversion, and the newly updated version of an object match-ing the search.

HyperDex provides the strongest form of consistency forkey operations, and a conservative and predictable consis-tency guarantees for search operations.

5. IMPLEMENTATIONHyperDex is fully implemented to support all the features

described in this paper. The implementation is nearly 44,000lines of code. The HyperDex software distribution containsan embeddable storage layer called HyperDisk, a hyperspacehashing library, the HyperDex server, the client library andthe HyperDex coordinator, as well as full client bindings forC, C++, and Python, and partial bindings for Java, Node.JSand Ruby.

This section describes key aspects of the HyperDex im-plementation.

5.1 HyperDex ServerHigh throughput and low latency access are essential for

any key-value store. The HyperDex server achieves high per-formance through three techniques; namely, edge-triggered,event-driven I/O; pervasive use of lock-free datastructuresand sharded locks to maximize concurrency; and carefulattention to constant overheads. The HyperDex server isstructured around a concurrent, edge-triggered event loopwherein multiple worker threads receive network messagesand directly process client requests. Whereas common de-sign patterns would distinguish between I/O threads andworker threads, HyperDex combines these two functions toavoid internal queuing and communication delays. Becausethe event-loop is edge-triggered, unnecessary interaction withthe kernel is minimized. Socket buffer management ensuresthat threads never block in the kernel when sending a mes-sage and consumes a constant amount of memory per client.

HyperDex makes extensive use of lock-sharding and lock-free datastructures to reduce the probability of contentionwhenever possible. Per-key datastructures are protected byan array of locks. Although nothing prevents two threadsfrom contending for the same lock to protect different keys,the ratio of locks to threads is high enough to reduce this oc-currence to 1 in 106. Global datastructures, such as lookuptables for the per-key datastructures, are concurrent throughthe use of lock-free hash tables. Our implementation ensuresthat background threads may safely iterate over global statewhile worker threads insert and remove pointers to per-keystate.

Finally, the use of cache-conscious, constant-time datastructures reduces the overheads associated with commonoperations such as linked-list and hash-table management.

5.2 HyperDisk: On-Disk Data StorageA key component of server performance for any key-value

store is the storage back end used to organize data on disk.Since hyperspace hashing is agnostic to the choice of theback end, a number of design options are available. At oneextreme, we could have used a traditional database to storeall the objects of a server in a single, large, undifferentiatedpool. While this approach would have been the simplestfrom an implementation perspective, it would make Hyper-Dex dependent on the performance of a traditional databaseengine, require manual tuning of numerous parameters, andsubject the system to the vagaries of a query optimizer.

Instead, HyperDex recursively leverages the hyperspacehashing technique to organize the data stored internally ona server. Called HyperDisk, this approach partitions theregion associated with a server into smaller non-overlappingsub-regions, where a sub-region represents a file on disk, andobjects are located in the file that contains their coordinate.Each file is arranged as a log, where insertions and deletionsoperate on the tail, and where a search operation linearlyscans through the whole file.

HyperDisk’s hashing scheme differs from the standard hy-perspace hashing in two ways: first, HyperDisk partitionsonly the region assigned to a HyperDex server; and second,HyperDisk may alter the mapping dynamically to accommo-date the number of objects stored within a region. Overall,recursive application of hyperspace hashing enables Hyper-Dex to take advantage of the geometric structure of the dataspace at every system layer.

5.3 Distributed CoordinationIn HyperDex, the hyperspace mapping is created and man-

aged by a logically centralized coordinator. Since a physi-cally centralized coordinator would limit scalability and posea single point of failure, the HyperDex coordinator is imple-mented as a replicated state machine. It relies on a co-ordination service [2, 9, 27] to replicate the coordinator onmultiple physical servers. The coordinator implementationensures that servers may migrate between coordinators sothat no coordinator failure leads to correlated failures in thesystem. The coordinator directs all failure recovery actions.Servers may report observed failures to the coordinator, orthe coordinator may directly observe failures through peri-odic failure detection pings to servers.

Overall, the replicated state machine implementation en-sures that the coordinator acts as a single, coherent com-ponent with well-defined state transitions, even though it iscomprised of fault-tolerant, distributed components.

6. EVALUATIONWe deployed HyperDex on both a small and medium-

size computational cluster and evaluated the performanceof each deployment using the Yahoo! Cloud Serving Bench-mark (YCSB) [15], an industry-standard benchmark for cloudstorage performance. Our evaluation also examines the per-formance of HyperDex’s basic operations, specifically, get,put, and search, using targeted micro-benchmarks. Thesemicro-benchmarks isolate specific components and help ex-pose the performance impact of design decisions. For both

30

0

5

10

20

30

40

Cassandra MongoDB HyperDex

Throughput(thousandop/s) Load

Workload AWorkload BWorkload CWorkload DWorkload FWorkload E

Figure 4: Average throughput for a variety of real-world workloads specified by the Yahoo! CloudServing Benchmark. HyperDex is 3-13 times fasterthan Cassandra and 2-12 times faster than Mon-goDB. Workload E is a search-heavy workload,where HyperDex outperforms other systems bymore than an order of magnitude.

YCSB and the micro-benchmarks, we compare HyperDexwith Cassandra [32], a popular key-value store for Web 2.0applications, and MongoDB [37], a distributed documentdatabase.

The performance benchmarks are executed on our small,dedicated lab-size cluster in order to avoid confounding is-sues arising from sharing a virtualized platform, while thescalability benchmarks are executed on the VICCI [42] test-bed. Our dedicated cluster consists of fourteen nodes, eachof which is equipped with two Intel Xeon 2.5 GHz E5420processors, 16 GB of RAM, and a 500 GB SATA 3.0 Gbit/shard disk operating at 7200 RPM. All nodes are running64-bit Debian 6 with the Linux 2.6.32 kernel. A single gi-gabit Ethernet switch connects all fourteen machines. Oneach of the machines, we deployed Cassandra version 0.7.3,MongoDB version 2.0.0, and HyperDex.

For all tests, the storage systems are configured to pro-vide sufficient replication to tolerate one node failure. Eachsystem was configured to use its default consistency set-tings. Specifically, both Cassandra and MongoDB provideweak consistency and fault-tolerance guarantees; because ac-knowledgments are generated without full replication, smallnumbers of failures can lead to data loss. In contrast, Hy-perDex utilizes value-depending chaining and, as a result,always provides clients with strong consistency and fault-tolerance, even in the presence of failures. Since MongoDBallocates replicas in pairs, we allocate twelve machines forthe storage nodes, one machine for the clients, and, whereapplicable, one node for the coordinator. HyperDex is con-figured with two subspaces in addition to the key subspaceto accommodate all ten attributes in the YCSB dataset.

6.1 Get/Put PerformanceHigh get/put performance is paramount to any cloud-

based storage system. YCSB provides six different work-loads that exercise the storage system with a mixture ofrequest types and object distributions resembling real-worldapplications (Table 1). In all YCSB experiments, the data-base is preloaded with 10,000,000 objects and each opera-tion selects the object and operation type in accordance withthe workload specification. Figure 4 shows the throughputachieved by each system across the YCSB workloads. Hy-

0

20

40

60

80

100

1 10 50

CDF

(%)

Latency (ms)

Cassandra (R)Cassandra (U)MongoDB (R)MongoDB (U)HyperDex (R)HyperDex (U)

Figure 5: GET/PUT performance. Latency distribu-tion for Workload A (50% reads, 50% updates, Zipfdistribution).

0

20

40

60

80

100

1 10 50

CDF

(%)

Latency (ms)

Cassandra (R)Cassandra (U)MongoDB (R)MongoDB (U)HyperDex (R)HyperDex (U)

Figure 6: GET/PUT performance. Latency distribu-tion for Workload B (95% reads, 5% updates, Zipfdistribution). HyperDex maintains low latency forreads and writes.

perDex provides throughput that is between a factor of twoto thirteen higher than the other systems. The largest per-formance gains come from improvements in search perfor-mance. Significant improvements in get/put performanceis attributable mostly to the efficient handling of get op-erations in HyperDex. Our implementation demonstratesthat the hyperspace construction and maintenance can berealized efficiently.

In order to gain insight into the performance of the system,we examine the request latency distributions of the differentsystems under all read/write workloads. HyperDex’s perfor-mance is predictable: all reads complete in under 1 ms, whilea majority of writes complete in under 3 ms. Cassandra’slatency distributions follow a similar trend for workloads B,C, D and F and show a slightly different trend for workloadA. MongoDB, on the other hand, exhibits lower latency thanCassandra for all operations. For all workloads, HyperDexcompletes 99% of operations sooner than either Cassandraand MongoDB. Figures 5 and 6 show the latency distribu-tions for workloads A and B respectively.

For completeness, we present the performance of all threesystems on a write-heavy workload. Figure 7 shows the la-tency distribution for inserting 10,000,000 objects to set upthe YCSB tests. Consistency guarantees have a significanteffect on the put latency. MongoDB’s default behavior con-siders a put complete when the request has been queued in

31

Name Workload Key Distribution Sample ApplicationA 50% Read/50% Update Zipf Session StoreB 95% Read/5% Update Zipf Photo TaggingC 100% Read Zipf User Profile CacheD 95% Read/5% Insert Temporal User Status UpdatesE 95% Scan/5% Insert Zipf Threaded ConversationsF 50% Read/50% Read-Modify-Write Zipf User Database

Table 1: The six workloads specified by the Yahoo! Cloud Serving Benchmark. These workloads modelseveral applications found at Yahoo! Each workload was tested using the same YCSB driver program andsystem-specific Java bindings. Each object has ten attributes which total 1 kB in size.

0

20

40

60

80

100

1 10 50

CDF

(%)

Latency (ms)

CassandraMongoDBHyperDex

Figure 7: PUT performance. Latency distribution for10,000,000 operations consisting of 100% insertions.Each operation created a new object under a unique,previously unused key. Although fast, HyperDex’sminimum latency is bounded by the length of thevalue-dependent chains.

a client’s send buffer, even before it has been seen by anyserver. Cassandra’s default behavior considers a put com-plete when it is queued in the filesystem cache of just onereplica. Unlike these systems, the latency of a HyperDexput operation includes the time taken to fully replicate theobject on f + 1 servers. Because MongoDB does not waitfor full fault tolerance, it is able to complete a majority ofoperations in less than 1 ms; however, it exhibits a long-tail (Figure 7) that adversely impacts average throughput.Similarly, Cassandra completes most operations in less than2 ms. Despite its stronger fault-tolerance guarantees, Hy-perDex completes 99% of its operations in less than 2 ms.

6.2 Search vs. ScanUnlike existing key-value stores, HyperDex is architected

from the ground-up to perform search operations efficiently.Current applications that rely on existing key-value storesemulate search functionality by embedding additional infor-mation about other attributes in the key itself. For exam-ple, applications typically group logically related objects byusing a shared prefix in the key of each object, and thenrely upon the key-value store to locate keys with the com-mon prefix. In Cassandra, this operation is efficient be-cause keys are stored in sorted order, and returning all log-ically grouped keys is an efficient linear scan. Fittingly, theYCSB benchmark calls this a scan operation. HyperDex’ssearch functionality is a strict superset of the scan opera-tion. Rather than using a shared prefix to support scans,HyperDex stores, for each object, the prefix and suffix ofthe key as two secondary attributes. Scans are then imple-

0

20

40

60

80

100

1 10 100 1000

CDF

(%)

Latency (ms)

CassandraMongoDBHyperDex

Figure 8: SEARCH performance. Latency distributionfor 10,000 operations consisting of 95% range queriesand 5% inserts with keys selected from a Zipf distri-bution. HyperDex is able to offer significantly lowerlatency for non-primary key range queries than theother systems are able to offer for primary-key rangequeries.

mented as a multi-attribute search that exactly matches aprovided prefix value and a provided range of suffix values.Thus, all YCSB benchmarks involving a scan operation op-erate on secondary attributes in HyperDex, but operate onthe key for other systems.

Despite operating on secondary attributes instead of thekey, HyperDex outperforms the other systems by an or-der of magnitude for scan operations (Figure 8). Seventyfive percent of search operations complete in less than 2 ms,and nearly all complete in less than 6 ms. Cassandra sortsdata according to the primary key and is therefore able toretrieve matching items relatively quickly. Although onecould alter YCSB to use Cassandra’s secondary indexingschemes instead of key-based operations, the result wouldbe strictly worse than what is reported for primary key op-erations. MongoDB’s sharding maintains an index; conse-quently, scan operations in MongoDB are relatively fast.The search performance of HyperDex is not attributableto our efficient implementation as search is more than anorder of magnitude faster in HyperDex, which eclipses the2-4× performance advantage observed for get/put through-put. Hyperspace hashing in HyperDex ensures that searchresults are located on a small number of servers; this en-ables effective pruning of the search space and allows eachsearch to complete by contacting exactly one host in ourexperiments.

An additional benefit of HyperDex’s aggressive search prun-ing is the relatively low latency overhead associated withsearch operations. Figure 9 shows the average latency of a

32

0.1

1

10

100

1000

1 10 50

Latency

(ms)

Percent scan/search

CassandraMongoDBHyperDex

Figure 9: The effect of an increasing scan workloadon latency. HyperDex performs significantly betterthan the other systems even as the scan workloadbegins to approach 50%.

01234567

Latency

(ms)

2468

101214

0 2 4 6 8 10

Throughput

(thousandops/s)

Number of Subspaces

HyperDex

Figure 10: Latency and throughput for put oper-ations as a function of non-key subspaces The er-ror bars indicate standard deviation from 10 exper-iments. Latency increases linearly in the length ofthe chain, while throughput decreases proportion-ally. In applications we have built with HyperDex,all tables have three or fewer subspaces.

single scan operation as the total number of scan operationsperformed increases. In this test, searches were constructedby choosing the lower bound of the range uniformly at ran-dom from the set of possible values, as opposed to workloadE which uses a Zipf distribution to select objects. Using auniform distribution ensures random access, and mitigatesthe effects of object caching. HyperDex consistently offerslow latency for search-heavy workloads.

A critical parameter that affects HyperDex’s search per-formance is the number of subspaces in a HyperDex table.Increasing the number of subspaces leads to additional op-portunities for pruning the search space for search opera-tions, but simultaneously requires longer value-dependentchains that result in higher put latencies. In Figure 10,we explore the tradeoff using between zero and ten addi-tional subspaces beyond the mandatory key subspace. Notethat adding ten additional subspaces increases the value-dependent chain to be at least 22 nodes long. As expected,HyperDex’s put latency increases linearly with each addi-tional subspace.

0

1

2

3

4

4 8 12 16 20 24 28 32

Throughput(m

illionops/s)

Nodes

4 Clients8 Clients

16 Clients32 Clients48 Clients

Figure 11: HyperDex scales horizontally. As moreservers are added, aggregate throughput increaseslinearly. Each point represents the average through-put of the system in steady state over 30 secondwindows. The error bars show the 5th and 95th per-centiles.

6.3 ScalabilityWe have deployed HyperDex on the VICCI [42] testbed to

evaluate its performance in an environment representative ofthe cloud. Each VICCI cluster has 70 Dell R410 PowerEdgeservers, each of which has 2 Intel Xeon X5650 CPUs, 48 GBof RAM, three 1 TB hard drives, and two 1 Gbit ethernetports. Users are provided with an isolated virtual machinefor conducting experiments. Each virtual machine comespreinstalled with Fedora 12 and runs the 2.6.32 Linux kernel.

We examined the performance of a HyperDex cluster asthe cluster increases in size. Increasing the number of serversin the cluster provides HyperDex with additional resourcesand leads to a proportional increase in throughput. In Fig-ure 11, we explore the change in system throughput as re-sources are added to the cluster. As expected, HyperDexscales linearly as resources are added to the cluster. Eachpoint in the graph represents the average throughput ob-served over a 30 second window and the error bars showthe 5th and 95th percentiles observed over any 1-second win-dow. At its peak, HyperDex is able to average 3.2 millionoperations per second.

The workload for Figure 11 is a 95% read, 5% write work-load operating on 8 B keys and 64 B values. The measure-ments reported are taken in steady state, with clients ran-domly generating requests according to the workload. Thisworkload and measurement style reflects the workload likelyto be encountered in a web application. The reported mea-surements exclude the warm-up time for the system. Inall experiments, 15 seconds was sufficient to achieve steadystate. Clients operate in parallel, and are run on separatemachines from the servers in all but the largest configura-tions. Clients issue requests in parallel, and each client main-tains an average of 1,000 outstanding requests per server.Increasing the number of clients does not significantly im-pact the achievable average throughput.

This experiment shows that a medium-sized HyperDexcluster is able to achieve high throughput for realisticallysized deployments [3]. Additional resources allow the clusterto provide proportionally better throughput.

33

7. RELATED WORK

Database system Storage systems that organize their datain high-dimensional spaces were pioneered by the databasecommunity more than thirty years ago [5,7,24,31,39,41,51].These systems, collectively known as Multi-Dimensional Da-tabases (MDB), leverage multi-dimensional data structuresto improve the performance of data warehousing and on-line analytical processing applications. However, unlike hy-perspaces in HyperDex, the data structures in MDBs aredesigned for organizing data on a single machine and arenot directly applicable to large-scale distributed storage sys-tems. Alternatively, more recent database systems [1, 17]have begun exploring efficient mechanisms for building andmaintaining large-scale, tree-based distributed indices. Incontrast, the mapping HyperDex constructs is not an index.Indices must be maintained and updated on object inser-tion. Hyperspace hashing, on other hand, is a mapping thatdoes not change as objects are inserted and removed.

Peer-to-peer systems Past work in peer-to-peer systemshas explored multi-dimensional network overlays to facili-tate decentralized data storage and retrieval. CAN [47] is adistributed hash-table that, much like HyperDex, organizespeers in a multi-dimensional space. However, CAN only pro-vides key inserts and lookups; the purpose of CAN’s multi-dimensional peer configuration is to limit a CAN node’speer-set size to provide efficient overlay routing. HyperDexprovides search, and does not do routing in the space.

MURK [21], SkipIndex [58], and SWAM-V [30] dynam-ically partition the multi-dimensional space into kd-trees,skip graphs, and Voronoi diagrams respectively to providemulti-dimensional range lookups. Although conceptuallysimilar to HyperDex, providing coordination and manage-ment of nodes for these dynamic space partitioning schemesis significantly more complex and error-prone than for Hy-perDex’s static space partitioning and require additional op-erational overhead. These systems also do not address sev-eral critical and inherent problems associated with mappingstructured data into a multi-dimensional space and provid-ing reliable data storage. Specifically, they are not efficientfor high dimensional data due to the curse-of-dimensionality,and they either lack data replication or provide only even-tually consistent operations on replicated data. Addressingthese problems by augmenting dynamic space partitioningschemes with subspaces and value-dependent chaining wouldfurther increase the complexity and overhead of node coor-dination and management. Mercury [8] builds on top of aChord [55] ring, and uses consistent hashing [29] on eachattribute as secondary indexes. Although Mercury’s imple-mentation is unavailable, Cassandra uses a similar designand its performance illustrates how a multi-ring-based sys-tem would perform. Arpeggio [13] provides search over mul-tiple attributes by enumerating and creating an index of all(

k

x

)

fixed-size subsets of attributes using a Chord ring. Bothof these approaches insert redundant pointers into rings with-out concern for consistency.

Space-filling curves A common approach to providingmulti-attribute search uses space-filling curves to partitionmulti-dimensional data across the storage nodes. This ap-proach uses the space filling curve to map the multi-di-mensional data into a single dimension, which then enablesthe use of traditional peer-to-peer techniques for perform-ing searches. SCRAP [21], Squid [52] and ZNet [53] are

examples of this approach with each node responsible fordata in a contiguous range of values. Similarly, MAAN [10]performs the same mapping, but uses uniform locality pre-serving hashing. Space-filling curves do not scale well whenthe dimensionality is high, because a single search querymay be partitioned into many one-dimensional ranges ofconsiderably varying size. Furthermore, unlike in Hyper-Dex, fully-qualified searches, where values for all attributesare specified, may involve contacting more than one node inspace-filling curve-based systems.

NoSQL Storage A new class of scalable storage systems,collectively dubbed “NoSQL”, have recently emerged withthe defining characteristic that they depart from the tradi-tional architecture and SQL interface of relational databases.It is common practice for NoSQL systems to make explicittradeoffs with respect to desirable properties. For instance,many NoSQL systems explicitly sacrifice consistency – evenin the common case without failures – to achieve availabil-ity under extreme failure scenarios. NoSQL systems in-clude document databases [16, 37] that offer a schema-freedata model, in-memory cache solutions [36, 43, 48] that ac-celerate application performance, and graph databases [38]which model interconnected elements and key-value storeswhich offer predictable performance. Still, some systems donot fit neatly into these categories. For example, Yahoo!’sPNUTS [14] system supports the traditional SQL selectionand projection, functions but does not support joins.

HyperDex explores a new point in the NoSQL design space.The rich HyperDex API provides qualitatively new function-ality not offered by other NoSQL systems, HyperDex sets anew bar for future NoSQL systems by combining strong con-sistency properties with fault-tolerance guarantees, a richAPI and high performance.

Key-Value Stores Modern key-value stores have roots inwork on Distributed Data Structures [23,36] and distributedhash tables [29,47,50,55]. Most open source key-value storesdraw heavily from the ring-based architecture of Dynamo [19]and the tablet-based architecture of BigTable [11]. For in-stance, Voldemort [45] and Riak [49] are heavily influencedby Dynamo’s design. Other systems like HBase [25] and Hy-perTable [28] are open source implementations of BigTable.Cassandra [32] is unique in that it is influenced by BigTable’sAPI and Dynamo’s ring structure. Like HyperDex, all ofthese systems are designed to run in a datacenter environ-ment on many machines.

Recent work on key-value stores largely focuses on improv-ing performance by exploiting underlying hardware or ma-nipulating consistency guarantees for a performance advan-tage. Fawn KV [4] builds a key-value store on underpoweredhardware to improve the throughput-to-power-draw ratio.SILT [33] eliminates read amplification to maximize readbandwidth in a datastore backed by solid-state disk. RAM-Cloud [40] stores data in RAM and utilizes fast network con-nections to rapidly restore failed replicas. TSSL [54] utilizesa multi-tier storage hierarchy to exploit cache-oblivious algo-rithms in the storage layer. Masstree [35] uses concatenatedB+ trees to service millions of queries per second. COPS [34]provides a high-performance geo-replicated key-value storethat provides causal+ consistency. Other systems [12, 19]trade consistency for other desirable properties, such as per-formance. Spanner [18] uses Paxos in the wide area to pro-vide strong consistency. Spinnaker [46] uses Paxos to build

34

a strongly consistent store that performs nearly as well asthose that are only eventually consistent [19,44,56].

Each of these existing systems improves the performance,availability or consistency of key value stores while retain-ing the same basic structure: a hash table. In HyperDex,we take a complementary approach which expands the key-value interface to support, among other operations, searchover secondary attributes.

8. DISCUSSIONSurprisingly, the open-source release of the HyperDex sys-

tem [20] uncovered various misunderstandings surroundingthe CAP Theorem [22]. The popular CAP refrain (“C, A, P:pick any two”) causes the subtleties in the definitions of C, Aand P to be lost, even on otherwise savvy developers. Thereexists no conflict between claims in our paper and the CAPTheorem. The failure model used in the CAP Theorem isunconstrained; the system can be subject to partitions andnode failures that affect any number of servers and networklinks. No system, including HyperDex, can simultaneouslyoffer consistency and availability guarantees using such weakassumptions. HyperDex makes a stronger assumption thatlimits failures to affect at most a threshold of servers and isthus able to provide seemingly impossible guarantees.

9. CONCLUSIONSThis paper described HyperDex, a second-generation No-

SQL storage system that combines strong consistency guar-antees with high availability in the presence of failures andpartitions affecting up to a threshold of servers. In addition,HyperDex provides an efficient search primitive for retriev-ing objects through their secondary attributes. It achievesthis extended functionality through hyperspace hashing, inwhich multi-attribute objects are deterministically mappedto coordinates in a low dimension Euclidean space. Thismapping leads to efficient implementations for key-based re-trieval, partially-specified searches and range-queries. Hy-perDex’s novel replication protocol enables the system toprovide strong consistency without sacrificing performance.Industry-standard benchmarks show that the system is prac-tical and efficient.

The recent trend toward NoSQL data stores has been fu-eled by scalability and performance concerns at the cost offunctionality. HyperDex bridges this gap by providing ad-ditional functionality without sacrificing scalability or per-formance.

10. ACKNOWLEDGMENTSWe would like to thank Pawel Loj for his contributions

on cluster stop/restart, Deniz Altınbuken for her ConCo-ord Paxos implementation, and members of the HyperDexopen source community for their code contributions, andthe VICCI team for granting us time on their cluster. Thismaterial is based upon work supported by National Sci-ence Foundation under Grants No. CNS-1111698 and CCF-0424422 and by the National Science and Engineering Re-search Council of Canada.

11. REFERENCES[1] M. K. Aguilera, W. M. Golab, and M. A. Shah. A

Practical Scalable Distributed B-Tree. In PVLDB,

1(1), 2008.

[2] D. Altınbuken and E. G. Sirer. CommodifyingReplicated State Machines with OpenReplica.Computing and Information Science, CornellUniversity, Technical Report 1813-29009, 2012.

[3] S. Anand. http://www.infoq.com/presentations/NoSQL-Netflix/.

[4] D. G. Andersen, J. Franklin, M. Kaminsky, A.Phanishayee, L. Tan, and V. Vasudevan. FAWN: AFast Array of Wimpy Nodes. In Proc. of SOSP, BigSky, MT, Oct. 2009.

[5] R. Bayer. The Universal B-Tree for MultidimensionalIndexing: General Concepts. In Proc. of WWCA,

Tsukuba, Japan, Mar. 1997.

[6] R. E. Bellman. Dynamic Programming. PrincetonUniversity Press, 1957.

[7] J. L. Bentley. Multidimensional Binary Search TreesUsed for Associative Searching. In CACM, 18(9), 1975.

[8] A. R. Bharambe, M. Agrawal, and S. Seshan. Mercury:Supporting Scalable Multi-Attribute Range Queries.In Proc. of SIGCOMM, Portland, OR, Aug. 2004.

[9] M. Burrows. The Chubby Lock Service forLoosely-Coupled Distributed Systems. In Proc. ofOSDI, Seattle, WA, Nov. 2006.

[10] M. Cai, M. R. Frank, J. Chen, and P. A. Szekely.MAAN: A Multi-Attribute Addressable Network forGrid Information Services. In Proc. of GRID

Workshop, Phoenix, AZ, Nov. 2003.

[11] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A.Wallach, M. Burrows, T. Chandra, A. Fikes, and R.Gruber. BigTable: A Distributed Storage System forStructured Data. In Proc. of OSDI, Seattle, WA, Nov.2006.

[12] J. Cipar, G. R. Ganger, K. Keeton, C. B. M. III,C. A. N. Soules, and A. C. Veitch. LazyBase: TradingFreshness for Performance in a Scalable Database. InProc. of EuroSys, Bern, Switzerland, Apr. 2012.

[13] A. T. Clements, D. R. K. Ports, and D. R. Karger.Arpeggio: Metadata Searching and Content Sharingwith Chord. In Proc. of IPTPS Workshop, La Jolla,CA, Feb. 2005.

[14] B. F. Cooper, R. Ramakrishnan, U. Srivastava, A.Silberstein, P. Bohannon, H.-A. Jacobsen, N. Puz, D.Weaver, and R. Yerneni. PNUTS: Yahoo!’s HostedData Serving Platform. In PVLDB, 1(2), 2008.

[15] B. F. Cooper, A. Silberstein, E. Tam, R.Ramakrishnan, and R. Sears. Benchmarking CloudServing Systems with YCSB. In Proc. of SoCC,Indianapolis, IN, June 2010.

[16] CouchDB. http://couchdb.apache.org/.

[17] A. Crainiceanu, P. Linga, J. Gehrke, and J.Shanmugasundaram. Querying Peer-to-Peer NetworksUsing P-Trees. In Proc. of WebDB Workshop, Paris,France, June 2004.

[18] J. Dean. Designs, Lessons, and Advice from BuildingLarge Distributed Systems. Keynote. In Proc. ofLADIS, Big Sky, MT, Oct. 2009.

[19] G. DeCandia, D. Hastorun, M. Jampani, G.Kakulapati, A. Lakshman, A. Pilchin, S.Sivasubramanian, P. Vosshall, and W. Vogels.Dynamo: Amazon’s Highly Available Key-ValueStore. In Proc. of SOSP, Stevenson, WA, Oct. 2007.

35

[20] R. Escriva, B. Wong, and E. G. Sirer. Hyperdex.http://hyperdex.org/.

[21] P. Ganesan, B. Yang, and H. Garcia-Molina. OneTorus to Rule Them All: Multidimensional Queries inP2P Systems. In Proc. of WebDB Workshop, Paris,France, June 2004.

[22] S. Gilbert and N. A. Lynch. Brewer’s Conjecture andthe Feasibility of Consistent, Available,Partition-Tolerant Web Services. In SIGACT News,

33(2), 2002.

[23] S. D. Gribble. A Design Framework and a ScalableStorage Platform to Simplify Internet ServiceConstruction. U.C. Berkeley, 2000.

[24] A. Guttman. R-Trees: A Dynamic Index Structure forSpatial Searching. In Proc. of SIGMOD, 1984.

[25] HBase. http://hbase.apache.org/.

[26] M. Herlihy and J. M. Wing. Linearizability: ACorrectness Condition for Concurrent Objects. InACM ToPLaS, 12(3), 1990.

[27] P. Hunt, M. Konar, F. P. Junqueira, and B. Reed.ZooKeeper: Wait-Free Coordination for Internet-ScaleSystems. In Proc. of USENIX, Boston, MA, June2010.

[28] Hypertable. http://http://hypertable.org/.

[29] D. R. Karger, E. Lehman, F. T. Leighton, R.Panigrahy, M. S. Levine, and D. Lewin. ConsistentHashing and Random Trees: Distributed CachingProtocols for Relieving Hot Spots on the World WideWeb. In Proc. of STOC, El Paso, TX, May 1997.

[30] F. B. Kashani and C. Shahabi. SWAM: A Family ofAccess Methods for Similarity-Search in Peer-to-PeerData Networks. In Proc. of CIKM, Washington, D.C.,Nov. 2004.

[31] A. Klinger. Patterns and Search Statistics. InOptimizing Methods in Statistics, Academic Press,1971.

[32] A. Lakshman and P. Malik. Cassandra: ADecentralized Structured Storage System. In Proc. ofLADIS, Big Sky, MT, Oct. 2009.

[33] H. Lim, B. Fan, D. G. Andersen, and M. Kaminsky.SILT: A Memory-Efficient, High-PerformanceKey-Value Store. In Proc. of SOSP, Cascais, Portugal,Oct. 2011.

[34] W. Lloyd, M. J. Freedman, M. Kaminsky, and D. G.Andersen. Don’t Settle for Eventual: Scalable CausalConsistency for Wide-Area Storage with COPS. InProc. of SOSP, Cascais, Portugal, Oct. 2011.

[35] Y. Mao, E. Kohler, and R. T. Morris. CacheCraftiness for Fast Multicore Key-Value Storage. InProc. of EuroSys, Bern, Switzerland, Apr. 2012.

[36] Memcached. http://memcached.org/.

[37] MongoDB. http://www.mongodb.org/.

[38] Neo4j. http://neo4j.org/.

[39] J. Nievergelt, H. Hinterberger, and K. C. Sevcik. TheGrid File: An Adaptable, Symmetric Multikey FileStructure. In ACM ToDS, 9(1), 1984.

[40] D. Ongaro, S. M. Rumble, R. Stutsman, J. K.Ousterhout, and M. Rosenblum. Fast Crash Recoveryin RAMCloud. In Proc. of SOSP, Cascais, Portugal,Oct. 2011.

[41] J. A. Orenstein and T. H. Merrett. A Class of DataStructures for Associative Searching. In Proc. ofPODS, 1984.

[42] L. Peterson, A. Bavier, and S. Bhatia. VICCI: AProgrammable Cloud-Computing Research Testbed.Princeton, Technical Report TR-912-11, 2011.

[43] D. R. K. Ports, A. T. Clements, I. Zhang, S. Madden,and B. Liskov. Transactional Consistency andAutomatic Management in an Application DataCache. In Proc. of OSDI, Vancouver, Canada, Oct.2010.

[44] D. Pritchett. BASE: An ACID Alternative. In ACM

Queue, 6(3), 2008.

[45] Project Voldemort.http://project-voldemort.com/.

[46] J. Rao, E. J. Shekita, and S. Tata. Using Paxos toBuild a Scalable, Consistent, and Highly AvailableDatastore. In PVLDB, 4(4), 2011.

[47] S. Ratnasamy, P. Francis, M. Handley, R. M. Karp,and S. Shenker. A Scalable Content-AddressableNetwork. In Proc. of SIGCOMM, San Diego, CA,Aug. 2001.

[48] Redis. http://redis.io/.

[49] Riak. http://basho.com/.

[50] A. I. T. Rowstron and P. Druschel. Pastry: Scalable,Decentralized Object Location, and Routing forLarge-Scale Peer-to-Peer Systems. In Proc. of ICDSP,volume 2218, 2001.

[51] H. Samet. Spatial Data Structures. In Modern

Database Systems: The Object Model, Interoperability,

and Beyond, Addison Wesley/ACM Press, 1995.

[52] C. Schmidt and M. Parashar. Enabling FlexibleQueries with Guarantees in P2P Systems. In Internet

Computing Journal, 2004.

[53] Y. Shu, B. C. Ooi, K.-L. Tan, and A. Zhou.Supporting Multi-Dimensional Range Queries inPeer-to-Peer Systems. In Proc. of IEEE International

Conference on Peer-to-Peer Computing, Konstanz,Germany, Aug. 2005.

[54] R. P. Spillane, P. J. Shetty, E. Zadok, S. Dixit, and S.Archak. An Efficient Multi-Tier Tablet Server StorageArchitecture. In Proc. of SoCC, 2011.

[55] I. Stoica, R. Morris, D. R. Karger, M. F. Kaashoek,and H. Balakrishnan. Chord: A Scalable Peer-to-PeerLookup Service for Internet Applications. In Proc. ofSIGCOMM, San Diego, CA, Aug. 2001.

[56] D. B. Terry, M. Theimer, K. Petersen, A. J. Demers,M. Spreitzer, and C. Hauser. Managing UpdateConflicts in Bayou, a Weakly Connected ReplicatedStorage System. In Proc. of SOSP, Copper Mountain,CO, Dec. 1995.

[57] R. van Renesse and F. B. Schneider. Chain Replicationfor Supporting High Throughput and Availability. InProc. of OSDI, San Francisco, CA, Dec. 2004.

[58] C. Zhang, A. Krishnamurthy, and R. Y. Wang.SkipIndex: Towards a Scalable Peer-to-Peer IndexService for High Dimensional Data. Princeton,Technical Report TR-703-04, 2004.

36