algorithms for load balancing in electricity markets and data centerscj... · 2021. 4. 13. · load...

Algorithms For Load Balancing In Electricity Markets And Data

Centers

A Dissertation Presented

by

Bochao Shen

to

The College of Computer and Information Science

in partial fulfillment of the requirements

for the degree of

Doctor of Philosophy

in

Computer Science

Northeastern UniversityBoston, Massachusetts

April 2018

To my family.

i

Contents

List of Figures iv

List of Tables v

Acknowledgments vi

Abstract of the Dissertation vii

1 Introduction 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.1.1 Temporal load balancing for electricity market . . . . . . . . . . . . . . . 21.1.2 Fault-tolerant spatial load balancing for data centers . . . . . . . . . . . . 31.1.3 Load balancing for multidimensional resources . . . . . . . . . . . . . . . 4

1.2 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Temporal Load Balancing for Electricity Market 92.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.3 Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.4 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132.5 Market model description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.6 SmartShift . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.7 Simulations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

3 Fault-tolerant Spatial Load Balancing in Data Centers 293.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.3 Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.4 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

3.4.1 VMPP and VMPP-AC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333.4.2 Exists?k-HA and Is?k-HA . . . . . . . . . . . . . . . . . . . . . . . . . . 34

3.5 k-HA is NP-complete . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

ii

3.5.1 Exists?k-HA is NP-hard . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.6 IID-IK and best heuristic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363.7 Analysis of heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

3.7.1 MTHM revisited . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.7.2 Gold-dust world: when VM sizes are small . . . . . . . . . . . . . . . . . 423.7.3 Doubling world: VM sizes in the form of 2i . . . . . . . . . . . . . . . . . 443.7.4 Performance evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453.7.5 Water-filling packs best . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4 Load Balancing for Multidimensional Resources 514.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.1.1 Motivation and Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 524.1.2 Our results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554.1.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

4.2 VITA(F) for linear F . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.3 VITA(min) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.3.1 Unconstrained, Bounded - exact . . . . . . . . . . . . . . . . . . . . . . . 594.3.2 Constrained, Bounded - strongly NP-hard . . . . . . . . . . . . . . . . . . 604.3.3 Unconstrained, Unbounded - inapproximable . . . . . . . . . . . . . . . . 604.3.4 Constrained, Unbounded - O(log n, log n) bicriteria . . . . . . . . . . . . 61

4.4 VITA(max) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 634.4.1 Unconstrained, Unbounded - exact . . . . . . . . . . . . . . . . . . . . . . 644.4.2 Constrained, Bounded - strongly NP-hard . . . . . . . . . . . . . . . . . . 644.4.3 Constrained, Unbounded - ⇥(log n) approximation . . . . . . . . . . . . . 65

4.5 VITA(2ndmax) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 664.5.1 Bounded, Unconstrained - weakly NP-hard . . . . . . . . . . . . . . . . . 664.5.2 Unweighted, Unconstrained, with number of buckets exceeding number of

dimensions - O(log n) approximation . . . . . . . . . . . . . . . . . . . . 664.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.6.1 Polynomial time heuristics for VITA . . . . . . . . . . . . . . . . . . . . . 684.6.2 Performance of VITA when vectors has constant number of dimensions . . 694.6.3 Performance of VITA when vectors has unbounded number of dimensions . 71

4.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

5 Conclusion 78

Bibliography 80

iii

List of Figures

2.1 Normalized revenue vs price volatility . . . . . . . . . . . . . . . . . . . . . . . . 242.2 Loss probability vs consumer tolerance . . . . . . . . . . . . . . . . . . . . . . . 252.3 Probability density function of profit . . . . . . . . . . . . . . . . . . . . . . . . . 262.4 Normalized social welfare vs price volatility . . . . . . . . . . . . . . . . . . . . . 272.5 Normalized social welfare and revenue of smartShift, flat-rate pricing and real-time

pricing vs Pareto distribution parameter �R . . . . . . . . . . . . . . . . . . . . . 28

3.1 Distribution of memory usage by running VMs . . . . . . . . . . . . . . . . . . . 483.2 Distribution of host memory capacity . . . . . . . . . . . . . . . . . . . . . . . . 493.3 Distribution of VM memory size . . . . . . . . . . . . . . . . . . . . . . . . . . . 503.4 Distribution of VM memory size after rounding . . . . . . . . . . . . . . . . . . . 50

4.1 VITA(min). The simplest unbounded case is inapproximable, and we give a bicri-teria guarantee for the hardest case. . . . . . . . . . . . . . . . . . . . . . . . . . . 56

4.2 VITA(max) and VITA(max�min). The unconstrained, cases are exactly solvableand we have tight logarithmic guarantees for the constrained unbounded case. . . . 57

4.3 Histogram of # of VCPUs requested . . . . . . . . . . . . . . . . . . . . . . . . . 714.4 Histogram of memory size requested . . . . . . . . . . . . . . . . . . . . . . . . . 724.5 Histogram of storage size requested . . . . . . . . . . . . . . . . . . . . . . . . . 734.6 Objective value of VITA(max) and three heuristics for minimizing bottleneck usage 734.7 Objective value of VITA(min) and three heuristics for minimizing maintenance

downtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 744.8 # of used buckets VS # of given buckets for minimizing maintenance downtime . . 744.9 Objective value with same increased number of buckets for minimizing maintenance

downtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 754.10 Objective value of VITA(max) with unbounded # of dimensions . . . . . . . . . . 754.11 Objective value of VITA(min) with unbounded # of dimensions . . . . . . . . . . 764.12 # of used buckets VS # of given buckets with unbounded # of dimensions . . . . . 764.13 Objective value with same increased number of buckets with unbounded # of di-

mensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

iv

List of Tables

3.1 One small host/VM size distribution . . . . . . . . . . . . . . . . . . . . . . . . . 453.2 Asymptotic metric PE for the small distribution . . . . . . . . . . . . . . . . . . . 463.3 Asymptotic metric PE for original host/VM size distribution from Nutanix . . . . . 473.4 Asymptotic metric PE for power-of-2 . . . . . . . . . . . . . . . . . . . . . . . . 47

v

Acknowledgments

First and foremost, I would like to thank my PhD advisor, Ravi Sundaram, for supporting andguiding me through my entire PhD study. Ravi’s insightful mind can always capture the essence ofproblems and enlighten me during our discussions. Working with Ravi and having him as my PhDadvisor is a truly amazing experience.

I would also like to thank my thesis committee members: Javed Aslam, Narayanaswamy Bal-akrishnan, Rajmohan Rajaraman. I am fortunate enough to work with each of them on differentprojects. Their comments and feedback are the valuable assets for me to have in my future careerand exploration.

I would like to thank all the lab mates during all these years at Northeastern University. Thereare so many names that I can not list them all here. But those days that we spent altogether aredeeply kept in my heart. They are vivid, warm and golden.

I would like to thank Megan Barry and Bryan Lackaye for being so patient answering myquestions about administrative processes in our college, and giving me so much help.

Finally, I would like to thank my parents for everything. I also would like to thank my wife,Die Sun, for supporting me all the time. I would never reach the moment of typing this line withouther love.

vi

Abstract of the Dissertation

Algorithms For Load Balancing In Electricity Markets And Data Centers

by

Bochao Shen

Doctor of Philosophy in Computer Science

Northeastern University, April 2018Dr. Ravi Sundaram, Adviser

Electricity and computers are two corner stones of this information era. Energy and compu-tation are critical resources in perennial short supply because our consumption continues to growday by day. Increasing supply in a substantial way requires fundamental advances in energy gen-eration and computing technology, but such advances are few and far between. Load balancingis an important technique for mitigating the issue of scarce resources. We broadly interpret loadbalancing to include both the optimization of resource distribution as well as the management ofend-user demand. In this thesis, we study algorithms for load balancing in electricity markets anddata centers.

First, we study the temporal load balancing problem in electricity markets where peak demandand supply-demand imbalance are major problems. It is often suggested that exposing consumers toreal-time pricing will incentivize them to change their usage and mitigate the problem. However, weshow that risk-averse electricity consumers react to price fluctuations by scaling back on their totaldemand, leading to the unintended consequence of an overall decrease in production/consumptionand reduced economic efficiency. Compared with the relatively fixed production mode of electric-ity power (the supply), the consumption pattern of end users (the demand) is more variable andpotentially changeable. This makes temporally shifting consumers’ electricity load possible. Wepropose SmartShift, a new scheme that allows households to move their demands from peak hoursin exchange for greater electricity consumption in non-peak hours. We show that our scheme notonly enables increased consumption and consumer welfare but also allows the distribution companyto increase profits.

We next consider the fault-tolerant spatial load balancing problem in data centers where com-putational loads get balanced by being assigned on different locations (machines - computationalresources). k-HA (high-availability), a fault tolerance property of virtual machine (VM) placement

vii

in clouds, represents the ability to tolerate up to k host failures by relocating VMs from failed hostswithout disrupting other VMs. It has long been assumed [15] that deciding the existence of a k-HAplacement is ⌃P

3 -complete. We show that k-HA reduces to multiple knapsack and hence is NP-complete. We propose a stochastic model for multiple knapsack that not only captures real-worldworkloads but also provides a uniform basis for comparing the efficiencies of different polynomial-time heuristics. We prove, using the central limit theorem and linear programming, that, in animportant special case, there exists a best polynomial-time heuristic. We turn to industry practiceand discuss the drawbacks of commonly used heuristics - First-fit, Best-fit, Worst-fit, MTHM andCSP. Based on a large real-world dataset of cluster workloads from industry we show that the naturalload-balancing heuristic - Water-filling - has several excellent properties. We compare and contrastWater-filling with MTHM using our stochastic model and find that Water-filling is a heuristic ofchoice.

Finally, we extend our study on load balancing from the single dimensional case to multidi-mensional resources. This is inspired by multiple questions from both electricity markets and datacenters. For example, what is the best way to assign VMs to data centers to minimize the disruptioncaused when data centers are powered down for maintenance? How companies distribute the loadto hybrid clouds? These related problems can be modeled as the Uncapacitated MultidimensionalLoad Assignment problem, where items (load demand) and buckets (supply capacity) are character-ized by a d-dimensional vector. Additional affinity constraints may restrict the subset of buckets aspecific item may be placed in. The cost of a bucket is obtained by aggregating the assigned itemsaccording to some metric, with different metrics (max, min, second-max, etc) representing differentapplication scenario for either electric markets or data centers. The goal is to minimize the total costacross all buckets. The temporal load balancing in electricity market and the spatial load balancingin data centers become the two applied scenarios of this problem. We provide hardness results andapproximation algorithms for this problem in a variety of settings.

viii

Chapter 1

Introduction

1.1 Motivation

Information technologies have kept making great impact on our daily lives ever since the com-

puter was invented. Nowadays, not only computers, but phones, tablets, even watches as well can

run impressive applications and computations inside. We can even speak to the smart home assis-

tants to control the home temperature, play music, switch on/off lights. The two invisible pillars

that support today’s prosperity of the information era are electricity and computation, undoubtedly.

In fact, electricity and computation represent the two critical resources that today’s worldwide

information infrastructure relies on. Like other natural resources, our growing consumption makes

energy and computation, the two critical resources, in perennial short supply. Admittedly, technolo-

gies have kept advancing to make more efficient and cleaner energy, produce the smaller and faster

chips. Such increase in supply of these two resources in a substantial way requires fundamental

advances in energy creation and computing technology.

Load balancing, a technique which aims to optimize the use of resource and improve the dis-

tribution of work loads, becomes useful when we need to consider how to efficiently consume

the resource, given the supply is limited. In this thesis, we first study two kinds of load balancing

techniques that are applied to two different scenarios respectively: temporal load balancing for elec-

tricity market; spatial load balancing for data centers. Then we move on our study to load balancing

problems with multidimensional resources.

1

CHAPTER 1. INTRODUCTION

1.1.1 Temporal load balancing for electricity market

Electricity power utilities have at least two major challenges. The first is Peak Demand - a

period in which the demand for power is significantly higher than average. In order to satisfy a large

peak demand, utilities (generation/distribution companies) have to make large capital investments

including new ‘peaking’ generation stations, larger capacity lines, transformers, and operational

expenditures including expensive purchases of electricity on the “spot market” [38]. For example,

it is estimated that a 5% lowering of demand would have resulted in a 50% price reduction during

the peak hours of the California electricity crisis in 2000-2001 [44]. As a result of the quadratic

dependence between resistive losses and transmitted current, peaks also lead to substantial energy

wastage. The second challenge faced by power utilities is that of Supply-Demand imbalance. [16]

states that ”the difficulties that have appeared in California and elsewhere are intrinsic to the design

of current electricity markets, in which demand exhibits virtually no price responsiveness and supply

faces strict production constraints”.

Without allowing consumption load be temporally shifted, economists and electricity compa-

nies traditionally have focused on pricing as a mechanism to solve these two problems in a sys-

tematic way. However, homes and small businesses have an inherent and systemic requirement for

stable electricity costs [18]. It is understood that ”Consumers generally shy away from markets

when products are complicated, supply is uncertain, prices are volatile, and information is lacking”

[19]. We demonstrate that exposing risk-averse consumers to the actual real-time costs of electricity

production, will result in a reduction in aggregate demand leading to reduced revenue for the gen-

erators and distributors with potential knock-on effects for the economy at large. At the same time,

the volatility of real-time pricing can have strong effects on grid stability [70]. A second concern

with real-time pricing, is that consumer prices may increase [8] or net electricity consumption may

decrease [9], depending on the model.

We notice that the consumption pattern of end users is more variable and potentially changeable

compared with the relatively fixed production mode of power supply. We initiate the study of a new

model of incentives and utility-customer interaction. To reduce volatility, whilst yet accounting

for end-user constraints (e.g., washer must be run only during the day), we introduce an inter-

temporal characteristic to the customer ‘bidding’ language. More importantly, we ensure that the

customer and the utility will both be no worse off under our scheme than under flat-rate pricing.

Our temporal load balancing mechanism, SmartShift, does this by rewarding consumers who shift

consumption with increased allocations for the same cost. Consumers are paid in kind and not cash,

2


advantaging both the consumer and the producer. For real-world adoption it is critical to devise

mutually beneficial schemes, such as Smartshift, that increase the economic pie within the world of

electrical power.

1.1.2 Fault-tolerant spatial load balancing for data centers

Cloud computing has established itself as a mainstay of modern computing infrastructures.

Clouds enable the efficient utilization of resources on an as-needed basis by dynamically configur-

ing these resources to accommodate varying workload needs. The core technology at the heart of

public cloud data-centers and private clusters is virtualization. Virtualization is the use of shared re-

sources to create and operate virtual machines (VMs) [61]; a VM is an operating system or software

that emulates the behavior of a computing system with a specified set of resource characteristics,

such as CPU and memory capacity. Virtualization allows for the execution of an application onto

heterogeneous systems as well as multiple applications in parallel. Virtualization also enables live

migration or the movement of running VMs between hosts. The enormous flexibility afforded by

virtualization enables the placement (mapping of VMs to physical hosts) and re-balancing of VMs

for a variety of reasons including performance and cost-efficient resource utilization [64].

In this thesis, we study spatial load balancing where computational loads (VMs) are distributed

to a set of host machines (locations for VMs). Our work is different in context with those works

on spatial load balancing in related literature [68, 53]. They consider a larger spatial/geographical

range. Instead of studying the inter data center load balancing problem, we focus on intra data

center load balancing with fault-tolerance concerns.

An important aspect of fault-tolerance in a cluster is High Availability (HA) [54]. In general,

HA is the property of ensuring continuity of services (applications) despite the failures of hosts,

VMs or the applications themselves. In the context of this paper we will interpret HA narrowly as

the ability to tolerate host failures by restarting the VMs (from the failed hosts) in back up hosts

without affecting any other existing running VMs. We define k-HA to be the property of a placement

to tolerate the failure of up to k hosts (sequentially or in parallel). In practice the response to a failure

or multiple failures the system could involve restarting of the failed VMs in other hosts; in platforms

with HwPFA (Hardware Predicted Failure Analysis alerts it could also involve live migration of the

VMs in advance of an impending failure. In either case the key algorithmic requirement is ensuring

the sufficiency of resources in the back up hosts to support the new VMs. In general resources are

multi-dimensional (such as CPU, memory, IO etc) but in this part of work we focus on a single

3


resource - memory.

HA has been studied extensively both in the algorithms and the systems communities. Un-

fortunately, even the special of case of 0-HA with a single resource is the probem of bin packing

and hence already NP-complete [34]. Partly as a reaction to the hardness of HA the systems com-

munity has turned to polynomial-time heuristics on the one hand or AI (artificial intelligence) and

CSP (Constraint Satisfaction Programming) on the other. Heuristics, the choice of industry, are

typically quick to return some solution but fail to provide guarantees on the quality of the solution.

AI and CSP, areas of academic research, allow much greater expressiveness involving additional

constraints such as affinity and co-location requirements as well as multiple resource dimensions.

However, both of these techniques are very slow, particularly CSP which solves SAT (satisfiability,

an NP-complete problem [34]) at its core.

In spite of the attention that HA has received the fundamental theoretical questions still remain

unanswered: Given VMs (with sizes) and hosts (with capacities) and k how easy/hard is it to decide

whether there exists a feasible k-HA placement? Given a placement and k how easy/hard is it to de-

cide whether it (the given placement) is k-HA? And, on the practical side, the fundamental questions

are: Given that typical inputs encountered in practice are not adversarially chosen (worst-case) how

do we meaningfully compare the quality of different heuristics? Given that load-balancing is a basic

customer requirement what is the best heuristic to use? In this thesis we answer these questions with

our theoretical study and experiment results.

1.1.3 Load balancing for multidimensional resources

The study of the load balancing problem with multidimensional resources in this thesis is

inspired by the following industrial scenarios.

Scenario 1. Minimizing bottleneck usage: Consider an enterprise customer that has a choice

of several different cloud providers at which to host their VMs (virtual machines). The requirements

of each VM can be characterized along several different resource dimensions such as compute

(CPU), network (latency, bandwidth), storage (memory, disk) and energy. When different virtual

machines are placed in the same elastic resource pool (cloud), their load across each dimension

is accrued additively (though, of course the different dimensions can be scaled suitably to make

them comparable). However, the various resources that cloud providers are equipped with are not

unlimited. They are capped in various settings. The question then arises - what is the optimal way

for the enterprise customer to distribute the load amongst the different cloud providers so as to

4


minimize bottleneck usage?

Scenario 2. Minimizing maintenance downtime: Hosts, and even data centers need to be

powered down every so often for maintenance purposes, e.g. installing a new HVAC system in a

data center. Given this reality, how should the application (collection of virtual machines and/or

containers collectively performing a task or service), be allocated to the different data centers so

as to minimize the aggregate disruption? This scenario also applies to industrial machines where

different factories (or floors of a factory) need to be shut down for periodical maintenance work.

Scenario 3. Preserving privacy: Consider a set of end-users each with its own (hourly)

traffic profile accessing an application. We wish to partition the application components across a

set of clouds such that by observing the (hourly volume of) traffic flow of any single cloud it is

not possible to infer which components are colocated there. This leads to the following question -

how should we distribute load across clouds in order to minimize the maximum hourly variation in

aggregate traffic? As an analogy, the situation here is similar to the problem of grouping households

such that the variation of energy usage of a group is minimized making it difficult for thieves to

infer who has gone on vacation.

Scenario 4. Burstable billing: Most Tier 1 Internet Service Providers (ISPs) use burstable

billing for measuring bandwidth based on peak usage. The typical practice is to measure bandwidth

usage at regular intervals (say 5 minutes) and then use the 95th percentile as a measure of the

sustained flow for which to charge. The 95th percentile method more closely reflects the needed

capacity of the link in question than tracking by other methods such as mean or maximum rate.

The bytes that make up the packets themselves do not actually cost money, but the link and the

infrastructure on either end of the link cost money to set up and support. The top 5% of samples

are ignored as they are considered to represent transient bursts. Burstable billing is commonly used

in peering arrangements between corporate networks. What is the optimal way to distribute load

among a collection of clouds, public and private, so as to minimize the aggregate bandwidth bill?

The above scenarios constitute representative situations captured by the uncapacitated multi-

dimensional load assignment problem framework - VITA. A host of related problems from a variety

of contexts can be abstracted and modeled as VITA(F): the input consists of n, d-dimensional load

vectors V = {Vi|1 i n} and m cloud buckets B = {Bj |1 j m} with associated weights

wj and assignment constraints represented by a bipartite graph G = (V [ B,E ✓ V ⇥ B) that

restricts load Vi to be assigned only to those buckets Bj with which it shares an edge. Here, F can

be any (projection) operator mapping a vector to a scalar, such as max, min, etc. Then the goal is

5


to partition the vectors among the buckets, respecting the assignment constraints, so as to minimize

X

j

wj ⇤ F (X

Vi2Bj

Vi)

where, in a slight abuse of notation, we let Bj also denote the subset of vectors assigned to bucket

Bj . VITA stands for Vectors-In-Total Assignment capturing the problem essence - vectors assigned

to each bucket are totaled. Unless otherwise specified we use i to index the load vectors, j to index

the cloud buckets and k to index the dimension. We let Vi(k) denote the value in the k’th position

of the vector Vi.

We now explain how VITA(F) captures the aforementioned scenarios. In general, dimensions

will either represent categorical entities such as resources (e.g., CPU, I/O, storage, etc.,) or time

periods (e.g., hours of the day or 5-minute intervals, etc.,). We gently remind the reader to note that

in each of the scenarios the elasticity of the clouds is a critical ingredient so that contention between

vectors is not the issue. The set of scenarios we present are but a small sample to showcase the

versatility and wide applicability of the VITA framework.

Scenario 1 is captured by having a vector for each VM, with each dimension representing its

resource requirement; constraints representing placement or affinity requirements [41], weights wj

representing the rates at different cloud providers. Then minimizing the sum of peak resource usage

at each cloud is just the problem VITA(max).

In Scenario 2 each dimension represents the resource (say, CPU utilization) consumed by the

application in a given time period, e.g. the vector for an application could have 24 dimensions one

for each hour in the day. Once the application is assigned to a data center (or cloud or cluster) it

is clear that disruption is minimized if the maintenance downtime is scheduled in that hour where

total resource utilization is minimum. Then minimizing the aggregate disruption is captured by the

problem VITA(min).

The dimensions in Scenario 3 are the hours of the day and the resource in question is the

traffic. To prevent leakage of privacy through traffic analysis the goal is to distribute the application

components across clouds so that the range between the peak and trough of traffic minimized. This

problem is exactly represented as VITA(max�min).

In Scenario 4, we have vectors for each application with 20 dimensions one for each 5th per-

centile [67, 66] or ventile of the billing period. Then minimizing the aggregate bandwidth bill under

the burstable billing method is VITA(2ndmax).

6


1.2 Contributions

This thesis first proposes an incentive mechanism for the temporal load balancing problem in

electricity market. For data centers, this thesis studies high available spatial load balancing with

a given expected faulty tolerance level. Further this thesis proposes a general framework studies

the case for load balancing problems when multidimensional resources are considered, where the

temporal load balancing in electricity market and the spatial load balancing in data centers are

naturally two applied scenarios of this framework. These three parts are described in more detail as

follows.

• Incentive mechanism based temporal load balancing for electricity market: Peak demand

and supply-demand imbalance are the two major problems in electricity markets. In order to

mitigate the peak demand problem, real-time pricing is advised to incentivize customers to

change their usage pattern. However, reacting to the real-time pricing, risk-averse electricity

consumers will scale back on their electricity demand. This will lead to the overall decrease

in production and consumption and reduced economic efficiency. We propose SmartShift, an

incentive mechanism that motivates consumers to move their demands from peak hours in ex-

change for an expanded electricity consumption in non-peak hours. We show that SmartShift

increases consumption and consumer welfare, meanwhile increases the profits of distribution

companies.

• High available spatial load balancing for data centers: k-HA (high availability) is a fault

tolerance property of virtual machine placement in clouds, characterizes the capability of

tolerating up to k host machine failures by relocating VMs from failed hosts to still-running

ones without disturbing other VMs. We show that k-HA is NP-complete. We propose a

stochastic model for multiple knapsack for comparing the efficiencies of different polynomial-

time heuristics. We also prove that there exists a best polynomial time heuristics. Based on

industrial cluster workload dataset, we show that Water-filling performs the best among a list

of common heuristics favored in industry.

• Load balancing for multidimensional resources: we study on load balancing for multidi-

mensional resources. We first present the Uncapacitated Multidimensional Load Assignment

problem, then we propose VITA(F) (Vectors-In-Total Assignment): the input consists of n,

d-dimensional load vectors V = {Vi|1 i n} and m cloud buckets B = {Bj |1 j

m} with associated weights wj and assignment constraints represented by a bipartite graph

7


G = (V [ B,E ✓ V ⇥ B) that restricts load Vi to be assigned only to those buckets Bj

with which it shares an edge. Here, F can be any operator mapping a vector to a scalar, such

as max, min, etc. Then the goal is to partition the vectors among the buckets, respecting the

assignment constraints, so as to minimizeP

j wj ⇤F (P

Vi2BjVi), where, in a slight abuse of

notation, we let Bj also denote the subset of vectors assigned to bucket Bj . We also let Vi(k)

denote the value in the k’th position of the vector Vi.

We characterize the complexity of VITA - providing approximation algorithms and hardness

results. Our approach involves clever rounding of carefully crafted linear programs and may

be of independent technical interest.

1.3 Outline

The rest of chapters are outlined as follows. Chapter 2 presents SmartShift, an incentive mech-

anism based temporal load balancing for electricity market. Chapter 3 studies high available spatial

load balancing for cloud computing in data centers. Chapter 4 presents the load balancing problems

with multidimensional resources. Chapter 5 concludes the thesis.

8

Chapter 2

Temporal Load Balancing for ElectricityMarket

Peak demand for electricity continues to surge around the world. The supply-demand imbal-

ance manifests itself in many forms, from rolling brownouts in California to power cuts in India.

It is often suggested that exposing consumers to real-time pricing, will incentivize them to change

their usage and mitigate the problem - akin to increasing tolls at peak commute times. We show

that risk-averse consumers of electricity react to price fluctuations by scaling back on their total

demand, not just their peak demand, leading to the unintended consequence of an overall decrease

in production/consumption and reduced economic efficiency. We propose a new scheme that allows

homes to move their demands from peak hours in exchange for greater electricity consumption in

non-peak hours - akin to how airlines incentivize a passenger to move from an over-booked flight in

exchange for, say, two tickets in the future. We present a formal framework for the incentive model

that is applicable to different forms of the electricity market. We show that our scheme not only

enables increased consumption and consumer social welfare but also allows the distribution com-

pany to increase profits. This is achieved by allowing load to be shifted while insulating consumers

from real-time price fluctuations. This win-win is important if these methods are to be embraced in

practice.

2.1 Introduction

Power utilities worldwide face at least two major challenges. The first is Peak Demand - a

period in which the demand for power is significantly higher than average. In order to satisfy a large

9

CHAPTER 2. TEMPORAL LOAD BALANCING FOR ELECTRICITY MARKET

peak demand, utilities (generation/distribution companies) have to make large capital investments

including new ‘peaking’ generation stations, larger capacity lines, transformers, and operational

expenditures including expensive purchases of electricity on the “spot market” [38]. For example,

it is estimated that a 5% lowering of demand would have resulted in a 50% price reduction during

the peak hours of the California electricity crisis in 2000-2001 [44]. As a result of the quadratic

dependence between resistive losses and transmitted current, peaks also lead to substantial energy

wastage. The second challenge faced by power utilities is that of Supply-Demand imbalance. [16]

states that ”the difficulties that have appeared in California and elsewhere are intrinsic to the design

of current electricity markets, in which demand exhibits virtually no price responsiveness and supply

faces strict production constraints”.

Research at the intersection of computer science and economics has established that incentives

are a powerful way to allocate scarce resources [24]. Incentive mechanisms incorporating infor-

mation exchange require back and forth communication between the consumers and the producers.

Conventional electricity infrastructure is not designed to support such a dialog, instead relying on

unilateral actions by the producers such as price-regulation and load-shedding. Fortunately, the in-

troduction of new communication [74] and control [31] infrastructure will allow increased consumer

participation in the smart grid.

Traditionally, economists and electricity companies have focused on pricing as a mechanism

to solve these two problems in a systematic way. However, homes and small businesses have an

inherent and systemic requirement for stable electricity costs [18]. It is understood that ”Consumers

generally shy away from markets when products are complicated, supply is uncertain, prices are

volatile, and information is lacking” [19]. We demonstrate that exposing risk-averse consumers to

the actual real-time costs of electricity production, will result in a reduction in aggregate demand

leading to reduced revenue for the generators and distributors with potential knock-on effects for the

economy at large. At the same time, the volatility of real-time pricing can have strong effects on grid

stability [70]. A second concern with real-time pricing, is that consumer prices may increase [8] or

net electricity consumption may decrease [9], depending on the model. We initiate the study of a

new model of incentives and utility-customer interaction. To reduce volatility, whilst yet accounting

for end-user constraints (e.g., washer must be run only during the day), we introduce an inter-

temporal characteristic to the customer ‘bidding’ language. More importantly, we ensure that the

customer and the utility will both be no worse off under our scheme than under flat-rate pricing. Our

mechanism, SmartShift, does this by rewarding consumers who shift consumption with increased

allocations for the same cost. Consumers are paid in kind and not cash, advantaging both the

10


consumer and the producer. For real-world adoption it is critical to devise mutually beneficial

schemes, such as Smartshift, that increase the economic pie within the world of electrical power.

We design consumer-supplier interaction protocols that respect the physical constraints and

characteristics of the grid. Such protocols must account for realistic models of consumer behavior

and requirements, while being computationally tractable to allow practical implementation in grids

with millions of customers. Our main contributions are:

• SmartShift - a novel incentive mechanism, consistent with utilities’ obligation to serve [19],

that allows risk-averse homes to move their demands from peak hours in exchange for greater

electricity consumption in non-peak hours (SmartShift section).

• demonstration of the practical benefits of SmartShift through extensive simulations (Simula-

tions section).

• characterization of the computational complexity of SmartShift (SmartShift section).

• formalization of the folklore that purely price driven approaches have negative consequences

in a world with risk-averse consumers (SmartShift section).

2.2 Background

We study a market model with distribution companies (or utilities - the suppliers of electric-

ity) and customers (consumers of electricity), each of whom is exposed to different prices, with

associated risks and properties.

Prices seen by customers: Flat-rate pricing is the de-facto standard for retail electricity, where

every unit costs the same fixed amount. Homes for example prefer this, particularly since, unlike

businesses that may experience revenues in line with consumption growth, they have a strong inbuilt

preference for stability in the costs. However, flat-rate pricing lacks any incentive for households to

make rational usage decisions, especially during peak hours. We show that under flat-rate pricing

risk-averse consumers will have high welfare due to lack of volatility but, (1) it cannot eliminate the

peak load problem, since there is no slot-wise price differentiation, and as a result (2) the distribution

company may face profit fluctuations.

With real-time pricing, peak reduction is achieved by charging consumers higher rates when

the overall electricity load is high, encouraging them to reduce consumption [60]. However, as we

will discuss in more detail, the fluctuating nature of market prices will discourage consumption by

11


risk-averse consumers, while also being known to increase grid volatility [70]. Real-time pricing

will result in a decrease of the absolute total consumption of electricity [27] and consequently, to-

tal revenue of the distribution company (utility). Participation in most voluntary real-time pricing

(RTP) programs has decreased, in particular ”Between 2000 and 2003, half of all programs in exis-

tence prior to 2000 lost 25% or more of their participants, while only two programs saw participation

increase” [12], likely due to price volatility.

Prices seen by the distribution company: The time varying market, procurement or generation

price of electricity can be modeled in different ways. The price can be modeled as endogenous

or exogenous. We use the term Endogenous Market Price to denote a price that depends on the

total electricity consumption at each time slot. This can be a useful model in situations where the

number of consumers is relatively large, with their total consumption affecting market price. The

resulting price is typically modeled as a quadratic function of the total electricity consumption; the

quadratic or piecewise quadratic nature of generation and distribution cost curves is usually due to

a multiplicity of fuel sources and generation modes. See [79, 38] for detailed justification of this

common model.

Exogenous Market Prices are independent of the actions of the distribution company, its cus-

tomers or their consumption. This is usually the case when the number of the responding consumers

are relatively small, or if there is a large background electricity load which dominates an endoge-

nous market price. In this case, we can just treat the real-time market price as a random variable

independent of the decisions made by customers. See [38, 19, 70] for some justification and analysis

of this stochastic pricing model.

Shortcoming of purely price driven approaches: We prove, in the exogenous price model, both

that (i) risk-averse consumers have a lower expected social welfare and that (ii) the distribution

company gets lower expected revenue under real-time pricing when compared to flat-rate pricing.

On the other hand, we point out that the distribution company never makes a loss under real-time

pricing, but can incur losses with some probability under flat-rate pricing. This leads us to develop

a scheme with the advantages of both.

2.3 Approach

We propose SmartShift, an expanded load shifting mechanism, where consumers are incen-

tivized to shift their electricity load from peak-time to non-peak-time in exchange for an expanded

electricity load, while still incurring the same cost. Since consumers self select, their social wel-

12


fare (of consuming electricity) is non-decreasing under this scheme. Observe that from a pragmatic

stand-point the unit of a home allows for the effective use of an in-kind incentive as a tool of be-

havior change. A single appliance/user may not be able/willing to consume the incentive, but an

aggregate of a household will be able to more effectively consume the incentive via coordination

among the household members [26].

In addition to the development of a new in-kind incentive model, our work is distinguished

by a focus on computational issues. We prove that, in the endogenous price model, calculating the

optimal shifted load is NP-complete for both real-time and flat-rate pricing. In contrast, we show in

the exogenous price case that we can calculate the optimal shifted load in polynomial time.

We prove that employing SmartShift increases the expected profit of the distribution company

both under real-time pricing and flat-rate pricing. We employ numerical experiments to show that

it also decreases the probability of scenarios where the distribution company incurs losses, i.e. its

risk. Thus, in the exogenous price model the use of SmartShift with flat-rate pricing provides a

solution to the problems of Peak Load and Supply-Demand imbalance; the resulting electricity load

allocation simultaneously increases consumers’ social welfare, and distribution companies’ profits

and thus is a win-win solution.

2.4 Related work

Evidence from the use of sophisticated usage-based pricing schemes [24] indicates there is a

strong case to be made for managing a grid via incentive schemes. Price based congestion control

schemes in the power systems literature (often based on successful protocols developed for internet

congestion control [51]) that discourage consumption when the grid is loaded, fall into four basic

categories: 1. day ahead or time-of-use (TOU) pricing, 2. dynamic pricing, 3. back-off strategies

and 4. tatonnement (or negotiation).

TOU pricing (particularly when users have significant storage, patience or flexibility) leads to

the herding problem, where large amounts of consumption are shifted to low price regions creating

new peaks [17]. Dynamic pricing, as we discuss, can lead to uncertainity and reduced consumption

by risk-averse customers. Randomized back-off type approaches, where users back-off on consump-

tion when the grid is loaded (motivated by CSMA protocols developed for wireless communication),

require trust that the agent will respond as expected to signals. This is not incentive compatible [14]

as the response strategies are unverifiable and it is in the agents’ interest to choose small back-off

windows which allow them to consume electricity with minimum inconvenience. In Tatonnement

13


[78], consumption agents and the distribution company exchange both price information and con-

sumption profile information until convergence. While they were not explicitly categorized as such,

tatonnement forms the basis of much work in this area including [59, 77, 52, 43]. Since typical

distribution grids could have millions of users and tens of millions of devices, computational com-

plexity is of paramount importance [46]. In this context, the use of in-kind incentives and attention

to computational complexity and risk-aversion makes our model and analysis novel.

2.5 Market model description

We now describe the overall electricity market model, explain the roles of the generation fac-

tory, the distribution company and the end consumers. We describe the two pricing mechanisms of

the distribution company and then explain the two models for the market price of electricity: (1)

Endogenous Market Price; (2) Exogenous Market Price.

Generation companies generate electricity, which is purchased by distribution companies. They

then transfer it to the (n) end consumers, and charge for this service. The distribution company payes

a price per unit of p(t)m in time slot t, and must decide how to charge users - i.e. decide on a pricing

mechanism.

We assume that both the distribution company and the end consumers know the statistical

properties of the market price p(t)m . For the gaussian model we use, the mean µm and the standard

deviation �m suffice. This can be learned, for example, based on the long run price history. In the

short term, i.e. over the period of a single billing cycle, say a day, only the distribution company is

assumed to know the market price p(t)m . A billing cycle is assumed to consist of k slots and without

loss of generality we index t in the next billing cycle as 1 t k. In this work, we study the market

behavior (i.e. social welfare, revenue, etc) in the short term given the known long term statistical

information.

At the beginning of each (short term) billing cycle, the distribution company will anounce

the specific pricing mechanism. The prices seen by the consumers, p(t)c will then be (1) real-time

pricing: p(t)c = p(t)m ; (2) Flat-rate pricing: p(t)c = µm, where µm is the mean of p(t)m .

Risk-averse consumers: utility, valuation: Each consumer i has a valuation function V (t)i : x(t)i !

R which reflects the value consumer i receives when she consumes x(t)i units of electricity at time

slot t. In general, the valuation function could be any concave increasing function and all the results

in this chapter continue to hold at a qualitative level. For sake of concreteness, we assume that

consumers’ valuation function V (t)i (·) has the form V (t)

i = ↵(t)i log x(t)i .

14


Intuitively, the marginal value of consuming a additional unit of electricity will increase more

slowly given the increased consumption of electricity. ↵(t)i can be seen as a parameter specific to

consumer i that pegs the logarithmically increasing valuation curve to the value that consumer i

puts on consuming x(t)i units of electricity at time slot t.

Under flat rate pricing, consumer i estimates her net utility U (t)i of consuming x(t)i units of

electricity at time slot t by U (t)i = V (t)

i � µcx(t)i .

As explained earlier, under uncertain real time pricing, consumers tend to be risk-averse. Risk-

averse consumers will take the additional risk of price fluctuation into consideration when they

estimate their net valuation, i.e., the net utility function for risk-averse consumers should be revised

as U (t)i = V (t)

i � (µc+�i�c)x(t)i ; �c is the standard deviation of the price p(t)c and µc is it’s average

over the k time slots. �c represents the degree of price fluctuation; �i (�i � 0) quantifies the level

of risk-aversion of consumer i. Specifically, �i = 0 captures the situation where consumer i is risk-

neutral while �i < 0 models the risk-averse consumer (�i > 0 models the risk-seeking consumer, a

type rarely seen in markets).

Electricity consumption, social welfare and profit: We apply our incentive mechanism to the two

different pricing mechanisms - real-time and flat-rate - of the distribution company and analyze the

resulting market behavior.

Real-time pricing: The distribution company charges the consumers a time varying price of p(t)c =

p(t)m over the time slots of the billing cycle. Clearly, µc = µm and �c = �m. With predictions

(possibly gleaned from long run observations) of the mean and variance of the real-time electricity

price, a rational consumer will consume x(t)i units of electricity so as to maximize the net utility

U (t)i . By the first order derivative condition dU (t)

i /dx(t)i = 0, the net utility U (t)i at time slot t

is maximized when x(t)⇤i = ↵(t)i /(µm + �i�m). Thus, risk-averse consumers will consume less

electricity when the real-time market price has a large standard deviation (large fluctuations and

uncertainty in price).

Flat-rate pricing: The distribution company charges the end consumers a fixed price p(t)c = µm

over the time slots of the billing cycle. Since the consumers know that they will be charged a flat-

rate price equal to the mean of the market price they will infer that µc = µm and �c = 0 and will

respond by consuming the amount x(t)⇤i = ↵(t)i /µm (even though they are risk-averse).

Social welfare of consumers: After each consumer is charged price p(t)c at time slot t, the actual

net utility she receives is U (t)0i = V (t)

i � p(t)c x(t)i . And, the social welfare W of all the consumers is

W =kP

t=1

nPi=1

U (t)0i .

15


The profit ⌘ that the distribution company gains is ⌘ =kP

t=1

nPi=1

(p(t)c � p(t)m )x(t)i . ⌘ < 0 means

that the distribution company incurs losses with respect to its expected margins; ⌘ > 0 means that

the distribution company profits. Note that in general, in a monopoly, a company can ensure profits

by overcharging (increasing p(t)c arbitrarily. In that case ⌘ quantifies the difference to expected

profits. In addition, in most markets, electricity companies are at minimum semi-regulated and

required to fix an operating margin. We avoid these complications by setting p(t)c = p(t)m .

Recall that the market price p(t)m can be either endogenous or exogenous.

Endogenous market price: When the total consumption of a group of end consumers is large

enough to dominate the overall electricity load of the grid, then the cost of generating electricity

depends on those end consumers’ total consumption. So does the market price p(t)m . Concretely,

we assume p(t)m has the form: p(t)m = Q(nP

i=1x(t)i ), and in particular that Q(·) has a quadratic form,

i.e. Q(x) = ax2 + bx + c. In this case, due to the convex nature of the problem, the market

equilibrium is the fixed point of the interaction between the consumers and the supplier. It can

hence be computed iteratively: starting with an initial estimate for mean µm and standard deviation

�m, the end consumers respond with their electricity demand {x(t)i }. Based on this response, i.e.

{x(t)i }, the endogenous market prices are generated by the quadratic function Q(nP

i=1x(t)i ). These

prices are then used as input for the next round of computation until convergence is achieved.

Exogenous market price: When the total consumption of a group of end consumers is too small to

meaningfully affect the overall electricity in the grid, then the market price p(t)m can be assumed to be

independent of consumers’ consumption. Under the exogenous model, we assume p(t)m (1 t k)

are independent random variables drawn from a normal distribution p(t)m ⇠ N(µm,�2m).

2.6 SmartShift

SmartShift is an alternative, expanded load shifting, incentive mechanism that addresses the

challenges of peak demand and the supply-demand imbalance detailed in the Introduction. In order

to guide the design of our mechanism, we first ask ourselves two questions: (1) What incentivizes

the consumers to shift their electricity load? (2) What goal will the distribution company achieve

after it offers the incentivies to the consumers? The answer to these two questions will provide a

win-win solution to both consumers and the distribution company. We then study how to achieve

the optimal solution under (1) Endogenous (2) Exogenous market price.

What is the incentive for consumers to shift load?: Each rational consumer i will only agree to

16


shift her consumption x(s)i from time slot s to time slot t, if her valuation does not decrease. This

gives us the following result,

Theorem 1. With the same payment p0, consumer i will have no decrease in utility, if she shifts x(s)i

from time slot s to time slot t with expansion, which results in an expanded load of m(s!t)i x(s)i at

time slot t. The necessary and sufficient condition, therefore is for the expansion ratio m(s!t)i to

satisfy the following:

m(s!t)i � max{exp(

(↵(s)i � ↵(t)

i ) log x(s)i

↵(t)i

), 1}. (2.1)

Proof. Since the payment does not change after consumer i gets her expanded load at a new time

slot, we only need to guarantee that after shifting her net utility does not decrease. By assuming

that the original electricity consumption at different time slots are heterogenous (the consumption

at different times can not be added up as a whole), we then just need to make sure the following is

satisified.

↵(t)i log(m(s!t)

i · x(s)i ) � ↵(s)i log x(s)i

m(s!t)i � exp(

(↵(s)i � ↵(t)

i ) log x(s)i

↵(t)i

)(2.2)

Meanwhile, m(s!t)i should be no less than 1, since the original electricity load demand should

be satisfied. Thus, we get Eq.(2.1).

This gives us a lower bound on the expansion ratio of a consumer. How close a user’s expansion

ratio is to her baseline is a measure of her tolerance or flexibility. The smaller the required expansion

ratio, the greater the tolerance. Conversely, the larger the expansion ratio the more intolerant is the

consumer of the slot change.

What is the benefit to the distribution company?: The distribution company will agree to shift

consumer i’s load x(s)i from time slot s to slot t with an expansion ratio m(s!t)i for the same payment

only if it benefits the distribution company to do so. For load x(s)i , the distribution company obtains

a profit ⌘ = (p(s)c � p(s)m )x(s)i . After the expanded load shifting to time slot t, the profit becomes

⌘0 = p(s)c x(s)i � p(t)m m(s!t)i x(s)i . So long as p(s)m > p(t)m ·m(s!t)

i , the distribution company stands to

increase its profit (⌘0 � ⌘ > 0) and hence will agree to the expanded load shifting.

17


Protocol 1 The message exchange protocol for SmartShiftSTEP 1: The distribution company and the end consumers exchange information, or observe com-

mon historical data, to achieve consensus on the long term mean µm and standard deviation �m of

the market price.

STEP 2: The n end consumers communicate their consumption {x(t)i |1 i n, 1 t k} over

the next period of k time slots, and their expansion ratios (which must be satisified if their loads

would be shifted) {m(s!t)i |1 i n, 1 s k, 1 t k} to the distribution company.

STEP 3: The distribution company computes the optimal load shifting and communicates the re-

sulting load allocation to the end consumers.

Our expanded load shifting incentive mechanism, SmartShift, meets both these requirements

and is supported by a protocol that exchanges messages between the distribution company and the

end consumers in the following sequence:

We now study how to achieve optimal load shifting in both cases - endogenous and exogenous

prices.

Endogenous market price: optimal load shifting

Theorem 2. Computing optimal load shifting under endogenous market price is (weakly) NP-

complete.

Proof. It is clear that the optimal shifting problem is in NP. So all that remains is to show that it is

NP-hard. The proof is by reduction from the Partition Problem [33] which is (weakly) NP-complete.

In the Partition problem we are given a set of positive numbers {a1, a2, . . . , an} summing to 2S and

decision question is whether there exists a partition of the set into two subsets such that the numbers

in each subset sums to S. To reduce a given instance of the partition problem to the endogenous price

case we assume there are n consumers and two time slots; the i’th consumer wishes to consume ai

units of electricity and is indifferent as to which slot she consumes it in; the price of electricity in

a slot is just the load. Now it is easy to see that a partition such that each subset sums to S exists

if and only if the total revenue is 2S2; if such a partition does not exist then the total revenue will

strictly exceed 2S2.

In the endogenous case, both real-time pricing and flat-rate pricing need to solve the following

quadratic integer programming problem. This problem is NP-complete[39].

Let s(t0!t)

i 2 {0, 1} denote that consumer i is assigned to shift her load from t0 to t if s(t0!t)

i =

1; otherwise she is not assigned. Then the total load L(t) at slot t is

18


L(t) =nX

i=1

kX

t0=1

m(t0!t)i x(t

0)i s(t

0!t)

i (2.3)

minimize{s

(t0!t)i }

kX

t=1

a(t)(L(t))2 + b(t)L(t) + c(t)

subject to s(t0!t)

i 2 {0, 1}, 8i, t, t0,X

t

s(t0!t)

i = 1, 8i, t0,

1 i n,

1 t, t0 k,

i, t, t0 2 Z.

(2.4)

where, a(t), b(t), c(t) are the coefficients of function Q(·) when endogeous market price is

determined.

With the optimal solution {s(t0!t)

i } of Eq. (2.4), we can get the real-time load {L(t)} by Eq.

(2.3). By applying function Q(·) on {L(t)}, we can get the real-time prices {p(t)m }. Then, under

real-time pricing, the customers are charged at the rate of p(t)m at time slot t; under flat-rate pricing

they are charged at the rate of µm.

The iterative process of generating the market equilibrium when employing SmartShift under

endogenous market prices can be described in Alg. 2.

It is easy to create examples in the endogenous market price case where an equilibrium (in

terms of prices and load distribution) may not exist. Consider the scenario where if a user doesn’t

consume a unit of electricity then the resultant lower price makes it favorable for her to in fact

consume the unit; but, in doing so, the load is driven up and along with it the price (which is

dependent quadratically on the load) thus making it unfavorable to consume the additional unit.

Thus, determining endogenous prices is an inherently hard problem.

We make a further comment, for the special setting where the consumption of electricity is

discrete, i.e. users may shift entire appliances or units but not arbitrary amounts. In this setting it

is not hard to create examples, where an equilibrium (in terms of prices and load distribution) may

not exist. Suppose utilities and prices are such that, if a user doesn’t consume a unit of electricity

then the resultant lower price makes it favorable for her to in fact consume the unit; but, in doing so,

the load is driven up and along with it the price (which is dependent quadratically on the load) thus

making it unfavorable to consume the additional unit. Thus, determining endogenous prices with

19


Algorithm 2 Optimal Load Shifting under Endogenous Market Price

Input: {↵(t)i |1 i n, 1 t k}, {m(s!t)

i |1 i n, 1 s k, 1 t k}, µm, �m.

Output: {x(t)i |1 i n, 1 t k}, {s(s!t)i |1 i n, 1 s k, 1 t k}

1: repeat2: for each consumer i do3: for each time slot t do4: Calculate x(t)i by real-time pricing or flat-rate pricing;

5: end for6: end for7: Solve the Nonlinear Integer Optimzation Problem in Eq. (2.4), get optimal solution {s(s!t)

i };

8: Calculate {L(t)} by Eq. (2.3);

9: Apply function Q(·) on {L(t)} to get the endogenous market prices {p(t)m };

10: update µm, �m11: until µm and �m converge

all or nothing or discrete consumption is an inherently hard problem where an equilibrium may not

exist.

Exogenous market price: optimal load shifting In the exogenous market price case, the load

shifting will never affect the market price {p(t)m }. To achieve the optimal load shifting, for each

consumer i, her load x(s)i should be shifted to the time slot t⇤ = z(t)i at which p(t⇤)

m ·m(s!t⇤)i is the

minimum over all time slots {t⇤}. Computing optimal load shifting under exogenous market price,

as Alg. 3, runs in O(nk2) time.

Exogenous market price: real-time vs flat-rate: We now study how the two pricing mechanisms

(real-time pricing and flat-rate pricing) have different performance on social welfare, revenue and

profit in the model of exogenous market price. Specifically, we have the following theorems.

Theorem 3. Given exogenous market prices, the expected social welfare W in real-time pricing is

less than the expected social welfare W 0 in flat-rate pricing.

Proof. We only need to show that at each time slot t each consumer i’s expected utility E(U (t)i )

under real-time pricing is less than her exptected utility E(U (t)0i ) under flat-rate pricing. Then, since

the expectation of social welfare is just the sum over the expectation of each consumer’s valuation

at each time slot, we get the conclusion trivially.

20


Algorithm 3 Optimal load shifting under exogenous market price

Input: {x(t)i |1 i n, 1 t k}, {m(s!t)i |1 i n, 1 s k, 1 t k}.

Output: {z(t)i |1 i n, 1 t k}.

1: for each consumer i do2: for each time slot t do3: z(t)i argmin

1t0k{m(t!t0)

i p(t0)

m };

4: end for5: end for

Under the real-time pricing mechanism, each consumer i will respond with the consumption

x(t)⇤i =↵(t)i

µm+�i�m. The expected utility for consuming x(t)⇤i electricity is

E(U (t)i ) = E(↵(t)

i log x(t)⇤i � p(t)m x(t)⇤i )

= ↵(t)i log x(t)⇤i � E(p(t)m )x(t)⇤i

= ↵(t)i log x(t)⇤i � µmx(t)⇤i

(2.5)

Under the flat-rate pricing mechanism, each consumer i will respond with the consumption

x(t)⇤0i =↵(t)i

µm. In this case, the expected utility is

E(U (t)0i ) = ↵(t)

i log x(t)⇤0i � µmx(t)⇤0i(2.6)

The function f(z) = ↵ log z � µz achieves maximum at z = ↵µ . Thus, we can conclude that

E(U (t)i ) < E(U (t)0

i ).

Theorem 4. Given exogenous market prices, the expected revenue RRT in real-time pricing is less

than the expected revenue RFR in flat-rate pricing.

Proof. We continue using the notation from the previous proof. The revenue RRT and RFR under

two different pricing mechanisms satisfy

21


E(RRT ) = E(nX

i=1

kX

t=1

p(t)m x(t)⇤i )

=nX

i=1

kX

t=1

µmx(t)⇤i

<nX

i=1

kX

t=1

µmx(t)⇤0i

= E(RFR)

(2.7)

Theorem 5. Given exogenous market prices, real-time pricing always leads to a zero profit (⌘RT =

0), while flat-rate pricing leads to a zero profit in expectation (E(⌘FR) = 0).

(Recall that profit and loss refer to money gained or lost beyond the expected or pre-set mar-

gins.)

Proof. In real-time pricing, the charging price p(t)c always equals p(t)m . Thus, ⌘RT is exactly zero. In

flat-rate pricing, the charging price p(t)0c = µm. Then, the expectation of ⌘FR is

E(⌘FR) = E(X

i

X

t

(µm � p(t)m ))x(t)i )

=X

i

X

t

(µm � E(p(t)m ))x(t)i

= 0

(2.8)

Theorem 5 shows that under real-time pricing the distribution company always achieves the

anticipated profit per electricity unit; under flat-rate pricing the distribution company achieves the

anticipated profit per electricity unit on average (in expectation). In other words, under flat-rate

pricing the distribution company will sometimes incur a loss. This actually happens roughly about

50% of time, if the exogenous market price is assumed to be drawn from a normal distribution. In

summary, flat-rate pricing mechanism has the advantage of encouraging consumers to consume the

electricity (larger social welfare) they need without being constrained by considerations of price

risk, but this risk is then shifted to the distribution company which could incur losses, i.e. ⌘ < 0, in

22


some cases. However, we show, using simulations, that employing our expanded load shifting incen-

tive mechanism, SmartShift, with flat-rate pricing reduces the probability of such (loss-incurring)

scenarios.

2.7 Simulations

We evaluate SmartShift using a real-time electricity load data set from Smart* [13]. Through

numerical experiments, we then study how our expanded load shifting mechanism can improve

the performance over flat-rate pricing while providing a win-win solution to both the distribution

company as well as the consumer.

We obtain real-time electricity load {x(t)i } from the Microgrid Data Set [13] which includes

average real power usage (kilowatts) at one minute sampling rate from 400 homes for 24 hours

(k = 24). We average the (per minute) sampled data points per hour to get the hourly averaged

electricity usage. We filter out households with zero power usage resulting in n = 395 profiles.

We simulate the real-time market price {p(t)m } using a normal distribution N(µm,�2m) moti-

vated by the model of [69]. We obtained similar results for other ditributions. We fix µm and vary

�m to study how the fluctuation of the price will affect the market. We generate valuation coeffi-cients {↵i} as follows: We assume that the electricity load profiles are generated under a flat-rate

pricing model with mean price µm = 50. Thus, ↵(t)i = x(t)i ·µm. Similar results were obtained with

other settings. We draw the risk-aversion parameter {�i}, from a Pareto distribution as follows:

fpdf (�i) =�R · �min

��R+1i

(2.9)

where fpdf gives the probability density function for the Pareto distribution; �min sets the minimum

value for �i; �R controls how much of the probability mass is close to �min, when �R is larger, there

is a larger probability mass of �i that is close to �min. To generate the expansion ratio {m(s!t)i },

we first use Eq. 2.1 to calculate the minimum value m(s!t)i,min . Then, we draw m(s!t)

i from a Pareto

distribution:

fpdf (m(s!t)i ) =

�G ·m(s!t)i,min

(m(s!t)i )�G+1

(2.10)

As before, the larger �G is, the larger the probability that m(s!t)i is close to the minimum

expansion ratio m(s!t)i,min . Thus, �G controls the dispersion in the tolerance of risk-averse consumers;

the larger �G is the more tightly concentrated is the population around a high tolerance level.

23


0 1 2 3 4 5 6 7 8 9 10 11

0.7

0.8

0.9

1

1.1

1.2

1.3

Standard Deviation of Market Price

Expe

cted

Rev

enue

(nor

mal

ized

) SmartShiftFlat�rateReal�time

Figure 2.1: Normalized revenue vs price volatility

Risk-aversion and pricing mechanism: In all our simulations, each point is averaged over 100

repeated experiments. 95% confidence interval is also shown. In Figure 2.1, we draw the risk-

aversion parameter {�i} from the Pareto distribution, Eq. (2.9), by setting �R = 1 and �min = 1.

These are fixed for subsequent repetitions. Figure 2.1 shows that, while real-time pricing induces

risk-averse consumers to contract usage with increased volatility (Theorem 4), SmartShift increases

total revenue by simultaneously buffering consumers from price shocks and exploiting the increased

volatility to more optimally shift the load.

In Figure 2.2, we fix the fluctation of the price by setting µm = 50,�m = 5, but vary the

tolerance of the consumers by varying the �G from 0.5 to 5 with step size of 0.5. For each �G, we

sample a set of prices from normal distribution N(µm,�2m) where µ = 50,�m = 5, then sample

the {m(s!t)i } from Eq. (2.10) with the fixed {m(s!t)

i,min } and the varying �G. We show that the

larger �G is (i.e. the more tightly clustered consumers are around a high tolerance level) the less

the probability of the distribution company incurring a loss ⌘ < 0. This relationship follows from

the fact that when the desired expansion ratio m(s!t)i is small, it is more probable for the expanded

load shift to be satisfied.

In Figure 2.3, we fix the price fluctuation, risk-aversion and tolerance of the consumers. We

sample prices from N(µm,�2m) where µm = 50,�m = 5, risk-aversion from Eq. (2.9) where

24


0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Pareto Distribution Parameter �G

Prob

abili

ty o

f Los

s

Flat�rateSmartShift

Figure 2.2: Loss probability vs consumer tolerance

�R = 1,�min = 1, and tolerance from Eq. (2.10) where �G = 1 and m(s!t)i,min computed by Eq.

(2.1). By sampling 1000 sets of prices, we compute the profit ⌘ distribution. Figure 2.3 shows

that the probability mass moves to the right after applying SmartShift. This also demonstrates that

SmartShift reduces the probability of the distribution company incurring a loss (⌘ < 0).

Fig. 2.4 shows that, whereas real-time pricing induces risk-averse consumers to contract usage

with increased volatility (Theorem 3), Smartshift increases total social welfare by simultaneously

buffering consumers from price shocks and exploiting the increased volatility to more optimally

shift the load. We see that for a fixed set of {�i}, the larger the standard deviation (fluctuation) of

the market price, the smaller the total social welfare of consumers under real-time pricing. Under

flat-rate pricing, the real-time change of the market price does not affect the social welfare.

In Fig. 2.5 we fix the extent of market price fluctuation. We draw the 24 slots of market prices

from the normal distribution N(µm,�2m) where µm = 50,�m = 5. We keep the prices fixed for

subsequent repetitions. We vary �R from 0.5 to 5 with step size of 0.5. For each �R we draw a set

of {�i}, then we compute the social welfare and revenue under both flat-rate and real-time pricing.

Every point is averaged over 100 repeated experiments. 95% confidence interval is also shown in

the results.

Fig. 2.5(a) shows that the larger the �R is, the closer the expected social welfare under real-

25


�2 �1 0 1 2 3x 105

0

0.01

0.02

0.03

0.04

0.05

Profit �

Prob

abili

ty

Flat�rateSmartShift

Figure 2.3: Probability density function of profit

time pricing will be to the expected social welfare under flat-rate pricing. This is because the larger

�R is, the more concentrated �i is around �min the maximal tolerance level, which makes the risk-

averse consumers use more electricity and leads to a larger social welfare. Under SmartShift the

same consumers utilize even more electricity because of the optimal shifting of load from peak to

non-peak slots, leading to an even larger social welfare. Fig. 2.5(b) shows the same trend for the

expected revenue under both the pricing models.

2.8 Summary

We have presented a general incentive-based mechanism, SmartShift, for reducing the load on

the electricity grid. Our scheme grants users increased consumption in exchange for reducing their

usage in peak periods. We have shown analytically that SmartShift under flat-rate pricing is a win-

win for both consumers (increased social welfare) and producers (enhanced profits). SmartShift

has elements of algorithms for iterative price setting [59, 46] with the added features of in-kind

incentives and a slot-pairwise bidding language. A separate study of each these effects is of interest.

26


0 2 4 6 8 100.9

0.95

1

1.05

1.1

1.15

1.2

1.25

Standard Deviation of Market Price

Expe

cted

Soc

ial W

elfa

re (n

orm

aliz

ed)

SmartShiftFlate�rateReal�time

Figure 2.4: Normalized social welfare vs price volatility

27


0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 60.75

0.8

0.85

0.9

0.95

1

1.05

1.1

1.15

Pareto Distribution Parameter �R

Expe

cted

Soc

ial W

elfa

re (n

orm

aliz

ed)

SmartShiftFlat�rateReal�time

(a) Expected Normalized Social Welfare

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.50.65

0.7

0.75

0.8

0.85

0.9

0.95

1

1.05

Pareto Distribution Parameter �R

Expe

cted

Rev

enue

(nor

mal

ized

)

SmartShiftFlat�rateReal�time

(b) Expected Normalized Revenue

Figure 2.5: Normalized social welfare and revenue of smartShift, flat-rate pricing and real-timepricing vs Pareto distribution parameter �R

28

Chapter 3

Fault-tolerant Spatial Load Balancing inData Centers

k-HA (high-Availability) is an important fault-tolerance property of VM placement in clouds

and clusters - it is the ability to tolerate up to k host failures by relocating VMs from failed hosts

without disrupting other VMs. It has long been assumed [15] that deciding the existence of a k-HA

placement is ⌃P3 -hard. In a surprising yet simple result we show that k-HA reduces to multiple

knapsack and hence is in NP= ⌃P1 .

We propose a stochastic model for multiple knapsack that not only captures real-world work-

loads but also provides a uniform basis for comparing the efficiencies of different polynomial-time

heuristics. We prove, using the central limit theorem and linear programming, that, there exists a

best polynomial-time heuristic, albeit impractical from the standpoint of implementation.

We turn to industry practice and discuss the drawbacks of commonly used heuristics - First-

fit, Best-fit, Worst-fit, MTHM and CSP. Load-balancing is a fundamental customer requirement in

industry. Based on a large real-world dataset of cluster workloads (from industry leader Nutanix)

we show that the natural load-balancing heuristic - Water-filling - has several excellent properties.

We compare and contrast Water-filling with MTHM using our stochastic model and find that Water-

filling is a heuristic of choice.

3.1 Introduction

Cloud computing has established itself as a mainstay of modern computing infrastructures.

Clouds enable the efficient utilization of resources on an as-needed basis by dynamically configur-

29

CHAPTER 3. FAULT-TOLERANT SPATIAL LOAD BALANCING IN DATA CENTERS

ing these resources to accommodate varying workload needs. The core technology at the heart of

public cloud data-centers and private clusters is virtualization. Virtualization is the use of shared re-

sources to create and operate virtual machines (VMs) [61]; a VM is an operating system or software

that emulates the behavior of a computing system with a specified set of resource characteristics,

such as CPU and memory capacity. Virtualization allows for the execution of an application onto

heterogeneous systems as well as multiple applications in parallel. Virtualization also enables live

migration or the movement of running VMs between hosts. The enormous flexibility afforded by

virtualization enables the placement (mapping of VMs to physical hosts) and re-balancing of VMs

for a variety of reasons including performance and cost-efficient resource utilization [64]. In this

chapter we will primarily be concerned with fault-tolerance, i.e. robustness to failures.

An important aspect of fault-tolerance in a cluster is High Availability (HA) [54]. In general,

HA is the property of ensuring continuity of services (applications) despite the failures of hosts,

VMs or the applications themselves. In the context of this chapter we will interpret HA narrowly

as the ability to tolerate host failures by restarting the VMs (from the failed hosts) in back up hosts

without affecting any other existing running VMs. We define k-HA to be the property of a placement

to tolerate the failure of up to k hosts (sequentially or in parallel). In practice the response to a failure

or multiple failures the system could involve restarting of the failed VMs in other hosts; in platforms

with HwPFA (Hardware Predicted Failure Analysis alerts it could also involve live migration of the

VMs in advance of an impending failure. In either case the key algorithmic requirement is ensuring

the sufficiency of resources in the back up hosts to support the new VMs. In general resources are

multi-dimensional (such as CPU, memory, IO etc) but in this chapter we focus on a single resource

- namely, memory.

HA has been studied extensively both in the algorithms and the systems communities. Un-

fortunately, even the special of case of 0-HA with a single resource is the probem of bin packing

and hence already NP-complete [34]. Partly as a reaction to the hardness of HA the systems com-

munity has turned to polynomial-time heuristics on the one hand or AI (artificial intelligence) and

CSP (Constraint Satisfaction Programming) on the other. Heuristics, the choice of industry, are

typically quick to return some solution but fail to provide guarantees on the quality of the solution.

AI and CSP, areas of academic research, allow much greater expressiveness involving additional

constraints such as affinity and co-location requirements as well as multiple resource dimensions.

However, both of these techniques are very slow, particularly CSP which solves SAT (satisfiability,

an NP-complete problem [34]) at its core.

In spite of the attention that HA has received the fundamental theoretical questions still remain

30


unanswered: Given VMs (with sizes) and hosts (with capacities) and k how easy/hard is it to decide

whether there exists a feasible k-HA placement? Given a placement and k how easy/hard is it to

decide whether it (the given placement) is k-HA? And, on the practical side, the fundamental ques-

tions are: Given that typical inputs encountered in practice are not adversarially chosen (worst-case)

how do we meaningfully compare the quality of different heuristics? Given that load-balancing is

a basic customer requirement what is the best heuristic to use? In this chapter we make substantial

progress towards answering such questions.

3.2 Related work

[54] is a comprehensive source on a variety of techniques leveraging virtualization for HA

including replication patterns such as active/active, active/passive and cold standby. Our work de-

rived inspiration from [15] that formally defines the k-HA problem - they call it k-resilient HA.

They observe that k-HA lends itself naturally to expression as a statement in Second Order Logic

and present a transformation as a statement in First-Order Logic thus allowing the constraints to

be solved by a generic CSP solver. However, this transformation is not exact in that there may be

solutions to the original problem even though the transformed input to the CSP solver may have

none. We note that they extend the notion of k-HA to VMs - a k-HA VM must be tolerant to the

failure of up to k hosts through relocation. Our result can be seen as an exact representation of k-

HA in First Order Logic, though our result does not apply to the notion of k-HA extended to VMs.

We represent our results in the language of complexity theory rather than in terms of logic systems,

which is the more accurate way to capture computational issues. [40] describes BtrPlace, a CSP

solver customized for handling VM placements that allows the user considerable expressiveness in

specifying custom requirements. Though BtrPlace is generally fast and scalable, the CSP solver

is slow to check feasibility of HA requirements. [80] consider the related problem of replica VM

placement constrained by bounded communication delays between the replicas. They prove the

NP-completeness of their problem and compare a variety of different heuristics. Replica placement

is different from the strategy of relocating VMs from failed hosts to other hosts considered in this

chapter.

From a systems perspective some related works include [47] which describes an HA-aware

scheduler based on openstack, [45] that considers the issue of hardware redundancy, [71] which

proposes an availability-aware scheduler that dynamically manages computing resources based on

user-specified requirements and [81] that describes a method for related VMs to share check point

31


images.

3.3 Approaches

The main technical contributions are as follows:

• k-HA is NP-complete Counter to intuition we prove that deciding the existence of a k-HA

placement is NP-complete by a simple reduction to MKP (Multiple Knapsack Problem) [57].

We note two things here 1. the clever part of the proof is in showing that k-HA is in NP

and not that it is NP-hard, which is why the reduction is to (and not from) MKP, and 2.

we do, however, believe that the problem of deciding whether a given placement is k-HA is

⇧P2 -complete.

• IID-IK and existence of a best heuristic We propose a simple stochastic model, IID-IK

(Independently and Identically Distributed Items and Knapsacks), for MKP, that captures the

essence of real-world workloads as well as synthetic generators. We define an asymptotic

metric PE for comparing the quality of different heuristics. We prove that the quality of the

best heuristic is computable.

• Water-filling is the best We discuss the shortcomings of some heuristics used in practice-

First-fit, Best-fit, Worst-fit, CSP and the industry leader MTHM. Based on a dataset of cluster

workloads from Nutanix we posit two natural subcases: the doubling world where VM sizes

are powers of 2 (this is indeed the case for Amazon’s AWS where VM sizes are restricted

to be powers of 2) and the gold-dust world where VM sizes are much smaller than host

capacities. We consider Water-filling, which smooths out the load by definition, and show that

it is optimal for the doubling world and near-optimal for the gold-dust world. We also show

that Water-filling provides a simple and easily computed criterion for determining the largest

k for which a given placement is k-HA. We also compare Water-filling and MTHM using

IID-IK with distributions instantiated from a large real-world dataset of cluster workloads

provided by Nutanix Inc. We conclude that Water-filling is the heuristic of choice, from

among the heuristics in consideration.

On a conceptual level we have shown that deciding the existence of a k-HA placement is easier

than checking whether a given placement is k-HA. We have presented a simple stochastic model

for workload generation that is both useful for comparing heuristics and tractable. We have shown,

32


through a variety of arguments, that Water-filling satisfies customer requirement for load-balancing

while possessing useful k-HA properties.

In this chapter we show that the problem of VM placement reduces to MKP (Multiple Knap-

sack Problem) an NP-complete problem [34]. [48] presents an EPTAS for MKP. Our stochastic

model, IID-IK is inspired by the model for bin packing investigated in [23] where items are drawn

from a continuous distribution. Subsequent to [23] there has been much work on distributions with

both continuous and discrete supports [72, 50]. [37] is a recent work with several related references.

our model IID-IK assumes a discrete support which we also refer to as a finite support since they

are the same for a bounded domain. MTHM is a polynomial-time heuristic for MKP presented in

[57] and deployed in some commercial providers of cloud infrastructure.

In Section 3.4 we state the k-HA VM placement problem. Section 3.5 proves k-HA placement

is NP-complete. We propose our stochastic model IID-IK model for multiple knapsack in Section

3.6. Section 3.7 studies the performance of different heuristics and points out that Water-filling is

the heuristic of choice. Section 3.8 summarizes this chapter.

3.4 Problem statement

3.4.1 VMPP and VMPP-AC

We start by defining the most basic VM placement problem (VMPP): Let V denote the set of

all n VMs and H denote the set of all m hosts. Each VM vj requires sj (GB) sized memory to run.

Each host hi has a total memory capacity ci (GB) configured for VMs. To each VM vj is attached

profit pj which reflects the value or reward for placing VMs. The goal is to place VMs on hosts,

so as to maximize the sum of the profits of the placed VMs, while satisfying the constraint that

for every host the sum of the memories of the VMs placed on it must not exceed its capacity, i.e.,

8iP

vj2hi

sj ci.

It is clear that VMPP is just MKP [48, 34] and therefore NP-complete. For the rest of this

chapter we will assume that all profits pj are the same and hence the goal of VMPP is to place the

maximum number of VMs on the hosts. This special version of VMPP (or MKP) continues to be

NP-complete, in fact strongly NP-complete as bin packing reduces to it.

At this point the reader might wonder why we connected VMPP to MKP rather than to bin

packing - the reason is that in the bin packing problem the bins are of uniform capacity whereas

in VMPP hosts can have different capacities just as knapsacks in MKP. As decision problems bin

33


packing and MKP with uniform profits are the same in complexity but as optimization problems

they are different - in bin packing the items are given and the goal is to minimize the number of bins

needed to place all of the items whereas in MKP both the items and knapsacks are given and the

goal is to to maximize the number of placed items without violating the capacity constraints of any

host. Thus MKP is in concordance with VMPP where, too, the VMs and hosts are given.

In the rest of this subsection we consider VMPP-AC (the VM placement problem with affinity

constraints). This problem is a digression in that we do not use VMPP-AC in the rest of the chapter,

nevertheless the placement of uniformly sized VMs with affinity constraints was mentioned in [15]

as a problem with undetermined complexity. We resolve its complexity in this subsection.

Three kinds of affinity constraints are mentioned in [15] (1) VM-VM co-location constraint:

e.g. vi and vj must always be placed on the same host (in order to minimize the communication

cost, etc); (2) VM-VM anti co-location constraint: e.g. vi and vj must always be placed on different

hosts (so as to be separate from each other due to privacy or security reasons); (3) VM-host affinity

constraint: e.g. vi can only be placed on a specific subset of hosts.

Theorem 6. VMPP-AC with uniform VM sizes is NP-complete.

Proof. We give a reduction from 3DM (3D-matching) which is NP-hard [34].

The input in 3DM is a tripartite graph with each partition having n vertices. The question is

whether the edges of the graph can be partitioned into exactly n triangles.

We transform a given instance of 3D-Matching into an instance of VMPP-AC as follows: let

the tripartite graph be G(V1,V2,V3, E), where V1, V2, V3 denote the three partitions of vertices, and

E denotes the edges crossing them. We interpret the three partitions as follows - the set of VMS V =

V1 [V2 and the set of hosts H = V3. We interpret the edge (vs, vt) between vs 2 V1 and vt 2 V2 to

mean that vs and vt can be placed together. (Note that these constraints are easily expressed using

VM-VM anti co-location constraints.) The edge (vs, hi) between vs 2 V1 or V2 and hi 2 V3 means

vs can be placed on hi. (Note that these constraints are easily expressed using VM-host affinity

constraints.) We set the size of each VM to be 1 and the capacity of each host to be 2. It is easy to

see that there exists a partition of the edges of the tripartite graph into n triangles iff all the VMs

can be packed.

3.4.2 Exists?k-HA and Is?k-HA

The HA property of a VM placement guarantees the system can tolerate some level of failure

in hosts. Basically, HA should satisfy the following two conditions: (1) VMs on failed hosts should

34


be re-instantiated quickly in other running hosts; (2) VMs originally on those running hosts should

not be affected, i.e. suspended, migrated, etc, at all.

This adds additional complexity in the VM placement problem. When VMs are initially placed,

one has to consider if there will be enough space on the still running hosts for the VMs assigned to

a failed host.

The k-HA property of a VM placement can quantify such fault tolerance. Specifically, a place-

ment is said to have the k-HA property, if for all sequences of up to k host failures there is enough

memory space on the running hosts for the VMs from the failed hosts to be re-instantiated (on the

running hosts).

We define two decision problems: Exists?k-HA and Is?k-HA. Exists?k-HA is the problem of

deciding whether a given instance of VMPP has any placement with the k-HA property. Is?k-HA

is the problem of deciding whether a given placement has the k-HA property.

Note that VMPP is the same as Exists?0-HA. Thus, we can say that deciding the existence of

a k-HA placement is at least as hard as the original VM placement problem.

3.5 k-HA is NP-complete

In a counter-intuitive result we show that the HA problem can be reduced to multiple knapsack

and hence is NP-complete.

3.5.1 Exists?k-HA is NP-hard

Naively, one would expect that Exists?k-HA lies in the third level of the polynomial hierarchy,

⌃P3 [63]. This is because it would seem to require 3 alternating quantifiers in the following form:

there-exists placement for-all upto k host failures there-exists rebalancing such-that the original

placement and rebalancing strategy are valid.

However, we show that in fact it lies in NP = ⌃P1 .

Theorem 7. For all k, Exists?k-HA reduces to MKP.

Proof. Given m knapsacks and n items pack the items into the smallest m � k knapsacks, i.e.

set aside the k largest capacity knapsacks and pack the items into the rest. The given system of

knapsacks and items is k-HA iff this packing is achievable. Clearly, if the system is k-HA then it

should be packable into the smallest m� k knapsacks in order to be resilient to the failure scenario

where the k largest knapsacks fail. And conversely, the packing into the m� k smallest knapsacks

35


can tolerate failure of any k knapsacks - if one of the smaller knapsacks fails then move its items

into one of the larger empty knapsacks and there will always be at least one larger empty knapsack

since we started out with k empty knapsacks.

The key point to note is that MKP (or VMPP) is equivalent to Exists?0-HA, hence it directly

follows that Exists?k-HA is at least NP-hard, and so the clever part of the above theorem is in

showing that Exists?k-HA is no harder by reducing it to MKP.

Corollary 1. Exists?k-HA in NP-complete.

Note that even though Exists?k-HA suddenly seems to have become easy we still believe that

Is?k-HA continues to be hard, i.e. ⇧P2 -complete, though we are unable to prove it yet.

3.6 IID-IK and best heuristic

We propose a stochastic model for multiple knapsack that not only captures real-world work-

loads but also provides a uniform basis for comparing the performance of different polynomial-time

heuristics. We prove that, in a fairly general sub-case, there is a best heuristic in our model, i.e., a

heuristic with optimal performance.

Having shown that HA reduces to the multiple knapsack problem which is NP-complete we

understand that we are limited to using polynomial-time heuristics. The question we are faced with

is which heuristic to use, i.e. how to measure the performance or efficiency of heuristics. One

standard way to do this is to look at their quality of approximation. However, even quality of

approximation is a worst-case measure. We argue that workloads drawn in practice have a lot more

regularity. In fact, based on a large dataset of client workloads drawn from the customer-base of one

of the largest providers of private cloud infrastructure we find that workloads can be characterized

as independent draws from distributions for knapsacks and items. Inspired by this finding as well

as by an early analysis of bin packing [23] we propose the following natural stochastic model of

multiple knapsack:

Items are drawn independently from a distribution of items - the item distribution specifies both

the size and the value of the draw ; knapsacks are drawn independently from a distribution of knap-

sacks - the knapsack distribution specifies the size of the knapsack. We assume that distributions for

both knapsacks and items have finite support, i.e., there are only a finite number of different types

of items and knapsacks. For simplicity we assume that all items have value 1 - our definitions and

results can easily be generalized to the case with arbitrary values.

36


This model gives us a solid mathematical basis for comparing different heuristics. We now

describe the measure PE(H) which captures the asymptotic packing efficiency of the heuristic

H . Draw n samples from the item distribution; then keep drawing samples one-by-one from the

knapsack distribution; after each knapsack draw check whether H packs all the items into the drawn

collection of knapsacks; let minH(n) denote the number of drawn knapsacks when H successfully

packs all items for the first time. Define the finite packing efficiency for n:

PE(H,n) , n

minH(n)

Define the packing efficiency of H to be:

PE(H) = lim infn!1

PE(H,n)

PE(H) is well-defined since lim sup exists for all bounded sequences and is the expected number

of items the heuristic packs per bin, in the asymptotic limit. We also note that the definition of PEis robust: instead of drawing bins until we can pack the n items we could also define it in terms of

the process where we draw items until they will no longer fit inside the m bins.

Through simulations and for simple distributions we are able to verify the existence of packing

efficiencies for Water-filling and MTHM. We are led to the natural question of whether, given the

item and knapsack distributions, there exists a best heuristic and surprisingly we can show that

the answer is yes for distributions with a finite support, which captures all real-world situations.

Informally the theorem below states that in the stochastic model of multiple knapsack where the

distributions have finite support, we can find the best heuristic and its packing efficiency (which is

optimal).

Theorem 8. There exists a heuristic H that achieves the maximum possible PE in the IID-IK model;

both H and PE(H) are explicitly computable.

Proof. We first give the intuition behind the proof to enable the reader to follow the formal argument

easily.

That the distributions have finite support means there are a finite number of different types of

item sizes and knapsack capacities. Further, by the Central Limit Theorem when the number of

items and knapsacks goes to infinity then we will get close to the expected proportion of each size

(for items) and capacity (for knapsacks). Now consider all possible ways of packing each of these

different knapsacks with different combinations of items - there are only a finite number of different

packed knapsack configurations. (This includes all such configurations, including for example the

37


empty knapsack). With one variable per packed knapsack configuration representing its proportion

in the final solution we can write a linear program to maximize packing efficiency constraining

the proportions of items sizes and knapsack capacities to fit the proportions dictated by the Central

Limit Theorem. Thus the maximum packing efficiency can be calculated in constant time since the

support sizes are finite.

Formally, let ◆ and index the finite support set of items and knapsacks respectively. Let the

◆’th item be generated in proportion P◆ and let the ’th knapsack be generated in proportion P. Let

w() denote the number of distinct ways that the ’th knapsack can be packed with items, and let

! be the corresponding index. Let N! (◆) denote the number of copies of item ◆ in the !’th packing

of the ’th knapsack. Variable p! is the proportion of the !’th packing of the ’th knapsack in the

optimal packing. Variable pe is the packing efficiency that we are trying to maximize. Then the

solution to the following LP gives the maximum possible PE in the IID-IK model:

maximize pe

subject toX

!,

p!(X

◆

N! (◆))= pe

8◆

X

!,kappa

p!N! (◆) = P ⇤ pe

8

X

!

p! = P

Since the support set is finite the above LP calculates the optimal PE in constant time. The optimal

heuristic H is also derivable in straightforward fashion from the values of the p! variables in the

optimal solution: given any input the optimal heuristic is to utilize the !’th packing of the ’th

knapsack to the extent of nPE(H) ⇤ p

! copies; any items left over can be given their own individual

knapsack; by Chernoff bounds we are guaranteed that there will be at most O(n0.5) additional knap-

sacks used so that asymptotically the packing efficiency will limit to PE since packing efficiency is

a ratio which has minH(n) = ⌦(n) in the denominator.

Note that PE(H) is the best packing efficiency achievable when both items and knapsacks

are generated using the stochastic model. However, given an arbitrary collection of items, we can

potentially achieve better packing efficiencies if we are allowed to select the knapsacks. In fact,

using an LP similar to the one above and the fact that fixed dimension integer programming is in P

[73] it is possible to compute the best packing in time polynomial in the number of items.

38


3.7 Analysis of heuristics

Given the reduction of k-HA to Multiple Knapsack Problem, k-HA is NP-complete. Unless

P = NP , to compute k-HA placement efficiently in polynomial time is impossible. Thus, to solve

a placement problem, we consider CSP and the following 5 polynomial time heuristics commonly

used in industrial practice: First-fit, Best-fit, Worst-fit, MTHM and Water-filling. Each of these 5

heuristics is a different variant of greedy algorithm.

In our scenario of VM placement, the hosts are sorted in increasing order of size before being

input into these heuristics. Given a sequence of VMs, First-fit and Best-fit will pack VMs one by

one until all the hosts can hold no more. At each step, First-fit will put the VM into the smallest

indexed host that has enough space, while Best-fit will put the VM into the tightest host that has

enough space. For Worst-fit, it puts VMs one by one into the emptiest host until no additional VM

can fit into any of the hosts.

MTHM is more complicated and we will give a more detailed description in the next subsection.

Water-filling first sorts the VM in decreasing order of size, then puts each VM into the lightest

loaded VM that also has enough space for that VM.

Load-balancing, i.e., ensuring the load is evenly spread among the hosts, is a critical property

desired by cloud clients. It is also a research topic that attracts much attention [55]. However, First-

fit, Best-fit and Worst-fit will overload some hosts with many VMs while underloading other hosts

with too few VMs.

Also, the time to compute a placement is critical both for initial placement of a set of VMs as

well as contingency placement of VMs from a failed host since cloud providers are contracted to

provide service with interruption/delay below some small threshold.

This makes a CSP solver impractical for industrial use. For example, the duration of a place-

ment computation for 32 hosts can take on the order of minutes [15]. But industrial requirement

demands the computation be at most on the order of milliseconds.

This leaves us with just two methods - Water-filling and MTHM - to consider. In the following

subsections, we first describe how MTHM works and use a simple example to show the limitation

of MTHM. Then we show that, in the Gold-dust World, Water-filling performs very close to the

optimal. We also develop simple bounds for the level of HA achievable in the Gold-dust World -

these bounds apply to all greedy variants though we state and prove them only for Water-filling. We

further show that in the Doubling World Water-filling actually achieves optimality.

In the end of this section, we evaluate the relative performance between Water-filling and

39


MTHM and conclude that Water-filling is the heuristic of choice.

3.7.1 MTHM revisited

We review a heuristic for Multiple Knapsack Problem – MTHM by Martello and Toth [57]. We

first go over the heuristic at a high level, then we use a small example to show the limit of MTHM.

3.7.1.1 A high level overview of MTHM

MTHM takes the input: (a) descending sorted list of items in unit profit (value over weight);

(b) ascending sorted list of knapsacks in sizes. It outputs the approximated maximized total value

of packable items with the information of which knapsack to pack each of them into.

In the case of our VM placement, we assign all the items with uniform value 1. This reduces

the goal to maximizing the number of the VMs that get placed. We just need to input a sorted VM

list in order of increasing size.

The overall procedure of MTHM is:

i) (Initialization): For each knapsack, run the greedy step in Algorithm 9, i.e. place as many

items from the unassigned sorted item list as possible. Label packed items as ”assigned”,

otherwise ”unassigned”.

ii) (Rearrangement): Clear all items from knapsacks but retain the ”assigned” or ”unassigned”

information. Run a cyclic allocation in Algorithm 5 only for those assigned items. During

this process, relabel these items if necessary. Finally, run the greedy step in Algorithm 9 again

over all the knapsacks with the rest of the ”unassigned” items.

iii) (Swap and fit): For each pair of assigned items, try to see if swapping them will create more

space for more item to fit in.

iv) (Delete and fit): For each assigned item, try to see if removing this item in its knapsack would

create space for more items to fit in.

The VMs are considered ”packable” if and only if the maximized packed value equals the total

value.

40


Algorithm 4 Greedy packing on one bin in MTHMGiven one knapsack, and a sorted list of unassigned items

1: for each item in the sorted list do2: if the size of this item is no greater than the bin’s capacity then3: Assign this item to the bin.

4: Remove this item from the unassigned list.

5: Subtract the bin’s capacity by the size of this item.

6: end if7: end for

3.7.1.2 One simple counter example

We use the following simple counter example to show that for certain lists of items, MTHM

will return “unpackable”, even though the items are truly “packable” into the same knapsacks.

Consider 2 knapsacks with capacity 100, a list of items sized [11, 12, 13, 14, 27, 44, 71]. There

is one packing solution: items sized [11, 13, 71] in the first knapsack, items sized [12, 14, 27, 44] in

the second knapsack. But MTHM will not return this solution.

Let us see each step of MTHM.

1. Initialization step will label all the item except ”71” as ”assigned”. The greedy result is:

[11, 12, 13, 14, 27] in the first knapsack, [44] in the second knapsack.

2. In rearrangement step, the cyclic allocation process will have the following. The first knap-

sack has [44, 14, 12], and the second knapsack has [27, 13, 11]. Then 71 can not fit into any

of the two in the final greedy step.

3. In swap and fit step, the largest possible space increase for the first bin can be made by

”swapping out 44, and swapping in 11”. This only increases the space from 30 to 63, which

is not enough for 71. Similarly, for the second knapsack, the largest possible space increase

can come from ”swapping out 27, and swapping in 12”. Then it only increases the space from

49 to 64 which is still not enough for 71.

4. The delete and fit step simply does not contribute at all towards getting a packing solution.

41


Algorithm 5 Cyclic allocation in MTHMGiven a sorted list of assigned items

1: s 0

2: for each item in the sorted list of assigned item do3: Let l be the first knapsack index in {s, s+1, . . . ,m� 1, 0, . . . , s� 1} such that knapsack l’s

capacity is no less than this item’s size.

4: if There is no such l then5: Label this item back to “Unassigned”.

6: else7: Assign this item to knapsack l.

8: Subtract knapsack l’s capacity by the size of this item.

9: if l + 1 == m then10: s 0.

11: else12: s l + 1.

13: end if14: end if15: end for

3.7.2 Gold-dust world: when VM sizes are small

We study the performance of Water-filling when the maximum VM size smax is much smaller

than the minimum host capacity cmin. Particularly, we show that the number of items that Water-

filling will allocate is close to the number achieved by OPT the optimal algorithm.

We show the following two results.

i) For packing, when smax ⌧ cmin, the number VMs of Water-filling can pack is close to the

number packed by OPT.

ii) Then for k-HA, when smax ⌧ cmin, instead of keeping k standby hosts, we can just keep as-

signing VMs to hosts by Water-filling, until the total remaining space reaches some threshold.

We state the corresponding claim and quantify the threshold in Theorem 10.

42


3.7.2.1 Bounds for packing when using Water-filling

Let there be m knapsacks with smallest of capacity cmin and let the largest of the items have

size smax with smax ⌧ cmin. Let total capacity of all knapsacks be C0, so C0 >= m · cmin.

Theorem 9. If NOPT items can be packed into the bins by OPT then Water-filling will pack at least

(1� smax/cmin) ⇤NOPT items.

Proof. First, Water-filling will always pack at least Nwaterfill = NOPT �m items. Consider the

items left over by Water-filling, the smallest left over item is bigger the largest residual space else

greedy would pack it. but we know the total volume of the residual space is greater than the total

volume of the leftover items because OPT is able to pack everything. So the number of remaining

items by greedy is at most m. (Otherwise, the total volume of the residual space must be less than

the total volume of the leftover items.)

So Nwaterfill >= NOPT � m, i.e. Nworstfit/NOPT >= 1 � m/NOPT . So Water-filling

achieves a ratio better than 1�m/NOPT . Now NOPT >= C/smax. Therefore 1�m/NOPT >=

1�m · smax/C >= 1�m · smax/m · cmin = 1� smax/cmin.

3.7.2.2 Bounds for k-HA when using Water-filling

Now given that a set of VMs that are packable into m hosts, what HA level will such packing

achieve? We answer this question by the following mechanism.

Let Ck denote the total capacity of all the hosts except the largest k hosts.

Theorem 10. k-HA is guaranteed, if k is the maximum number such that the inequality in Eq.(3.1)

holds.

Ck � S > (m� k)(smax � 1) (3.1)

Proof. Given a k, Ck � S denotes the residual capacity of all the hosts except the largest k hosts

when all the VMs are packed into these m � k hosts. Now, if Ck � S > (m � k)(smax � 1), by

Pigeon Hole Principle, then there is at least one host that has residual capacity that is strictly greater

than smax � 1 (at least smax), i.e. one other VM can fit into the residual space.

Now, given a set of m hosts, and a sequence of incoming VMs, we can actually keep allocating

VMs to hosts by Water-filling. At each assignment, we need to update all m inequalities, the number

of such inequalities that hold gives the number of HA level.

43


3.7.3 Doubling world: VM sizes in the form of 2i

We show that when the VM sizes are of the form: 2i(i = 0, 1, 2, . . . ), VMP can be solved

exactly in polynomial time. In particular, Water-filling not only finds the feasible packing if it exists

but it also achieves load balancing.

We first give the intuition for why packability is decidable in polynomial-time in the doubling

world. We are given a list of n VMs in decreasing order of size, the VM sizes being of the form

2i(i = 0, 1, 2, . . . ), i.e. 256GB, 512GB, etc., and a fixed number of k hosts with arbitrary capacity

(does not need to be of any special form). Consider the first VM in the list; one of the two following

cases must happen.

i) There is no space on any host to accommodate this VM. Then clearly, this set of VMs is

simply not able to be packed/repacked.

ii) There are multiple hosts that can accommodate this VM. Then, it does not matter which host

we place this VM on, because the remaining VMs can always well-utilize the empty space

which the first could have been put into.

Then we claim that given a set of k hosts Water-filling can exactly check the packability of the

list of VMs, whose sizes are in the form of some power of 2.

We develop the following definitions and theorems to support the above argument.

Definition 1 (General packing). Any packing that just satisfies the capacity constraint is a general

packing.

Definition 2 (Canonical packing). The packing solution returned by Water-filling is called canonical

packing.

Definition 3 (Packing level of each VM). Let j denote the host to which VM i is assigned by Water-

filling. Let lij denote the total size of VMs packed on host j before VM i is placed.

Canonical packing has the following necessary and sufficient condition:

Property 1. A general packing is a canonical packing, if and only if for any packed item on any

host, there is no strictly smaller item on another host that has a strictly smaller packing level.

Theorem 11. There is a general packing, if and only if there is a canonical packing.

44


Proof. ”(”: this direction is trivial since any canonical packing is a general packing.

”)”: Let VMs in each host be ordered from largest at the bottom to smallest at the top; Then

across different hosts, for any two different hosts s, t, if a larger VM on host s is at a higher packing

level than a smaller VM on host t, doing a swap still gets a valid packing. Do such swaps until no

more swap can be done. We have a canonical packing.

Corollary 2. If Water-filling decides no canonical packing exists, then there is no general packing.

This guarantees no false negative when Water-filling returns a non-packable decision.

3.7.4 Performance evaluation

We evaluate Water-filling vs MTHM by our IID-IK model. Given host capacity and VM size

distribution, repeat the process of random sampling from both of the two distributions described in

Section 3.6 for 500 times, we get the asymptotic metric PE in each setting.

3.7.4.1 A small dataset

First, we show how much worse MTHM performs while Water-filling achieves optimality. We

compute PE for Water-filling and MTHM given the following host/VM size distribution in Table

3.1.

Table 3.1: One small host/VM size distribution

Host size (GB) Frequency (%) VM size (GB) Frequency (%)

100 100 2 16.664 16.668 16.6616 16.6632 16.6664 16.66

Table 3.2 shows MTHM performs ⇠ 21% worse than Water-filling which achieves optimality

in this dataset with a small support for the distributions.

3.7.4.2 Nutanix workload & asymptotic metric PE results

In Nutanix’s data center, host-nodes (hosts) are deployed inside clusters. Each cluster contains

a number of hosts. VMs share physical resources on each host.

45


Table 3.2: Asymptotic metric PE for the small distribution

Number of hosts PE for Water-filling Ratio for MTHM

10 4.84 3.9820 4.72 3.6830 4.77 3.7340 4.74 3.6550 4.74 3.6560 4.77 3.6870 4.77 3.6980 4.78 3.6890 4.75 3.69100 4.76 3.67

In this snapshot of a typical workload at Nutanix, there are 5499 clusters. 2687 of them have 4

host nodes, the remaining 2872 of them have 3 host nodes.

Fig. 3.1 shows the memory in percentage consumed by those running VMs on their host nodes.

We can see the majority of memory consumption on those host nodes are under 40%.

Fig. 3.2 shows the host capacity distribution. Fig. 3.3 shows the VM size distribution.

Table 3.3 shows the asymptotic metric PE for host/VM size distribution from the unmodified

Nutanix workload. As we can see from Fig. 3.2 and 3.3, the VM sizes are much smaller than the

bin sizes. This causes the asymptotic metric PE of Water-filling to be slightly better than MTHM’s,

which also supports our previous conclusion on Gold-dust World in Section 3.7.2.

We round VM sizes up to the closest 2i GB, and merge the distribution entries with the same

rounded-up sizes. We get the rounded-up memory size distribution for VMs in Fig. 3.4.

Table 3.4 presents the asymptotic metric for this setting. Water-filling achieves optimality due

to the conclusion on Doubling World in Section 3.7.3. Water-filling still performs slightly better

than MTHM due to the large gap in size between hosts and VMs.

3.7.5 Water-filling packs best

To summarize, Water-filling has the following nice properties.

• Water-filling places VMs into the emptiest host at each step, which leads to a most balanced

load distribution among hosts.

• Given sorted lists of VMs and hosts in order of size, Water-filling runs in ✓(mn) time, while

MTHM takes ✓(mn + n2). In industrial practice, the number of hosts is often a constant.

46


Table 3.3: Asymptotic metric PE for original host/VM size distribution from Nutanix


10 23.253 22.90520 23.996 23.73230 23.136 22.89840 23.101 22.85550 23.798 23.59160 23.670 23.45370 23.668 23.46280 23.484 23.27590 23.265 23.060100 23.666 23.454

Table 3.4: Asymptotic metric PE for power-of-2


10 21.831 21.83120 21.730 21.73030 21.881 21.87840 21.399 21.39850 21.485 21.48360 21.141 21.13870 21.973 21.97180 21.407 21.40790 21.464 21.457100 21.555 21.551

47


0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100

Memory usage percentage of running VMs (%)

0

5

10

15

20

25

Fre

quency

(%

)

Figure 3.1: Distribution of memory usage by running VMs

Thus, Water-filling satisfies the stringent real-time requirements of commercial applications.

• Water-filling is proved to perform very close to the optimal in Gold-dust World where VM

sizes are much smaller than host sizes.

• Also in Gold-dust World, we develop simple inequalities to check the level of HA when

applying Water-filling in Theorem 10, which makes the computation of k-HA placement even

faster.

• Simulation on industrial workload from Nutanix shows Water-filling has better asymptotic

metric than MTHM in our IID-IK model.

In other words, Water-filling is the best.

3.8 Summary

In this chapter, we study the HA problem in VM placement and prove novel complexity-

theoretic results, particularly the surprising result that the existence of a k-HA placement is in NP .

We leave open the problem of deciding the exact complexity of deciding whether a given placement

48


0 200 400 600 800 1000 1200 1400 1600

Host memory capacity (GB)

0

5

10

15

20

25

30

Fre

quency

(%

)

Figure 3.2: Distribution of host memory capacity

is k-HA or not. We propose a natural stochastic model - IID-IK (Independently and Identically

Distributed Items and Knapsacks) - which provides a uniform basis for comparing heuristics. We

also show that interestingly there exists a best heuristic in this model whose packing efficiency is

computable. We utilize this model and analyze a variety of heuristics used in practice; we also

consider natural special cases of input distributions that occur in the real world and conclude that

Water-filling is the best.

49


0 1 2 3 4 5 6 7 8 9

VM memory size in log scale (2i GB)

0

5

10

15

20

25

30

35

Fre

quency

(%

)

Figure 3.3: Distribution of VM memory size

-0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9 9.5 10

VM memory size in log scale (2i GB)

0

5

10

15

20

25

30

35

40

Fre

qu

en

cy (

%)

Figure 3.4: Distribution of VM memory size after rounding

50

Chapter 4

Load Balancing for MultidimensionalResources

The enterprise customer today has to decide how to distribute her load among multiple clouds

- between on-premise private clouds and public clouds (such as AWS, Azure and Google Cloud).

What is the best way to assign load to different clouds so as to minimize the disruption caused by

routine maintenance? How should services be distributed to take advantage of different network

infrastructure, or to minimize bandwidth or energy costs, in different clouds? How should load be

placed so that the resulting aggregate traffic profile to/from a cloud protects the privacy of end-users?

A defining feature of clouds is their elasticity or ability to scale with load. Motivated by

the aforementioned use cases we consider the uncapacitated multidimensional load assignment

problem - VITA(F) (Vectors-In-Total Assignment): the input consists of n, d-dimensional load




be any operator mapping a vector to a scalar, such as max, min, etc. Then the goal is to partition

the vectors among the buckets, respecting the assignment constraints, so as to minimize

X

j

wj ⇤ F (X

Vi2Bj

Vi)


Bj . We also let Vi(k) denote the value in the k’th position of the vector Vi.

In this chapter we characterize the complexity of VITA(min) and VITA(max) - providing

51

CHAPTER 4. LOAD BALANCING FOR MULTIDIMENSIONAL RESOURCES

approximation algorithms and hardness results. Our approach involves clever rounding of carefully

crafted linear programs and may be of independent technical interest. We also consider VITA(F)

for some additional F such as max�min and 2ndmax.

4.1 Introduction

The launch of EC2 in 2006 by AWS [1] heralded the explosive growth in cloud computing.

Cloud computing is an umbrella term for computing as an utility. It enables 24x7 Internet-based

access to shared pools of configurable system resources and real-time provision-able higher-level

services. Public clouds enable organizations to focus on their core businesses instead of spending

time and money on IT infrastructure and maintenance. One of the major benefits of clouds is that

they are elastic, i.e. they are uncapacitated. This allows enterprises to get their applications up

and running quicker, and rapidly adjust resources to meet fluctuating and unpredictable business

demand.

Today, in addition to AWS, Microsoft’s Azure [5] and the Google Cloud [3] are the other

major public cloud platforms. But the advent of multiple clouds means that enterprises are faced

with new questions: How much of their load should they keep on-premise and how much should

they move to public clouds? How should they mix and match the various options to save money

without sacrificing customer satisfaction? etc. A number of enterprise software companies such

as HPE [4] and startups such as Turbonomic [7], Datadog [2] and RightScale [6] are beginning to

provide software and sevice solutions to these problems.

At the same time this is also a fertile area for new problems with the potential for clever

theoretical solutions to have practical impact. In this chapter we provide a framework - VITA :

Vectors-In-Total Assignment - that captures a variety of interesting problems in the area of hybrid

clouds with interesting theoretical challenges. We answer some of these challenges with novel

algorithmic solutions and leave others as open questions. In the subsection that follows we list a

few typical use cases captured by the VITA framework.

4.1.1 Motivation and Model

Scenario 1. Minimizing peak pricing: Consider an enterprise customer that has a choice of

several different cloud providers at which to host their VMs (virtual machines). The requirements of

each VM can be characterized along several different resource dimensions such as compute (CPU),

52


network (latency, bandwidth), storage (memory, disk) and energy. When different virtual machines

are placed in the same elastic resource pool (cloud), their load across each dimension is accrued

additively (though, of course the different dimensions can be scaled suitably to make them compa-

rable). A typical pricing contract will charge based on the most bottle-necked dimension since peak

provisioning is the biggest and most expensive challenge for the resource provider. And different

providers may have different rates based on differing infrastructure and their cost for installation and

maintenance. The natural question then arises - what is the optimal way for the enterprise customer

to distribute the load amongst the different cloud providers so as to minimize total cost?

Scenario 2. Minimizing maintenance downtime: Hosts, and even data centers need to be

powered down every so often for maintenance purposes, e.g. installing a new HVAC system in a

data center. Given this reality, how should the application (collection of virtual machines and/or

containers collectively performing a task or service), be allocated to the different data centers so

as to minimize the aggregate disruption? This scenario also applies to industrial machines where

different factories (or floors of a factory) need to be shut down for periodical maintenance work.

Scenario 3. Preserving privacy: Consider a set of end-users each with its own (hourly)

traffic profile accessing an application. We wish to partition the application components across a

set of clouds such that by observing the (hourly volume of) traffic flow of any single cloud it is

not possible to infer which components are colocated there. This leads to the following question -

how should we distribute load across clouds in order to minimize the maximum hourly variation in

aggregate traffic? As an analogy, the situation here is similar to the problem of grouping households

such that the variation of energy usage of a group is minimized making it difficult for thieves to

infer who has gone on vacation.

Scenario 4. Burstable billing: Most Tier 1 Internet Service Providers (ISPs) use burstable

billing for measuring bandwidth based on peak usage. The typical practice is to measure bandwidth

usage at regular intervals (say 5 minutes) and then use the 95th percentile as a measure of the

sustained flow for which to charge. The 95th percentile method more closely reflects the needed

capacity of the link in question than tracking by other methods such as mean or maximum rate.

The bytes that make up the packets themselves do not actually cost money, but the link and the

infrastructure on either end of the link cost money to set up and support. The top 5% of samples

are ignored as they are considered to represent transient bursts. Burstable billing is commonly used

in peering arrangements between corporate networks. What is the optimal way to distribute load

among a collection of clouds, public and private, so as to minimize the aggregate bandwidth bill?

The above scenarios constitute representative situations captured by the uncapacitated multi-

53


dimensional load assignment problem framework - VITA. A host of related problems from a variety

of contexts can be abstracted and modeled as VITA(F): the input consists of n, d-dimensional load




be any (projection) operator mapping a vector to a scalar, such as max, min, etc. Then the goal is

to partition the vectors among the buckets, respecting the assignment constraints, so as to minimize

X

j

wj ⇤ F (X

Vi2Bj

Vi)


Bj . VITA stands for Vectors-In-Total Assignment capturing the problem essence - vectors assigned

to each bucket are totaled. Unless otherwise specified we use i to index the load vectors, j to index

the cloud buckets and k to index the dimension. We let Vi(k) denote the value in the k’th position

of the vector Vi.

We now explain how VITA(F) captures the aforementioned scenarios. In general, dimensions

will either represent categorical entities such as resources (e.g., CPU, I/O, storage, etc.,) or time

periods (e.g., hours of the day or 5-minute intervals, etc.,). We gently remind the reader to note that

in each of the scenarios the elasticity of the clouds is a critical ingredient so that contention between

vectors is not the issue. The set of scenarios we present are but a small sample to showcase the

versatility and wide applicability of the VITA framework.

Scenario 1 is captured by having a vector for each VM, with each dimension representing its

resource requirement1; constraints representing placement or affinity requirements [41], weights wj

representing the rates at different cloud providers. Then minimizing the sum of prices paid for peak

resource usage at each cloud is just the problem VITA(max).

In Scenario 2 each dimension represents the resource (say, CPU utilization) consumed by the

application in a given time period, e.g. the vector for an application could have 24 dimensions one

for each hour in the day. Once the application is assigned to a data center (or cloud or cluster) it

is clear that disruption is minimized if the maintenance downtime is scheduled in that hour where

total resource utilization is minimum. Then minimizing the aggregate disruption is captured by the

problem VITA(min).1In general, these requirements vary over time, and if we wish to avoid migration, then we can model the problem by

having a dimension for each resource, for each time period, which has the effect of further increasing the dimensionalityof the problem.

54


The dimensions in Scenario 3 are the hours of the day and the resource in question is the

traffic. To prevent leakage of privacy through traffic analysis the goal is to distribute the application

components across clouds so that the range between the peak and trough of traffic minimized. This

problem is exactly represented as VITA(max�min).

In Scenario 4, we have vectors for each application with 20 dimensions one for each 5th per-

centile [67, 66] or ventile of the billing period2. Then minimizing the aggregate bandwidth bill

under the burstable billing method is VITA(2ndmax).

4.1.2 Our results

All the problems we consider are in NP [35]. For VITA(min) and VITA(max) we present our

results as a lattice - see Figs 4.1 and 4.2. For any given F, VITA(F) can be partitioned into a lattice of

4 different problem spaces based on the following 2 criteria: 1. constraints, and 2. dimensionality.

The 4 different problem spaces arise from the Cartesian product: {unconstrained, constrained} X

{bounded, unbounded}. Unconstrained refers to the situation where there is no bipartite graph

representing constraints, i.e. any load vector may be placed in any bucket. And, Bounded refers to

the situation where each load vector has a fixed dimension (independent of n). It should be clear

that the simplest of the 4 spaces is unconstrained, bounded VITA(F) and the most general is the

constrained, unbounded version of VITA(F). We present our results, algorithms and hardness, for

the different F, in the form of a lattice. In each of the figures, the algorithmic results are displayed

only at the highest possible node in the lattice, since it automatically applies to all nodes in the

downward-closure; similarly, hardness results are presented at the lowest possible node since they

apply to all nodes in the upward-closure. Further, our hardness results use only uniform weights

whereas our algorithmic results work for general weights.

Our results are as follows:

• VITA(F) for F linear. We show that when F is linear then the problem is solvable exactly in

polynomial-time. In particular VITA(avg) is in P.

• VITA(min). Our results are summarized in Fig. 4.1. We show that VITA(min) is inapprox-

imable when the dimensions are unbounded, i.e. it cannot be approximated to any finite factor.

Since it is inapproximable we counter-balance this result by providing an O(log n, log n)-

bicriteria approximation algorithm [56]. Our bicriteria algorithm produces an assignment of2This is a modeling approximation and does not exactly capture 5 minute samples.

55


cost within O(log n) of the optimal while using no more than O(log n) copies of each bucket.

The bicriteria result, which is based on rounding an LP (linear program) [65] is the technical

center-piece and contains the central ideas used in the other LP-based results in this chapter.

• VITA(max). Our results are summarized in Fig. 4.2. Our results for VITA(max) also apply

to VITA(max�min).

• VITA(2ndmax). 2ndmax turns out to be a particularly difficult problem from the stand-

point of characterizing its computational complexity. We consider the unweighted (or uni-

form weights) case and the requirement that the number of buckets exceeds the number of

dimensions. With these restrictions we are able to demonstrate an LP-based approximation

algorithm that achieves a logarithmic factor of approximation in the constrained case. We

also show that unconstrained, bounded VITA(2ndmax) is weakly NP-hard [35].

Constrained

Unconstrained Bounded

UnboundedExact

NP-hard(strong)

Inapproximable

O(log n, log n)(bicriteria)

min

Figure 4.1: VITA(min). The simplest unbounded case is inapproximable, and we give a bicriteriaguarantee for the hardest case.

56


Constrained

Unconstrained Bounded

Unbounded

ExactNP-hard(strong)

(log n)

max max - min

Figure 4.2: VITA(max) and VITA(max�min). The unconstrained, cases are exactly solvable andwe have tight logarithmic guarantees for the constrained unbounded case.

4.1.3 Related Work

There is extensive literature on multidimensional versions of scheduling and packing problems.

[20] is an informative survey that provides a variety of new results for multidimensional generaliza-

tions of three classical packing problems: multiprocessor scheduling, bin packing, and the knapsack

problem. The vector scheduling problem seeks to schedule n d-dimensional tasks on m machines

such that the maximum load over all dimensions and all machines is minimized. [20] provide a

PTAS for the bounded dimensionality case and poly-logarithmic approximations for the unbounded

case, improving upon [42]. For the vector bin packing problem (which seeks to minimize the num-

ber of bins needed to schedule all n tasks such that the maximum load on any dimension across all

bins is bounded by a fixed quantity, say 1), they provide a logarithmic guarantee for the bounded di-

mensionality case, improving upon [25]. This result was subsequently further improved by [10]. A

PTAS was provided for the multidimensional knapsack problem in the bounded dimension case by

[32]. The key distinction between the vector scheduling problem of [20] and our framework is that

57


they seek to minimize the maximum over the buckets and the dimensions whereas (in VITA(max))

we seek to maximize the weighted sum over buckets of the maximum dimension in each bucket. The

multidimensional bin packing knapsack problems are capacitated whereas this chapter deals with

uncapacitated versions. There has also been a lot of work on geometric multidimensional packing

where each vector is taken to represent a cuboid [22, 11].

There is a fair bit of recent literature substantiating the motivating scenarios we provide in the

introduction (4.1.1) to this chapter. Peak provisioning is an area of active research in the systems

community [21, 75]. Fairness in provisioning multi-dimensional resources is studied in [36]. The

use of CSP (Constraint Satisfaction Programming) in placement has been investigated [41]. Energy

considerations in placement have also been explored [29, 30] [28, 67, 66]. Building scalable systems

that provide some guarantee against traffic analysis is an area of ongoing active research [76, 49, 58].

To the best of our knowledge our formulation is novel - our attempts to find prior literature on

the exact same problems have been futile. This seems surprising since the VITA(F) framework is

simple and natural. One reason could be that the 1-D version is trivial and so may have forestalled

investigation into higher dimensions.

We present our results in the sections that follow. Section 4.2 presents results for linear F. Sec-

tion 4.3 presents our results for VITA(min) while Section 4.4 contains our results for VITA(max).

VITA(2ndmax) results are presented in Section 4.5.

4.2 VITA(F) for linear F

By linear F we mean one of the following two situations:

• F is a vector and F (V ) = F · V (where we abuse notation slightly and use F as a function

and a vector).

• F is a matrix and the weights are vectors with ⇤ representing an inner-product so that wj ⇤ F

is a scalar.

Lemma 1. VITA(F) can be solved exactly in polynomial time for linear F.

Proof. Using the linearity of F the value of the objective function can be simplified thusX

j

wj ⇤ F (X

Vi2Bj

Vi) =X

j

X

Vi2Bj

wj ⇤ F (Vi)

Hence minimizing the value of the objective function is simply a matter of finding the j that mini-

mizes wj ⇤ F (Vi) for each feasible Vi.

58


Corollary 3. VITA(avg) can be computed exactly in polynomial time.

Proof. Set F =< 1d ,

1d , . . . ,

1d > where d is the dimension. It is straightforward to see that F · V =

Pi Vi

d .

Note that many real-world pricing situations are captured by linear F, such as charging sepa-

rately for the usage of each resource (dimension).

4.3 VITA(min)

4.3.1 Unconstrained, Bounded - exact

First, we prove two lemmas about the optimal solution which will help us constrain the search

space for our exact algorithm.

Without loss of generality assume that the bucket index j is sorted in order of increasing weight

wj .

Lemma 2. There exists an optimal solution which uses only the first b buckets, for b d. Further,

let min(j) be the dimension with the minimum value in bucket j; then, the set {min(j)|1 j b}

has all distinct elements.

Proof. It is clear that if in a solution two buckets have the same dimension with the minimum value

then the bucket with the larger weight can be emptied into the smaller without increasing the value

of the objective function. Thus the set of dimensions with the minimum value must be distinct

across buckets and therefore the optimal solution need have at most d buckets. It is also clear that if

the optimal solution does not involve a bucket j but does involve a bucket j0 > j then all the items

in bucket j0 can be moved to bucket j without increasing the value of the objective function. Thus

the optimal solution may consist only of the first b buckets, for b d.

We remind the reader that Vi(k) denotes the value in the k’th position of the vector Vi.

Lemma 3. There exists an optimal solution in which item i is placed in that bucket j for which

wj ⇤ Vi(min(j)) is minimized, amongst the first d buckets.

Proof. Suppose not. Let item i be placed in bucket j0. Now if we move it to bucket j then the value

of the objective function is changed by �wj0 ⇤ Vi(min(j0)) + wj ⇤ Vi(min(j)) which by definition

is non-positive. Contradiction, and hence proved.

59


Algorithm 6 Exact Algorithm for Unconstrained Bounded VITA(min)1: for each permutation ⇧ of the first d buckets do2: for each load vector Vi do3: Place load vector Vi in that bucket j which minimizes w⇧(j) ⇤ Vi(min(⇧(j)))

4: end for5: Compute the value of the objective function for this permutation

6: end for7: Output the best value over all permutations and the corresponding assignment

The above two lemmas give rise to a straightforward search, Algorithm .

Theorem 12. Unconstrained, Bounded VITA(min) can be computed exactly in time O(m ⇤ n ⇤ d!)

Proof. The correctness of Algorithm 4.3.1 follows from the prior two lemmas. The running time

follows from the fact that the algorithm searches over d! permutations and for each permutation it

takes O(m) time to assign each of the n load vectors.

4.3.2 Constrained, Bounded - strongly NP-hard

Theorem 13. Constrained, Bounded VITA(min) is strongly NP-hard.

Proof. The proof is by reduction from Bin Packing [35] which is strongly NP-hard. In an instance

of Bin Packing we are given m bins of the same (constant) size S and a collection of n items ai

such thatP

i ai = m ⇤ S and we need to decide if these n items can be packed into the m bins.

Given the instance of Bin Packing we create m buckets and m + n load vectors of dimension

2. m of the load vectors are of the form < S, 0 > and the vectors are matched up with the buckets

so that each such vector is necessarily assigned to its corresponding bucket. Then for each item ai

there is a load vector < 0, ai > and these vectors are unconstrained and can be assigned to any

bucket. All weights are set to 1. Now, it is easy to see that the given instance of Bin Packing is

feasible if and only if the value of the objective function of VITA(min) is m ⇤ S.

4.3.3 Unconstrained, Unbounded - inapproximable

Theorem 14. Unconstrained, Unbounded VITA(min) is inapproximable unless P = NP .

Proof. The proof is by reduction from Set Cover [35].

60


In Set Cover we are given a collection of m sets over a universe of n elements and a number

C and we need to decide whether there exists a subcollection of size C that covers all the elements.

We reduce the given instance of Set Cover to Unconstrained, Unbounded VITA(min) as fol-

lows: we let m be the dimension size as well as the number of buckets, one for each set. And, for

each element i, we have an m-dimensional load vector:

Vi(j) =

8<

:1 if element i 2 set j

1 otherwise

We set the weights of C of the buckets to be 1 and the weights of the remaining buckets to be1.

It is easy to see that the value of the objective function for Unconstrained, Unbounded VITA(min)

is C if and only if there exist C sets covering all the elements, otherwise the value of the objective

function is 1. Thus, Unconstrained, Unbounded VITA(min) cannot be approximated to any fac-

tor.

4.3.4 Constrained, Unbounded - O(log n, log n) bicriteria

Given that the problem is inapproximable (unless P=NP) we relax our expectations and settle

for the next best kind of approximation - a bicriteria approximation, [56] where we relax not just

the objective function but also the constraints. In this particular situation we will find a solution that

uses at most O(log n) copies of each bucket while obtaining an assignment whose value is no worse

than an O(log n) factor worse than the optimal solution which uses at most 1 copy of each bucket.

Consider the following LP (Linear Program). Let yjk denote the fraction bucket j gives to

dimension k, and xijk denote the weight vector i gives to dimension k of bucket j. Gjk (= 1 or 0)

denotes the affinity constraint.

minX

j

wj

X

i

X

k

xijkvik min-LP

s.t.X

k

yjk = 1 8j

xijk yjk ·Gij 8i, j, kX

j

X

k

xijk � 1 8i

xijk � 0 8i, j, k

yjk � 0 8j, k

61


Lemma 4. The above LP is a valid relaxation of Constrained, Unbounded VITA(min).

Proof. Suppose we have a solution of the original problem. Let min(j) be the minimum dimension

of bucket j, and �(i) be the bucket assigned for load vector i. The value of the objective function

for this solution isP

j wjP

i:�(i)=j Vi(min(j)). Now construct the integer solution of the LP. Let

yjk =

8<

:1 if k = m(j)

0 otherwise

and

xijk =

8<

:1 if j = �(i), k = m(j)

0 otherwise

Because each bucket only has one minimum dimension, the first constraint is satisfied. And

each vector is assigned to one bucket, so the second and third constraints are satisfied also. On the

other hand, if we have the integer solution, we can assign min(j) = k and �(i) = j to have a valid

solution of the original problem. So there is a one to one relation between the integer solutions of

the LP and the solutions of the original problem. Furthermore, the objective function of the LP is

the same as the objective function of the original problem. So the optimal integer LP solution must

map to the optimal solution of the original problem, and vice versa.

Let x⇤ijk and y⇤jk be the optimal solution of the LP. The algorithm is as follows.

Theorem 15. Algorithm 7 is an O(log n, log n) bicriteria approximation algorithm for Constrained

Bounded VITA(min).

Proof. Notice that, in our algorithm we assume that x⇤ijk = y⇤jk or x⇤ijk = 0. This is not hard to

achieve. For each item, it will order its favorite bin-dimension pair by y⇤jk values. And maximize

the corresponding x⇤ijk values in order. So there is only one x⇤ijk value that is not equal to y⇤jk value

or 0. If this x⇤ijk value is greater or equal to 12y

⇤

jk, we can round it to y⇤jk. Our new objective value

is within twice the LP value. If not, we could round it to 0, and double all the previous non-zero

x⇤ijk values. Then our value is still within twice the LP value. Even if we don’t double the previous

x⇤ijk values, we still haveP

j,k x⇤

ijk � 1/2, which we could use to bound the value output by our

algorithm.

The expected value of the solution obtained by the (above randomized) Algorithm 7 is exactly

the same as the optimum value of the LP. The expected number of copies of each bucket we make

62


Algorithm 7 Bicriteria Approximation for Constrained Bounded VITA(min)1: for each vector do2: Order its bucket-dimension pair by y⇤jkwjvikGij values. And maximize the corresponding

x⇤ijk values in order. So there will be only one x⇤ijk value that is neither equal to y⇤jk nor 0.

3: if this x⇤ijk value is greater or equal to 12y

⇤

jk, then4: round it to y⇤jk

5: else6: round it to 0, and double all the previous non-zero x⇤ijk values.

7: end if8: end for9: for ln n

" times do10: for each dimension k in each bucket j do11: With probability y⇤jk make a copy of bucket j in dimension k. And assign all the vectors

with x⇤ijk = y⇤jk to this bucket.


isP

k yjk = 1. And the probability that vector i is not assigned to one of the buckets is: (where

s = m ⇤ d),

⇧j,k(1� x⇤ijk)

1�

Pj,k x

⇤

ijk

s

!s

=

✓1�

1

s

◆s

e�1

So, if we repeat for t = ln n" times, then

Pr[some vector is not assigned]

X

i

Pr[vector i is not assigned] =n

et= "

The expected value of the solution is OPTLP · ln n" . The expected number of copies of a bucket is

ln n" . Thus Algorithm 7 gives a (log n, log n)-approximation to Constrained Bounded VITA(min).

4.4 VITA(max)

Max�Min and Max are very similar, in that for the lower bound we can use the same log-

hardness result since min is 0 and for the upper bound we can set the y variable to be greater than

the difference of two dimensions for every pair of dimensions.

63


4.4.1 Unconstrained, Unbounded - exact

Theorem 16. Unconstrained, Unbounded VITA(max) can be computed exactly in time O(m + n)

time by placing all items into the bucket with the smallest weight.

Proof. We first show that the bucket with the smallest weight will always be used in the optimal

solution. If the bucket with smallest weight is not used in the optimal solution, we can always

move all the items in one bucket with non-smallest weight to the bucket with the smallest weight to

improve the solution.

Now, we show that if we move all the items in the buckets with non-smallest weight to the

bucket with smallest weight, the objective value of this new solution will not increase.

To see this, let the bucket B0with the smallest weight w0. Let the aggregated vector in B0 be

V0. Let the bucket Bi with a non-smallest weight wi in the solution, the aggregated vector in Bi be

Vi.

It is easy to see that w0 ·max(V0 + Vi) w0 · (max(V0) +max(V1)) w0 ·max(V0) +wi ·

max(Vi).

Thus, moving all items from Bi to B0 will not increase the objective value of the current

solution.

Moving all items to the smallest weighted buckets is optimal.

4.4.2 Constrained, Bounded - strongly NP-hard

Theorem 17. Constrained, Bounded VITA(max) is strongly NP-complete even when the number of

dimension equals 2.

Proof. We prove by making reduction from bin packing. For k bins with capacity c, we correspond-

ingly assign k buckets. As part of input vectors, we will have k 2-dimensional vectors (c, 0). Each

of them are strictly constrained to each bucket. Then for each item i with size si in the problem

of bin packing, we create a 2-dimensional vector (0, si) which can be put into any bucket. We fur-

ther let each bucket have uniform weight of 1. Then there exists k bins that can hold all the items

in the bin packing problem if and only if the objective value of this VITA(max) that equals kc is

reachable.

64


4.4.3 Constrained, Unbounded - ⇥(log n) approximation

Lemma 5. Constrained, Unbounded VITA(max) is strongly NP-complete, and can not be approxi-

mated within O(log n).

Proof. We prove by making reduction from set cover. First we let the number of dimensions of

input vector in VITA(max) be the number of elements in the set cover problem. For each element

si(i = 1 ⇠ n), we correspondingly let vector Vi have value one on dimension i, and value zero on

all the other dimensions. Thus, there are no two element vectors that have value of 1 on the same

dimension.

Each subset Sj maps to a bucket Bj . If element si 2 Sj , then Vi can be placed at bucket Bj .

Thus, there exist k subsets that cover all the elements if and only if the objective value of this

VITA(max) that equals k is reachable.

Lemma 6. Constrained, Unbounded VITA(max) is O(log n) approximable.

Proof. Consider the following LP. Let xij be the fraction of item i assigned to bucket j.

minmX

j=1

wj ⇤ yj max�LP

s.t. yj �nX

i=1

xij · vik 8j, k

xij Gij , 8i, jmX

j=1

xij � 1 8i

It is easy to see that this max-LP is a valid relaxation of constrained, unbounded VITA(max).

Then we need to repeat rounding {xij} O(log n) times to make sure that all items are placed to

some buckets with high probability. The proof is similar to the part in min-LP.

Directly from Lemma 5 and 6, we get the following.

Corollary 4. Constrained, Unbounded VITA(max) is ⇥(log n) approximable.

65


4.5 VITA(2ndmax)

We found VITA(2ndmax) to be a qualitatively harder problem and thus were forced to con-

sider the restricted version where the weights are uniform and the number of buckets exceeds the

(bounded) number of dimensions.

4.5.1 Bounded, Unconstrained - weakly NP-hard

Theorem 18. Bounded, Unconstrained VITA(2ndmax) is weakly NP-hard.

Proof. The proof is by reduction from Partition [35]. in Partition we are given an array of numbers

a1, a2, . . . , an such thatPn

i=1 ai = 2B, and we are required to decide whether there exist a partition

of these numbers into two subsets such the sum of numbers in either subset is B.

Our reduction will use 3 dimensions. for each number we construct a load vector < 0, 0, ai >.

And we add another two vectors, < L,B, 0 > and < B,L, 0 > (L >> B), to the collection of

vectors. There are totally two buckets. Each bucket has 3 dimensions and their costs are the same

(let’s say c). From our construction, it is easy to see there is a partition of that array of numbers if

and only if the minimum cost of this bin packing (measured by maximum dimension values of the

bins) is 2Bc. Because L >> B, we cannot afford to put items < L,B, 0 > and < B,L, 0 > in the

same bucket. They are separated in the two buckets. If there is a partition, then those two buckets

will be in the shapes of < L,B,B > and < B,L,B >. If there is not, then one of the bucket has

a larger maximum dimension value then B, which means the objective value has to be larger than

2Bc.

4.5.2 Unweighted, Unconstrained, with number of buckets exceeding number of di-mensions - O(log n) approximation

Consider the following LP. Let xij be the fraction of vector i assigned to bucket j.

66


minmX

j=1

yj 2ndmax�LP

s.t. yj �nX

i=1

xij · vik 8j, k (j 6= k)

mX

j=1

xij � 1 8i

Lemma 7. The above LP 2ndmax-LP is a valid relaxation of unconstrained VITA(2ndmax) where

the number of buckets exceeds the number of dimensions.

Proof. First, let us explain the constraints. For each bucket Bj , we name the dimension j of Bj a

“taken-out” dimension. The value of yj for Bj will obviously take the minimum dimension value

of the vector sum of Bj from all Bj’s dimensions except the “taken-out” dimension.

Now, it is easy to see that the value of yj can only be the maximum dimension value or the 2nd

max dimension value of Bj’s vector sum. Then, if yj represents the maximum dimension value of

Bj . Let dj 6= j denote the the dimension of the maximum dimension of Bj which is different with

the “taken-out” dimension.

Then the two of the following cases can happen.

1. The bucket Bdj ’s maximum dimension is the same as its “taken-out” dimension dj . Then

in this case, simply by moving all the vectors in Bj to Bdj . We can achieve a solution

with objective no greater than the previous one. Also, now ydj will represent the 2nd max

dimension value of bucket Bdj .

2. The bucket Bdj ’s maximum dimension is different with its “taken-out” dimension dj . Then in

this case, we claim we can always find the subset of buckets, for each of which, the maximum

dimension is different to the “taken-out” dimension. Then we can do reassignment of “taken-

out” dimensions for this subset of buckets. The a solution in which yj represents 2nd max

dimension of each Bj also achieves an objective value no greater than the original one.

Thus, Minimizing over the feasible solutions will get the minimum vector placement based on

2nd max dimension of each bucket.

67


Theorem 19. Unweighted, Unconstrained, VITA(2ndmax) with number of buckets exceeding num-

ber of dimensions can be approximated to factor O(log n).

Proof. As with the algorithm and proof for min-LP, we need to repeat rounding {xij} O(log n)

times to make sure that all vectors are placed in some bucket with high probability.

4.6 Experiments

In this section, we run numerical experiments to test the performance for VITA(max) and

VITA(min) against three heuristics with industrial trace data from Nutanix and synthesized data.

We first give an introduction to these three heuristics for VITA: (1) Conservative vector placement;

(2) Greedy-based vector placement; and (3) Local search vector placement.

We set up data sets and test the two following cases:

• When the number of vector dimensions is a constant: In this case, VITA(max) or VITA(min)

will both correspond to a real world application. VITA(max) will be useful in minimizing

bottleneck usage. VITA(min) will be useful in minimizing downtime.

• When the number of vector dimensions is unbounded: In this case, the performance of VITA

is tested to verify our theory results.

4.6.1 Polynomial time heuristics for VITA

4.6.1.1 Heuristic 1 - conservative vector placement

Each vector Vi will be placed to the bucket j such that the product of the value of F (Vi) · wj

is the smallest among all the the affinity constrained buckets, where F is still max(·) for the max

case, but avg(·) for the min case.

4.6.1.2 Heuristic 2 - greedy-based vector placement

Greedy-based heuristics are often very efficient and also lead to quite good performance [62].

The greedy heuristics for VITA can be found in Algorithm 9.

4.6.1.3 Heuristic 3 - local search vector placement

Local search based vector placement in Algorithm 10 starts from a random feasible placement

and keeps swapping vector from two different bins to decrease the total cost. Since the size of search

68


Algorithm 8 Heuristic 1 - Conservative1: for each vector do2: Place the vector to the bucket so that the product of the maximum (average) dimension value

and bucket weight is the smallest in max (min) case.

3: end for

Algorithm 9 Heuristic 2 - Greedy1: Shuffle the order of vectors;

2: for each vector do3: Place the vector to the bucket so that the current objective value is raised the least;

4: end for

space is exponential to the number of vector (size of input), we only do such swapping for poly(n)

times.

4.6.2 Performance of VITA when vectors has constant number of dimensions

In this subsection, we evaluate VITA’s performance when the input vectors only have constant

number of dimensions. When the input vectors have constant number of dimensions, we show

two real world applications in cloud industry for VITA(max) and VITA(min) correspondingly: (1)

minimizing bottleneck usage; (2) minimizing maintenance downtime.

4.6.2.1 Experiment for minimizing bottleneck usage

We get one subset of trace data from Nutanix. It consists of 12, 371 job time stamped requests

of a month. Each requests resources of CPU, memory and storage, which can be represented as a 3

dimensional vector.

Fig. 4.3, Fig. 4.4, Fig. 4.5 show the request histograms of VCPU, memory, storage.

As we can see, different dimension will have different units of measure. The industrial practice

of cloud data centers is to equip each individual physical machines’ parts in a balanced way. For

example, there is no reason to equip with more CPUs with small memory, vice versa. Thus, in order

to make the value comparable across dimensions, we scale each resource request value by a factor

of 100 over the maximum possible value of each dimension, then get a normalized value between 0

and 100.

69


Algorithm 10 Heuristic 3 - Local search1: for each vector do2: Randomly assign it to a feasible bucket by affinity constraint;

3: end for4: for 1 to poly(n) steps do5: for every two buckets do6: Swap any pair of two vectors if the swap will reduce the objective value;


We pick the number of buckets ranging from 2 to 20. For each bucket number k, we sample 5k

vectors from the request data from Nutanix and try to place them into these buckets at the minimized

objective value. We randomly generate affinity constraints between these 5k vectors and k buckets.

Each bucket is randomly assigned a weight with an integral value between 1 and 5.

For each vector number from 10 to 100, we repeat VITA(max) against the three heuristics on

such input 100 times and get an averaged value for objective value. Fig. 4.6 shows the result such

experiment. For this scenario, it shows that VITA(max) performs similarly to greedy, slightly better

than conservative. Local search performs worst.

4.6.2.2 Experiment for minimizing maintenance downtime

Instead of representing various kinds of resource request, each vector in this data set represents

a load profile for one job continuously over time. We sample from request data of 24 hours, and

use the CPU dimension as evidence of activity at each hour. Each bucket represents one integral

physical host facility, i.e. one host cluster, that can be shut down at a certain time slot.

By following the same sampling procedure as before, we get Fig. 4.7 for the scenario of

minimizing maintenance downtime. Fig. 4.7 shows that VITA(min) performs the best. Conservative

performs the worst. Since VITA(min) is a (log n, log n) bi-criteria approximation algorithm, Fig.

4.8 shows the number of used buckets vs the number of given buckets. Fig. 4.9 shows that given

the same number of extra buckets Greedy and Local search will perform better than our bicriteria

VITA(min).

70


0 100 200 300 400 500 600 700 800 900 1000 1100

# of VCPU

0

20

40

60

80

100

120

140

160

180

# o

f re

quest

s

Figure 4.3: Histogram of # of VCPUs requested

4.6.3 Performance of VITA when vectors has unbounded number of dimensions

In order to evaluate our algorithm for the case when vectors can have variable number of

dimensions, for both experiments of VITA(max) and VITA(min), we synthesize the data set to do

the experiments.

We let the number of buckets range from 2 to 20. We let the number of dimension of each

vector equal to the number of buckets. We further let the number of vectors be 5 times the number

of buckets.

For each vector, we sample uniformly from 0 to 100 for each dimension. Then we test

VITA(max)/VITA(min) against the three heuristics. The experiment results are presented as fol-

lows.

4.6.3.1 VITA(max) for vectors with unbounded number of dimensions

Fig. 4.10 shows VITA(max) performs similarly to greedy, conservative. Local search performs

worst.

71


0 100 200 300 400 500 600

Memory (GB)

0

500

1000

1500

2000

2500

# o

f re

quest

s

Figure 4.4: Histogram of memory size requested

4.6.3.2 VITA(min) for vectors with unbounded number of dimensions

Fig. 4.11 shows VITA(min) performs in terms of average objective value, at the cost of using

extra number of buckets as shown in Fig. 4.12, while in Fig. 4.13 shows Greedy and Conservative

perform better than VITA(min) given the same number of extra buckets.

4.7 Summary

We have proposed a new framework VITA that captures several naturally occurring problems

in the context of hybrid clouds. We presented hardness results and approximation algorithms (using

LP rounding) for several problems. We also presented experimental results comparing our approxi-

mation algorithm to several natural heuristics.

72


0 500 1000 1500 2000 2500 3000 3500 4000 4500

Storage size (GB)

0

500

1000

1500

2000

2500

3000

3500

# o

f re

quest

s

Figure 4.5: Histogram of storage size requested

10 20 30 40 50 60 70 80 90 100

# of vectors

0

5000

10000

15000

Ave

rage o

bje

ctiv

e v

laue

VITA(max)

Conservative

Greedy

Local search

Figure 4.6: Objective value of VITA(max) and three heuristics for minimizing bottleneck usage

73


10 20 30 40 50 60 70 80 90 100

# of vectors

500

1000

1500

2000

2500

3000

3500

4000

4500

Ave

rage o

bje

ctiv

e v

alu

e

VITA(min)

Conservative

Greedy

Local search

Figure 4.7: Objective value of VITA(min) and three heuristics for minimizing maintenance down-time

2 4 6 8 10 12 14 16 18 20

# of given bins

0

5

10

15

20

25

30

35

40

45

# o

f use

d b

ins

VITA(min)

Figure 4.8: # of used buckets VS # of given buckets for minimizing maintenance downtime

74


10 20 30 40 50 60 70 80 90 100

# of vectors

0

500

1000

1500

2000

2500

3000

3500

4000

4500

Ave

rag

e o

bje

ctiv

e v

alu

e

VITA(min)

Conservative

Greedy

Local search

Figure 4.9: Objective value with same increased number of buckets for minimizing maintenancedowntime

10 20 30 40 50 60 70 80 90 100

# of vectors

0

2000

4000

6000

8000

10000

12000

14000

16000

18000

Ave

rage o

bje

ctiv

e v

alu

e

VITA(max)

Conservative

Greedy

Local search

Figure 4.10: Objective value of VITA(max) with unbounded # of dimensions

75


10 20 30 40 50 60 70 80 90 100

# of vectors

500

1000

1500

2000

2500

3000

3500

4000

4500

Ave

rage o

bje

ctiv

e v

alu

e

VITA(min)

Conservative

Greedy

Local search

Figure 4.11: Objective value of VITA(min) with unbounded # of dimensions

2 4 6 8 10 12 14 16 18 20

# of given bins

0

5

10

15

20

25

30

35

40

45

# o

f use

d b

ins

VITA(min)

Figure 4.12: # of used buckets VS # of given buckets with unbounded # of dimensions

76


10 20 30 40 50 60 70 80 90 100

# of vectors

500

1000

1500

2000

2500

3000

3500

4000

4500

Ave

rag

e o

bje

ctiv

e v

alu

e

VITA(min)

Conservative

Greedy

Local search

Figure 4.13: Objective value with same increased number of buckets with unbounded # ofdimensions

77

Chapter 5

Conclusion

Electricity and computation are the two critical resources to the information era. This thesis

studies load balancing problems that aims to efficiently use these resources. Specifically, for tem-

poral load balancing for electricity market, fault-tolerant load balancing for data centers and load

balancing problem with multidimensional resources, we have the following results.

• Temporal load balancing for electricity market: We have presented a general incentive-

based mechanism, SmartShift, for reducing the load on the electricity grid. Our scheme

grants users increased consumption in exchange for reducing their usage in peak periods. We

have shown analytically that SmartShift under flat-rate pricing is a win-win for both con-

sumers (increased social welfare) and producers (enhanced profits). SmartShift has elements

of algorithms for iterative price setting [59, 46] with the added features of in-kind incentives

and a slot-pairwise bidding language.

• Spatial load balancing for data centers: We study the HA problem in VM placement and

prove novel complexity-theoretic results, particularly the surprising result that the existence

of a k-HA placement is in NP . We leave open the problem of deciding the exact complexity

of deciding whether a given placement is k-HA or not. We propose a natural stochastic model

- IID-IK (Independently and Identically Distributed Items and Knapsacks) - which provides

a uniform basis for comparing heuristics. We also show that interestingly there exists a best

heuristic in this model whose packing efficiency is computable. We utilize this model and

analyze a variety of heuristics used in practice; we also consider natural special cases of input

distributions that occur in the real world and conclude that Water-filling is the best.

78

CHAPTER 5. CONCLUSION

• Load balancing with multidimensional resources: We have proposed a new framework

VITA that captures several naturally occurring problems in the context of hybrid clouds.

We presented hardness results and approximation algorithms (using LP rounding) for several

problems. We also presented experimental results comparing our approximation algorithm to

several natural heuristics.

Our work suffers from a number of limitations which can also be seen as opportunities for

future work. One natural direction for future investigation is devising distributed and on-

line algorithms. Our experimental work indicates that VITA(max) works similarly to greedy

and conservative, VITA(min) works better than all the three heuristics at the cost of us-

ing extra buckets. Our theoretical work has left some obvious open gaps including con-

strained bounded VITA(min) and VITA(max) and removing the restrictions from our results

for VITA(2ndmax).

79

Bibliography

[1] Amazon web services - cloud computing services. https://aws.amazon.com/.

[2] Datadog - modern monitoring and analytics. https://www.datadoghq.com/.

[3] Google cloud platform. https://cloud.google.com/.

[4] Hewlett packard enterprise - hybrid it with cloud. https://www.hpe.com/us/en/

home.html.

[5] Microsoft azure cloud computing platform and services. https://azure.microsoft.

com/en-us/.

[6] Rightscale. https://www.rightscale.com/.

[7] Turbonomic. https://turbonomic.com/.

[8] Hunt Allcott. Real time pricing and electricity markets. Harvard University, 2009.

[9] Hunt Allcott. Rethinking real-time electricity pricing. Resource and Energy Economics,

33(4):820–842, 2011.

[10] Nikhil Bansal, Alberto Caprara, and Maxim Sviridenko. A new approximation method for

set covering problems, with applications to multidimensional bin packing. SIAM J. Comput.,

39(4):1256–1278, 2009.

[11] Nikhil Bansal and Arindam Khan. Improved approximation algorithm for two-dimensional

bin packing. In Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete

Algorithms, SODA 2014, Portland, Oregon, USA, January 5-7, 2014, pages 13–25, 2014.

[12] Galen L. Barbose, Charles A. Goldman, and Bernie Neenan. A survey of utility experience

with real time pricing. page 127, 12/2004 2004.

80

BIBLIOGRAPHY

[13] Sean Barker, Aditya Mishra, David Irwin, Emmanuel Cecchet, Prashant Shenoy, and Jeannie

Albrecht. Smart*: An open data set and tools for enabling research in sustainable homes. In

Proceedings of the 2012 Workshop on Data Mining Applications in Sustainability (SustKDD

2012), Beijing, China, August 12, 2012.

[14] Tamer Basar, Geert Jan Olsder, GJ Clsder, T Basar, T Baser, and Geert Jan Olsder. Dynamic

noncooperative game theory, volume 200. SIAM, 1995.

[15] Eyal Bin, Ofer Biran, Odellia Boni, Erez Hadad, Eliot K Kolodner, Yosef Moatti, and Dean H

Lorenz. Guaranteeing high availability goals for virtual machine placement. In Distributed

Computing Systems (ICDCS), 2011 31st International Conference on, pages 700–709. IEEE,

2011.

[16] Severin Borenstein. The trouble with electricity markets: Understanding California’s restruc-

turing disaster. The Journal of Economic Perspectives, 16(1):191–211, 2002.

[17] T. Carpenter, S. Singla, P. Azimzadeh, and S. Keshav. The impact of electricity pricing

schemes on storage adoption in ontario. In e-Eenergy, 2012.

[18] Hung-po Chao. Price-responsive demand management for a smart grid world. The Electricity

Journal, 23(1):7–20, 2010.

[19] Hung-po Chao. Competitive electricity markets with consumer subscription service in a smart

grid. Journal of Regulatory Economics, 41(1):155–180, 2012.

[20] C. Chekuri and S. Khanna. On multi-dimensional packing problems. In SODA: ACM-SIAM

Symposium on Discrete Algorithms (A Conference on Theoretical and Experimental Analysis

of Discrete Algorithms), 1999.

[21] Gong Chen, Wenbo He, Jie Liu, Suman Nath, Leonidas Rigas, Lin Xiao, and Feng Zhao.

Energy-aware server provisioning and load dispatching for connection-intensive internet ser-

vices. In Proceedings of the 5th USENIX Symposium on Networked Systems Design and Im-

plementation, NSDI’08, pages 337–350, 2008.

[22] Henrik I. Christensen, Arindam Khan, Sebastian Pokutta, and Prasad Tetali. Multidimensional

bin packing and other related problems: A survey. https://people.math.gatech.

edu/˜tetali/PUBLIS/CKPT.pdf.

81

BIBLIOGRAPHY

[23] Edward G Coffman, Kimming So, Micha Hofri, and AC Yao. A stochastic model of bin-

packing. Information and Control, 44(2):105–115, 1980.

[24] C. Courcoubetis and R. Weber. Pricing Communication Networks: Economics, Technology

and Modeling. Wiley, 2003.

[25] W. Fernandez de la Vega and G. S. Lueker. Bin packing can be solved within 1+✏ in linear

time. Combinatorica, 1:349–355, 1981.

[26] J. de Vries. The Industrious Revolution: Consumer Behavior and the Household Economy,

1650 to the Present. Cambridge University Press, 2008.

[27] Valeria Di Cosmo, Sean Lyons, and Anne Nolan. Estimating the impact of time-of-use pricing

on Irish electricity demand. MPRA Paper 39971, University Library of Munich, Germany,

July 2012.

[28] Xenofontas A. Dimitropoulos, Paul Hurley, Andreas Kind, and Marc Ph. Stoecklin. On the

95-percentile billing method. In Passive and Active Network Measurement, 10th International

Conference, PAM 2009, Seoul, Korea, April 1-3, 2009. Proceedings, pages 207–216, 2009.

[29] Corentin Dupont, Fabien Hermenier, Thomas Schulze, Robert Basmadjian, Andrey Somov,

and Giovanni Giuliani. Plug4green: A flexible energy-aware VM manager to fit data centre

particularities. Ad Hoc Networks, 25:505–519, 2015.

[30] Corentin Dupont, Thomas Schulze, Giovanni Giuliani, Andrey Somov, and Fabien Hermenier.

An energy aware framework for virtual machine placement in cloud federated data centres. In

e-Energy, page 4. ACM, 2012.

[31] H. Farhangi. The path of the smart grid. IEEE Power and Energy Magazine, 8(1):18–28, 2010.

[32] A.M. Frieze and M. Clarke. Approximation algorithms for the m-dimensional 0-1 knapsack

problem: worst-case and probabilistic analyses. European J. Oper. Res., 15(1):100–109, 1984.

[33] M. R. Garey and D. S. Johnson. Computers and Intractability: A Guide to the Theory of

NP-Completeness. W. H. Freeman & Co., New York, NY, USA, 1979.

[34] Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to the Theory

of NP-Completeness. W. H. Freeman & Co., New York, NY, USA, 1979.

82

BIBLIOGRAPHY

[35] Michael R. Garey and David S. Johnson. Computers and Intractability: A Guide to the Theory

of NP-Completeness. W. H. Freeman & Co., New York, NY, USA, 1979.

[36] Ali Ghodsi, Matei Zaharia, Benjamin Hindman, Andy Konwinski, Scott Shenker, and Ion

Stoica. Dominant resource fairness: Fair allocation of multiple resource types. In Proceedings

of the 8th USENIX Symposium on Networked Systems Design and Implementation, NSDI 2011,

Boston, MA, USA, March 30 - April 1, 2011, 2011.

[37] Varun Gupta and Ana Radovanovic. Online stochastic bin packing. CoRR, abs/1211.2687,

2012.

[38] Chris Harris. Electricity Markets: Pricing, Structures and Economics. Wiley, Sussex, England,

2006.

[39] R. Hemecke, M. Koeppe, J. Lee, and R. Weismantel. Nonlinear integer programming. In

M. Juenger, editor, 50 years of Integer Programming 1958-2008: The Early Years and State-

of-the-Art Surveys, pages 561–618. Springer-Verlag, Berlin, 2009.

[40] Fabien Hermenier, Julia Lawall, and Gilles Muller. Btrplace: A flexible consolidation manager

for highly available applications. IEEE Transactions on dependable and Secure Computing,

10(5):273–286, 2013.

[41] Fabien Hermenier, Julia L. Lawall, and Gilles Muller. Btrplace: A flexible consolidation

manager for highly available applications. IEEE Trans. Dependable Sec. Comput., 10(5):273–

286, 2013.

[42] D.S. Hochbaum and D.B. Shmoys. Using dual approximation algorithms for scheduling prob-

lems: theoretical and practical results. Journal of the ACM, 34:144–162, 1987.

[43] MD Ilic, L. Xie, and J.Y. Joo. Efficient coordination of wind power and price-responsive

demand Part I: Theoretical foundations. IEEE Transactions on Power Systems, 26(4):1875–

1884, 2011.

[44] International Energy Agency. The power to choose - enhancing demand response in liberalised

electricity markets findings of IEA demand response project. 2003.

[45] Azadeh Jahanbanifar, Ferhat Khendek, and Maria Toeroe. Providing hardware redundancy

for highly available services in virtualized environments. In Software Security and Reliability,

2014 Eighth International Conference on, pages 40–47. IEEE, 2014.

83

BIBLIOGRAPHY

[46] Shweta Jain, Narayanaswamy Balakrishnan, Yadati Narahari, Saiful A Hussain, and

Nyuk Yoong Voo. Constrained tatonnement for fast and incentive compatible distributed de-

mand management in smart grids. In Proceedings of the fourth international conference on

Future energy systems, pages 125–136. ACM, 2013.

[47] Manar Jammal, Ali Kanso, and Abdallah Shami. Chase: Component high availability-aware

scheduler in cloud computing environment. In Cloud Computing (CLOUD), 2015 IEEE 8th

International Conference on, pages 477–484. IEEE, 2015.

[48] Klaus Jansen. A fast approximation scheme for the multiple knapsack problem. In Inter-

national Conference on Current Trends in Theory and Practice of Computer Science, pages

313–324. Springer, 2012.

[49] Rob Jansen, Kevin S. Bauer, Nicholas Hopper, and Roger Dingledine. Methodically modeling

the tor network. In 5th Workshop on Cyber Security Experimentation and Test, CSET ’12,

Bellevue, WA, USA, August 6, 2012, 2012.

[50] Edward G. Coffman Jr., Costas Courcoubetis, M. R. Garey, David S. Johnson, Lyle A. Mc-

Geoch, Peter W. Shor, Richard R. Weber, and Mihalis Yannakakis. Fundamental discrepancies

between average-case analyses under discrete and continuous distributions: A bin packing case

study. In Proceedings of the 23rd Annual ACM Symposium on Theory of Computing, May 5-8,

1991, New Orleans, Louisiana, USA, pages 230–240, 1991.

[51] S. Keshav and C. Rosenberg. How internet concepts and technologies can help green and

smarten the electrical grid. ACM SIGCOMM Computer Communication Review, 41(1):109–

114, 2011.

[52] N. Li, L. Chen, and S.H. Low. Optimal demand response based on utility maximization in

power networks. In IEEE Power and Energy Society General Meeting, pages 1 –8, july 2011.

[53] Zhenhua Liu, Minghong Lin, Adam Wierman, Steven H Low, and Lachlan LH Andrew. Green-

ing geographical load balancing. In Proceedings of the ACM SIGMETRICS joint international

conference on Measurement and modeling of computer systems, pages 233–244. ACM, 2011.

[54] Scott Loveland, Eli M Dow, Frank LeFevre, Duane Beyer, and Phil F Chan. Leverag-

ing virtualization to optimize high-availability system configurations. IBM Systems Journal,

47(4):591–604, 2008.

84

BIBLIOGRAPHY

[55] Siva Theja Maguluri, R Srikant, and Lei Ying. Stochastic models of load balancing and

scheduling in cloud computing clusters. In INFOCOM, 2012 Proceedings IEEE, pages 702–

710. IEEE, 2012.

[56] Madhav V. Marathe, R. Ravi, Ravi Sundaram, S. S. Ravi, Daniel J. Rosenkrantz, and Harry

B. Hunt III. Bicriteria network design problems. J. Algorithms, 28(1):142–171, 1998.

[57] Silvano Martello and Paolo Toth. Heuristic algorithms for the multiple knapsack problem.

Computing, 27(2):93–112, 1981.

[58] Nick Mathewson and Roger Dingledine. Practical traffic analysis: Extending and resisting

statistical disclosure. In Privacy Enhancing Technologies, 4th International Workshop, PET

2004, Toronto, Canada, May 26-28, 2004, Revised Selected Papers, pages 17–34, 2004.

[59] A. Mohsenian-Rad, V.W.S. Wong, J. Jatskevich, R. Schober, and A. Leon-Garcia. Au-

tonomous demand-side management based on game-theoretic energy consumption scheduling

for the future smart grid. IEEE Transactions on Smart Grid, 1(3), 2010.

[60] A-H Mohsenian-Rad and Alberto Leon-Garcia. Optimal residential load control with price

prediction in real-time electricity pricing environments. Smart Grid, IEEE Transactions on,

1(2):120–133, 2010.

[61] Susanta Nanda and Tzi cker Chiueh. A survey of virtualization technologies. Technical report,

StonyBrook University, 2005.

[62] Rina Panigrahy, Kunal Talwar, Lincoln Uyeda, and Udi Wieder. Heuristics for vector bin

packing. research. microsoft. com, 2011.

[63] Christos M. Papadimitriou. Computational complexity. Addison-Wesley, Reading, Mas-

sachusetts, 1994.

[64] Ilia Pietri and Rizos Sakellariou. Mapping virtual machines onto physical machines in cloud

computing: A survey. ACM Comput. Surv., 49(3):49:1–49:30, October 2016.

[65] Prabhakar Raghavan and Clark D. Thompson. Randomized rounding: a technique for provably

good algorithms and algorithmic proofs. Combinatorica, 7(4):365–374, 1987.

[66] Vamseedhar Reddyvari Raja, Amogh Dhamdhere, Alessandra Scicchitano, Srinivas Shakkot-

tai, kc claffy, and Simon Leinen. Volume-based transit pricing: Is 95 the right percentile? In

85

BIBLIOGRAPHY

Passive and Active Measurement - 15th International Conference, PAM 2014, Los Angeles,

CA, USA, March 10-11, 2014, Proceedings, pages 77–87, 2014.

[67] Vamseedhar Reddyvari Raja, Srinivas Shakkottai, Amogh Dhamdhere, and kc claffy. Fair,

flexible and feasible ISP billing. SIGMETRICS Performance Evaluation Review, 42(3):25–28,

2014.

[68] Lei Rao, Xue Liu, Le Xie, and Wenyu Liu. Minimizing electricity cost: Optimization of

distributed internet data centers in a multi-electricity-market environment. In INFOCOM,

2010 Proceedings IEEE, pages 1–9. IEEE, 2010.

[69] M. Roozbehani, Munther Dahleh, and S. Mitter. On the stability of wholesale electricity

markets under real-time pricing. In 49th IEEE Conference on Decision and Control (CDC),

2010, pages 1911–1918, Dec 2010.

[70] Mardavij Roozbehani, Munther A Dahleh, and Sanjoy K Mitter. Volatility of power grids

under real-time pricing. Power Systems, IEEE Transactions on, 27(4):1926–1940, 2012.

[71] Siqi Shen, Alexandru Iosup, Assaf Israel, Walfredo Cirne, Danny Raz, and Dick Epema. An

availability-on-demand mechanism for datacenters. In Cluster, Cloud and Grid Computing

(CCGrid), 2015 15th IEEE/ACM International Symposium on, pages 495–504. IEEE, 2015.

[72] Peter W. Shor. the average-case analysis of some on-line algorithms for bin packing. Combi-

natorica, 6(2):179–200, 1986.

[73] Ionica Smeets, Arjen K. Lenstra, Hendrik Lenstra, Laszlo Lovasz, and Peter van Emde Boas.

The history of the lll-algorithm. In The LLL Algorithm - Survey and Applications, pages 1–17.

2010.

[74] VK Sood, D. Fischer, JM Eklund, and T. Brown. Developing a communication infrastructure

for the smart grid. In Electrical Power & Energy Conference (EPEC), 2009 IEEE, pages 1–7.

Ieee, 2009.

[75] Mark Stillwell, Frederic Vivien, and Henri Casanova. Virtual machine resource allocation for

service hosting on heterogeneous distributed platforms. In 26th IEEE International Parallel

and Distributed Processing Symposium, IPDPS 2012, Shanghai, China, May 21-25, 2012,

pages 786–797, 2012.

86

BIBLIOGRAPHY

[76] Jelle van den Hooff, David Lazar, Matei Zaharia, and Nickolai Zeldovich. Vuvuzela: scalable

private messaging resistant to traffic analysis. In Proceedings of the 25th Symposium on Oper-

ating Systems Principles, SOSP 2015, Monterey, CA, USA, October 4-7, 2015, pages 137–152,

2015.

[77] P. Vytelingum, T.D. Voice, S.D. Ramchurn, A. Rogers, and N.R. Jennings. Agent-based micro-

storage management for the smart grid. In AAMAS, pages 39–46, 2010.

[78] D.A. Walker. Walras’s theories of tatonnement. The Journal of Political Economy, 95(4):758–

774, 1987.

[79] Allen J Wood and Bruce F Wollenberg. Power generation, operation, and control. John Wiley

& Sons, 2012.

[80] Song Yang, Philipp Wieder, and Ramin Yahyapour. Reliable virtual machine placement in dis-

tributed clouds. In Resilient Networks Design and Modeling (RNDM), 2016 8th International

Workshop on, pages 267–273. IEEE, 2016.

[81] Ao Zhou, Shangguang Wang, Zibin Zheng, Ching-Hsien Hsu, Michael R Lyu, and Fangchun

Yang. On cloud service reliability enhancement with optimal resource usage. IEEE Transac-

tions on Cloud Computing, 4(4):452–466, 2016.

87

algorithms for load balancing in electricity markets and data centerscj... · 2021. 4. 13. · load...

Documents