enabling renewable energy powered sustainable high … · 2015. 1. 8. · enabling renewable energy...

ENABLING RENEWABLE ENERGY POWERED SUSTAINABLE HIGH-PERFORMANCE COMPUTING

By

CHAO LI

A DISSERTATION PRESENTED TO THE GRADUATE SCHOOL OF THE UNIVERSITY OF FLORIDA IN PARTIAL FULFILLMENT

OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY

UNIVERSITY OF FLORIDA

2014

This work is dedicated to my family.

4

ACKNOWLEDGMENTS

First and foremost I would like to express my sincerest gratitude to my advisor,

Dr. Tao Li, who has been a pillar of support in my graduate study. He is an inspiring

mentor who teaches me how to break through various obstacles and get things done

with intelligence, creativity, and perseverance. I could not make such a strong record of

achievements without his guidance, patience, and encouragement.

My supervisory committee guided me throughout my proposal and thesis. I want

to thank Dr. Jose A.B. Fortes, Dr. Renato Figueiredo, and Dr. Prabir Barooah, for their

valuable feedback and comments. It has been a highly enjoyable learning experience

from Dr. Jose A.B. Fortes, who taught me Autonomic Computing, and Dr. Renato

Figueiredo, who taught me Virtual Computers.

Dozens of people have helped me immensely in the IDEAL Research group.

Thanks to them, my working experience at the University of Florida is productive and

enjoyable. I want to thank Dr. Wangyuan Zhang and Dr. Xin Fu, who provided

enormous amount of help when I first joined the group. I want to thank Dr. James Poe II

and Dr. Clay Hughes, who helped with proofreading in my first year of graduate study.

Thank you to those who have spent a lot of time on the development of our 2nd and 3rd

generation system prototype: Yang Hu, Juncheng Gu, Longjun Liu, and Jingling Yuan.

Also thank you to Ruijin Zhou, Ming Liu, and Amer Qouneh, who have supported me in

many ways such as experimental setup and system trouble-shooting.

This research is generously funded by a University of Florida Alumni Fellowship

Award, a Yahoo! Key Scientific Challenges Program Award in Green Computing, and a

Facebook Graduate Fellowship Award in Computer Architecture.

5

TABLE OF CONTENTS

page

ACKNOWLEDGMENTS .................................................................................................. 4

LIST OF TABLES ............................................................................................................ 7

LIST OF FIGURES .......................................................................................................... 8

LIST OF ABBREVIATIONS ........................................................................................... 11

ABSTRACT ................................................................................................................... 12

CHAPTER

1 INTRODUCTION AND BACKGROUND ................................................................. 13

2 RELATED WORK ................................................................................................... 16

3 SERVER-LEVEL RENEWABLE POWER MANAGEMENT: RETHINKING LOAD ADAPTATION SCHEMES ...................................................................................... 19

3.1 PV Systems and Load Matching ....................................................................... 21

3.2 Managing Solar Energy Powered Multi-core System ........................................ 22

3.2.1 Multi-core Aware MPP Tracking .............................................................. 24

3.2.2 Dynamic Per-Core Load Tuning .............................................................. 26

3.3 Experimental Methodology ............................................................................... 29

3.4 Evaluation Results ............................................................................................ 32

3.4.1 Power Tracking Accuracy ........................................................................ 33

3.4.2 Fixed Power Budget ................................................................................ 34

3.4.3 Solar Energy Utilization ........................................................................... 36

3.4.4 Performance Improvement ...................................................................... 37

3.5 Summary of SolarCore Design ......................................................................... 39

4 CLUSTER LEVEL RENEWABLE POWER MANAGEMENT: UNDERSTANDING THE CONTROL OVERHEAD ................................................................................. 40

4.1 Power Management Regions ............................................................................ 41

4.2 An Overview of iSwitch Architecture ................................................................. 44

4.3 Optimizing Load Tuning Activities ..................................................................... 47

4.3.1 Lazy Power Supply Tracking ................................................................... 48

4.3.2 Dynamic Power Demand Smoothing ....................................................... 49



4.6 Summary of iSwitch Design .............................................................................. 57

6

5 DATA CENTER LEVEL GREEN POWER MANAGEMENT: TOWARDS LOAD-FOLLOWING BASED DESIGN .............................................................................. 58

5.1 The Load Following Challenge ......................................................................... 60

5.2 Distributed Generation Powered Data Centers ................................................. 61

5.2.1 Power Demand Shaping Mechanism ...................................................... 62

5.2.1 Adaptive Load Performance Scaling ....................................................... 65



5.5 Summary of PDS Control .................................................................................. 72

6 ENABLING DATA CENTER SERVERS TO SCALE OUT SUSTAINABLY ............. 73

6.1 Oasis Scale-Out Model ..................................................................................... 76

6.1.1 Modular Power Sources .......................................................................... 77

6.1.2 Distributed Integration ............................................................................. 79

6.2 Implementation of Oasis ................................................................................... 80

6.2.1 Power Control Hub .................................................................................. 80

6.2.2 Bridging Server and Power Supply .......................................................... 82

6.2.3 Dynamic Energy Source Switching.......................................................... 83

6.3 Optimized Oasis Operation (O3) ....................................................................... 85

6.3.1 Managing Battery Lifetime ....................................................................... 86

6.3.2 Managing Backup Capacity ..................................................................... 87

6.3.3 Managing Server Performance ................................................................ 87



6.5.1 Load Performance and Energy Efficiency ............................................... 91

6.5.2 Battery Service Life and Backup Capacity ............................................... 92

6.6 Cost Analysis of Oasis ...................................................................................... 95

7 SUSTAINABLE COMPUTING IN THE SMART GRID AND BIG DATA ERA .......... 98

7.1 Green Data Centers Powered by Hybrid Renewable Energy Systems ............. 99

7.1.1 Hierarchical Power Management Framework ........................................ 100

7.1.2 Multi-Source Driven Power Management .............................................. 101

7.1.3 Verification Platform and Results .......................................................... 103

7.2 Towards Sustainable Power Provisioning for In-situ Server Systems ............. 105

7.2.1 Standalone Green Server Clusters ........................................................ 107

7.2.2 System Implementation and Verification ............................................... 109

7.2.3 Analyzing In-Situ Green Computing ...................................................... 110

8 CONCLUSIONS ................................................................................................... 114

LIST OF REFERENCES ............................................................................................. 115

BIOGRAPHICAL SKETCH .......................................................................................... 123

7

LIST OF TABLES

Table page 3-1 The impact of load tuning on output power behaviors ........................................ 25

3-2 The evaluated different solar power traces ......................................................... 30

3-3 Performance levels of battery-equipped PV systems ......................................... 30

3-4 Simulated SolarCore machine configuration ...................................................... 31

3-5 The simulated multi-programmed workloads ...................................................... 32

3-6 The evaluated solar power management schemes ............................................ 32

4-1 Configurations of heterogeneous clusters .......................................................... 53

4-2 The evaluated data center workload sets ........................................................... 53

4-3 The evaluated wind power supply traces ............................................................ 54

4-4 The evaluated server cluster power management schemes .............................. 54

5-1 The evaluated HPC data center workload traces ............................................... 68

5-2 The evaluated data center power management schemes .................................. 68

5-3 Mean job turnaround time of tracking based design (ST) ................................... 69

6-1 Computing platform setup for Oasis ................................................................... 89

6-2 Evaluated cloud workloads for Oasis .................................................................. 90

7-1 Key variables collected from the inSURE system log ....................................... 112

8

LIST OF FIGURES

Figure page 3-1 Three typical PV systems ................................................................................... 20

3-2 The current-voltage and power-voltage characteristics of PV cells .................... 21

3-3 The characteristics of load matching .................................................................. 22

3-4 An overview of SolarCore power management architecture ............................... 23

3-5 The flowchart of SolarCore maximum power point tracking ............................... 25

3-6 The per-core load adaptation policies of SolarCore ............................................ 29

3-7 The per-core load adaptation demonstration of SolarCore ................................. 29

3-8 Pseudo code for SolarCore per-core load adaptation ......................................... 29

3-9 MPP tracking results for different workloads in a sunny day .............................. 33

3-10 MPP tracking results for different workloads in a cloudy day .............................. 33

3-11 The histogram of the power tracking error .......................................................... 33

3-12 Operation durations on different power budget thresholds ................................. 34

3-13 Normalized solar energy usage under different fixed power budget ................... 35

3-14 Normalized performance time product (PTP) ..................................................... 35

3-15 Energy utilization across different locations ........................................................ 37

3-16 Normalized PTP with different load scheduling methods .................................... 38

4-1 Power management overhead vs. renewable energy utilization ......................... 41

4-2 Wind power output characteristics ...................................................................... 42

4-3 The probability distribution of wind speed ........................................................... 43

4-4 Power variation in wind powered server clusters (x-axis: minutes) ..................... 43

4-5 The iSwitch load power balancing scenario ........................................................ 46

4-6 The iSwitch global (facility-level) power management ........................................ 47

4-7 The iSwitch local (rack-level) power management ............................................. 47

9

4-8 Structure of the iSwitch scheduler ...................................................................... 48

4-9 The histogram of switching frequency using round-robin virtual machine selection during load migration ........................................................................... 50

4-10 Optimization timeline of iSwitch .......................................................................... 50

4-11 The average network traffic ................................................................................ 55

4-12 The peak network traffic ..................................................................................... 55

4-13 The average execution time increase (ETI) of job requests ............................... 57

4-14 The normalized overall wind energy utilization ................................................... 57

5-1 Distributed generation (DG) powered data centers ............................................ 59

5-2 Load following scenario ...................................................................................... 59

5-3 Distributed generation powered data centers ..................................................... 61

5-4 Power demand shaping demonstration .............................................................. 63

5-5 Power demand shaping algorithm ...................................................................... 65

5-6 Average job turnaround time under varying load following interval ..................... 70

5-7 Execution time of the worst 5% jobs ................................................................... 70

5-8 Dependence on utility power .............................................................................. 71

5-9 Battery lifetime degradation comparison (with distributed UPS) ......................... 72

5-10 Battery life degradation comparison (with centralized UPS) ............................... 72

6-1 Conventional data center power architecture is not ‘scale-out friendly’ .............. 74

6-2 Oasis power provisioning architecture ................................................................ 77

6-3 Distributed energy storage system (battery) ....................................................... 78

6-4 Prototype of Oasis node ..................................................................................... 81

6-5 Power management agent ................................................................................. 83

6-6 The control flow of power supply switching ........................................................ 85

6-7 Battery charging/discharging trace ..................................................................... 85

6-8 The power control decision tree of Ozone .......................................................... 88

10

6-9 The execution latency due to server performance scaling .................................. 92

6-10 The ratio of green energy usage to overall power consumption ......................... 92

6-11 The estimated battery lifetime calculated from battery usage profile .................. 93

6-12 Battery discharging profile and power demand traces ........................................ 94

6-13 The observed average battery backup capacity in our UPS systems ................. 95

6-14 A cost breakdown of Oasis ................................................................................. 95

6-15 Cost trends of deploying Oasis for scaling out ................................................... 97

7-1 High-level system architecture for GreenWorks ............................................... 100

7-2 The GreenWorks power management hierarchy .............................................. 101

7-3 Details of our three-layer simulation platform ................................................... 103

7-4 The impact of GreenWorks on job execution time ............................................ 104

7-5 Battery lifetime estimation based on its usage log ............................................ 105

7-6 Average UPS backup time (normalized to the rated value) .............................. 105

7-7 Cumulative distribution function (CDF) for UPS autonomy time ....................... 105

7-8 In-situ server system as an ancillary to future cloud ......................................... 107

7-9 In-situ server cluster with reconfigurable battery array ..................................... 108

7-10 The energy flow scenarios for InSURE ............................................................. 109

7-11 Operating modes transition of InSURE energy buffer ....................................... 109

7-12 The structure of our verification platform .......................................................... 110

7-13 Solar power budget trace and battery voltage trace ......................................... 111

7-14 Expected optimization results from our spatio-temporal management ............. 113

11

LIST OF ABBREVIATIONS

ATS Autonomic Transfer Switch

CSC Central Switch Controller

DG/DGR Distributed Generation/Distributed Generation Resource

DVFS Dynamic Voltage and Frequency Scaling

EMM Energy Management Module

ETI Execution Time Increase

GHG Greenhouse Gas

HMI Human Machine Interface

IT Information Technology

MPPT Maximum Power Point Tracking

PCH Power Control Hub

PCPG Per-Core Power Gating

PDU Power Distribution Unit

PDC Power Demand Controller

PDS Power Demand Shaping

PLC Programmable Logic Controller

PV Photovoltaic Module

REU Renewable Energy Utilization

SAB Switch Allocation Buffer

TCO Total Cost of Ownership

UPS Un-interruptible Power Supply

VM Virtual Machine

12

Abstract of Dissertation Presented to the Graduate School of the University of Florida in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy

ENABLING RENEWABLE ENERGY POWERED SUSTAINABLE HIGH-

PERFORMANCE COMPUTING

By

Chao Li

August 2014

Chair: Tao Li Cochair: José A.B. Fortes Major: Electrical and Computer Engineering

Computer system inevitably enters the landscape of design for sustainability. As

global server power demand hits a record level of thirty gigawatts, modern data centers

must develop efficient power provisioning and management schemes to alleviate their

escalating energy needs, reduce the skyrocketing power expenditures, and resolve the

looming environmental problems. To maintain the performance scaling trend of

emerging workload applications, and to eventually realize carbon-free computing, this

dissertation extends the benefits of renewable power supply to the computing system

design area. The main obstacle of coordinating renewable power supply and server

power demand is the lack of cross-layer power management solutions that span

software, hardware, and energy sources. This observation is the driving insight that

defines the scope of this dissertation. By synergistically integrating renewable energy

characteristics with emerging hardware/software supports at different layers of the

hierarchical structure of computing infrastructures, one can greatly improve the

demand-supply coordination effectiveness, and ultimately open the door for a new class

of sustainable high-performance computing platform.

13

CHAPTER 1 INTRODUCTION AND BACKGROUND

The rapid adoption of cloud computing and data analysis is deemed a powerful

engine for the growth of installed server capacity. Data centers housing thousands of

servers have become essential to the operation of businesses, academic, and

governmental institutions. To support emerging data-analytic workloads that tend to

scale well with large number of compute nodes, modern datacenters are continually

adding computing resources (i.e., scaling out) to their existing sites. In turn, the global

server market is projected to triple in 2020, accounting for over 1000 terawatt hour (109

kWh) annual energy consumption [1] [2].

Consequently, the environmental impact of information technology (IT) has

become a growing concern worldwide. In the past ten years, Google server’s electricity

demand has increased almost 20-fold [2]. The worldwide data centers also run the risk

of doubling their energy consumption every 5 years [2]. These large-scale computing

infrastructures, if operated entirely using power generated from conventional fossil fuel,

will unavoidably increase the total cost of ownership (TCO) and irreparably damage the

environment. According to a McKinsey Quarterly report [3], the annual CO2 emissions

of computing systems will research 1.54×109 metric tons within eight years, which could

make IT company among the biggest greenhouse gas emitters by 2020. Consequently,

many environmental activist groups such as the Greenpeace [4] have called on data

center operators to make renewable energy a priority.

Faced with a growing concern about the projected rise in both server energy

expenditures and greenhouse gas emissions, academia and industry today are focusing

more attention than ever on the sustainability of data centers. Nevertheless, simply

14

capping peak power demand and total energy consumption in data centers can hardly

improve overall sustainability. On the contrary, it can significantly limit the workload

performance that we will be able to achieve on current and future cloud computing

platforms. This is because the major source of unsustainability – a heavy dependence

on conventional fossil fuel based electricity – still exists in current designs. As a result,

renewable energy (e.g., solar and wind energy) has attracted considerable attentions

from different sectors in the IT industry. Several companies, including Microsoft, IBM,

Google, Facebook, Apple, HP, and eBay, all start to explore renewable energy

integration in data centers as part of their long-term energy strategies and corporate

environmental goals. According to their plans, these enterprise data centers will draw

power from onsite renewable energy generators, nearby co-located clean energy plants,

or a combination of both.

While it is fairly easy to incorporate green energy sources into a data center’s

energy portfolio, managing renewable energy powered computing system can be a

great undertaking. The variability of renewable power generation, together with the

stochastic workload fluctuation, can cause varying degrees of power mismatch between

the supply and demand. Due to their sensitivity to power disturbances, computing

systems must maintain a continuous balance between the fluctuating renewable power

budget and the variable IT power demand. The power mismatch can have a significant

adverse impact on energy efficiency and productivity. Conventionally, one can smooth

out the temporal variation in renewable power with substantial support from the utility

grid or uninterruptable power supplies (UPS). However, increasing the standby capacity

of utility grid and over-provisioning the UPS system often increase the cost overhead

15

significantly. In addition, the power mismatch problem can cause frequent and

excessive battery discharging events, which not only increase UPS failure rate but can

also quickly deplete the stored energy that is crucial for handling emergencies. Further,

both grid-inverter and UPS battery incur significant efficiency problems. For example,

the round-trip energy loss of battery is typically around 25% [5]. Therefore, appropriate

power management and workload scheduling is the crux of eliminating power

disruptions and ensuring high quality of services.

This dissertation focuses on the question of how to leverage clean energy

technologies and emerging hardware and software techniques to tackle the grand

energy and environmental challenges faced by the power-hungry computing systems

today and tomorrow. The driving insight of this work is the observation that the main

obstacle of green energy integration in computing systems is the lack of cross-

layer power management solutions that spans workload application, computer

hardware, and energy sources.

In the past, energy-efficient computing system designs only emphasize reducing

load power consumption and maximizing resource utilization. They are not aware of the

intrinsic behaviors of renewable power supply, and consequently overlook the interplay

between power systems and computing systems. By synergistically integrating

renewable energy characteristics with novel hardware/software supports, one can

greatly improve the demand-supply coordination effectiveness, and ultimately open the

door for a new class of sustainable high-performance computing platforms.

16

CHAPTER 2 RELATED WORK

While energy-aware computer system has become a research focus since long

time ago, designing green energy (e.g., solar energy and wind energy) powered

computing systems gained its popularity only recently.

Conventional power management techniques for processors and server systems

all assume constant power supply [6]. The goal of these techniques is either reducing

load power consumption or maximizing the performance under fixed power budget. For

example, Teodorescu et al. [7] propose a power management scheme that uses linear

programming to optimize the average throughput with given power budget. Isci et al. [8]

evaluates different power management policies for a given power budget. Lee et al. [9],

analyzes the throughput impact of applying dynamic voltage and frequency scaling

(DVFS) and per-core power gating to power-/thermal- constrained multi-core processors.

Different from conventional energy source, renewable power generators are not a

constant power source. Existing proposals that aim at improving energy efficiency under

fixed power budget cannot guarantee the same energy utilization when the power

budget is variable and unpredictable.

At the data center level, three different renewable energy management

approaches have emerged in recent years. The first design approach focuses on

hardware control techniques. For example, Blink [10] dynamically adjusts the on/off

power cycles of server motherboard to match server power demand to wind power

budget. Deng et al. [11] leverages distributed grid-tie inverters for managing renewable

power integration in data centers. In contrast, the second approach leverages workload

adaptation schemes [12] [13]. The main idea is to shift deferrable workloads to a time

17

window in which renewable generation is sufficient (temporal adaptation), or to relocate

workloads to a different system where power budget is abundant (spatial adaptation).

For the third approach, the gap between power supply management and workload

management starts to diminish. By jointly tune both energy sources and workloads, one

can achieve a better overall performance and efficiency [14].

Many recent papers have discussed the role of battery in server clusters and data

centers. For example, Govindan et al. [15] investigates the use of energy storage

(particularly the UPS batteries) to manage data center peak power. In [16], the authors

also use UPS batteries as the major tuning knob for minimizing datacenter power cost

in aggressively under-provisioned datacenter. Kontorinis et al. [17], has explored the

benefits of deploying distributed UPS system architecture in data centers.

Several recent proposals focus on optimizing cost and energy utilization in green

data centers. For example, Liu et.al [18] models and evaluates dynamic load scheduling

scheme in geographically distributed systems for reducing data center electricity prices.

Deng et.al [19] explores algorisms for optimizing clean energy cost for green Web

hosts. Ren et.al [20] demonstrates that intelligently leveraging renewable energy can

lower data center costs, not just cap carbon footprints.

In addition, several studies have demonstrated the feasibility of renewable energy

powered data centers. These designs typically employ energy storage devices, grid-tie

power controller, or a combination of both to manage renewable power. For example,

HP Labs [21] tests a renewable energy powered micro-cluster called Net-Zero for

minimizing the dependence on traditional utility grid. Their scheduling considers

shedding non-critical workload to match the time-varying solar energy output. In [14],

18

the authors have developed a workload and energy source co-management scheme on

a prototype green data center called Parasol. This work highlights data center free

cooling, low-power server nodes, and net-metering mechanism.

This dissertation explores efficient renewable energy integration at different layers

of the hierarchical structure of modern data centers. In contrast to prior work, it looks at

four novel system architecture with fundamentally different design perspectives: 1) a

solar energy powered multi-core processor that aims at jointly improving renewable

energy utilization and workload throughput [22]; 2) a wind energy powered server

cluster that aims at harvesting wind energy utilization while minimizing power

management overhead [23]; 3) a distributed generator powered data center that targets

near-optimal workload performance without sacrificing system reliability [24]; 4) a

modular green data center design that enables modern power-/carbon- constrained

data centers to scale out economically and sustainably [25].

19

CHAPTER 3 SERVER-LEVEL RENEWABLE POWER MANAGEMENT:

RETHINKING LOAD ADAPTATION SCHEMES

At the server system level, a cross-layer power management scheme that can

jointly manages power supply and server load is of paramount importance.

Although solar energy is an abundant energy source, its efficient utilization

presents a new challenge. A photovoltaic (PV) cell (a.k.a. solar cell is the basic building

block of PV systems, where the electromagnetic radiation of sun can be directly

converted to electricity through the photovoltaic effect [26]. PV cells have a non-linear

relationship between their output photocurrent and terminal voltage. Under uniform

irradiance (solar energy per unit area of the solar panel’s surface) and temperature,

photovoltaic arrays exhibit a current-voltage characteristic with a unique point, called the

maximum power point (MPP) [26], where the module produces maximum output power.

Due to the high fabrication cost and low conversion efficiency (typically 13~19%) of

present solar cells, it is crucial to operate the solar panel at the MPP since doing so can

help PV system users recover their investment more quickly.

Nevertheless, the power generated by PV systems changes over time due to the

unpredictable nature of weather pattern. The unstable working environment along with

the existence of optimum operating points necessitate the rethinking of server power

management policy to achieve higher efficiency in utilizing solar energy. Specifically,

this work focuses on multi-core servers which are the mainstream hardware design

choice for today’s IT industries and demand increasing amount of green power to

unleash their full computation potential. The multi-core processor load should be

regulated and tuned in such a way that the overall workload performance is also

optimized during the solar energy harvesting process.

20

Figure 3-1. Three typical PV systems. A) Grid-connected systems. B) Direct-coupled systems. C) Battery-equipped systems.

There are various ways to connect solar panels to a computer load, as shown in

Figure 3-1. For instance, solar-powered systems can employ energy storage elements

(e.g., batteries or super-capacitors) [27] to reduce voltage fluctuations and perform

maximum power tracking. However, energy storage elements introduce many

limitations in PV system. For example, the maximal energy that can be delivered is

limited by the battery capacity and this unavoidably affects the performance of multi-

core systems. The large current drawn by multi-core chips requires battery with large

capacity which is both bulky and expensive. In addition, the turn-around efficiency of

battery is about 75% due to internal resistance and self-discharge [5]. It causes

significant loss in overall solar energy utilization. Last but not the least, existing

rechargeable batteries all have limited lifetime. Frequent charge/discharge and self-

discharge further aggregate the aging effect. Without proper maintenance (which

requires human intervention and causes loss of availability), the aging problem will

directly lead to the capacity reduction and output voltage change. As a result, over the

lifetime of the solar-powered systems, batteries cost (e.g. initial and maintenance cost)

can be the most expensive component of a renewable energy system [28].

In this study we consider cost-effective direct-coupled PV systems. Different from

the conventional design, we propose techniques that allow multi-core processors to

autonomously harvest renewable energy to the maximal degree.

21

A B

Figure 3-2. The current-voltage (I-V) and power-voltage (P-V) characteristics of PV cells. A) I-V curve and MPP. B) P-V curve and MPP.

3.1 PV Systems and Load Matching

For a given current-voltage curve (i.e., I-V curve), there is a single operating point

where the values of the voltage and current of the cell result in the maximum power

output, which is the optimal operation point (maximum power point, MPP) for the

efficient use of the solar energy. As shown in Figure 3-2, a load connected PV cell may

operate (as determined by the intersection of the I-V curve and the load line) far from its

optimal condition, resulting in inefficient solar power utilization.

When connected to electrical loads, solar panels generally employ power

converters to achieve appropriate output voltage. Assuming 𝑃𝑖𝑛 = 𝑃𝑜𝑢𝑡, the converters

can be described as 𝑉𝑜𝑢𝑡 = 𝑉𝑖𝑛/𝑘 and 𝐼𝑜𝑢𝑡 = 𝑘 ∙ 𝐼𝑖𝑛, where k is the transfer ratio. The

actual operating point of the PV system occurs at the intersection of the electrical

characteristics of the solar panel and the load. By load matching, a maximum power

point tracker (MPPT) is able to extract the optimal power from the panel under varying

atmospheric conditions [26]. Such load matching can be achieved by either adjusting

the DC-DC converter transfer ratio, k, or tuning multi-core load, w. For a multi-core

processor, its electrical characteristics are largely dependent on parameters such as

clock frequency and supply voltage. For instance, as clock frequency increases, the

processor will exhibit lower impedance and draw more current and power.

22

A B

Figure 3-3. The characteristics of load matching. A) I-V curve with operating point on the right. B) I-V curve with operating point on the left.

Figure 3-3 shows how the operating point changes if we tune the multi-core load

w and transfer ratio k. The PV output voltage decreases when the load line moves

counterclockwise and increases when the load line moves clockwise. However, the

actual output power may either increase or decrease as the load line moves, depending

on whether the operating point is approaching the MPP or not.

Note that simply increasing the multi-core processor utilization rate and voltage

level will not automatically draw maximal power because the required load adaptation

scheme (i.e. increasing or decreasing the load) varies at different operating points. On

the other hand, tuning the transfer ratio alone cannot increase the multi-core

performance either due to the lack of effective multi-core load adaptation schemes.

Moreover, without appropriate coordination between the power converter and load

adaptation, the system will suffer from severe voltage fluctuations.

3.2 Managing Solar Energy Powered Multi-core System

SolarCore is a multi-core architecture power management scheme that can

dynamically track the maximal solar energy generation while improving workload

performance. One of the key benefits of using SolarCore is that it reduces the power

dependence on conventional fossil-fuel-based power supply.

23

Figure 3-4. An overview of SolarCore power management architecture.

Figure 3-4 provides an overview of the SolarCore architecture. The system is

powered by solar energy with utility grid as backup. An automatic transfer switch (ATS)

is employed to seamlessly select between the primary (i.e. the solar panel) and backup

power sources and an uninterruptible power supply (UPS) ensures continuous power

delivery to the load. An AC/DC converter is used only if the ATS switches to the utility. A

tunable power-conservative matching network is used to convert the PV output voltage

to the level that is compatible to existing multi-core processors and systems.

SolarCore assumes that the processor, which contributes to the significant power

dissipation in typical multi-core systems, is the only component powered by the

renewable energy while the rest of the system is power by utility grid. At the front end,

both load current and voltage are measured via I/V sensors and the results are fed-back

to a SolarCore controller, which is responsible for identifying and tracking the maximal

power point. The controller adjusts the DC/DC converter through MPPT control signal

and communicates with the processor through an adapter, which is responsible for per-

core load tuning and workload optimization. The harvested solar energy is allocated

across all running cores with the objective of maximizing the overall performance.

24

SolarCore decomposes solar energy management into 1) multi-core aware MPP

tracking and 2) dynamic server load adaptation. It uses per-core dynamic voltage and

frequency scaling (DVFS) [29] and per-core power gating (PCPG) techniques for load

adaptation. The purpose of this load adaptation is to move the operating point of solar

array closer to the MPP under changing atmospheric conditions. To achieve per-core

DVFS, SolarCore leverages on-chip voltage-regulator modules (VRM) [30] to provide

the supply voltage for each core. The supply voltage information is communicated

among the SolarCore controller, the VRMs, and cores via a number of bits called the

Voltage Identification Digital (VID), which has been implemented in commercial

microprocessors. For example, the Intel Xeon processor employs a 6-bit VID to specify

the input voltage from 0.84V DC to 1.6V DC with 32 different voltage steps.

3.2.1 Multi-core Aware MPP Tracking

The multi-core aware MPP tracking of SolarCore relies on successive tuning of

both the DC/DC converter transfer ratio k and the multi-core load w. The SolarCore

controller aims at coordinating the power supply converter and the multi-core load

adaptation so as to achieve maximal power drawn. In addition, this technique ensures

1) a correct tracking direction and 2) a stable load operating voltage 𝑉𝑙.

Table 3-1 summarizes the electrical behavior of load tuning. Tuning k or w can

increase output power 𝑃𝑜𝑢𝑡, load voltage 𝑉𝑙 and load current 𝐼𝑙, depending on the

location of the operating point. By simultaneously adjusting k and w, one can increase

the output current while maintaining constant output voltage. As a result, the operating

point moves closer to MPP and the processor can utilize more power. SolarCore uses a

three-step control to perform MPP tracking, as shown in Figure 3-5.

25

Table 3-1. The impact of load tuning on output power behaviors. The position of current operating point affects the way we perform MPP tracking.

Position MPPT Operation Power Load voltage Load current

Left of MPP 𝑘 ↑ (𝑐𝑜𝑛𝑠𝑡. 𝑙𝑜𝑎𝑑) 𝑃𝑜𝑢𝑡 ↑ 𝑉𝑙 ↑ 𝐼𝑙 ↑

Left of MPP 𝑤 ↑ (𝑐𝑜𝑛𝑠𝑡. 𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟 𝑟𝑎𝑡𝑖𝑜) 𝑃𝑜𝑢𝑡 ↓ 𝑉𝑙 ↓ 𝐼𝑙 ↑

Left of MPP 𝑘 ↑ + 𝑤 ↑ 𝑃𝑜𝑢𝑡 ↑ ∆𝑉𝑙 ≈ 0 𝐼𝑙 ↑

Right of MPP 𝑘 ↓ (𝑐𝑜𝑛𝑠𝑡. 𝑙𝑜𝑎𝑑) 𝑃𝑜𝑢𝑡 ↑ 𝑉𝑙 ↑ 𝐼𝑙 ↑

Right of MPP 𝑤 ↑ (𝑐𝑜𝑛𝑠𝑡. 𝑡𝑟𝑎𝑛𝑠𝑓𝑒𝑟 𝑟𝑎𝑡𝑖𝑜) 𝑃𝑜𝑢𝑡 ↑ 𝑉𝑙 ↓ 𝐼𝑙 ↑

Right of MPP 𝑘 ↓ + 𝑤 ↑ 𝑃𝑜𝑢𝑡 ↑ ∆𝑉𝑙 ≈ 0 𝐼𝑙 ↑

Figure 3-5. The flowchart of SolarCore maximum power point tracking.

Step 1. The algorithm starts with a normal operating voltage (i.e. 𝑉𝑙 = 𝑉𝑑𝑑). Due

to supply variation, 𝑉𝑙 may not equal to 𝑉𝑑𝑑 at each beginning phase of periodically

triggered MPP tracking. In this step, the SolarCore controller will restore the output

voltage to 𝑉𝑑𝑑 by decreasing or increasing the load to an appropriate level. This step

avoids system over-loading and serves as preparation for other steps.

Step 2. As Table 3-1 shows, the transfer ratio tuning direction that approaches

MPP depends on the system’s current operating position. To determine the tuning

direction, our techniques set the transfer ratio of the DC/DC converter from 𝑘 + ∆𝑘 to

and observe the current. An increase in output current suggests that the panel

generates more power and the system is approaching to the MPP. In this case, the

actual operating point is on the left side of the MPP and our algorithm proceeds to step

(3) for load matching. On the other hand, a decrease in output current suggests a wrong

26

tuning direction. In this case, the system further decreases the transfer ratio by 2∆𝑘

which results in a net change of −∆𝑘 in transfer ratio. Consequently, our technique

resumes the correct direction and proceed to step (3) to perform load matching.

Step 3. In this step, we tune the load until 𝑉𝑙 equals 𝑉𝑑𝑑. Due to the adjustment of

the transfer ratio in step (2), the output voltage changes as well. By increasing the load,

we can change the PV operating point and decrease 𝑉𝑙 until it reaches 𝑉𝑑𝑑. During each

control period, SolarCore increases the load successively with the aid of tuning transfer

ratio k as discussed in steps (2) and (3). Through the stepwise tuning of transfer ratio

and load matching, this method progressively adapts multi-core power consumption and

eventually reaches to the new MPP. The goal of SolarCore power management is

appropriate coordination of the variable power supply and demand.

3.2.2 Dynamic Per-Core Load Tuning

SolarCore uses DVFS to perform successive load tuning. To be more specific, it

applies per-core DVFS to manage multi-core power at a fine granularity and achieve

more effective maximum power tracking. There has been prior work [7] that formulates

DVFS control as linear programming problem and using online linear programming to

optimize power and performance trade-offs has been proven to be efficient.

The processor dynamic power model used in SolarCore is 𝑃 = 𝛼𝐶𝑉2𝑓 [6]. Our

system further assumes that: (1) the voltage scales approximately linearly with

frequency within typical operating ranges, i.e., 𝑓𝑖 = 𝜇𝑉𝑖 + 𝜆, 𝑉𝑖 ∈ {𝑉1, 𝑉2, … , 𝑉𝑛}, where n is

the number of voltage levels; (2) the total power drawn by a core is approximately a

cubic function of its voltage level, i.e. 𝑃𝑖 = 𝛼𝐶𝑉𝑖2(𝜇𝑉𝑖 + 𝜆) ≈ 𝑎𝑖𝑉𝑖

3+ ci; (3) the voltage

scaling has little impact on IPC and the throughput 𝑇𝑖 = 𝐼𝑃𝐶𝑖 × 𝑓𝑖 of a core is

27

proportional to its frequency; and (4) the available power for an individual core does not

exceed its maximum power and all cores are homogeneous. Given the above

assumptions, the per-core throughput can be expressed as 𝑇𝑖 = 𝐼𝑃𝐶𝑖 × (𝜇𝑉𝑖 + 𝜆) =

𝑏𝑖𝑉𝑖 + 𝑑i, and the average throughput across all cores is 𝑇𝑖̅̅ ̅ =

1

𝑁∑(𝑏𝑖𝑉𝑖 + 𝑑i), where N is

total number of cores. Since the system working duration, which is subject to the solar

power generation level, plays a key role in determining the overall performance, our

goal is to maximize 𝑇𝑖̅̅ ̅ × 𝑅𝑢𝑛𝑡𝑖𝑚𝑒/𝑑𝑎𝑦. This performance-time product (PTP) can be

measured as the total instructions committed per day. Intuitively, to maximize the PTP,

one needs to increase both the throughput and the system operation duration.

In contrast to conventional power management problem, solar power is not fixed,

i.e., 𝑃𝑡𝑜𝑡𝑎𝑙 = ∑ 𝑃𝑖 = ∑(𝑎𝑖

𝑉𝑖3 + 𝑐𝑖) ≤ 𝑃𝑀𝑃𝑃 and 𝑃𝑀𝑃𝑃 = 𝑓(𝐼𝑟𝑟𝑎𝑑𝑖𝑎𝑡𝑖𝑜𝑛, 𝑇𝑒𝑚𝑝𝑒𝑟𝑎𝑡𝑢𝑟𝑒). The

unpredictable nature of PV output makes performance optimization much more

challenging. In this work, SolarCore uses a throughput-power ratio (TPR) optimization –

a PTP optimization method that can be performed along with load adaptation. The

throughput-power ratio specifies the throughput speedup of a processor when additional

power is available. When adjusting the core voltage by ∆𝑉, a power consumption

change of ∆𝑃 = 3𝑎𝑖𝑉𝑖2∆V will occur. The relationship between throughput change and

voltage scaling is given by ∆𝑇 = 𝑏𝑖∆𝑉. Hence the throughput power ratio can be defined

as ∆𝑇

∆𝑃=

𝑏𝑖

3𝑎𝑖𝑉𝑖2 . Both 𝑏𝑖 and 𝑎𝑖 can be derived using profiling information obtained from

the performance counters and the current/voltage sensors at each beginning of MPP

tracking. Given the throughput power ratio, our scheme identifies the appropriate core

voltage setting through a heuristic process. The cores that exhibit large throughput-

power ratio will have higher priority to receive the available power.

28

SolarCore stores the core ID and per-core voltage level in a table sorted by the

throughput-power ratio, as shown in Figure 3-6. As described previously, load matching

is achieved by increasing or decreasing the voltage level and frequency of the selected

core. When we increase the load, we choose the core with higher throughput-power

ratio, which will maximize the performance by using additional power. When the load

needs to be decreased, our scheme chooses the core with lower throughput-power

ratio. Doing so allows us to meet the power budget while minimizing the impact on the

overall system performance. The above step is performed iteratively until the

aggregated multi-core power approximates the new power budget.

Figure 3-7 shows the per-core load adaptation scenario. The optimum computing

load is achieved when the inflection point (the point where the slop of power trace

changes sign) is reached. Since the inflection point can be stable operation states or

transition states (i.e. depending on whether the inflection point is under the power

budgets), the MPPT controller should not stop tuning immediately once an inflection

point is met. For example, if the inflection point is above the power budget, the MPPT

controller needs to decrease the load further. As a result, the system returns back to the

stable load level and the optimum power consumption is met.

Note that there is still power headroom even if we perform maximum power

budget racking. However, the existence of such a power margin is actually beneficial

since it improves the robustness of the system. Figure 3-8 shows the pseudo code of

our per-core load-tuning algorithm. The processor starts tuning its load when the

controller detects a change in PV power supply. Successive server load adaptation is

performed at the beginning of each triggered tracking period.

29

Figure 3-6. The per-core load adaptation policies of SolarCore.

Figure 3-7. The per-core load adaptation demonstration of SolarCore.

Figure 3-8. Pseudo code for SolarCore per-core load adaptation.

3.3 Experimental Methodology

SolarCore is evaluated through heavily extended simulators and real-world solar

traces. Our PV power model takes irradiance and temperature profiles as input and

generates the I-V and power profiles. Table 3-2 shows the meteorological data from the

Measurement and Instrumentation Data Center (MIDC) [31] of the National Renewable

Energy Laboratory. The MIDC provides real-time records from measurement stations

located throughout the US. Table 3-2 lists different locations that have different solar

energy resource potentials. For each location, we evaluated our design with daytime

solar irradiance data (i.e. 7:30AM ~ 5:30 PM) across different seasons (e.g. the middle

of Jan., Apr., Jul. and Oct. in the year of 2009). We trigger the maximum power point

tracking at each of 10-minute interval. We observed that the tracking duration within

each interval is less than 5ms. We applied sampling techniques to capture the

irradiance changes at a large time scale.

30

Table 3-2. The evaluated different solar power traces. Station

Location

KWh/m2/day on Average Solar Energy Potential

PFCI Phoenix, AZ > 6.0 Excellent

BMS Golden, CO 5.0 ~ 6.0 Good

ECSU Elizabeth City, NC 4.0 ~ 5.0 Moderate

ORNL Oak Ridge, TN < 4.0 Low

Table 3-3. Performance levels of battery-equipped PV systems.

Level Range MPP Efficiency Battery Efficiency Overall de-rating factors

High 81~ 92% 97% 95% 92% Moderate 70~ 81% 95% 85% 81% Low < 70% 93% 75% 70%

We also compared the total solar energy drawn by SolarCore with that of battery-

equipped standalone PV system. We assume that the battery is optimally charged using

MPPT circuit and the multi-core processor runs with full speed using stable power

supply. We assume that dynamic power monitor is used to ensure that all the solar

energy stored in the battery is consumed in our simulation. We use de-rating factors to

estimate the upper bound of the utilization on conventional battery-equipped solar

power systems. Typically, battery loss ranges from 5% to 25% [5] and a modern

maximum power tracking controller has conversion efficiency between 93% ~ 97%. We

consider three levels of performance ranges, as shown in Table 3-3.

We simulated a multi-core system comprised of 8 Alpha 21264 processors where

each core has private L1 and L2 caches. We assume a frequency of 2.5GHz and a

90nm technology with a maximum supply voltage of 1.45V. Table 3-4 summarizes our

core configuration. We used a cycle-accurate multi-core simulator integrated with the

modified power models from Wattch [32] and CACTI [33]. Both dynamic and leakage

power are modeled. We assume that the dynamic voltage and frequency scaling

schemes employed in each core are similar to Intel’s SpeedStep techniques [34]: each

core supports 6 frequency and voltage operating points and the frequency can be

31

scaled from 2.5GHz down to 1GHz with a 300MHz stepping. The voltage scales

approximately linearly with frequency within typical operating ranges; six voltages

ranging from 1.45V to 0.95V with a 0.1V stepping are used. To support DVFS, the

Wattch and CACTI power models are extended to use the voltage and frequency as

input and then adjust the power estimation for a core when the operating status

changes. For the performance analysis, we used the performance-time products rather

than IPC since the frequency varies across the entire program execution.

We use workloads from the SPEC2000 benchmark suite shown in Table 3-5. To

form multi-programming workloads, we use a program’s average energy-per-instruction

(EPI) and categorize the benchmarks as being high-EPI (EPI 15nJ), moderate-EPI

(15nJ EPI 8nJ), or low-EPI (EPI 8nJ). We run each benchmark in their

representative execution intervals and the EPI is obtained by calculating the average-

energy consumed per-instruction. We generated both homogeneous and

heterogeneous workloads, as shown in Table 3-5.

Table 3-4. Simulated SolarCore machine configuration. Parameters Configurations

Frequency 2.5/2.2/1.9/1.6/1.3/1.0GHz Voltage 1.45/1.35/1.25/1.15/1.05/0.95V Width 4-wide fetch/issue/commit Issue Queue 64 ITLB 128 entries, 4-way, 200 cycle miss Branch Predictor 2K entries Gshare10-bit global history BTB 2K entries, 4-way Return Address Stack 32 entries RAS ROB Size 98 entries LSQ 48 entries Integer ALU 4 I-ALU, 2 I-MUL/DIV, 2 Load/Store FP ALU 2 FP-ALU, 2 FP-MUL/DIV/SQRT DTLB 256 entries, 4-way, 200 cycle miss L1 I-/D- Cache 64KB, 4-way, 64 B/line, 2 ports, 3 cycle access L2 Cache 2MB, 8-way, 128 B/line, 12 cycle access Memory Access 64 bit wide, 400 cycles access latency

32

Table 3-5. The simulated multi-programmed workloads. EPI Value Workload Set 1 Workload Set 2

High H1(art×8) H2 (art×2, apsi×2, bzip×2, gzip×2) Moderate M1(gcc×8 ) M2 (gcc×2, mcf×2, gap×2, vpr×2) Low L1(mesa×8) L2 (mesa×2, equake×2, lucas×2, swim×2) High-Moderate HM1(bzip×4, gcc×4) HM2 (bzip, gzip, art, apsi, gcc, mcf, gap, vpr) Moderate-Low ML1(gcc×4, mesa×4) ML2 (gcc, mcf, gap, vpr, mesa, equake, lucas, swim)

Table 3-6. The evaluated solar power management schemes. Algorithm MPPT Load Tuning Scheduling Method

Fixed-Power No DVFS Linear programming with a fixed power budget MPPT & IC Yes DVFS Concentrating solar power onto a few cores MPPT & RR Yes DVFS Round-robin scheduling MPPT& Opt Yes DVFS Optimized scheduling based on throughput-power ratio

3.4 Evaluation Results

We compare the efficiency of SolarCore with various power management

policies, as summarized in Table 3-6. The Fixed-Power is a non-tracking power

management scheme which assumes a constant power budget during the entire

workload execution. In contrast to SolarCore, MPPT&IC and MPPT&RR both apply

tracking control techniques but employ different scheduling policy to perform load

adaptation. MPPT&IC keeps tuning individual core until reaching its highest or lowest

V/F level, while MPPT&RR distributes the power budget variation evenly across all the

cores in a round-robin fashion. The MPPT&Opt (i.e. default configuration for SolarCore)

selects cores using the proposed throughput-power ratio optimization method. We also

compared the power efficiency to that of the battery-equipped PV systems.

The operation duration we refer to in this work will always be the daytime

duration. In addition, the effective operation duration means the duration that multi-core

processor successfully performs MPP tracking and workload optimization with

SolarCore power management technique. Due to the intermittency of solar energy,

effective operation duration will always be shorter than the operation duration.

33

A B C

Figure 3-9. MPP tracking results for different workloads in a sunny day. A) Tracking result for H1. B) Tracking results for HM2. C) Tracking results for L1.

A B C

Figure 3-10. MPP tracking results for different workloads in a cloudy day. A) Tracking result for H1. B) Tracking results for HM2. C) Tracking results for L1.

Figure 3-11. The histogram of the power tracking error.

3.4.1 Power Tracking Accuracy

Figures 3-9 and 3-10 provide a graphic view of the tracking results. Each figure

shows the maximum power trace and the actual power harvested with tracking. We

present results for sunny and cloudy weather patterns. For high EPI workloads (e.g.

H1), large ripples in power tracking are generated due to the high variation in load

power. Power ripples, together with the unpredictable nature of the environment, can

affect the use of renewable energy and the system reliability. Low EPI workloads and

heterogeneous workloads are observed to have small ripples. The former manifests

small power ripples since they have lower average energy per instruction, which results

in relatively small power fluctuation amplitude; the latter exhibits small ripples because a

mix of diverse workloads can smooth the power demand fluctuation.

100 200 300 400 500 6000

20

40

60

80

100

Time (min)

Pow

er

(watt)

Maximal Power Budgets

Actual Power Consumptions

100 200 300 400 500 6000

20

40

60

80

100

Time (min)

Pow

er

(watt)



100 200 300 400 500 6000

20

40

60

80

100

Time (min)

Pow

er

(watt)


Actual Power Consumption

100 200 300 400 500 6000

50

100

150

Time (min)

Pow

er

(watt)



100 200 300 400 500 6000

50

100

150

Time (min)

Pow

er

(watt)



100 200 300 400 500 6000

50

100

150

Time (ms)

Pow

er

(watt)



34

A B C

Figure 3-12. Operation durations on different power budget thresholds. A) Declines slowly. B) Declines linearly. C) Declines rapidly.

Our dynamic load adaptation scheme reduces the impact of load power ripples

by leaving a small power margin between the actual load power consumption and the

solar power budget. This power margin slightly degrades the tracking accuracy but

improves the reliability. To quantify the impact of power margin on the tracking

accuracy, we calculated the relative tracking error. In each tracking period t, the relative

tracking error is defined as �̅� = |𝑃 − 𝑃′| 𝑃′⁄ , where 𝑃 is the actual power consumption

and 𝑃′ is the power budget. Figure 3-11 shows the histogram of power tracking errors

across 12 locations (different weather patterns) and 10 workload sets. Due to the

reduced power margin, the low EPI workloads (e.g. L1) show relatively small tracking

errors [22]. Similarly, compared with H1, heterogeneous workloads (e.g. HM2) have

small tracking error as well [22]. The majority of tracking errors fall into 8% to 16%.

3.4.2 Fixed Power Budget

For the direct-coupled PV systems, the load starts to operate using solar power

when the amount of renewable energy exceeds a power-transfer threshold. We

evaluated the solar energy utilization on multi-core systems that use power-transfer

threshold as a fixed power budget. When the renewable power output falls below the

power-transfer threshold, we switch to utility grid.

35

A A

B B

C C

D D

Figure 3-13. Normalized solar energy usage under different fixed power budget. A) Arizona. B) Colorado. C) North Carolina. D) Tennessee. (x-axis: watts)

Figure 3-14. Normalized performance time product (PTP) under different fixed power budget. A) Arizona. B) Colorado. C) North Carolina. D) Tennessee. (x-axis: watts)

We simulated both homogeneous and heterogeneous workloads with high and

low EPI. For each weather pattern, we calculated the effective operation duration under

different power budget thresholds, as shown in Figure 3-12. The threshold (x-axis:

watts) affects processor performance-time product (PTP) in both throughput and

effective operation duration. A higher power-transfer threshold will make the multi-core

processor run at higher voltages and frequencies but only for short durations. A

36

conservative, low power-transfer threshold operating mode will have longer duration but

lower clock frequencies. Intuitively, the processor should run under higher (lower) power

budget to maximize its performance when the effective operation duration declines

slowly (rapidly). Extensive simulations shown in Figures 3-13 and 3-14 further proof this.

We calculated the accumulated energy drawn by the multi-core processor and

the performance in terms of PTP. All the results are normalized to those obtained on

SolarCore. As shown in Figures 3-13 and 3-14 (average results across all benchmarks),

the maximal solar energy drawn does not guarantee maximal performance (e.g. July

@CO and Apr @TN). The maximal workload performance may occur under a high

power budget (e.g. Apr @AZ), a moderate power budget (e.g. Jan @TN) or even a low

power budget (e.g. Apr @NC). Therefore, a single, optimal fixed power budget for the

multi-core system does not exist. Even under the optimal fixed power budget, the best

energy utilization and PTP that the Fixed-Power schemes can achieve is less than 70%

of that yielded on maximum power tracking. In other words, SolarCore outperforms

Fixed-Power control scheme by at least 43% (i.e. (100%-70%)/70%) in terms of both

energy utilization and workload performance.

3.4.3 Solar Energy Utilization

The energy utilization of SolarCore depends on the solar resource potential of

the geographic location. We calculated the energy utilization (i.e. actual solar energy

consumed / overall solar energy generated) with various load adaptation scenarios. As

shown in Figure 3-15, the average energy utilization drops when the renewable energy

potential is low. For locations with abundant solar resource, SolarCore draws 5% more

power compared to a typical battery equipped PV system which has an energy

utilization upper bound of 81%.

37

Figure 3-15. Energy utilization across different locations. The upper bound of energy

utilization on battery-based system is estimated using de-rating factors.

In addition to the weather patterns, the load adaptation methods and workload

characteristics affect the utilization as well. In Figure 3-15, MPPT&Opt load adaptation

method has 2.6% lower energy utilization compared with MPPT&RR. This is because

MPPT&Opt relies on successively tuning low-power, high-throughput cores to improve

performance. Compared with the high EPI workloads, low EPI workloads exhibit higher

energy utilization due to the fine-grained load tuning and the reduced power margin.

3.4.4 Performance Improvement

We compared SolarCore with the high efficiency battery-equipped systems.

Battery-L denotes the lower bound of a high-end battery-equipped system which has a

total energy conversion efficiency of 0.81. Battery-U denotes the upper bound of a high-

end battery-equipped system which has a total energy conversion efficiency of 0.92.

Figure 3-10 shows the performance of SolarCore with different load scheduling methods

across different weather patterns. All the results are normalized to Battery-L.

In Figure 3-16, MPPT&IC shows the lowest performance because it concentrates

the solar power into fewer cores. The throughput-power ratio ∆𝑇

∆𝑃=

𝑏𝑖

3𝑎𝑖𝑉𝑖2 shows that the

performance return decreases when we allocate the limited available power to those

high V/F level cores. MPPT&RR increases the performance by distributing the solar

power to each individual core in a round-robin fashion. In this case, each core runs in a

38

moderate V/F level with a relatively high performance per watt. The average

performance of MPPT&IC, MPPT&RR and MPPT&Opt is 0.82, 1.02 and 1.13,

respectively. The normalized performance of Battery-U is 1.14. As can be seen, with

runtime optimization, MPPT&Opt improves the performance by 37.8% compared with

MPPT&IC and 10.8% compared with MPPT&RR. Compared with the most efficient

battery-based systems, SolarCore yields less than 1% performance degradation.

Figure 3-16. Normalized PTP with different load scheduling methods.

39

3.5 Summary of SolarCore Design

Photovoltaic power generation are one of the most promising technologies for the

future of green computing in IT infrastructures. SolarCore represents the first step on

exploring the utilization of solar energy for powering multi-core processors, which are

the core hardware building blocks of current and future IT infrastructures and demand

increasing amounts of power to unleash their computation potential. Unlike conventional

energy sources, PV modules exhibit non-linear I-V characteristics and their output

power changes with the variation of light intensity and ambient temperature. Therefore,

the best design practice of solar energy driven high-performance computing system

requires a cross-layer power management scheme: 1) appropriate power provisioning

control to harvest the renewable energy and reduce the impact of supply variation and

2) dedicated load adaptation schemes that optimize the workload performance.

While existing techniques seek to minimize multi-core power dissipation under

performance constraints or to maximize throughput for a given power budget, SolarCore

harvests the maximum amount of solar energy to maximize the power budget for

optimal throughput without using the expensive storage elements. Furthermore,

SolarCore applies load optimization based on the workload throughput-power ratio to

improve the power application effectiveness. Because of its ability to extract additional

solar energy and its ability for load optimization, SolarCore boosts multi-core processor

performance by 43% compared with conventional fixed power budget scheme and

10.8% compared with relatively straightforward round-robin load adaptation.

40

CHAPTER 4 CLUSTER LEVEL RENEWABLE POWER MANAGEMENT:

UNDERSTANDING THE CONTROL OVERHEAD

This chapter investigates the efficiency issue of incorporating renewable energy

resources into existing utility grid powered server clusters. At the server cluster level, we

primarily focus on wind energy since it is widely used to provide affordable and

abundant green energy for large scale facilities [35].

Due to the intermittent nature of renewable energy, existing designs typically use

on-site renewable generation to compensate part of their total server power

requirement. Consequently, many server clusters can be powered by hybrid energy

sources, i.e., part of the server clusters is powered by wind energy and the remaining

part of the server clusters is powered by utility grid. When renewable energy contributes

a large portion of the load power demand (e.g., > 15%), variations in renewable power

supply can still have a significant impact on load operation. To maximize the utilization

of green energy sources, it is desirable to place as much workload as possible on wind

energy powered clusters. Therefore, dynamic load balancing between the two types of

clusters is important to maintain the desired green energy utilization. However,

aggressively matching server power demand to renewable power budget can degrade

system response time (due to load power capping or control overhead) and provides

very limited energy efficiency return.

Many research efforts are focused on reducing idle server power [36], lowering

provisioned power capacity [37], managing computing power and cooling power [38],

and optimizing power allocation [39] [40]. However, the impact of renewable power

variation behavior on server cluster operation is still an unexplored area.

41

A B

Figure 4-1. Power management overhead vs. renewable energy utilization. A) Normalized control overhead. B) Average energy utilization.

We simulate a wind energy powered server cluster which tracks the renewable

power output surge with a pre-defined coverage factor (CF). It tracks the renewable

power budget whenever renewable generation decreases to avoid brownout. When

CF=1, it means we aggressively track the wind power variability. As shown in Figure 4-

1A, compared to no-tracking (i.e., CF=0), always tracking the power variation (i.e., CF =

1) increases the load tuning overhead by 2X. Nevertheless, the improvement of energy

utilization is less than 15%, as shown in Figure 4-1B.

To improve load matching efficiency, this chapter introduces iSwitch, a light-

weight power management scheme that maintains a desirable balance between

renewable energy utilization and server cluster control overhead. Its novelty is two-fold.

First, it is a supply-aware power tracking mechanism that applies different power

management strategy under different wind variation scenarios. Second, it also features

a load-aware optimization that is able to further minimize the performance overhead.

4.1 Power Management Regions

A wind turbine generates electrical power by extracting kinetic energy from the

air flow. Figure 4-2 shows the output characteristics of a GE wind turbine, whose power

curve is divided into three regions by the designated operating wind speeds. The cut-in

speed is the minimum speed at which the blade starts to rotate. The cut-off speed is the

wind speed at which the turbine shuts down for protecting the blade assembly.

42

Figure 4-2. Wind power output characteristics.

In Figure 4-2, we refer to the three regions as intermittent power outage period

(Region-I), variable power generation period (Region-II) and stable power generation

period (Region-III), respectively. In Region-I, wind power is intermittently unavailable

because the wind speed is either too low or too high. In Region-II, the mechanical

power delivered to the turbine generator is given by 𝑝 = 0.5𝜌𝐴𝑣3𝐶 [35], where 𝜌 is the

air density, A is the swept area of the blades, v is the wind speed and C is called power

coefficient factor. In Region-III, the wind turbine operates at its rated power.

The variations in wind speed are described by the Weibull distribution [35]. In

Equation 4-1, k is the shape parameter, c is the scale parameter and v is the wind

speed. At most wind farm sites, the wind speed has the Weibull distribution with k = 2,

which is specifically known as the Rayleigh distribution [35].

𝑓(𝑣) = (𝑘

𝑐) (

𝑣

𝑐)

𝑘−1𝑒−(

𝑣

𝑐)

𝑘

, 𝑣 ∈ [0, ∞) (Equation 4-1)

As shown in Figure 4-3, the distribution function in Region-II is not monotonic. In

this region, wind speed has equally high possibilities at a wide range of values. As a

result, we are more likely to incur time-varying wind speed in Region-II. In addition, the

turbine output is a steep curve due to the cubic relation between wind power and wind

speed. A small change of the wind speed can lead to large output fluctuation.

43

Figure 4-3. The probability distribution of wind speed.

Figure 4-4. Power variation in wind powered server clusters (x-axis: minutes).

Figure 4-4 shows real traces of wind power generation and server cluster power

demand. It illustrates the aforementioned three scenarios, namely, intermittent wind

power outage period (Region-I), low renewable generation with frequent fluctuation

(Region-II), and full renewable generation with relatively stable output (Region-III).

Similar region partition method is also applicable for intermittent renewable energy

sources such as solar power and tidal power.

Region-I: tune wisely. During the low generation period, it is wise to shift server

cluster load from renewable energy supply side to utility power. Existing practices either

put servers into low power states or apply power cycling techniques [10] on the

hardware. Although these approaches show impressive power control capability, they

sacrifice the computing throughput. In addition, it typically takes a long time (about tens

of minutes as we observed) for renewable energy output to resume. As a result, putting

servers into sleep state and waiting for the renewable energy to resume may not be

wise, especially for parallel clusters that have inter-node workload dependency.

44

Region-II: track wisely. Whenever the load power fluctuates or renewable

energy generation varies, load matching is performed as a common practice to handle

the power discrepancy [10] [22]. In Region-II, the wind generation oscillates severely.

The load power tuning is largely a result of the power supply variation, as shown in

Figure 4-4. However, aggressively matching the load to the supply results in little energy

benefits but disturbs the normal server operation and degrades the performance of

parallel workload. Therefore, seeking appropriate tracking is especially important.

Region-III: schedule wisely. When the renewable energy is stable, frequent

load fluctuation contributes to a number of load tuning operations. In Figure 4-4 (Phase

III), although the data center power has a relatively small dynamic range, frequent

variation invokes a large number of back-and-forth load migration operations (e.g., via

virtual machine migration). Those tuning activities have little contribution to the overall

energy utilization but increase network traffic significantly.

A renewable energy powered data center frequently experiences the

aforementioned three power management regions throughout its lifetime. To improve

overall quality of service, a cooperative power management scheme is required to

handle the above power management regions.

4.2 An Overview of iSwitch Architecture

We propose iSwitch, a server cluster coordination and optimization scheme that

ensures high renewable energy utilization and low operational overhead. It is designed

to provide autonomic load power balancing between conventional utility grid and

renewable energy generation. As shown in Figure 4-5, the on-site renewable power

supply provides the data center with clean energy through a separate power line. We

choose not to synchronize renewable power to the grid because the fluctuating nature

45

of renewable energy often challenges the grid stability [41]. Although one can leverage

dual-corded servers to utilize two power supplies simultaneously, it is not energy-

efficient since the efficiency of power supply units decreases when the computing load

is low [42]. As an alternative approach, iSwitch explores load migration.

The basic idea behind iSwitch is switching, or the operation of performing a

switch. In this work, the switch is defined as a load migration event that leads to

redistribution of load power between two different group of server clusters (and

consequently, between different energy sources) Instead of throttling load power when

renewable energy generation decreases, iSwitch intelligently shifts the computing load

from one energy source to another to achieve best load power matching. We use virtual

machine (VM) live migration to implement iSwitch since it is the most convenient way to

perform load power shifting in a virtualized computing environment.

Note that in this work we consider server clusters that have reserved computing

capacity. In other words, in Figure 4-5, iSwitch does not require increasing the number

of servers to meet the workload surge. In addition, when power emergency happens,

we still use backup energy storage to temporarily support the load. According to Fan et

al. [43], a typical server clusters (not under-provisioned) can spend more than 80% of

the time within 80% of their peak power, and 98% of the time within 90% of their peak

power. Therefore, the chance of workload triggered emergency is small. In this study,

we assume that the number of renewable energy powered servers is less than 40% of

the overall deployed machines since a data center typically consumes about 60% of its

actual peak power [43]. In this case, even if the wind power is extremely low, the utility

grid can still take over most of the load power demand.

46

Figure 4-5. The iSwitch load power balancing scenario. The server clusters have reserved computing capacity and are powered by hybrid power supplies (i.e., wind turbine + utility grid).

To handle the time-varying, intermittent renewable power, iSwitch dynamically

allocates and de-allocates (i.e., “switches”) the renewable energy powered server load.

The supply/load variability makes the switch tuning challenging since the control should

1) globally respect the time-varying renewable budget and 2) locally avoid any power

failure induced by load fluctuation. To this end, iSwitch uses a hierarchical switching

control scheme, which can be easily incorporated into existing hierarchical power

management methods such as [44].

Facility level. Figure 4-6 shows a global view of iSwitch control mechanism.

The switching operation is controlled by a central switch controller (CSC), which

communicates with a central power controller (a typical facility-level data center power

controller), a switch scheduler and multiple cluster level switch controllers. CSC

performs switch tuning based on the discrepancy between the load power consumption

and the RES budget. Whenever needed, switching operations are scheduled by the

switch scheduler, which stores profiling information for each server load.

47

Figure 4-6. The iSwitch global (facility-level) power management.

Figure 4-7. The iSwitch local (rack-level) power management.

Cluster level. The switching allocation is assigned to computing nodes via

cluster-level switching controllers, which are counterparts to PDU-level power

controllers. It collects switching outcomes (i.e., the number of switching operations

accomplished/failed) of each computing node and feeds the information to the CSC for

switching scheduler updates.

Rack level. As shown in Figure 4-7, a rack-level switch controller executes

power supply switching and sends the execution outcomes to the CSC via a cluster-

level switch controller. It also interacts with the rack-level power controller throughout

the switching process to avoid any brownout. For example, whenever the power

consumption of a server rack reaches its local renewable power budget, the power

controller will signal the rack-level switch controller to throttle the switching activities.

4.3 Optimizing Load Tuning Activities

The switching scheduler is the key architecture for iSwitch, as shown in Figure

4-8. It monitors the power provisioning status (i.e. powered by renewable energy or

utility grid) of each server load (i.e., VMs). All the running loads within each cluster are

indexed consecutively in a switch allocation buffer (SAB). A switching history table is

used to store the switching frequency for each virtual machine. An optimizer computes

the optimal switching assignment and a tracking module initiates the switching process.

48

Figure 4-8. Structure of the iSwitch scheduler.

To make load tuning decisions, iSwitch scheduler needs profiling information

such as server utilization data from the load history table. The central power controller

invokes scheduling activities in response to variations in renewable power supply and

load power demand. Whenever necessary, the scheduler sends a sequence of load

switching commands to the central switch controller for execution.

4.3.1 Lazy Power Supply Tracking

The first idea of iSwitch is to avoid tracking the severely fluctuant renewable

power in Region-II. We call it lazy tracking because the module only harvests the

relatively stable renewable energy generation. Note that iSwitch carefully distributes the

switching activities across all the loads evenly to avoid local traffic jam.

Lazy tracking. At each fine-grained interval, when switching is triggered by the

CSC, an estimated switching assignment will be sent to the scheduler for calculating the

switch operation balance (e.g., estimated assignment minus the baseline). If the switch

balance indicates a reduced number of servers to RES connection, the scheduler

signals the CSC to schedule the estimated assignment to avoid brownout. On the other

hand, if the switching balance suggests an increased number of servers to RES

connection (e.g., due to temporally decreased load or increased supply), the scheduler

will signal CSC only if the balance is larger than a preset threshold (e.g., 10% of the

baseline). In this case, we ignore the high-frequency switching surge which brings little

benefit on renewable energy utilization but leads to excessive load migration.

49

LRU distribution. Within each cluster, iSwitch allocates switch operations with

least recently used (LRU) method, which avoids aggressively tuning a small set of

computing nodes. Note that a naive switching allocation can result in unbalanced

switching allocation. In Figure 4-9 we show the switching distribution as a result of

round-robin scheduling. The average switching frequency is 200 times per day per VM.

A small group of VMs receives up to 400 times per day. As a result, some racks may

incur more performance penalty due to high communication traffic.

To implement LRU, iSwitch uses the switch frequency record stored in the

switching history table. The operation of iSwitch scheduler relies on the load history

record of the previous control period. This record can be implemented using a round-

robin database (circular buffer) with constant storage occupation over time.

4.3.2 Dynamic Power Demand Smoothing

Optimizing the supply-side fluctuation alone cannot achieve significant overhead

mitigation. To this end, iSwitch leverages the heterogeneity of server clusters to

minimize load fluctuation-induced overhead in power management of region III.

Figure 4-10 illustrates the switch management timeline of iSwitch. The controller

re-shuffles the renewable energy powered servers at a coarse-grained time interval R

(e.g., 15 minutes as the default value in our experiments). During each re-shuffling

interval, the average load utilization is recorded in a fine-grained time interval (e.g., 1

minute) and is used to predict the load for the next period. Upon rescheduling, the

optimizer in the iSwitch scheduler updates the baseline switch operations of each server

cluster in SAB with the goal of mitigating the likelihood of severe load power fluctuation

in the next control period. Each switch tuning invoked by the CSC will be assigned

based on the updated SAB.

50

Figure 4-9. The histogram of switching frequency using round-robin virtual machine selection during load migration.

Figure 4-10. Optimization timeline of iSwitch. Lazy tracking is fine-grained

and demand smoothing is coarse-grained.

At the beginning of each control period, iSwitch recalculates the optimal switch

assignment. To simplify the problem, we assume the servers are logically divided into c

clusters and the load is balanced within each cluster (i.e., homogeneous utilization).

Let 𝑈𝑖 = [ 𝑢𝑖1 𝑢𝑖2… 𝑢1𝑐] denotes the average utilization of each cluster at

time stamp 𝑖. The utilization history record for the previous control period that consists of

𝑚 time stamps is:

𝑼 = [

𝑈1

𝑈2

⋮𝑈𝑚

] = [

𝑢11 𝑢12

𝑢21 𝑢22

… 𝑢1𝑐

𝑢11 𝑢2𝑐

⋮ ⋮𝑢𝑚1 𝑢𝑚2

⋱ ⋮𝑢11 𝑢𝑚𝑐

] (Equation 4-2)

Assuming a total number of 𝑁 virtual machines is to be connected to the

renewable power supply in the next control period. The migration decision for the next

control period is 𝑺 = [ 𝑠1 𝑠2 … 𝑠𝑘 … 𝑠𝑐], where 𝑠𝑘 is the number of VMs selected

to be tuned for cluster 𝑘. To reduce unnecessary load tuning in the future, we want the

51

aggregate power consumption of the selected VMs to have small oscillations in the next

control period. In other words, the standard deviation of the aggregate utilization should

be minimized. The aggregate utilization is given by:

[𝑎𝑖𝑗]𝑚×1

= 𝑼 × 𝑺𝑇 (Equation 4-3)

The standard deviation of the expected utilization in the next control period can

be calculated as:

𝛿 = √1

𝑚∑ (𝑎𝑖 − 𝜇)2𝑚

𝑖=1 = √1

𝑚∑ (𝑎𝑖)2𝑚

𝑖=1 − (1

𝑚∑ 𝑎𝑖

𝑚𝑖=1 )2 (Equation 4-4)

𝑎𝑖 = ∑ 𝑢𝑖𝑘𝑠𝑘𝑐𝑘=1 (Equation 4-5)

In Equation-4, 𝑎𝑖 is the aggregate utilization of renewable energy powered load

and 𝜇 is the mean utilization during the past control period 𝑅. The re-shuffling problem is

therefore formulated as:

Objective: min {1

𝑚∑ (𝑎𝑖)2𝑚

𝑖=1 − (1

𝑚∑ 𝑎𝑖

𝑚𝑖=1 )2} (Equation 4-6)

Constraints: ∑ 𝑠𝑘 ≤ 𝑁 (Equation 4-7)

We solve the above non-linear minimization problem with simulated annealing

(SA). Given the utilization history records, our SA solver is capable of finding the

desired global extreme very fast. Note that the renewable power supply fluctuation

typically occurs on a coarse-grained time interval (several minutes). As a result, the

execution time (several seconds in our experiments) of SA solver does not affect the

optimization effectiveness. At the beginning of the re-shuffling period, the switching

operations are assigned in proportion to the number of servers in the cluster. The SA

solver iteratively generates a stochastic perturbation for the switching assignment and

checks whether or not the optimum solution is reached.

52


We evaluate our design with trace-driven simulation. We have developed a

framework that simulates dynamic load tuning and hierarchical power control in

renewable energy powered data centers. For each scheduled job requests, we calculate

its contribution to the overall data center power consumption based on the number of

nodes requested and the job’s specific resource utilization statistics. Our framework

takes realistic wind energy generation traces as supply-side input.

We assume a raised floor data center consisting of 4,800 servers and the server

we modeled resembles an HP ProLiant DL 360 G6. The peak and idle power of the

modeled server are 186W and 62W respectively. We convert the server utilization

traces to its power consumption using the published SPECPower results [45], which

have been widely used for data center power evaluation [37] [38]. Since the

SPECPower results only reflect the server power at intervals of 10% utilization, we use

linear interpolation to approximate the power across different load levels.

Our heterogeneous data center configuration is based on an academic high-

performance computing (HPC) center in the department of electrical and computer

engineering at the University of Florida, which hosts over 600 servers. The HPC center

has five major clusters (C-I~C-V) with different service targets and loads, as detailed in

Table 4-1. Those clusters have 20 to 400 computing nodes and their average utilization

ranges from 25% to more than 90%. Since we have limited access to industrial data

center traces, we collect real-world workload logs from a well-established online

repository [46]. The workload logs provide information such as job arrival times, start

time, completion time, and number of requested nodes.

53

Table 4-1. Configurations of heterogeneous clusters.

Cluster ID % of Overall Deployed Servers Average Cluster Loading

C-I 5% 97% C-II 63% 60%

C-III 17% 57%

C-IV 3% 54%

C-V 12% 25%

Table 4-2. The evaluated data center workload sets. Workload Set Workload Trace Combination

Mix-High “HPC2N” + ”LANL CM5” + ”LPC EGEE” + “SDSC BLUE” + “LLNL Thunder”

Mix-Low “DAS- fs0” + “DAS2-fs1” + “DAS2-fs2” + “DAS2-fs3” + “DAS2-fs4”

Mix-Stable “HPC2N” + “KTH SP2” + “LANL CM5” + “DAS2-fs0” + “SDSC BLUE”

Mix-Bursty “DAS2-fs2” + “DAS2-fs3” + “DAS2-fs4” + ”LPC EGEE” + “OSC Cluster”

Mix-Rand “LLNL Thunder” + “OSC Cluster” + ”LPC EGEE” + ”LANL CM5” + “KTH SP2”

Dept-HPC Five traces collected from departmental high-performance computing center

Table 4-2 summarizes our evaluated workload trace combinations. In additional

to the load traces from our local HPC center (i.e., Dept-HPC), we also build various

workload mixes to mimic co-located clusters. Each workload set in Table 4-2 consists of

five traces which run on the five clusters (C-I to C-V) respectively. We characterize

workload traces based on their average job size and runtime. In Table 4-2, Mix-High

includes traces that have larger job size (resulting in >30% average data center

utilization) and Mix-Low contains traces that have small job size (resulting in <10%

utilization). On the other hand, Mix-Stable consists of five traces that feature relatively

longer job runtime and Mix-Bursty consists of traces that have very short job runtime.

We use wind power data traces from The Wind Integration Datasets [47]. In

Table 4-3, capacity factor is the ratio of the actual wind turbine output to the theoretical

peak output. We have selected a group of traces with various capacity factor values.

Typically, the capacity factor is 30% [35]. The total installed wind power capacity in this

study equals to the nameplate power of data centers. The actual power budget is

therefore only affected by the capacity factor.

54

In Table 4-4 we list our evaluated power management schemes, Utility and

Battery are two conventional schemes which do not adjust server load to match the

renewable power budget. Green is the most sustainable design that only uses wind

energy to power server clusters. Tracking represents emerging design approaches

which adjusts server load to actively track every joule of renewable energy generation

(i.e., place as much workload as possible on green clusters) [10] [22].

Table 4-3. The evaluated wind power supply traces. Traces Wind Energy Potential Locations (ID) Capacity Factor Power Density

W1 Low California (9250) 15% 195 W/m2 W2 Medium Arizona (6107) 25% 338 W/m2 W3 Typical Colorado (10563) 35% 581 W/m2 W4 High Texas (1360) 45% 708 W/m2

Table 4-4. The evaluated server cluster power management schemes. Abbr. Design Philosophy Power Tracking Stored Energy Load Deferment

Utility Fully relies on utility grid No No No Battery Relies on battery No Yes No Green 100% renewable energy Yes No Yes Tracking Aggressive tracking Yes Yes No iSwitch Low overhead and latency Yes Yes No


We first characterize the impact of supply/load power variability on data center

load matching using homogeneous workload traces. Frequent load matching activities

result in operational overhead, which is our primary concern in the design of renewable

energy powered computing systems. In a virtualized environment, iSwitch could

effectively reduce the VM migration rate and help to save data center network

bandwidth significantly. The data migration traffic is calculated at rack-level. Each VM

live migration transfers approximately the size of the VM’s memory between hosts [48].

We assume a VM memory size of 1.7GB in our calculation, which is the default memory

size of Amazon EC2 standard instance.

55

Figure 4-11. The average network traffic. All the results are normalized to Tracking.

Figure 4-12. The peak network traffic. All the results are normalized to Tracking.

Figure 4-11 shows the average communication traffic across various workload

configurations and wind power supply levels. All the results are normalized to Tracking.

We do not show the results of Green because it has the same power tracking frequency

as Tracking. As can be seen, on average, iSwitch could reduce 75% of the rack-level

traffic and therefore significantly releases the network bandwidth burden. The results

are even more impressive for peak traffic hours when the renewable energy fluctuates

severely. In Figure 4-12 we calculate the communication traffic during the top 1% high-

traffic hours. Because iSwitch puts a limit on the power tracking activities during

fluctuant supply period, it shows only 5% network traffic compared with Tracking.

Another advantage of iSwitch is that it reduces the migration frequency of each

VM instance and thereby improves the job turnaround time. We calculate an execution

time increase (ETI) for each job to evaluate the impact of load shedding and workload

deferment on job execution time. Even for utility-connected systems such as iSwitch

56

and Tracking, ETI exists due to the data migration. For example, the time needed for a

1GB VM migration takes about 20 seconds to complete [49].

We use ETI per hour (EPH) to evaluate the performance overhead. For example,

an EPH of 10 means each individual VM instance experiences 10 seconds waiting time

per hour on average. Figure 4-13 shows the average EPH across the entire data center

servers. The average EPH of iSwitch is about 30 seconds per hour while the average

EPH of Tracking reaches 126 seconds per hour. The average EPH of Green, however,

is about 1500 seconds per hour – 50 times that of iSwitch. Therefore, waiting for

renewable energy to resume (e.g. Green) should be the last resort.

We evaluate the renewable energy utilization (REU) of data centers with different

wind power provisioning capacities and workload behaviors. The REU is defined as the

amount of wind energy that is actually utilized by the load divided by the total amount of

wind energy generation. A higher REU indicates better supply/demand coordination,

which reduces on-site energy storage capacity, improves return-on-investment (ROI)

and data center sustainability, and eases the initial infrastructure planning.

While iSwitch uses a lazy power tracking scheme, it does not sacrifice energy

utilization significantly. As shown in Figure 4-14, iSwitch can achieve an average

renewable energy utilization of 94% – higher than Green (92%) but lower than Tracking

(98%). The reason why Tracking outperforms iSwitch on energy utilization is that

Tracking tracks every joule of wind energy generation aggressively. Note that a 4%

decrease in energy utilization does not mean that iSwitch is less preferable in our study;

iSwitch significantly reduces network traffic and improves the performance by 4X. In

contrast to Tracking, iSwitch trades off energy utilization for better turnaround time.

57

Figure 4-13. The average execution time increase (ETI) of job requests.

Figure 4-14. The normalized overall wind energy utilization.

4.6 Summary of iSwitch Design

Matching the variable load power consumption to the intermittent power supply

appropriately is the crux of designing a renewable energy powered data center.

Conventional workload-driven power management has less adaptivity to the power

supply variation while existing power-driven system control schemes are not aware that

they are experiencing unnecessary and redundant tuning activities. As a result, we

either miss the opportunity of utilizing precious renewable energy or incur significant

load tuning overhead and performance degradation.

This chapter proposes iSwitch, a renewable energy-driven power tuning scheme

that addresses a two-fold challenge: the first is to manage intermittent renewable power

without power throttling; the second is to ensure high energy utilization with minimized

load matching activities. To the best of our knowledge, this is the first work that digs into

renewable power variation characteristics and introduces a supply/load cooperative

optimization scheme that minimizes power management overhead.

58

CHAPTER 5 DATA CENTER LEVEL GREEN POWER MANAGEMENT:

TOWARDS LOAD-FOLLOWING BASED DESIGN

Supply-tracking (a.k.a. supply-following) can eliminate supply-demand power

mismatches in renewable energy powered computing systems, as demonstrated in

previous SolarCore design and iSwitch design. In this chapter we discuss a radically

different design approach, which allows near-optimal performance of computing

systems on green energy resources. The key idea is to put emphasis on enabling

renewable energy supply to follow power demand, rather than forcing server systems to

track the variable power budget. To achieve this, we leverage distributed generation

initiated by the smart grid technology [50].

Distributed generation (DG) refers to a variety of small, modular electric

generators near the point of use. In recent years, DG has gained tremendous interest

as an alternative source of power for IT industry. According to the U.S. Environmental

Protection Agency (EPA), using DG in data center design could achieve great energy

savings, significant environmental benefits, and high power reliability [51].

As shown in Figure 5-1, DG system encompasses a wide range of green energy

technologies, such as solar panels, wind turbines, fuel cells, and bio-fuel based gas

turbines. While solar/wind power heavily depends on environmental condition, the

outputs of fuel cells and gas turbines are tunable. They can provide a key supporting

service called load following [52], which refers to the use of online generation equipment

to track the changes in customer loads. Therefore, one can take advantage of the load

following capabilities of these tunable DG systems to meet the time-varying server

power demand. Such design is non-trivial because it enables computing system to run

on renewable energy sources without compromising workload performance.

59

Figure 5-1. Distributed generation (DG) powered data centers. Some of the tunable

DG systems can be controlled to follow the load power demand.

Figure 5-2. Load following scenario. Fine-grained server power demand fluctuation

hinders efficient load following on distributed generators.

When employing distributed generation to build a better data center power

provisioning architecture, challenges arise due to the unpredictable and fluctuating data

center load. Figure 5-2 shows typical load following scenario that tracks customer load

every 30 minutes. As can be seen, DG systems mainly provide coarse-grained load

demand following due to limited response speed of onsite generators. To handle the

moment-to-moment load power demand, DG typically relies on large on-site energy

storage elements [52]. Such design not only increase the TCO (due to storage cost and

maintenance), but also incurs up to 25% roundtrip energy loss.

We propose data center power demand shaping (PDS), a novel power

management approach that enables high-performance low-overhead data center

operation on pure renewable energy sources. The novelty of PDS is two-fold. First, PDS

intelligently trims data center load power and enables DG systems to follow the power

60

demand efficiently. Second, PDS features two adaptive load tuning schemes that could

boost data center performance and enable near-oracle operation during power demand

trimming process. As a cross-layer power optimization scheme, our power management

module resides between front-end distributed generation and back-end computing

facilities to provide a coordinated tuning between the supply and load.

5.1 The Load Following Challenge

Different from conventional bulk grid which has large amount of capacity inertia,

distributed generators does not have reserved capacity [51]. The supply and storage of

energy must be planned carefully to ensure instantaneous energy balance [50].

Most energy storage devices have very fast response speed that could release

power almost immediately. As a result, they are widely used to handle moment-to-

moment load oscillation and disturbances, which is referred to as regulation [52]. In

contrast, gas turbines and fuel cells are typically too slow to meet the load power

variation since the change of the engine speed or the chemical reaction in the fuel

requires time. Therefore, they are used to track the intra- and inter-hour changes in

customer loads, which is referred to as load following [52]. Although there is no strict

rule to define the temporal boundary between regulation and load following, typically

load following occurs every 10~15 minutes or more [52]. It is not economically feasible

to frequently adjust the power output of distributed generators due to the increased

performance cost and decreased fuel utilization efficiency.

The energy balance issue arises due to the fluctuating load in data centers and

other computerized environments. Dynamic power tuning techniques (e.g., DVFS),

frequent on/off power cycles, stochastic user requests, and data migration activities can

cause varying degree of load power variation. Since distributed generators are generally

61

placed near or at the point of energy consumption, they are often exposed to the full

fluctuation of local IT loads rather than experiencing the averaging effect seen by larger,

centralized power system. As a result, energy balancing becomes rather difficult in

distributed generation powered data centers.

5.2 Distributed Generation Powered Data Centers

Conventional design always focus on managing data center power demand and

overlooks the capability of power supplies. In contrast, our system has two tuning

knobs: the onsite generation level and the server power demand.

We adopt typical microgrid power provisioning hierarchy for managing various

distributed energy resources, as shown in Figure 5-3. The design consists of a group of

radical feeders to provide power to electrical loads. Microsources are connected to the

feeder through circuit breakers and coordinated by the microgrid energy management

module (EMM), which provides real-time monitoring and autonomic control of the power

generation [53]. Although microgrid can connect to the utility through a single point

called point of common coupling (PCC), we focus on standalone microgrid due to

sustainability and cost-effectiveness reasons. Such islanded mode is also a remarkable

feature of microgrid to avoid power quality issue in the main grid [53].

Figure 5-3. Distributed generation powered data centers. A power demand controller is used to coordinate onsite generators, batteries, and data center loads.

62

The connected electrical loads may be critical or non-critical. HPC servers are

typically sensitive and mission-critical loads that require stringent power quality and

sufficient power budget. It is preferable to assign these loads with controllable and

stable microsources such as gas turbines and fuel cells.

We use a power demand controller (PDC) to manage green data centers. It

keeps monitoring the job running status and the overall power demand. It also

coordinates with the job scheduler for the purpose of better computing resource

allocation. Meanwhile, it also communicates with the microgrid EMM to obtain current

DG levels and dynamically inform the EMM for necessary adjustment.

A microgrid system typically uses frequency droop control to monitor and adjust

the on-site power generation [54]. This scheme uses a small change in the power bus

frequency to determine the amount of power that each generator should put into the

power bus. In addition, the microgrid also includes energy storage devices to provide

necessary startup time for generators’ output to ramp up when system changes. In this

study we leverage uninterruptible power supplies (UPS) system to provide necessary

stored energy. Note that although our proposed technique also applies to centralized

UPS systems, we choose emerging distributed UPS architecture which is known to

have higher energy efficiency and better scalability [17].

5.2.1 Power Demand Shaping Mechanism

While most of the related work on green data center focuses on managing the

load, we have identified a non-conventional supply-side opportunity. We propose power

demand shaping (PDS) technique to better utilize onsite distributed generation in

modern HPC data centers. The idea is to aptly influence data center power demand to

match the planned power usage curve that yields high load following effectiveness.

63

Although one can transform the IT power demand into many different shapes, we

found that a square-wave-like demand curve is the most convenient, as shown in Figure

5-4. In this case, the distributed generation system only needs to increase (ramp up) its

generation at each end of the load tuning cycle (e.g., every 15 minutes) to meet the new

power budget goal. Since the load power demand becomes less bursty and fluctuating

after tuning, there is no need to install bulky energy storage devices onsite. In this work

we mainly leverage the UPS to handle the ramping period and the temporary power

discrepancy between supply and load. By eliminating the unnecessary energy storage

capacity, our design also becomes more cost-effective and sustainable.

Figure 5-4 demonstrates our power demand shaping mechanism in terms of

power supply behavior and load performance status. The proposed scheme uses a

three-step control process in which data center load is adaptively managed and the

distributed energy is carefully scheduled.

Figure 5-4. Power demand shaping demonstration. The idea is to gracefully

influence the data cater power demand to generate a planned, DG-friendly power usage curve.

64

Step 1: Maintain constant power demand. During each control period, the

power demand controller uses power capping to maintain a constant power demand.

Assume the original load power demand (i.e., the expected power consumption without

any load tuning) is D, we scale down the load performance whenever D exceeds current

power demand goal. If the power demand is below the power supply (rarely occurs

since we under-provision the system), we store the excess power in UPS batteries.

Depending on the system’s hardware characteristics and BIOS support, the

actual performance scaling policy may vary. In this work we take advantage of

performance states (a.k.a. P-states) to dynamically adjust CPU frequency to match the

required budget. This is a technique to manage power by running the processor at a

less-than-maximum clock frequency. We do not use voltage scaling because it may not

yield notable cubic power saving in the real measurement [40]. It has been shown that

the power-to-frequency curve for both DFS and DVFS can be approximated as a linear

function [40]. As a result, we use only frequency scaling for simplicity.

Step 2: Re-schedule DG generation. In contrast to existing power capping

scenario, PDS deals with time-varying power capping budget. At the end of each control

period, the controller will inform the distributed generation EMM to update the

generation level. To calculate the amount of generation adjustment, PDS calculates the

mean value of the original power demand in the past control period. The actual DG

power adjustment is selected from the mean value and current demand measurement,

depending on whichever is greater. After that, the distributed generation will gradually

increase its output and we assume 1-min power ramping for the generators. During this

period, we leverage onsite UPS storage to provide necessary power support.

65

Step 3: Adjust node performance. At the beginning of each new period, we

update the node performance level using the newly scheduled energy. There are

several different scaling schemes. One can improve the frequency of every running

node to achieve a fair performance tuning. In addition, we can use a priority-based

scheme. It associates each node with a priority and makes a scheduling choice based

on the priorities. For example, a node with the highest priority gets highest frequency.

In Figure 5-5 we show the pseudo code of power demand shaping. The controller

monitors the data center load every tick (i.e., every second in our design) and adjust

node frequency based on the discrepancy between the budget and the original demand.

At the end of each cycle TL, the controller adjusts the budget for a new demand goal.

Definition: TL: Load following interval; D: Maximal power demand of

current load; P: Mean demand in the last control interval

1:

2:

3:

4:

5:

6:

7:

8:

9:

10:

11:

for Each load tuning timestamp tick

if tick % TL == 0 // update power budget

Budget ← max(P, D);

else // maintain constant power demand

D ← load power measurement

if load demand changes

ΔPLF ← (Budget – D)

Adjust load performance based on ΔPLF

end if

end if

end for

Figure 5-5. Power demand shaping algorithm.

5.2.1 Adaptive Load Performance Scaling

Strict power demand shaping facilitate load following but comes with

performance degradation. To this end, we also propose adaptive load performance

scaling that helps PDS to improve job performance while maintaining high load following

effectiveness. The key idea is to use a relaxed power demand shaping policy [24].

66

In addition to the scheduled generation adjustment, the controller also assigns a

bonus power budget for two reasons. First, after a period of time, the UPS requires

recharge to retain its full capacity. In this case, the PDS will assign additional generation

in the next period based on the required charging power. Second, PDS features an

adaptive load performance scaling scheme which intelligently leverages DG energy and

UPS energy for boosting the load performance.

DGR boost. The first optimization scheme helps to fine-tune the distributed

generation level [24]. Since we have no oracle knowledge of the incoming task,

adjusting DG generation based on historical average unavoidably incurs prediction

error. All the active jobs under this circumstance suffer low processor speed and the

negative impact can last as long as the load tuning interval. To solve this problem, our

DGR Boost optimization dynamically monitors the normalized job execution progress

and the overall system throughput, and leverages the over-clocking capabilities of

modern processor nodes to minimize the job delay. To enable DGR Boost, the PDS

controller needs to dynamically monitor the execution status of each active job.

UPS boost. This scheme fine-tunes intra-cycle load power [24]. The main idea is

that for short-running jobs (jobs that finish within one power demand control interval)

that cannot gain the benefits of DGR Boost, we can leverage the UPS stored energy to

provide additional power support and avoid significant performance degradation. In

typical HPC data centers, users are required to submit their job runtime estimations to

enable backfilling, which can help maximize cluster utilization and throughput. In this

study we use job runtime to sort out short jobs. The short jobs here are defined as

tasks that will complete before current load following cycle ends.

67


We develop a HPC data center simulation infrastructure that takes real-world

workload traces as input. Our discrete-event simulator puts each user’s request into a

queue and waits to grant allocation if computing nodes are available. Each job request

in the trace has exclusive access to its granted nodes for a bounded duration. Such

trace-driven discrete-event simulation has been adopted by several prior studies on

investigating data center behaviors and facility-level design effectiveness [37] [38].

We simulate the power behavior of distributed generation system on per-second

time scale which is in tune with our data center simulator. The distributed generator

adjusts its output on a coarse-grained interval (10-minute as the default value) and

batteries respond to the fluctuating computing load. We adopt microsource optimization

model HOMER in our simulator [55]. HOMER simulates both conventional and

renewable energy technologies and evaluates the technical feasibility of a large number

of power system options. We assume a 1-minute constant duration for the distributed

generation to ramp up when perform load following. The battery cycle life is set to be

10,000 times and we calculate the capacity degradation of battery cell based on its

discharging profile which includes detailed information of each discharging event.

We still use real-world workload traces from [46]. We use five key task

parameters of each trace file: job arrival time, job start time, job completion time,

requested duration, and job size in number of granted computing nodes. As shown in

Table 5-1, we select six 1-week workload traces that have different mean utilization

level and mean job runtime. Our simulator uses batch scheduling and the job

scheduling policy is first come first serve (FCFS). We examine the data center load and

distributed generation budget at each fine-grained simulated timestamp.

68

Table 5-1. The evaluated HPC data center workload traces.

Trace Description Avg. Load

Mean Inter-arrival Time

Avg. Job Run Time

LANL Los Alamos Lab’s 1024-node machine 56% 4.9 min 31.6 min SDSC San Diego Supercomputer Center’s Blue Horizon 60% 3.7 min 33.0 min Atlas Lawrence Livermore Lab’s 9216-CPU cluster 46% 10.6 min 36.8 min Seth A 120-node European production system 85% 20.5 min 6.2 h MCNG A 14-cluster data center, 806 processors in total 89% 2.2 min 11.1 h RICC A massively parallel cluster with 8k+ processors 52% 0.9 min 16.6 h

Table 5-2. The evaluated data center power management schemes.

Schemes Descriptions

Oracle Ideal power provisioning with no performance overhead LF Existing load following based design ST Existing supply tracking based design PDS-s PDS without optimization ( i.e., uses strict power budget ) PDS-r PDS + adaptive load tuning (DGR Boost & UPS Boost)


In this section we discuss the benefits of applying power demand shaping to

distributed generation powered data centers. In Table 5-2, Oracle is an ideal design that

has a priori knowledge of load patterns and could always meet the fluctuating data

center power demand. It represents the ideal energy balance scenario that one can

achieve with renewable energy resources. Different from Oracle, LF is a conventional

load following based scheme that has heavy reliance on energy storage. ST represents

existing power supply driven design that aims at managing the computational workload

to match the renewable energy supply [10] [12]. PDS-s is our proposed power demand

shaping mechanism without optimization. PDS-r is the relaxed power demand shaping

that features adaptive workload-aware performance scaling.

Performance is one of the key driven forces of our power demand shaping

technique. Although tracking renewable power budget has shown great success on

reducing IT carbon footprint, it cannot ensure performance and sustainability

69

simultaneously. Table 5-3 evaluates the performance degradation of supply tracking

based design under different wind energy intensity. We have scaled the wind power

traces so that the average wind power budget equals the average data center power

demand. When renewable energy generation drops or intermittently unavailable, we

scale down the node frequency and defers job execution if it is necessary.

Table 5-3. Mean job turnaround time of tracking based design (ST) under different renewable generation levels. All the results are normalized to Oracle.

LAN SDSC Atlas HPC MCNG RICC

200 W/m2 2.92 1.60 1.64 7.25 1.31 1.24 400 W/m2 2.10 1.51 1.66 9.27 1.12 1.29 600 W/m2 2.89 1.41 1.37 5.91 1.08 1.10 800 W/m2 5.27 1.34 1.30 7.00 1.06 1.08

The results in Table 5-3 show that data centers designed to track the variable

renewable power supply could incur up to 8X job runtime increase. The geometric mean

value of turnaround time increase is 59%. It also shows that workload performance

heavily depends on renewable power variation behavior and data center load pattern.

We have observed that occasionally the renewable energy variation pattern is totally

uncorrelated with the load power demand. There is no guaranteed performance lower

bound since both the user behavior and environment conditions are stochastic.

In contrast to supply tracking based design, load following fundamentally

changes this situation. Figure 5-6 shows the mean job turnaround time under varying

load following interval. All the results are normalized to Oracle, which has no

performance scaling and maintains full-speed server operation with sufficient power

budget. The average execution time increase (ETI) of PDS-s is 8.0%. Our optimized

scheme, PDS-r, shows 1.2% mean performance degradation compared to Oracle. This

means that power demand shaping with adaptive load tuning (i.e., DGR Boost and UPS

Boost) could yield a 37% improvement over existing supply-tracking based design.

70

Figure 5-6. Average job turnaround time under varying load following interval. All the

results are normalized to Oracle.

A B

Figure 5-7. Execution time of the worst 5% jobs. A) Results of PDS-s. B) Results of PDS-r. All the results are normalized to Oracle.

In addition to the mean performance statistics, Figure 5-7 further investigates the

worst-case scenario in terms of the mean job turnaround time of the worst 5% jobs. We

observe that PDS-s and PDS-r shows 47% and 9.6% performance degradation,

respectively. Therefore, PDS-r shows better performance guarantee in terms of the

worst-case performance degradation.

Conventional green energy-aware scheduling schemes normally rely on utility

grid. To achieve the same performance as PDS-r, these designs have to increase their

reliance on the utility grid, as shown in Figure 5-8. The percentage of time that utility

power is used is between 30% and 70%. The geometric mean value across all

workloads and renewable generation levels is 51%. Heavy reliance on utility means

increased carbon footprint and utility power cost, which goes against the original

intention of sustainability.

71

Figure 5-8. Dependence on utility power. To compete with PDS-r, even state-

of-the-art designs have to heavily rely on the utility power.

Compared to conventional load following, PDS results in much lower UPS stress.

In Figure 5-9B, both PDS-s and PDS-r shows less than 3% battery lifetime degradation.

Although PDS-r explores additional stored energy to perform performance boost, its

impact on UPS system is very similar to that of PDS-s. Note that for PDS, the battery

wear out problem is alleviated when we increase the interval of LF cycles. This trend is

different from what we observed in LF. The reason is that batteries are mainly used to

provide power shortfall during generator’s ramp-up period in PDS based system. When

we increase the LF intervals, we also lower the frequency of generator ramp-up, the

dominant factor of battery discharging.

Another advantage of PDS is that it allows one to use conventional centralized

UPS system to assist power demand shaping. The drawbacks of centralized UPS

system is that it cannot provide a fraction amount of energy – all the data center load

has to be switched between the main supply and UPS. Consequently, during each UPS

ride-through, UPS batteries have to experience much higher discharge current. The

immediate result of this is significantly decreased UPS lifetime. As shown in Figure 5-

10, LF in this case results in 35% lifetime degradation on average. The lifetime

degradation of PDS-s and PDS-r is 6% and 7%, respectively. Therefore, we believe

PDS maintains acceptable overhead and can be applied to centralized UPS systems.

72

A B

Figure 5-9. Battery lifetime degradation comparison (with distributed UPS). A) Load following based design. B) PDS design.

Figure 5-10. Battery life degradation comparison (with centralized UPS).

5.5 Summary of PDS Control

Distributed generation (DG) systems are gaining popularity and their installed

capacity is projected to grow at a faster pace. We expect this trend to continue, as the

energy crisis and environmental problem become increasingly crucial to our planet.

In this work we investigate how the incoming smart grid incentive would impact

the design and optimization of data centers. We propose distributed generation

powered data center that leverages the load following capability of onsite renewable

generation to achieve low carbon footprint without compromising performance. The

unique feature of load following based design makes this work a step forward toward

the goal of incorporating a variety of clean energy resources into computing systems.

We have developed novel power demand shaping technique (PDS) to improve the load

following efficiency in data centers while boosting the workload performance with two

adaptive load tuning schemes: DGR Boost and UPS Boost. Overall, PDS could achieve

98.8% performance of an ideal oracle design.

73

CHAPTER 6 ENABLING DATA CENTER SERVERS TO SCALE OUT SUSTAINABLY

As cloud applications proliferate and data-processing demands increase, server

resources must grow to unleash the performance of emerging workloads that scale well

with large number of compute nodes. Over time, the constant influx of server resources

in data centers will eventually become power-constrained. According to a recent

industry survey from the Uptime Institute [56], 30% of the enterprise data center

managers expected to run out of power capacity within 12 months.

While leasing collocated racks and deploying cloud services fit small server

clusters on a budget, this study mainly focuses on large-scale enterprise data centers

and approaches that aim at improving computing capability and capacity. Server

consolidation is mature techniques that can free up power capacity. However, even

consolidated servers have to limit their performance using either software-based (i.e.,

virtual CPU allocation) or hardware-based (i.e., DVFS) control knobs to avoid tripping

circuit-breakers and causing costly down-time.

Upgrading power systems is a radical solution that allows one to add more

servers, racks, and even onsite containers. However, like building a new data center,

re-sizing data center initial power capacity can be a great undertaking since

conventional centralized power provisioning scheme does not scale well. In a typical

data center, the power delivery path involves multiple power equipment across several

layers, as shown in Figure 6-1. Upgrading power infrastructure often requires re-design

of the entire power delivery path which is not only costly but also time-consuming.

Worse, utility power feeds are often at their capacity in some urban areas and additional

electricity access for data center is restricted.

74

In addition, modern scale-out data centers are not only power-constrained, but

also carbon-constrained. As server power demand increases, the associated carbon

footprint expansion poses significant challenges for scale-out data centers. It has been

shown that global greenhouse gas (GHG) emissions could easily exceed 100 million

metric tons per year if we keep using utility grid power that is generated from

conventional fossil fuel [57]. An emerging solution to the carbon reduction problem is to

leverage green energy sources to lower the environmental impact of computing

systems. Unfortunately, to the best of our knowledge, existing green energy integration

schemes often employ centralized power integration architecture and largely overlook

the modularity of renewable power supplies. As shown in Figure 6-1, facility-level

integration allows renewable power supplies to be synchronized to the utility grid.

However, the operation of such grid-connected renewable power system often relies on

the availability of utility power. Meanwhile, the centralized power integration not only

results in single-point of failure, but also makes future capacity expansion expensive.

Figure 6-1. Conventional data center power architecture is not ‘scale-out friendly’.

75

In this chapter we ask one important question: “Can we enable a power-

constrained data center to gracefully scale out while lowering the carbon

footprint with high efficiency?” Faced with the ever-growing computing demand, the

skyrocketing server power expenditures, and the looming global environmental crisis,

this question is of significant importance to data center operators who have to embrace

efficient power provisioning to survive economically and sustainably.

We propose Oasis, a unified power provisioning scheme that synergistically

integrates energy source control, power system monitoring, and architectural support for

power-aware computing. Oasis leverages modular renewable power integration and

distributed battery architecture to provide flexible power capacity increments. It allows

power-/carbon- constrained systems to stay on track with horizontal scaling and

computing capacity expansion.

We implement Oasis as a prototype research platform for exploring key design

tradeoffs in multi-source powered data center. Our first generation prototype is a micro

server rack (12U) that draws power from onsite solar panels, conventional utility power,

and local energy storage devices. These energy sources are coordinated by a micro-

controller based power control hub (PCH) built from scratch. We customize the PCH to

make it a rack-mounted, interactive system for easy installation and diagnosis.

We further propose Ozone, a power management scheme for Optimized Oasis

Operation (O3). Transcending the boundaries of traditional primary and secondary

power, Ozone creates a smart power source switching mechanism that enables server

system to deliver high performance while maintaining desired system efficiency and

reliability. Ozone is able to dynamically distill crucial runtime statistics of different power

76

sources to avoid unbalanced usage of different power sources. It could identify the most

suitable control strategies and adaptively adjusts the server speed via dynamic

frequency scaling to maximize the benefits of Oasis system.

To summarize, Oasis and Ozone constitute an interesting platform from many

perspectives: (1) it enables data centers to scale-out and facilitates partial green power

integration; (2) it links power supply management and server system control, and could

enable real-time power supply driven workload scheduling; (3) its power provisioning

architecture (i.e., hybrid power supplies + distributed control domain) could bring us

improved data center availability in many power failure scenarios; (4) its decentralized

multi-source power management could offer the flexibility of offering different green

computing services based on different customer expectations.

6.1 Oasis Scale-Out Model

We classify existing power capacity scale-out model as either utility power over-

provisioning or centralized power capacity expansion. In the first case, the utility grid

and data center power infrastructure is designed to support the maximal power the load

may ever draw. Although such design provides abundant power headroom, it inevitably

increases the carbon footprint. In the second case, the power delivery infrastructure is

provisioned for a certain level of anticipated future power drawn and the scaling out is

handled entirely by data center-level power capacity integration. However, installing

large-scale renewable power system often results in high expansion cost.

Oasis explores a “pay-as-you-grow” model for scale-out data-centers, which we

refer to as distributed green power increments model. In this model, the utility grid and

data center power delivery infrastructure are provisioned for a fixed level of load power

demand. When data center reaches its maximum capacity, however, we add renewable

77

power capacity by small increments in a distributed manner. This approach not only

provides carbon-free power capacity expansion to a power-constrained data center, but

also avoids over-committing its capitals. As shown in Figure 6-2, Oasis consists of a

number of green energy powered computing racks, called Oasis nodes. Each Oasis

node is attached to a power control hub (PCH), which further connects to renewable

power supply, power distribution unit (PDU), and distributed energy storage devices.

6.1.1 Modular Power Sources

Scale-out data centers need both modular standby power sources that provide

incremental backup power capacity and modular primary power sources that generate

additional electrical power. Today, distributed battery architecture is emerging to

improve data-center efficiency [17] [58]. It provides us a convenient way to add

additional backup power as computing demand increases. In addition, renewable power

supplies such as solar panels are usually modular and highly scalable in their capacity.

They are ideal power supplies that support carbon-free server scale-out.

Figure 6-2. Oasis power provisioning architecture. It leverages the modularity of both

battery systems and renewable power systems.

78

Figure 6-3 illustrates the distributed battery architectures in the Open Compute

Project [58] led by Facebook. In the coarse-grained integration scheme, a battery

cabinet populated with commodity lead-acid batteries is used to provide standby power

for a rack triplet. Each triplet consists of three column racks and each rack is further

divided into three power zones. The battery cabinet also includes several breakers,

quick fuse disconnects, sensors, and a high current DC bus bar. In the fine-grained

integration scheme, the battery cabinet is replaced by a high-density lithium-ion battery

backup unit (BBU) in each rack power zone. In both cases, battery provides 48V DC

backup power and could provide around 45 seconds of runtime at full load.

Figure 6-3. Distributed energy storage system (battery) emerge in

recent implementation of server clusters.

On the other hand, wind turbine and solar panel are both modular power

sources. Compared to wind turbines, solar panels can provide even smaller capacity

increments. In this study we look at solar panels that use micro-inverter [59] to provide

incremental power capacity. Conventionally, solar power system uses string inverters

which require several panels to be linked in series to feed one inverter. String inverters

79

are big, prone to failure from heat, and shows low efficiency. In contrast, micro-inverters

are smaller in size and can be built as an integrated part of the panel itself. In Figure 6-

2, each panel has its own micro-inverter; solar panels are further connected in parallel

to offer larger power capacity. Compared to centralized design, solar panel with micro-

inverter shows much better scalability, reliability and efficiency. The only disadvantage

is its relatively high cost per watt compared to centralized inverter. However, the

amortized cost can be lower than a centralized system (Section 6.5).

6.1.2 Distributed Integration

Oasis adds renewable power capacity at power distribution unit (PDU) level. As

shown in Figure 6-2, the PDU-level renewable energy integration allows us to scale out

server system on a per-rack basis. We do not use centralized power integration since it

does not support fine-grained server expansion very well. Today’s power-constrained

data center typically over-subscribes pieces of its power systems such as the PDU,

thereby creating power delivery bottlenecks. Adding renewable power capacity at data

center-level does not guarantee increased power budget for the newly added server

racks since the associated PDU probably has already reached its capacity limit.

Oasis does not synchronize renewable power to utility power due to three

considerations. First, at the PDU level, renewable power synchronization induces

voltage transients, frequency distortions, and harmonics. Data center servers are

susceptible to the power anomalies. As the impact of power quality issue on server

efficiency and reliability is still an open question, massively adding grid-tied renewable

power at rack level is debatable. Second, even if one chooses to synchronize

renewable power at data center facility-level, grid-tied renew-able power system can

cause efficiency problems. The newly added renewable power incurs many levels of

80

redundant power conversion before reaching server racks, resulting over 10% energy

loss [60]. Third, as required by UL 1741 and IEEE 1547, all grid-tied inverters must

disconnect from the grid if they detect power islanding. That is, these renewable power

systems must shut down if the grid power no longer present. As reported in a recent

survey, the U.S. data center experiences 3.5 times of utility power loss per year with an

average duration over 1.5 hours [60]. Thus, with synchronization based power

provisioning, data centers may lose power supply even when they need it most.

The intention of Oasis project is to facilitate server scale-out in power-/carbon-

constrained data centers. Oasis aims to become a non-intrusive, modularized

renewable power integration scheme which allows data center operators to increase

power capacity on-demand and reduce data center carbon footprint gradually.

6.2 Implementation of Oasis

In this section we describe our implementation of Oasis starting from scratch

(Figure 6-4A). Specifically, we focus on Oasis nodes, the key element of Oasis design.

As shown in Figure 6-4B, its structure consists of a power control hub, a server rack,

and a power management agent that coordinates them.

6.2.1 Power Control Hub

The power control hub (PCH) is the major hardware addition of Oasis nodes. It

integrates battery charger, power inverter, power supply switch panel, programmable

logic controller (PLC), and a human-machine interface (HMI) that allows easy system

diagnosis. The PCH is designed to manage multiple energy sources (i.e., renewable

power, utility power, and distributed battery devices). On its backside we provide three

electrical sockets that allows easy connection to utility power (AC), battery systems

(DC), and solar panels (DC).

81

A B Figure 6-4. Prototype of Oasis node. A) System prototype. B) Node architecture.

Our prototype system is powered by an array of solar module. Each module is a

270 Watt Polycrystalline panel from Grape Solar. The panel output is a complex

function of the solar irradiation, ambient temperature, and the load connected to it. To

harvest the solar energy, we use a maximal power point tracker (MPPT) in our power

control hub. The MPPT samples the output of the solar cells and applies proper control

to obtain the optimal solar energy generation. However, it is not an essential part of the

PCH. The MPPT is often a build-in module of the solar micro-inverter.

Oasis leverages distributed battery devices to provide temporary solar energy

storage. Energy storage devices, such as uninterruptable power supplies (UPS), are

widely used in data centers to address the risk of interruptions to the main grid. We

have customized a stack of nine 2Ah sealed lead-acid batteries for the power control

hub (scalable in the future). Our battery chassis can provide 5~10 minutes of backup

time, depending on the server load.

82

We convert the DC solar power to AC to match the output level of utility power.

The AC solar power and AC utility power are merged (but isolated) at a switch panel in

the PCH. We use high volt-age Omron relay to perform power switch and a Mitsubishi

FX2N programmable logic controller (PLC) is used to manage the switch behavior.

Finally, the PCH routes power to rack-level power distribution strip which further feeds a

cluster of server nodes. The entire rack is either powered by the utility power or

renewable power, depending on the status of the internal power switch.

We measured several key technical data of our current Oasis prototype. The

maximum charging current of our system is 4A, which is limited by the battery charger

for battery lifetime considerations. The PCH itself consumes static power (about 9

Watts) due to the HMI and PLC operations. The dynamic power loss is due to the heat

dissipation of power conversion systems.

6.2.2 Bridging Server and Power Supply

We enrich Oasis with sensing components that keep monitoring real-time solar

power output voltage and battery terminal voltage. These hardware agents could inform

us the renewable energy re-source conditions and the health status of batteries. Our

systematic checkup of power supply behaviors also offers a real-time profile of the

system’s energy utilization. This feature makes it possible to pin-point areas of high

energy usage in server farms. To facilitate real-time configuration and diagnosis, we

have designed human-machine interface (HMI) for each power control hub. As shown in

Figure 6-4A, it is a touch screen panel with built-in microprocessors that could display

graphics, animation, and interchange data to and from the PLC. Besides, the HMI

device also serves as an important portal for communication between the PLC and

power management agent.

83

HMI

PLC

(ModBus TCP/Server)(ModBus TCP/Client)

EthernetRS-232/485

Server OS

Workload Workload

Power Management Agent

Workload

Server System (The Load) Power Control Hub

Figure 6-5. Power management agent.

A key feature of the PCH is that it sets up the communication gateway between

power supply layer and server workload layer. This design is partly enlightened by the

energy management strategy in future smart grid, which focuses on intelligent control,

communication, and coordination across electrical loads, power electric interfaces, and

generators. Figure 6-5 illustrates the data communication scheme of Oasis. The power

management agent is a middleware lies between operating system and workloads.

We use Modbus protocol [61] for communication between the power

management agent and the power control hub. It is a widely used serial communication

protocol for industrial electronic devices due to its robustness and simplicity. There are

several other benefits of using Modbus. For example, the Modbus allows us to easily

scale up our deployment of nodes and share information among them. Besides, there is

no need to worry about the transmission failure of the Mod-bus instruction as the lower

layer TCP/IP protocol has provided the redundant checksum. Typical Modbus TCP

communication includes a Modbus server and a Modbus client. In this study, the power

management agent is the Modbus client (master) which initiates the communication

requests periodically through socket to the Modbus server (slave), i.e., the HMI.

6.2.3 Dynamic Energy Source Switching

Oasis nodes are able to dynamically switch between green power supply and

utility power. The PCH offers two power switching modes.

84

Autonomous Mode. This is the default mode. In this mode, Oasis nodes intend

to run autonomously, i.e., switching the load between solar power and utility grid based

on the given solar and utility power budget. The PLC in our power control hub defines

two atomic modules: SupplySense and SupplySwitch, as shown in Figure 6-6. While the

former module focusing on setting parameters that are used for making energy source

management decision, the latter one executes energy source switch.

Coordinated Mode. Oasis also provides servers the option of establishing their

own power supply switching policies. Our system currently allows two user-defined

operations: Utility Power Enforcement and Solar Power Enforcement. Users could

specify their preferred energy source at runtime, by calling the power management

agent. However, the execution of a power supply switching signal depends on battery

status, solar power output, and utility power availability. The user’s switching request

will be ignored if it violates the power budget or causes safety issues.

For every switch operation, the controller will first check the output of power

supplies to ensure that they work normally. The PCH configuration allows an overlap

between solar power and utility power when performing power supply switching. We

disconnect one energy source only when the other has been successfully working for 5

seconds. This mechanism helps to avoid potential switching failure that interrupts server

operation. Our controller also maintains appropriate voltage thresholds to prolong the

lifetime of backup power system. Figure 6-7 shows the measured battery voltage during

power switching. The battery starts to discharge at 12.5V (charging threshold) and stop

discharging when its voltage drops to 11.8V (discharging threshold). These thresholds

prevent batteries from entering deep-discharging or over-charging mode.

85

A

B

Figure 6-6. The control flow of power supply switching. A) The SupplySense module. B) The SuppleSwitch module.

Figure 6-7. Battery charging/discharging trace. The power control hub uses threshold

to avoid over charge/discharge a battery.

6.3 Optimized Oasis Operation (O3)

The flexibility of Oasis power provisioning architecture provides data center

owners plenty of room for improving the efficiency of their servers. In this section we

introduce Ozone, a power management scheme for Optimized Oasis Operation (O3).

Ozone enables Oasis users to take one step further towards the aim of efficient scale-

out data centers. Ozone seeks a balance between power supply management and

server load management. In addition to the basic function of Oasis, Ozone features a

new set of policies that intend to maximally take advantage of multiple energy sources

86

without heavily relying on any single energy sources. In addition, Ozone adaptively

adjusts server load based on power supply status to find the best design tradeoffs.

While many factors may affect Oasis operation, Ozone picks up six

representative factors for making control decisions. Three of them are power-related:

Utility Budget, Solar Budget, and Load Demand; two of them are battery-related:

Discharge Budget and Remaining Capacity; and one Switch parameter that specifies

which energy source is in use as primary power.

6.3.1 Managing Battery Lifetime

Batteries have a lifespan (typically 5~10 years) estimated by their manufactures.

These energy storage devices become no longer suitable for mission-critical data

centers after their designed service life. Aggressively discharging batteries will

significantly degrade their lifetime. However, even if batteries are not frequently used,

they still suffer aging and self-discharging problems.

Ozone uses an Ah-Throughput Model [62] for battery lifespan optimization. This

model assumes that there is a fixed amount of aggregated energy (overall throughput)

that can be cycled through a battery before it requires replacement. The model provides

a reasonable estimation of the battery lifetime and has been used in software developed

by the National Renewable Energy Lab [55].

During runtime, Ozone monitors battery discharging events and calculates

battery throughput (in Ah) based on the Peuket’s equation [63]: 𝐷𝑖𝑠𝑐ℎ𝑎𝑟𝑔𝑒 = 𝑇 ∙ 𝐼𝑎𝑐𝑡𝑢𝑎𝑙 ∙

(𝐼𝑎𝑐𝑡𝑢𝑎𝑙 𝐼𝑛𝑜𝑚𝑖𝑛𝑎𝑙⁄ )𝑝𝑐−1, where 𝐼𝑎𝑐𝑡𝑢𝑎𝑙 is the observed discharging current; 𝐼𝑛𝑜𝑚𝑖𝑛𝑎𝑙 is the

nominal discharging current, 𝑝𝑐 is the Peukets coefficient, and 𝑇 is the duration. Over

time, the aggregated battery throughput is given by 𝐷𝑎𝑔𝑔𝑟 = ∑ 𝐷𝑖𝑠𝑐ℎ𝑎𝑟𝑔𝑒𝑖𝑖 . To avoid

87

battery over-utilization, Ozone carefully sets a soft limit on battery usage at any given

time. At the beginning of each control cycle, Oasis nodes receive a Discharge Budget

which specifies the maximum amount of stored energy uses that will not compromise

battery lifetime. Assuming the overall battery throughput is 𝐷𝑡𝑜𝑡𝑎𝑙, our system calculates

a Discharge Budget as 𝐷𝐵 = 𝐷𝑡𝑜𝑡𝑎𝑙 − 𝐷𝑎𝑔𝑔𝑟. When the Discharge Budget is inadequate,

Ozone will switch server to utility grid or decrease load to avoid over-use of batteries.

6.3.2 Managing Backup Capacity

Distributed battery systems not only provide Oasis basic support for managing

time-varying renewable power, but also serve as uninterruptible power supplies for

scale-out servers. Maintaining necessary backup capacity is critical. It has been shown

that UPS capacity exceeded is one of the top root causes of unplanned outages in data

centers [60]. The backup capacity is the primary factor that determines UPS autonomy

(a.k.a. backup time). It is a measure of the time for which the UPS system will support

the critical load during power failure. In addition to the Discharge Budget, Ozone also

sets a limit on the minimum remaining capacity of batteries. Our system only uses 40%

of the installed capacity for managing renewable power shortfall (referred to as Flexible

Capacity). The remaining 60% battery capacity (i.e., Reserved Capacity) is only used

for emergency handling purpose. When the battery capacity drops below 60%, Ozone

will switch the server from renewable power supply to utility power supply.

6.3.3 Managing Server Performance

When renewable power output drops, there are always two power management

options available: (1) decrease server performance level to lower server power demand,

or (2) continue to operate at full speed and use energy storage to compensate the

power shortfall. To find the best design tradeoff between server performance and

88

system reliability, Ozone cooperatively controls power supply switching and server

speed. The idea is to adaptively select one of the two power management options

based on the observed Discharge Budget and the amount of Flexible Capacity.

When both Discharge Budget and Flexible Capacity are adequate, Ozone gives

high priority to server performance (i.e., run workload at the highest frequency) with the

support of stored green energy. If the system runs out of Discharge Budget but still have

Flexible Capacity, Ozone allows the server to keep using green energy with reduced

processing frequency. If the stored energy drops below 60% of its installed capacity,

Ozone switches the load to the utility power side to avoid low battery backup.


We develop an evaluation framework using our prototype system. The framework

is configured into three layers: the Power Budget Control Layer, the Oasis Operation

Layer, and the Data Collection and Analytic Layer.

In the Power Budget Control Layer, we feed the system with pre-defined power

budget. We use the peak server power demand as default utility power budget of our

system. To ensure fare comparison, we collect real-world solar power traces and use it

as renewable power budget for all the experiments.

Figure 6-8. The power control decision tree of Ozone. ‘Y’: switch the power supply;

‘N’: do not switch; ‘TBD’: server speeds depends on actual power budget.

89

Table 6-1. Computing platform setup for Oasis. Computing Nodes Front-End Server

CPU Intel Core i7-2720QM, 2.2G, 45W AMD Opteron 4256 EE, 2.5G, 32W RAM 8 GB, registered 16 GB, registered Disk Seagate Barracuda 7200RPM, 500 G Seagate Barracuda 7200RPM, 1000 G M/B SuperMicro ITX Socket G2 M/B SuperMicro ATX Socket C32 M/B Network N/A TP-Link TL-SF1024 24-Port 10/100M

In the Oasis Operation Layer, we setup four 1U rack-mounted servers as

computing load, as shown in Table 6-1. They are high-performance lower-power

computing nodes that use Intel Core i7-2720QM 4-core CPU as processing engine.

With the Intel Turbo Boost Technology, these processors support up to 3.3 GHz

operating frequency. We deploy Xen 4.1.2 with Linux kernel 2.6.32.40 on each server

node. Both para-virtualization and hardware virtualization are used to support different

virtual machines with various memory size. Multiple virtual machines are booted to

execute different workloads on each server. We enable the relocation feature of VM in

Xen and live migrate the virtual machine by using command (xm migrate DOMID IP -l).

Xen power management is also enabled to dynamically tune the vCPU frequency. Our

system is configured with the on-demand frequency scaling governor. We set the

minimum frequency as 0.8GHz and the normal frequency as 2.2 GHz.

We choose various data center workloads from Hibench [64] and CloudSuite

[65], Hibench consists of a set of representative Hadoop programs. CloudSuite is a

benchmark suite designed for emerging scale-out applications in datacenters. As shown

in Table 6-2, we select ten workloads from five roughly classified categories.

In the Data Collection and Analytic Layer, we deploy front-end network server to

communicate with the server cluster through a TP-Link 10/100M rack-mounted switch.

The network server uses an AMD low power 8-core CPU with 16GB installed memory.

We write system drivers using Linux socket to enable data communication between

90

front-end server and the Oasis node. We store the collected battery charging and

discharging statistics and the measured server power consumption data in a log file. We

assume the battery has a cycle life of 5000 times and a maximal service life of ten

years. We also use Watts UP Pro power meter [66] for our measurement in energy-

related experiments. This power meter is able to display instantaneous power

consumption with relatively high accuracy (1.5%). It also provides internal memory for

storing up to 120K history power data.

Table 6-2. Evaluated cloud workloads for Oasis.

Abbreviation Workload Category

Sort Sort program on Hadoop Micro Benchmarks WC Word count program on Hadoop Micro Benchmarks Rank Page rank algorithm of Mahout Web Search Nutch Apache Nutch indexing Web Search Bayes Bayesian classification Machine Learning KM K-means clustering Machine Learning Web Web serving Internet Service Media Cline-server media streaming Internet Service YCSB Yahoo! cloud serving benchmark Cloud Application Test Software testing Cloud Application


In this section we evaluate the impact of various power supply switching and

server adaptation schemes. To be more specific, we evaluate three kinds of Oasis

power management schemes: Oasis-B, Oasis-L, and Ozone. Oasis-B is a battery

oriented design which aggressively uses battery to store solar energy for powering

server clusters. The Oasis-L mainly focuses on the load power scaling capability of

servers. It first uses frequency scaling to decrease load power, then leverage battery to

compensate the power shortfall. In contrast to Oasis-B and Oasis-L, Ozone focuses on

balanced usage of server load adaptation and the stored renewable energy. In the

following sub-sections, we first investigate the performance of various control schemes.

91

We then discuss their impacts on battery lifetime and emergency handling capabilities.

Afterwards, we evaluate the energy usage profile of different schemes. Finally, we

discuss the cost issue of Oasis.

6.5.1 Load Performance and Energy Efficiency

Job turn-around time is a crucial metrics for emerging data-analytic workload in

scale-out data centers. Figure 6-9 shows the in-crease in job turn-around time under

both high-variability and low-variability solar power generation scenarios. In Figure 6-

9A, the mean turn-around time of Oasis-B, Oasis-L and Ozone is increased by 0.5%,

5.4%, 0.6%, respectively. In Figure 6-9B, the turn-around time of Oasis-B, Oasis-L and

Ozone is increased by 0.4%, 4.7%, 1%, respectively. As can be seen, Oasis-B shows

the best performance. This is because Oasis-B trades off battery lifetime for server

performance. In contrast, Oasis-L shows much higher performance degradation as it

frequently lowers the processing frequency to match the inadequate renewable power

budget. In Figures 6-9A and 6-9B, the results of Ozone are very close to Oasis-B. On

average, Ozone yields less than 1% performance degradation compared to Oasis-B,

which heavily uses battery to provide power shortfall.

Data centers with heavy data-processing computing task often consume large

amount of energy. Leveraging green energy to provide additional power could save

utility power bills considerably and lower the negative environmental impact of carbon-

constrained data centers. In Figure 6-10 we evaluate data center green energy

utilization as the ratio of renewable energy usage to overall IT energy consumption.

While Ozone yields impressive system performance, battery lifetime, and battery

backup capacity, it shows relatively lower green energy utilization. Compared to Oasis-

B, Ozone yields 11% less renewable power rate when renewable power varies

92

significantly and 8% less renewable power rate when renewable power generation is

high. The reason Oasis-B shows high green energy usage is that it heavily uses battery

to harvest renewable energy. In contrast, Oasis-L aggressively scales load power

demand to match the renew-able power generation.

A B

Figure 6-9. The execution latency due to server performance scaling. A) High renewable power variability. B) Low renewable power variability.

A B

Figure 6-10. The ratio of green energy usage to overall power consumption. A) High renewable power variability. B) Low renewable power variability.

6.5.2 Battery Service Life and Backup Capacity

In Figure 6-11 we estimate the battery service life based on detailed battery

profiling information. When renewable power supply varies significantly, the operation of

server nodes typically requires substantial support from the energy storage. As a result,

the predicted lifetime is much shorter than the designed service life, as shown in Figure

6-11A. On average, the battery lifetime of Oasis-B, Oasis-L, and Ozone is 3.9 years, 6.2

years, and 6.0 years, respectively. Due to the over-use of battery systems, the battery

lifetime of Oasis-B is only 63% of Oasis-L. In contrast, the battery lifetime of Ozone is

97%. Compared to Oasis-B, Ozone increase the battery lifetime by more than 50%.

93

When renewable power supply becomes adequate, as shown in Figure 6-11B,

the battery service of all the three power management schemes increases significantly.

This happens because batteries are not discharged for most of time. However,

commercial batteries cannot last for 20 years even if they are under-utilized. Many other

issues such as aging and leakage will become the dominant factors for lifetime

estimation if the batteries are used for an extended duration. In fact, batteries are used

more frequently since renewable power system does not maintain peak output.

A B

Figure 6-11. The estimated battery lifetime calculated from battery usage profile. A) High renewable power variability. B) Low renewable power variability.

Figure 6-12 illustrates the problem of uneven battery usage. Note that when

renewable output is constantly high, the distributed battery system is rarely used. In

contrast, when renewable power fluctuates severely, batteries start to show heavy

charging and discharging activities. Therefore, one can actually save plenty of

Discharging Budget in scenarios like Figure 6-12B. This amount of Discharging Budget

saving can be further used in scenarios like Figure 6-12A to provide necessary load

power support. In fact, the Oasis user can opportunistically leverage stored energy

(even if current Discharging Budget is zero) to boost system performance without

significantly affecting the battery service life.

94

A

B

Figure 6-12. Battery discharging profile and power demand traces. The workload used is Nutch page indexing. A) High renewable power variability. B) Low renewable power variability.

When data center servers scale out, their backup up power should be increased

accordingly. While instantaneous workload performance boost is important, maintaining

necessary backup capacity is even more important. A low battery backup capacity can

pose significant risk since the backup generator may not be ready to pick up the load. In

Figure 6-13 we show the average UPS capacity during runtime for different power

management mechanisms. Ozone could maintain around 73% backup capacity when

renewable power variability is high and about 98% backup capacity when renewable

power variability is low. Oasis suffers increased numbers of charging and discharging

cycles in circumstances that renewable power varies significantly. Therefore, the

backup capacity is low for all the three power management schemes in Figure 6-13A.

Without setting a limit on the battery usage and the minimum battery stored capacity,

the battery backup time can drops by 75% (i.e., Oasis-B).

95

A B

Figure 6-13. The observed average battery backup capacity in our UPS systems. A) High renewable power variability. B) Low renewable power variability.

A B Figure 6-14. A cost breakdown of Oasis. A) Summary of our prototype. B) Estimation

for a 5KW standard server rack.

6.6 Cost Analysis of Oasis

This section discusses the system cost of Oasis. Figure 6-14 presents two pie

charts that show the cost breakdown. In Figure 6-14A we evaluate our Oasis node

prototype. As can be seen, solar panel is the most expensive component in our system,

followed by the PLC module, the power inverter, and battery. In Figure 6-14B we

estimate the cost of Oasis design for a 40U standard server with moderate density

(<10KW). We assume the rack is populated with 20 SuperMicro 1017R Xeon 1U

servers. While PLC and HMI are both major cost components in our prototype system,

they do not need to scale up when more servers are added. In contrast, solar panels,

batteries, and inverters need to increase their capacities.

96

Centralized renewable power integration has relatively low initial cost due to the

scale effect. Recent report estimates that small-scale PV panel (around 5KW, for

residential use) has an installed price of $5.9/W, while large-scale PV panel (several

hundred KW) has a lower price of around $4.74/W [68]. In addition, solar power inverter

accounts for about 10%~20% of the initial system cost [69]. Central inverters in several

MWs level are often cheaper compared to micro inverters (typically < 10KW) used in

Oasis. The former costs around $0.18/W, while the later costs around $0.5/W [69].

Oasis allows users to gradually increase the installed renewable power capacity,

thereby eliminating over-provisioning. Oasis users can also take advantage of the ever-

decreasing component cost to further lower costs. It has been shown that the installed

prices of U.S. residential and commercial PV systems declined 5%~7% per year, on

average, from 1998~2011, and by 11%~14% from 2010~2011, depending on system

size [69]. The cost of micro-inverter also de-creases by 5% yearly [69].

Figure 6-15 illustrates how Oasis design helps to improve the overall cost-

effectiveness of renewable energy powered scale-out data centers. We assume that

Oasis users evenly increase their deployment of Oasis node with a ten-year scale-out

plan. (e.g., equip 10% of the data center servers with solar power system every year).

For the cost of solar power system, we use a conservative decline rate of 6% per year,

and an optimistic decline rate of 12% per year. We calculate electricity cost savings of

renewable energy powered systems based on real historical solar power traces. We use

hourly solar irradiance measurement data (January 2003 ~ December 2012, 24 hours a

day) provided by the NREL Solar Radiation Research Laboratory [70]. We assume the

utility power is $0.1/kWh and data centers sell excess renewable power to the utility.

97

Figure 6-15. Cost trends of deploying Oasis for scaling out.

Figure 6-15 shows the cost overhead (total additional cost due to renewable

power integration) for different design scenarios. The results are normalized to the one-

time expenditure of data center-level renewable power integration. For conventional

green data center that uses centralized renewable power integration, one can expect to

get 17% investment return (due to the electricity cost savings) after ten years. However,

this estimation is optimistic as the utility grid typically uses negotiated renewable power

feed-in tariff that has a lower purchase price. In addition, for safety reasons, there is

also a limit on the maximum amount of renewable power that can be synchronized with

the utility power. The overall cost of Oasis is close to the conventional centralized

design if solar power cost decreases by a conservative rate of 6% per year. If solar

power cost declines faster, namely, 12% per year, Oasis could result in 25% less

overhead cost, compared to a centralized design.

Note that there is no one-for-all universal design. Green data-centers with

centralized solar power system may be a better choice if one has a firm goal of

renewable power integration capacity and a confidence in future load power demand.

However, if one wants to enhance data center operational resiliency and seeks to

gradually incorporate renewable energy to avoid over-committing capital, Oasis

provides an attractive alternative.

98

CHAPTER 7 SUSTAINABLE COMPUTING IN THE SMART GRID AND BIG DATA ERA

Looking ahead, the attributes of workload application and energy source will

undoubtedly influence the way we compute. In the future when computer architecture

meets big data workload and smart grid techniques, it requires rethinking of

conventional computing system design approaches.

First, as we move toward a smarter grid, data centers are expected to be

powered by hybrid renewable energy systems that combine multiple power generation

mechanisms [71]. For example, it may include intermittent power supplies such as wind

turbines and solar panels, controllable power supplies such as fuel cells and gas

turbines, and various backup power system such as batteries and super-capacitors.

With an integrated mix of complementary power provisioning methods, one can

overcome the limitations of each single type of power supply, thereby achieving better

energy reliability and efficiency. However, a common limitation of prior proposals is that

they mainly focus on certain specific type of green power supplies. As a result, they can

hardly gain the maximum benefits from hybrid renewable energy systems, and

consequently yield sub-optimal design tradeoffs.

Second, although much of the computing resources today are hosted in data

centers, a tremendous amount of datasets are generated from distributed machines,

monitors, instruments, and various types of sensor networks. For example, a large

network that consists of ten thousands tiny nodes can easily generate over 5 terabytes

(TB) of data within a week [72]. In addition, today’s fast-growing scientific datasets (e.g.,

climate data and genome data) are typically distributed among many stations and

research institutions around the world. Such wide-area collaboration on location-

99

dependent datasets normally requires a routine data sharing of tens of petabytes (PB)

every year [73]. Consequently, the challenges of analyzing these geographically

dispersed data sets are increasing due to significant data storage overhead, time-

consuming data aggregation and the escalating energy needs. Rather than constantly

move a tremendous amount of raw data to remote warehouse-scale systems, it would

be beneficial to bring computation to where data is located.

This chapter contains two case studies that explore the opportunities to benefit

from various green computing systems in the incoming smart grid and big data era. We

first propose GreenWorks, a power management framework for green high-

performance computing data centers powered by renewable energy mix. Afterwards, we

present InSURE, an intelligent design of stand-alone green server clusters for locally

processing distributed big datasets generated in the field.

7.1 Green Data Centers Powered by Hybrid Renewable Energy Systems

In the smart grid era, data centers must increase their awareness of the

attributes of power supplies to achieve the best design trade-offs. There are three types

of renewable power supplies that we can leverage to power a data center: Some green

power supplies, such as solar panels and wind turbines, are affected by the availability

of ambient natural resources. They are referred to as intermittent power supply. Several

emerging power supplies – including fuel cells, bio-fuel based generators – can offer

controllable output by burning various green fuels. We refer to them as baseload power

supply since they can be used to provide stable renewable power to meet the basic

data center power demand (e.g., idle power). In addition, energy storage devices such

as batteries and super-capacitors are also critical components that provide backup

power supply. They can be used to temporarily store green energy.

100

Figure 7-1. High-level system architecture for GreenWorks.

Data centers in the smart grid era are expected to be powered by hybrid

renewable energy systems that combine all the three types of power supplies. They are

typically implemented as small, modular electric generators (called micro-sources) near

the point of use. This section explores diversified multi-source power provisioning for

green high-performance data-centers. The intention of this work is to provide an initial

power management framework for data centers powered by renewable energy mix.

7.1.1 Hierarchical Power Management Framework

GreenWorks is a hierarchical power management scheme that is tailored to the

specific timing and utilization requirements of different energy sources. It provides

coordinated power management across intermittent renewable power supplies,

controllable base-load generators, onsite batteries, and data center servers.

As shown in Figure 7-1, GreenWorks comprises two key elements: the green

workers and the green manager. The former are platform-specific power management

modules for managing different types of micro-sources and the later coordinates these

modules. In this study we define three types of green workers: baseload laborer (B),

energy keeper (E), and the load broker (L). We adopt typical micro-grid power

distribution scheme for managing various renewable energy resources.

101

Figure 7-2. The GreenWorks power management hierarchy.

Although the hybrid renewable energy systems are often centrally installed at the

data center facility level, improving the overall efficiency requires a multi-level,

cooperative power management strategy, as shown in Figure 7-2.

Baseload Laborer. In the top tier of the hierarchy is the baseload laborer. It

controls the output of controllable generators. We put the load laborer at the data center

facility level since it is where the baseload generator is integrated.

Server Load Broker. The load broker manages the intermittent renewable

power supply at the cluster/PDU level. At this level, dynamic voltage and frequency

scaling (DVFS) shows impressive peak power management capabilities [43] and could

be leveraged to manage the supply-load power mismatch.

Battery Energy Keeper. The energy keeper resides in the third tier of the

hierarchy. This allows us to provide backup power directly to server racks if local

demand surge happens or power budget drops. It also monitors the capacity utilization

and the health status of the battery packs for optimization purpose.

7.1.2 Multi-Source Driven Power Management

The main advantage of our multi-level power management framework is that it

facilitates cross-source power optimization. GreenWorks features a novel three-stage

coordination scheme to balance the usage of different types of energy sources [74].

102

Stage I: Adequate Power Supply Budget. When allocating the additional

renewable power budget across server nodes, the green manager will always give

priority to jobs that have higher job execution time increase (ETI). Specifically, our green

manager uses a job acceleration scheme which opportunistically boosts the processing

speed/frequency (i.e., over-clocking) to take advantage the additional renewable power

budget [74]. It allows a processor to enter a performance state higher than the specified

frequency if there is enough thermal/power headroom and if it is enabled by the system.

Stage II: Moderate Power Supply Drop. Our system enters Stage-II when it

senses inadequate power supply. Unlike prior designs which heavily rely on either load

shedding or backup power, we use a balanced power management. GreenWorks

choose to decrease server processing speed or request stored energy from the energy

keeper, depending on whichever yields the best design tradeoffs [74].

Stage III: Significant Power Supply Drop. The Stage-III is an emergency state

since in this scenario the green manager might put the load into minimum power state

or use reserved UPS capacity to avoid server shutdown. GreenWorks uses a deadline-

aware load shedding to achieve a better tradeoff between UPS energy and job

execution time increase (ETI). The green manager first checks the current ETI values of

all the jobs for load shedding opportunities. It calculates a Time Budget which evaluates

if a job could meet its deadline in the future with frequency boosting techniques [74]. If

the given job has enough Time Budget, our controller will incrementally reduce its CPU

frequency until it reaches its lowest speed. It will put server nodes into low power states

in a round-robin fashion if the demand-supply discrepancy still exists. Finally, we

release the reserved UPS energy if necessary.

103

Figure 7-3. Details of our three-layer simulation platform.

7.1.3 Verification Platform and Results

Our simulation is configured into three layers for modeling the entire system from

the job dispatching behavior to the renewable energy mixes. It uses discrete-event

simulation to process a chronological sequence of job submissions. It also simulates the

power system on per-second time scale which is in tune with our workload scheduler.

We compare GreenWorks to two baselines: Shedding and Boosting. Shedding

always defers workload when renewable energy drops [12] [21]; Boosting emphasis the

role of energy storage devices [15] [16]. Both baselines adjust baseload output at each

end of the control period. The only difference between the two is that Shedding gives

priority to load scaling, while Boosting gives priority to UPS stored energy.

Execution Time. Figure 7-4 shows the average job execution time increase. On

average, the job execution time increase of Shedding, Boosting and GreenWorks are

5.4%, 2.1%, and 2.4%, respectively. Compared to Shedding, Boosting shows less

execution time increase since it trades off UPS capacity for performance. Figure 7-5

shows the maximum increase of job turnaround time which is calculated as the average

execution time increase of the 5% worst cases. The worst-case result of Shedding is

28%. Because GreenWorks apply a performance limit when shedding load, it reduces

the maximum job execution time increase by 33%, compared to Boosting (18%).

104

A B

Figure 7-4. The impact of GreenWorks on job execution time. A) Average ETI across all the job requests. B) Maximum ETI for the worst 5% jobs.

Battery Lifetime. In Figure 7-5, GreenWorks shows a near-threshold battery life

(8.3 years). It means our power management can maximally leverage batteries without

degrading their life significantly. In contrast, Boosting shows a mean lifetime of 6.7

years; and Shedding shows a mean lifetime of 19.7 years. Typically, the battery lifetime

is not likely to exceed 10 years [75]. The reason Shedding over-estimates battery life is

that the system underutilizes batteries. Since batteries may fail due to aging problems

and self-discharging issues, it is better to fully utilize it.

UPS Backup Time. GreenWorks can further optimize the mean UPS autonomy

time (a.k.a. backup time). It is a measure of the time for which the UPS system will

support the critical load during an unexpected power failure. Figure 7-6 shows the mean

normalized UPS autonomy time throughout the operation duration. On average, the

mean autonomy time is: Shedding (88%), Boosting (70%), and GreenWorks (78%). In

Figure 7-7 we plot the cumulative distribution function (CDF) for the normalized UPS

autonomy time. The CDF curve of GreenWorks lays nicely between our two baselines.

GreenWorks could ensure rated backup time (the discharge time of a fully charged

UPS) for 20% of the time. Shedding maintains its rated backup time for 50% of the time,

while the number for Boosting is only 10% due to aggressive battery usage.

105

Figure 7-5. Battery lifetime estimation

based on its usage log. Figure 7-6. Average UPS backup time

(normalized to the rated value).

Figure 7-7. Cumulative distribution function (CDF) for UPS autonomy time.

7.2 Towards Sustainable Power Provisioning for In-situ Server Systems

Although much of the computing resources today are hosted in data centers, a

tremendous amount of datasets are generated from distributed machines, monitors,

instruments, and various types of sensor networks.

Rather than constantly move a huge amount of data to a central data warehouse

for aggregation and processing, we instead explore a fundamentally different approach:

tapping into in-situ server systems. The idea is to bring servers to where data is located

to pre-process part, if not all, of the datasets. For instance, these servers can be used to

eliminate duplicate copies, compress logs, normalize data formats or classify data

types. The concept of in-situ data processing has been used in the high-performance

computing (HPC) community to solve the I/O overhead for compute-/memory- intensive

workloads [76] [77]. In this work, we repurpose this concept to design onsite server

systems that can accelerate or facilitate pre-processing of distributed big datasets.

106

Our interest in in-situ server systems also arises out of the fact that modern data

centers are power-constrained, particularly when they employ power over-subscription

techniques to reduce cost. In the past five years, 70% companies have to build new

data centers or significantly renovate existing facilities to handle the ever-growing traffic

[56]. Meanwhile, recent studies are forced to aggressively discharge backup batteries to

provision more server under existing power budget [17] [78]. As data continues to flood

into data centers, it is not unusual that the utility power feeds are at full capacity and

data centers do not have enough power to accommodate their growth [79] [80].

There are also significant challenges associated with power provisioning for in-

situ servers. Different from sensor nodes, computer servers require much higher power

budget and cannot solely rely on batteries. In remote areas or rural locations that are

environmentally sensitive, the construction and operation of transmission lines could be

prohibitive [81]. Even if utility power access is not an issue, conventional server power

provisioning methods often violate local environmental quality regulations [82].

We present InSURE: in-situ server systems using renewable energy. Such in-situ

server system is well complementary to cloud computing in the incoming big data era. It

can greatly reduce data processing overhead in public cloud and private network, with

zero carbon emission. As Figure 7-8 shows, we explore the opportunity to benefit from

data pre-processing using a group of inexpensive servers that are placed near the data.

Specifically, we focus on standalone systems powered by local renewable energy

sources. It has been shown that about 85% data processing tasks can be deferred by a

day; 40% task can be deferred by a month [83]. Therefore, even if the renewable power

is intermittent and time-varying, we can still leverage it for data pre-processing.

107

Figure 7-8. In-situ server system as an ancillary to future cloud.

7.2.1 Standalone Green Server Clusters

Since batteries are widely used as energy buffer that links renewable power

supply with load [27], their efficient use is the crux of high data processing throughput.

Without appropriate control, battery charging process (using green energy) could incur

unnecessary slowdown and, consequently, cause data processing delay. Besides, peak

load could cause significant battery voltage drop even if there is still enough renewable

energy stored. This can disrupt server operation and greatly degrade productivity. To

this end, we have designed a cross-layer power management scheme that is tailored to

the specific behavior of standalone in-situ systems. It highlights two techniques:

Reconfigurable distributed energy storage: We synergistically integrates

existing power switch network with emerging distributed energy storage architecture. It

allows us to dynamically adjust the size of energy buffer for better design trade-offs.

Joint spatial-temporal power management: Our spatial control could

intelligently balance the longevity of battery systems and ensure the optimal charging

effectiveness under varying renewable power generation conditions. Our temporal

control scheme cooperatively manages server power demand and battery discharge

current to improve battery discharge efficiency and reduce load shedding frequency.

108

Figure 7-9. In-situ server cluster with reconfigurable battery array. Although we focus on stand-alone system, our design also supports a secondary power (e.g., local micro-grid, if available).

Our reconfigurable energy buffer is shown in Figure 7-9, three power switches

(P1, P2, and P3) are used for managing the battery cabinets to provide different voltage

outputs and ampere-hour ratings to the load. For example, if P1 and P3 are closed

while P2 is open, batteries are connected in parallel. If switches P1 and P3 are open

while P2 is closed, batteries are connected in serial. A set of sensors S1 is used to

monitor the runtime operation condition of each battery cabinet. All switches and

sensors are managed by a PLC controller.

The energy buffer of InSURE supports a variety of operating modes, as shown in

Figure 7-10. Based on the states of energy storage, we categorize them into four types:

Offline, Charging, Standby, and Discharging. In the Offline mode, batteries are

disconnected from the server load for system protection purposes. In the Charging

mode, onsite renewable power, if available, is used for charging batteries with the best

achievable efficiency. Our system put batteries online when they are charged to a pre-

determined capacity (i.e., 90%). In the Standby and Discharging modes, our system

mainly relies on renewable energy (either directly generated from the onsite green

generator or previously stored in batteries) to power green server clusters.

109

Figure 7-10. The energy flow scenarios for InSURE.

Figure 7-11. Operating modes transition of InSURE energy buffer.

The transition between various operating modes is shown in Figure 7-11. The

energy buffer is able to adjust its operating mode based on the stored renewable energy

budget, server power demand, and battery health conditions.

7.2.2 System Implementation and Verification

We have built a full-system prototype of InSURE as our verification platform. It is

a three-tier hierarchical system, as illustrated in Figure 7-12. It is composed of three

primary modules built from scratch: (1) a reconfigurable battery array, (2) a real-time

monitoring module, and (3) a supply-load power management/coordination node.

Reconfigurable Battery Array. Our customized battery system is composed of

six 12 V lead-acid batteries and a relay network. We use six 10A/24V DC relays as the

power switches for reconfiguring the battery array. Each battery is managed

independently using a pair of two relays (charging switch + discharging switch). These

relays provide satisfactory mechanical life (10M operations). We also use a Siemens

S7-200 CPU224 PLC module as the controller for our reconfigurable battery system. Its

digital output is connected to the relay network and can energize or de-energize the coil

of relay to perform battery switching.

110

Figure 7-12. The structure of our verification platform.

Real-Time System Monitoring. The monitoring module notifies our system

whenever the battery configuration profile changes. Each battery in the system is

equipped with a voltage transducer and a current transducer. Their outputs are further

sampled by two PLC extension modules. All the analog readings will be stored in

several specific registers in the PLC. We use an HMI panel to link battery system and

the coordination server node via Ethernet using the Modbus TCP protocol [61].

Power and Load Coordination. The top hierarchy of our design consists of a

power coordination module. This module is implemented on a separate server node.

We have designed a power supply switching API and a server control API. The former

API provides necessary communication interface that allows the system to select its

power source during runtime, if it is necessary. The latter API is used to manage load

power through frequency scaling, power state control, and virtual machine migration

7.2.3 Analyzing In-Situ Green Computing

To understand the power behavior of inSURE when handling bulk data using

self-generated solar energy. We investigate a typical system power behavior trace that

are collected from our prototype, as shown in Figure 7-13. We have marked five typical

regions (Region-A ~ Region-D) that frequently appears in our everyday operation.

111

Figure 7-13. Solar power budget trace and battery voltage trace Top figure shows a

full-day operation and the bottom provides a zoomed view.

Our standalone in-situ system starts to charge a selected subset of battery in the

morning, as shown in Region-A. During this period, the battery terminal voltage

gradually increases until it researches a preset value.

Figure 7-13 also shows a zoomed view of a fraction of the system trace. As we

can see, Region-B incurs a great deal of solar usage surges/peaks. This is because our

system uses a Perturb and Observe (P&O) maximum renewable power tracking

mechanism [84]. In Region-B, each green peak is a result of our peak power tracking

control. It will not cause overloading since the battery can temporarily provide the power

short fall. Region-C shows our temporal control. In this region the solar power demand

of our in-situ system is significantly larger than the maximum solar budget, and battery

is in discharging mode. If the discharge current is too high, our system will trigger power

capping. As a result, the power demand drops at the end of this region.

In the Region-D of Figure 7-13, we are in the most desired region. Renewable

power is adequate and our in-situ server system can get the most benefit from

renewable energy powered data-preprocessing. In contrast, Region-E is an unfavorable

regions as severely frequenting solar power budget can cause many load power peaks.

We will adjust our peak power tracking mechanism to address this issue.

checkpoint and suspend servers

initial battery charging

supply-demand matching

severely fluctuating solar power

6:54 AM 7:59 PM

9:28 AM 11:18 AM

A

D

E

Cusage

usage

Bpower tracking

b

b v

v

112

Table 7-1. Key variables collected from the inSURE system log. We show three typical renewable power generation scenarios.

kwh Usage Total(Useful)

Power Ctrl. Freq.

Server on/off Cycles

VM Ctrl. Freq.

Battery Volt. σ

Sunny & Aggr. Ctrl. 6.7 (6.4) 12 8 8 1.05 Sunny & Non-Aggr.* 6.5 (5.9) 47 16 42 0.93 Cloudy & Aggr. Ctrl. 5.5 (5.2) 10 8 15 1.03 Cloudy & Non-Aggr. 5.0(4.2) 51 20 51 0.92 Rainy & Aggr. Ctrl. 2.8(2.5) 10 8 11 1.04 Rainy & Non-Aggr. 2.6(2.1) 33 15 38 0.93

*A non-aggressive battery management has a limit on battery discharge throughput.

We further investigate the system behaviors of InSURE by looking at the detailed

system logs. As we can see from Table 7-1, the effective energy consumption usually

does not equal to the overall load energy consumption. This is because VM check-

pointing operations and the on/off cycles of server systems also consume energy. On

average the effective energy utilization is about 89% of the total load energy

consumption. The non-aggressive server power management yields about 86%

effective energy consumption, and about 94% total load energy consumption, both

compared to the aggressive schemes. This is because the nonaggressive server power

management trades off energy utilization for battery life-time.

In Figure 7-14 we further evaluate the overall impact of our spatio-temporal

power management on in-situ server systems. We compare our system with

conventional design that cyclically charge and discharge battery (i.e., aggressive battery

management). As shown in the figure, our spatio-temporal control has bigger impact on

datacenter availability and battery capacity (i.e., SOC). The improvement over a con-

design mainly comes from two sources. First, the temporal control prevents battery from

sudden, fast capacity drop (voltage drop). Second, by dynamically concentrates the

limited renewable power generation on a smaller set of batteries to achieve fast battery

charging, our spatial control can greatly accelerate the charging speed of batteries.

113

A B C

Figure 7-14. Expected optimization results from our spatio-temporal management. A) Service availability. B) Battery state of charge. C) Battery lifetime.

Another observation is that the power budget level has a significant influence on

the optimization effectiveness of our technique. Our system shows about 41% service

availability improvement compared to conventional design under low solar generation,

while the improvement can reach 51% under high solar generation. If we look at each

individual workload, we can find that our system always shows larger availability

improvement under low solar generation level. This is mainly because the benefit of our

spatial power management often increases when renewable power is inadequate.

We also observe a battery lifetime improvement of approximately 21~24% across

our workload mix. The optimization of battery service life is mainly a result from battery

discharge capping and battery discharge balancing across the battery cabinet. The

significant lifetime improvement also indicates that with our design, a larger amount of

solar energy are directly used by server load (i.e., bypasses the battery), thereby

reducing the electrical ware-out on batteries.

114

CHAPTER 8 CONCLUSIONS

This dissertation envisions the long-term competitiveness of designing green

energy aware computing systems. We demonstrate that a cross-layer and cross-

component power management scheme that span workload applications, computer

hardware, and energy sources is the crux of achieving high efficiency and high

performance in such design. The proposed techniques can greatly contribute to

enabling computing systems to stay on track with their historic performance scaling

trend, and thus, paves the way for the ultimate solution to the energy and environmental

problems of modern IT infrastructures.

This work is only a glimpse into the possibilities opened by emerging computing

systems and energy systems. The expected growth in data center heterogeneity and

power provisioning diversity will unavoidably increase the design complexity of green

computing platforms. Future work on a holistic load/power management scheme within

and across green data centers will allow us to maximize the benefits we can get from

the synergy of computing facility and energy systems.

115

LIST OF REFERENCES

[1] C. Belady, (March 2011). Projecting Annual New Data center Construction Market Size, accessed on May 2014. Global Foundation Services, White Paper, [Online]. Available: http://www.business-sweden.se/PageFiles/9118/Projecting_Annual_New_Data_Center_Construction_Final.pdf

[2] J. Koomey, (August 2011). Growth in Data Center Electricity Use 2005 to 2010, accessed on May 2014. Analytics Press, White Paper, [Online]. Available: http://www.analyticspress.com/datacenters.html

[3] G. Boccaletti, M. Löffler, and J. Oppenheim, (October 2008), How IT can cut carbon emissions, accessed on May 2014. [Online]. Available: http://storage.globalcitizen.net/data/topic/knowledge/uploads/2011110295634705.pdf

[4] Greenpeace International, accessed on May 2014. [Online]. Available: http://www.greenpeace.org/usa/en/

[5] IEEE Guide for Array and Battery Sizing in Stand-Alone Photovoltaic (PV) Systems, IEEE Standard 1562™-2007.

[6] S. Kaxiras and M. Martonosi, Computer Architecture Techniques for Power-Efficiency, Morgan & Claypool, 2008, pp. 1-185.

[7] R. Teodorescu and J. Torrellas, “Variation-aware application scheduling and power management for chip multiprocessors,” in Proc. 35th Int. Symp. on Computer Architecture (ISCA’08), 2008, pp. 363-374.

[8] C. Isci, A. Buyuktosunoglu, C.-Y. Chen, P. Bose and M. Martonosi, “Analysis of efficient multi-core global power management policies: maximizing performance for a given power budget,” in Proc. 39th Int. Symp. on Microarchitecture (MICRO’06), 2006, pp. 347-358.

[9] J. Lee and N. Kim, “Optimizing throughput of power- and thermal-constrained multicore processors using DVFS and per-core power-gating,” in Proc. 46th Design Automation Conference (DAC’09), 2009, pp. 47-50.

[10] N. Sharma, S. Barker, D. Irwin, and P. Shenoy, “Blink: Managing server clusters on intermittent power,” in Proc. Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS’11), 2011, pp. 185-198

[11] N. Deng, C. Stewart, and J. Li. “Concentrating renewable energy in grid-tied data centers,” in Proc. Int. Symp. on Sustainable Systems and Technology (ISSST’11), 2011, pp. 1-6.

http://www.business-sweden.se/PageFiles/9118/Projecting_Annual_New_Data_Center_Construction_Final.pdf



http://www.analyticspress.com/datacenters.html

http://storage.globalcitizen.net/data/topic/knowledge/uploads/2011110295634705.pdf

http://storage.globalcitizen.net/data/topic/knowledge/uploads/2011110295634705.pdf

http://www.greenpeace.org/usa/en/

116

[12] I. Goiri, K. Le, T. Nguyen, J. Guitart, J. Torres, and R. Bianchini, “GreenHadoop: Leveraging green energy in data-processing frameworks,” in Proc. 7th European Conference on Computer Systems (EuroSys), 2012, pp. 57-70.

[13] Y. Zhang, Y. Wang, and X. Wang. “GreenWare: Greening cloud-scale data centers to maximize the use of renewable energy,” in Proc. 12th International Conference on Middleware (Middleware’11), 2011, pp 143-164.

[14] I. Goiri, W. Katsak, K. Le, T. Nguyen, and R. Bianchini. Parasol and GreenSwitch: Managing data centers powered by renewable energy, in Proc. Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS’13), 2013, pp. 51-64.

[15] S. Govindan, A. Sivasubramaniam, and B. Urgaonkar. Benefits and Limitations of Tapping into Stored Energy for Data centers, in Proc. 38th Int. Symp. on Computer Architecture (ISCA’11), 2011, pp 341-352.

[16] S. Govindan, D. Wang, A. Sivasubramaniam, and B. Urgaonkar, “Leveraging stored energy for handling power emergencies in aggressively provisioned data centers, in Proc. Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS’12), 2012 , pp. 75-86.

[17] V. Kontorinis, L. Zhang, B. Aksanli, J. Sampson, H. Homayoun, E. Pettis, T. Rosing and D. Tullsen, “Managing distributed UPS energy for effective power capping in data centers,” in Proc. 39th Int. Symp. on Computer Architecture (ISCA’12), 2012, pp. 499-499.

[18] Z. Liu, M. Lin, A. Wierman, S. Low, and L. Andrew. “Greening geographical load balancing,” in Proc. Int. Conf. on Modeling and Measurement of Computer Systems (SIGMETRICS’11), 2011, pp. 233-244.

[19] N. Deng, C. Stewart, D. Gmach, M. Arlitt, and J. Kelley, “Adaptive green hosting,” in Proc. Int. Conf. on Autonomic Computing (ICAC’12), 2012, pp. 135-144.

[20] C. Ren, D. Wang, B. Urgaonkar, and A. Sivasubramaniam, “Carbon-aware energy capacity planning for data centers,” in Proc. Int. Symp. on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS’12), 2012, pp. 391-400.

[21] P. Banerjee, C. Patel, C. Bash, A. Shah, and M. Arlitt, “Towards a net-zero data center,” ACM J. on Emerging Technologies in Computing Systems, vol. 8, issue 4, October 2012, Article No.8

[22] C. Li, W. Zhang, C. Cho, and T. Li. “SolarCore: Solar energy driven multi-core architecture power management,” in Proc. 17th Int. Symp. on High-Performance Computer Architecture (HPCA’11), 2011, pp. 205-216.

117

[23] C. Li, A. Qouneh, and T. Li, “iSwitch: Coordinating and optimizing renewable energy powered server clusters,” in Proc. 39th Int. Symp. on Computer Architecture (ISCA’12), 2012, pp. 512-523.

[24] C. Li, R. Zhou, and T. Li, “Enabling distributed generation powered sustainable high-performance data center,” in Proc. 19th Int. Symp. on High-Performance Computer Architecture (HPCA’13), 2013.

[25] C. Li, Y. Hu, R. Zhou, M. Liu, L. Liu, J. Yuan, and T. Li, “Enabling datacenter servers to scale out economically and sustainably,” in Proc. 46th Int. Symp. on Microarchitecture (MICRO’13), 2013.

[26] A. Luque and S. Hegedus. Handbook of Photovoltaic Science and Engineering, John Wiley & Sons, 2003.

[27] D. Linden and T. Reddy. Handbook of Batteries, 3rd Edition, McGraw Hill, 2002

[28] A. Jossen, J. Garche and D.U. Sauer, “Operation conditions of batteries in PV applications,” Solar Energy, Vol. 76, Issue 6, 2004, pp. 759-769.

[29] W. Kim, M.S. Gupta, G.Y. Wei and D. Brooks, “System level analysis of fast, per-core DVFS using on-chip switching regulators,” in Proc. 14th Int. Symp. on High-Performance Computer Architecture (HPCA’08), 2008.

[30] VRM Design Guideline, (April 2002), accessed on May 2014. White paper, Intel Corporation [Online]. Available: http://www.intel.com/design/pentium4/guides/249205.htm

[31] Measurement and Instrumentation Data Center, accessed on May 2014. [Online]. Available: http://www.nrel.gov/midc/

[32] D. Brooks, V. Tiwari and M. Martonosi, “Wattch: A framework for architectural-level power analysis and optimizations,” in Proc. 27th Int. Symp. on Computer Architecture (ISCA’00), 2000, pp. 83-83.

[33] N. Muralimanohar, R. Balasubramonian and N.P. Jouppi, (April 2009). CACTI 6.0: A tool to model large caches, accessed on May 2014. Technical Report HPL-2009-85, HP Labs, [Online]. Available: http://www.hpl.hp.com/techreports/2009/HPL-2009-85.html

[34] Enhanced Intel SpeedStep technology for the Intel Pentium M processor (March 2004), accessed on May 2014. White Paper, Intel Corporation, [Online] Available: ftp://download.intel.com/design/network/papers/30117401.pdf

[35] M. Patel. Wind and Solar Power Systems, CRC Press, 1999.

http://www.intel.com/design/pentium4/guides/249205.htm

http://www.nrel.gov/midc/

http://www.hpl.hp.com/techreports/2009/HPL-2009-85.html

ftp://download.intel.com/design/network/papers/30117401.pdf

118

[36] D. Meisner, B. Gold and T. Wenisch, “PowerNap: Eliminating server idle power,” in Proc. Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS’09), 2009, pp. 205-216.

[37] S. Pelley, D. Meisner, P. Zandevakili, T. Wenisch and J. Underwood, “Power Routing: Dynamic power provisioning in the data center,” in Proc. Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS’10), 2010, pp. 231-242.

[38] F. Ahmad and T.N. Vijaykumar, “Joint optimization of idle and cooling power in data centers while maintaining response time,” in Proc. Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS’10), 2010, pp. 243-256.

[39] E. Pakbaznia and M. Pedram. “Minimizing data center cooling and server power costs,” in Proc. Int. Symp. on Low Power Electronics and Design (ISLPED’09), 2009, pp. 145-150.

[40] A. Gandhi, M. Harchol-Balter, R. Das and C. Lefurgy. “Optimal power allocation in server farms,” in Proc. Int. Conf. on Modeling and Measurement of Computer Systems (SIGMETRICS’09), 2009, pp. 157-168.

[41] J. Peças Lopesa,N. Hatziargyrioub, J. Mutalec, P. Djapicc, and N. Jenkinsc. “Integrating distributed generation into electric power systems: a review of drivers, challenges and opportunities,” Electric Power Systems Research, Vol. 77, Issue 9, 2007, pp. 1189-1203.

[42] M. Ton and B. Fortenbery, (March 2008). DC power for improved data center efficiency, accessed on May 2014. Lawrence Berkeley National Laboratory, Technical Report, [Online]. Available: http://hightech.lbl.gov/documents/data_centers/dcdemofinalreport.pdf

[43] X. Fan, W. Weber and L. Barroso, “Power provisioning for a warehouse-sized computer,” in Proc. 34th Int. Symp. on Computer Architecture (ISCA’07), 2007, pp. 13-23.

[44] X. Wang, M. Chen, C. Lefurgy, and T. Keller. “SHIP: Scalable hierarchical power control for large-scale data centers, in Proc. 18th Int. Conf. on Parallel Architecture and Compilation Techniques (PACT’09), 2009, pp. 91-100.

[45] SPECPower Results, accessed on May 2014. [Online]. Available: http://www.spec.org/power_ssj2008

[46] Logs of Real Parallel Workloads from Production Systems, accessed on May 2014. [Online]. Available: http://www.cs.huji.ac.il/labs/parallel/workload/logs.html

http://hightech.lbl.gov/documents/data_centers/dcdemofinalreport.pdf

http://www.spec.org/power_ssj2008

http://www.cs.huji.ac.il/labs/parallel/workload/logs.html

119

[47] Western Wind and Solar Integration Study, accessed on May 2014. [Online]. Available: http://www.nrel.gov/wind/systemsintegration/wwsis.html

[48] V. Soundararajan and J. Anderson, “The impact of management operations on the virtualized data center,” in Proc. 37th Int. Symp. on Computer Architecture (ISCA’10), 2010, pp. 326-337.

[49] M. Zhao and R. Figueiredo, “Experimental study of virtual machine migration in support of reservation of cluster resources,” in Proc. Int. Workshop on Virtualization Technology in Distributed Computing, 2007, Article No.5

[50] R. Lasseter and P. Piagi. “Microgrid: A conceptual Solution,” in Proc. IEEE Annual Power Electronics Specialists Conference, 2004, pp. 4285-4290.

[51] The role of distributed generation and combined heat and power (CHP) systems in data centers, (August 2007), accessed on May 2014. Technical Report, EPA, [Online]. Available: http://www.epa.gov/chp/documents/datactr_whitepaper.pdf

[52] B. Kirby and E. Hirst, (January 2000). Customer-specific metrics for the regulation and load-following ancillary services, accessed on May 2014. Technical Report, Oak Ridge National Lab, [Online]. Available: http://energy.lbl.gov/ea/certs/pdf/ornl-con-474.pdf

[53] S. Chowdhury and P. Crossley. Microgrid and active distribution networks, The Institute of Engineering and Technology, 2009

[54] P. Piagi and R. Lasseter. “Autonomous control of microgrids,” in Proc. IEEE Power Engineering Society General Meeting, 2006, pp. 8-8

[55] HOMER Energy Modeling Software, http://homerenergy.com/

[56] M. Stansberry and J. Kudritzki, (April 2012). The 2012 Uptime Institute data center industry survey, accessed on May 2014. Write Paper, The Uptime Institute, [Online]. Available: http://uptimeinstitute.com/images/stories/Uptime_Institute_2012_Data_Industry_Survey.pdf

[57] D. Bouley, (2011). Estimating a data center’s electrical carbon footprint, accessed on May 2014. White Paper #66, Schneider Electric, [Online]. Available: http://www.apcmedia.com/salestools/DBOY-7EVHLH/DBOY-7EVHLH_R0_EN.pdf

[58] P. Sarti, (April, 2011). Battery cabinet hardware v1.0, accessed on May 2014. Write Paper, Facebook Open Compute Project, [Online], Available: http://www.opencompute.org/wp/wp-content/uploads/2011/07/DataCenter-Battery-Cabinet-Specifications.pdf

http://www.nrel.gov/wind/systemsintegration/wwsis.html

http://www.epa.gov/chp/documents/datactr_whitepaper.pdf

http://energy.lbl.gov/ea/certs/pdf/ornl-con-474.pdf

http://homerenergy.com/

http://uptimeinstitute.com/images/stories/Uptime_Institute_2012_Data_Industry_Survey.pdf

http://uptimeinstitute.com/images/stories/Uptime_Institute_2012_Data_Industry_Survey.pdf

http://www.apcmedia.com/salestools/DBOY-7EVHLH/DBOY-7EVHLH_R0_EN.pdf

http://www.apcmedia.com/salestools/DBOY-7EVHLH/DBOY-7EVHLH_R0_EN.pdf

http://www.opencompute.org/wp/wp-content/uploads/2011/07/DataCenter-Battery-Cabinet-Specifications.pdf

http://www.opencompute.org/wp/wp-content/uploads/2011/07/DataCenter-Battery-Cabinet-Specifications.pdf

120

[59] H. Sher, and K. Addoweesh, “Micro-inverters - promising solutions in solar photovoltaics,” Energy for Sustainable Development, Elsevier, 2012, pp. 389-400

[60] National survey on data center outages, (September 2010), accessed on May 2014. White Paper, Ponemon Institute, [Online]. Available: http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white%20papers/data-center-optimization_24655-r09-10.pdf

[61] Modbus Protocol, http://www.modbus.org/

[62] H. Bindner, T. Cronin, P. Lundsager, J. Manwell, U. Abdulwahid, I. Gould, (April 2005). Lifetime modelling of lead acid batteries, accessed on May 2014. Risø National Laboratory, Technical Report, [Online]. Available: http://orbit.dtu.dk/fedora/objects/orbit:88309/datastreams/file_7710966/content

[63] D. Doerffel, and S. Sharkh. “A critical review of using the Peukert equation for determining the remaining capacity of lead-acid and Lithium-ion batteries.” Journal of Power Sources, vol. 155, issue 2, 2006, pp. 395-400.

[64] S. Huang, J. Huang, J. Dai, T. Xie, and B. Huang, “The HiBench benchmark suite: Characterization of the MapReduce-based data analysis. Data Engineering Workshops, IEEE Int. Conf. on Data Engineering, 2010, pp. 41-51.

[65] CloudSuite 2.0, accessed on May 2014. [Online]. Available: http://parsa.epfl.ch/cloudsuite

[66] Watts Up? Meters, https://www.wattsupmeters.com

[67] D. Feldman, G. Barbose, R. Margolis, R. Wiser, N. Darghouth, and A Goodrich, (November 2012). Photovoltaic (PV) pricing trends: historical, recent, and near-term projections, accessed on May 2014. Technical Report, National Renewable Energy Laboratory and Lawrence Berkeley National Laboratory, [Online]. Available: http://www.nrel.gov/docs/fy13osti/56776.pdf

[68] Navigant Consulting, (January 2006). A review of PV inverter technology cost and performance projections, accessed on May 2014. National Renewable Energy Lab, Technical Report, [Online]. Available: http://www.nrel.gov/docs/fy06osti/38771.pdf

[69] R. Simpson, (February 2012). Levelized cost of energy from residential to large scale PV, accessed on May 2014. in Proc. the Applied Power Electronics Conference and Exposition, [Online]. Available: http://www.psma.com/sites/default/files/uploads/tech-forums-alternative-energy/presentations/2012-apec-256-evolution-photovoltaic-inverter.pdf

http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white%20papers/data-center-optimization_24655-r09-10.pdf



http://www.modbus.org/

http://orbit.dtu.dk/fedora/objects/orbit:88309/datastreams/file_7710966/content

http://parsa.epfl.ch/cloudsuite

https://www.wattsupmeters.com/

http://www.nrel.gov/docs/fy13osti/56776.pdf

http://www.nrel.gov/docs/fy06osti/38771.pdf

http://www.psma.com/sites/default/files/uploads/tech-forums-alternative-energy/presentations/2012-apec-256-evolution-photovoltaic-inverter.pdf

http://www.psma.com/sites/default/files/uploads/tech-forums-alternative-energy/presentations/2012-apec-256-evolution-photovoltaic-inverter.pdf

121

[70] SRRL Baseline Measurement System, accessed on May 2014. [Online]. http://www.nrel.gov/midc/srrl_bms/

[71] G. Burch, (August 2001). Hybrid renewable energy systems, accessed on May 2014. in Proc. The Natural Gas / Renewable Energy Workshops, U.S. Department of Energy, [Online]. Available: http://www.netl.doe.gov/publications/proceedings/01/hybrids/Gary%20Burch%208.21.01.pdf

[72] Information optimization: Harness the power of big data, (December 2012), accessed on May 2014. White Paper, Hewlett-Packard, [Online]. Available: http://www.hp.com/hpinfo/newsroom/press_kits/2012/HPDiscoverFrankfurt2012/IO_Whitepaper_Harness_the_Power_of_Big_Data.pdf

[73] W. Johnston, E. Dart, M. Ernst, and B. Tierney, (2003). Enabling high throughput in widely distributed data management and analysis systems: lessons from the LHC, accessed on May 2014. Technical Report, Lawrence Berkeley National Laboratory, [Online]. Available: http://www.es.net/assets/pubs_presos/High-throughput-lessons-from-the-LHC-experience.Johnston.TNC2013.pdf

[74] C. Li, R. Wang, T. Li, J. Yuan, and D. Qian, “Managing green datacenters powered by hybrid renewable energy systems,” in Proc. Int. Conf. on Autonomic Computing, 2014.

[75] S. McCluer, (2011). Battery technology for data centers and network rooms: lead-acid battery options, accessed on May 2014. White Paper #30, APC, [Online]. Available: http://www.apcmedia.com/salestools/SADE-5TNQZD/SADE-5TNQZD_R12_EN.pdf

[76] F. Zhang, S. Lasluisa, T. Jin, I. Rodero, H. Bui, and M. Parashar, “In-situ feature-based objects tracking for large-scale scientific simulations,” SC Companion: High Performance Computing, Networking Storage and Analysis (SCC), 2012, pp. 736-740.

[77] S. Lakshminarasimhan, D. Boyuka, S. Pendse, X. Zou, J. Jenkins, V. Vishwanath, M. Papka, and N. Samatova, “Scalable in-situ scientific data encoding for analytical query processing,” in Proc. int. Symp. on High-Performance Parallel and Distributed Computing (HPDC’13), 2013

[78] B. Aksanli, P. Eddie, and R. Tajana, “Architecting efficient peak power shaving using batteries in data centers,” in Proc. IEEE Int. Symp. on Modelling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS’13), 2013, pp. 242-253.

http://www.nrel.gov/midc/srrl_bms/

http://www.netl.doe.gov/publications/proceedings/01/hybrids/Gary%20Burch%208.21.01.pdf

http://www.netl.doe.gov/publications/proceedings/01/hybrids/Gary%20Burch%208.21.01.pdf

http://www.hp.com/hpinfo/newsroom/press_kits/2012/HPDiscoverFrankfurt2012/IO_Whitepaper_Harness_the_Power_of_Big_Data.pdf

http://www.hp.com/hpinfo/newsroom/press_kits/2012/HPDiscoverFrankfurt2012/IO_Whitepaper_Harness_the_Power_of_Big_Data.pdf

http://www.es.net/assets/pubs_presos/High-throughput-lessons-from-the-LHC-experience.Johnston.TNC2013.pdf

http://www.es.net/assets/pubs_presos/High-throughput-lessons-from-the-LHC-experience.Johnston.TNC2013.pdf

http://www.apcmedia.com/salestools/SADE-5TNQZD/SADE-5TNQZD_R12_EN.pdf

http://www.apcmedia.com/salestools/SADE-5TNQZD/SADE-5TNQZD_R12_EN.pdf

122

[79] M. Ebbers, A. Galea, M. Schaefer, M. Khiem, (August 2008). The green data center: Steps for the journey, accessed on May 2014. White Paper, IBM, [Online]. Available: ftp://ftp.boulder.ibm.com/software/uk/itsolutions/optimiseit/energy-efficiency-solutions/2-new-enterprise-data-centre-the-green-new-enterprise-data-centre-red-books.pdf

[80] Z. Kishimoto, (August, 2013). IT controls power at data centers, accessed on May 2014. Technical Blog, AltaTerra Research Network, [Online]. Available: http://www.altaterra.net/blogpost/587092/168544/IT-Controls-Power-at-Data-Centers--part-1

[81] Environmental impacts of transmission lines, accessed on May 2014. White Paper, Public Service Commission of Wisconsin, [Online]. Available: http://psc.wi.gov/thelibrary/publications/electric/electric10.pdf

[82] J. Glanz, (September 2012). Power, pollution and the internet, accessed on May 2014. [Online]. Available: http://www.nytimes.com/2012/09/23/technology/data-centers-waste-vast-amounts-of-energy-belying-industry-image.html?pagewanted=all

[83] C. Bange, T. Grosser, and N. Janoschek, (February 2013). Big data survey Europe: Usage, technology and budgets in European best-practice companies, accessed on May 2014. White Paper, BARC, [Online]. Available: http://www.pmone.com/fileadmin/user_upload/doc/study/BARC_BIG_DATA_SURVEY_EN_final.pdf

[84] N. Femia, G. Petrone, G. Spagnuolo, and M. Vitelli. “Optimization of perturb and observe maximum power point tracking method”, IEEE Transactions on Power Electronics, Vol. 20, No. 4, July 2005, pp. 963-973.

ftp://ftp.boulder.ibm.com/software/uk/itsolutions/optimiseit/energy-efficiency-solutions/2-new-enterprise-data-centre-the-green-new-enterprise-data-centre-red-books.pdf



http://www.altaterra.net/blogpost/587092/168544/IT-Controls-Power-at-Data-Centers--part-1

http://www.altaterra.net/blogpost/587092/168544/IT-Controls-Power-at-Data-Centers--part-1

http://psc.wi.gov/thelibrary/publications/electric/electric10.pdf

http://www.nytimes.com/2012/09/23/technology/data-centers-waste-vast-amounts-of-energy-belying-industry-image.html?pagewanted=all



http://www.pmone.com/fileadmin/user_upload/doc/study/BARC_BIG_DATA_SURVEY_EN_final.pdf

http://www.pmone.com/fileadmin/user_upload/doc/study/BARC_BIG_DATA_SURVEY_EN_final.pdf

123

BIOGRAPHICAL SKETCH

Chao Li was born and raised in Taiyuan, Shanxi, China. He earned his B.S. (with

Chu Kochen Certificate of Honors) in electronic and information engineering from

Zhejiang University, China. He was a visiting student at UCLA in August 2008.

Chao joined the Intelligent Design of Efficient Architectures Laboratory (IDEAL)

at the University of Florida in August 2009. His research was generously supported by

several prestigious awards, including a four-year UF Alumni Graduate Fellowship Award

from 2009 to 2013, a Yahoo! Key Scientific Challenges Program Award in Green

Computing in 2012, and a Facebook Graduate Fellowship in Computer Architecture in

2013. He was a winner of the IEEE HPCA 2011 Best Paper Award.

Chao was a Ph.D. recipient of the 2010 University of Florida Gator Engineer

Recognition Award for proposing sustainable solutions to the computer energy

challenge and global climate change. He was named an Outstanding International

Student by the UF International Center in 2010 and 2013. He received the ACM A.M.

Turing Centenary Celebration Student Scholarship in 2012 and attended the once-in-a-

lifetime ceremony with 34 Turing Award winners participating in the event. He was also

awarded the prestigious 2013 Chinese Government Award for Outstanding Self-

Financed Students Studying Abroad from the China Scholarship Council (CSC).

During his graduate studies, Chao served as the organizing committee member

for the 2013 Annual IEEE International Conference on Networking, Architecture, and

Storage (NAS) and the 2014 Annual IEEE International Symposium for High-

Performance Computer Architecture (HPCA). He was a student member of IEEE, IEEE

Computer Society, ACM, and ACM SIGARCH.

enabling renewable energy powered sustainable high … · 2015. 1. 8. · enabling renewable energy...

Documents