thermal aware server provisioning (tasp) and workload distribution (tawd) for internet data centers...
TRANSCRIPT
Thermal Aware Server Provisioning (TASP) and Workload Distribution (TAWD) for Internet Data Centers (IDCs)
Zahra Abbasi, Georgios Varsamopoulos and Sandeep Gupta
Impact Laboratory
School of Computing, Informatics and Decision Systems Engineering
Arizona State University
http://impact.asu.edu/
Funded in parts by NSF CNS grants#0834797 and #0855277, and by Intel Corp
Introduction-Motivation The magnitude of data center
energy consumption
Internet users’ growth in the world from 2000-2009: 400% [http://www.internetworldstats.com/stats.htm]
Data center energy consumption grew 20-30% annually in 2006 and 2007 [ Uptime Institute research]
Addressing energy saving for internet Data Center Thermal awareness to improve
energy consumption
2
Historical energy use Future energy use projection- current efficiency trend
Projected Electricity Use of data centers\, 2007 to 2011
Typical data center energy end use
[Source: EPA]
[Source: Department of energy]
The BlueTool project
Design and on-site assessment services
BlueWeb: On-line data and simulation services
Online services:• Measurement archive• Profile archive• Model archive• Energy Calculator
authorized userBackend data andservice access
Research:• Model development• Scheduling testing• Design methodology development• Alternative, eco-friendly cooling technologies
Researchers at ASU
Consulting services:• Energy and efficiency
assessment• Design and online
solutions• Expert advising
BlueCenter:Experimental testebd
BlueSense: on-site monitoring
http://impact.asu.edu/BlueTool/
Talk outline Why thermal awareness for data centers? Opportunities for energy saving in IDCs TASP and TAWD Modeling IDCs:
Software: two tier architecture Hardware: performance and power consumption
Heuristics for TASP and TAWD Simulation model and results Experimental validation of TASP and TAWD
4
Typical layout of a data center Rack outlet temperature Tout
Rack inlet temperature Tin
Computing Room Air conditioner (CRAC) supply temperature Tsup Tout
Tin
Tsup
Redline temperature:Tred of Tin
Fah
renh
eit
[Source: Uptime Institute research ]
Heat recirculation
Possible ways to save energy in IDCs Resizing the active server set
Dynamically changing number of active servers according to the long traffic fluctuation (couple of hours) [Chase et al. SOSP ’01] , [Chen et al. NSDI ’08]
DVFS Adapting CPU speed with respect to incoming workload during fine time slots
[Ranganathan et al. ISCA ’06]
Virtualization Consolidating applications in a few physical machines with respect to their
performance requirement [Kusic et al. CCJ ’09]
Thermal awareness Considering servers’ thermally impact on the cooling system. Servers’ thermal
impact is tightly related to the heat recirculation.
6
Why thermal awareness? PUE (Power Usage Effectiveness):
A metric to measure data center energy efficiency
Large value for PUE is indication of large value for cooling energy
Current state of PUE
7
Total Facility Power {cooling, IT equip., lighting, other}
PUE=IT equipment Power
Scenario PUE
Current Trends 1.9
Improved Operations
1.7
Best Practices 1.3
State-of-the-Art 1.2
EPA estimated value of PUE for 2011(2007 report)
Improving cooling energy by thermal aware task placement [Moore et al. ATEC’ 05], [Tang et al. T-PDS ’08]
Task placement determines IDCs’ thermal profile Servers thermally interfere with each other by
recirculated heat Heat recirculation is uneven and creates hot
spots CRAC must supply sufficient cooling to keep hot
spots under the redline temperature Thermal aware task placement can reduce heat
recirculation and hot spots and improve cooling efficiency
8
Thermal aware task placement
9
[Source: HP]
Heat removedCoP= Work required to remove the heat
Co
effi
cie
nt
of
Pe
rfo
rma
nce
Tsup: CRAC Supply Temperature
= + ×
Tin Tsup D P
inlettemperatures
supplied airtemperatures
heat distribution powervector
Cooling = Pcomputing /CoP(Tsup)=Pcomputing /CoP(Tred – maxi(DiP))
N1 AC
N3N2
d21 d31
d11
d12 d13
Tsup TAC, inTin
Recirculation
Servers thermally interfere with each other by recirculated heat
Linear model for the heat recirculation [Tang et al. T-PDS ’08]
CRAC ‘s CoP
Directly affected by task placement
Measuring thermal efficiency: LRH Thermal efficiency: least contribution on the heat recirculation
LRH: A metric of thermal efficiency of a server [Tang et al. T-PDS ’08]
10
Based on a two-layer rank calculation Rank the servers as recipients of heat
recirculation Rank the servers as contributors of heat
recirculationLRH weight of S =
Σrecipients recipient value amount of heat from S to recipient
LRH rank of Server B is worse than AB
The direction and amount of heat recirculation
A
Example: LRH ranking of servers A and B
Measuring thermal efficiency: LRH Thermal efficiency: least contribution on the heat recirculation
LRH: A metric of thermal efficiency of a server [Tang et al. T-PDS ’08]
11
Based on a two-layer rank calculation Rank the servers as recipients of heat
recirculation Rank the servers as contributors of heat
recirculationLRH weight of S =
Σrecipients recipient value amount of heat from S to recipient
Low Medium High
Server A: Low low low medium
Server B: Medium medium medium high
Server C: high medium high high
Contribution on
the heat recirculation
Incoming heat to the recipients of heat recirculation
(Low means better LRH rank)
Example: LRH ranking of servers A, B and C with respect to their heat recipients
TASP and TAWD TASP
Saving energy by choosing active server set according to
servers’ computing power efficiency(Joules/MIPS) AND thermal efficiency (e.g. LRH)
Doing TASP in long time intervals (couple of hours) called
epochs TAWD
Saving more energy by skewing workload toward thermally
efficient and computing power efficient servers in fine time slots Constraints
Maintain performance [response time] Prevent redlining of servers
12
13
Server 1 Server 2 Server 3 Server N
Load Dispatcher
……
TASP Tier (Epochs)
{λi} TAWD Tier
(Slots)
λ request/sec
On/Off Control
Traffic flow
Parameters
Control data
Server N-1
Two-Tier architecture for IDCs
0 100 200 300 400 500 600 700 800 900 1000550
600
650
700
750
800
850
900
950
1000
Num
ber
of
request
arr
ivals
every
5 s
econds
Time index (every half an hour) over one month
Peak arrival rate
Time index (every 5 second)
Nu
mb
er
of
req
ue
sts
HTTP requests over time, 1998 FIFA World Cup
Server 1 Server 3 Server N-1
λ1=0 λ2 λ3=0 λNλN-1=0
Heat recirculation contribution
Computing capabilities of machines
Computing power efficiency
∑λi= λ
TASP and TAWD: Problem statements TASP:
Input: Data center server set S with N servers, epoch lengths (T), history of arrival rate
Find: Active server set: Ŝ S , where, | Ŝ|=n ≤ N⊆ Objective: Minimizing total energy Constraint: Performance requirements [i.e. response time]
TAWD: Input: Active server set Ŝ , L fine time slots of length t (T=Lt),
history of arrival rate Find: For time slot m (1 ≤ m ≤ L) the workload distribution among
active servers : λim s∀ i Ŝ ∈ Objective: Minimizing total energy Constraint: Performance requirements [i.e. response time]
14
TASP analytical formulation : Prerequisites
15
Observed relationship b/w CPU utilization, arrival rate and turnaround time
Model:
Energy consumption modeling Assumption: Dynamic CRAC temperature setting Etotal = Ecomputing + Ecooling (=Ecomputing/CoP(Tsup))
Performance modeling
This
0 500 1000 1500 2000 2500 3000 35000
10
20
30
40
50
60
70
80
90
100
Throughput, Request Per Second (rps)
Serv
er C
PU u
tiliza
tion
(%)
0 500 1000 1500 2000 2500 3000 3500400
500
600
700
800
900
1000
Turn
arou
nd T
ime
( s
)CPU Utilization(%)Turn-around time( sec)
λmax
uthresh
Sscu iiithreshi ,max
dual-CPU dual-core E7520- chipset “Sossaman” Xeon LV systems
Arrival rate (Requests/Sec)
TASP analytical formulation: Prerequisites Power Consumption Modeling
Linear relationship between power consumption and utilization
Workload Prediction
Request_ArrivalPeak = Request_arrivalAvg. + Request_arrivalStd. dev
16
ωIdle power
power
ω + αMaximum power
Utilization0 1
Active server set size
overestimation factor (>1)Kalman Filtering
iiicompi up
Λ peak
Formulating TASP: Optimization problem Unknown variable
How many servers are required? Which servers among all servers should be chosen as active server set?
Objective: Minimizing total energy consumption:
Constraints: Meet the capacity requirement:
x is a binary vector:
17
Defining a binary vector as the variable. Each element determines if a server should be chosen or not. x: 1 0 0 1 0 1 …..
peaki
threshi
N
ii c
ux
1
tuxuxDTCoP iiiiiiiii
i
red)(
))(((
11
max
Computing powerHeat recirculation
Nixi ...1},1,0{
Heuristic approaches for TASP MinMax Approximation Numerical approx.:SQP (Seq. Quad. Prog.) Ŝ determined by discretization real solution High time complexity (QP: O(n5))
sLRH (scaled LRH) Ranking servers based on heat recirculation and
computing performance:
Ŝ = High ranking server sufficient for peak request arrival
CP-sLRH (Computing Power efficiency and sLRH) servers are first ranked according to their
computing power efficiency (J/MIPS) and then according to sLRH
Ŝ determined similar to sLRH
18
Series1Series2
Series1
Series2
Heat recirculation
Computing power efficiency: Series1>Series2
Series1
Series2
Least recirculated heat (LRHi)
Computing efficiency (MIPS)
sLRHi=
Heuristic solution for TAWD Ranking servers based on CP-sLRH Giving the maximum affordable workload to the highest ranking servers
Skewing workloads toward thermal efficient server rather than performance oriented Load Balancing (LB)
Solution for TAWD
19
10 20 30 40 500
20
40
60
80
100
Total uAverage
=30%
uThreshold
=60%
Servers are ranked according to heat recirculation
Util
iza
tion
10 20 30 40 500
20
40
60
80
100
uAverage
=30%
uThreshold
=60%
Servers are ranked according to heat recirculation
Util
iza
tion
Utilization in LB Utilization in TAWD
Evaluation Baseline algorithms
TASP versus CPSP (Computing Power efficient Server Provisioning) TAWD versus LB
Simulation Heterogeneous DC Model ASU HPCI (PUE>1.3)
Heat recirculation using CFD 50 computing nodes (1000 cores)
Experimental validation Using carton boxed Sossaman
systems
20
ASU HPCI data center
Validating thermal awareness TASP Validation: NoSP: All servers are equally utilized (25%)
TASP(sLRH): Two thermal efficient servers are utilized 50% and the other two machines are turned off
CPSP (Thermally oblivious): Two non thermal efficient ate utilized 50% and the other two are turned off
TAWD Validation LB: All servers are equally utilized such that their
utilization fluctuate over fine time slots (30 second) TAWD: Workload is skewed toward thermal efficient
servers in fine time slots, such that the total workload in any moment equals to LB scenario
21
Simulation: Workload model
SPECweb2009 benchmark (e-commerce ) suite Synthesizing SPECweb2009 + FIFA World CUP 1998
22
HTTP requests over time, 1998 FIFA World Cup
Request arrival rate over time of SPECweb2009 epoch-level peaks are obtained from the 1998 FIFA World Cup traces
23
Simulation results: TASP versus CPSP
Energy saving with respect to CPSP for different TASP schemes over different values for
Simulation results- TASP-TAWD versus TASP-LB
24
Energy saving of TAWD with respect to LB.
Conclusion Extra energy saving by choice of servers
TASP approaches have nothing to do with QoS violations
TASP MiniMax approach yields to the maximum energy saving
CP-sLRH, the low complex heuristic approach, can be used for large sale data center
More energy saving by combining TASP with TAWD TAWD improved cooling energy by 3%
25
Future Works
BlueTool Project (Ongoing project) http://impact.asu.edu/BlueTool/wiki/index.php/Main_Page
Enhancing Thermal Modeling Considering the dynamic behavior of cooling
systems
Virtualization, internet multi-tier applications
26
THANKSQuestions?
27
References [Moore et al. ATEC ’05] J. Moore, J. Chase, P. Ranganathan, and R. Sharma, “Making
scheduling "cool": temperature-aware workload placement in data centers,” in ATEC ’05: Proceedings of the annual conference on USENIX Annual Technical Conference.
[Tang et al. T-PDS ’08] Q. Tang, S. K. S. Gupta, and G. Varsamopoulos, “Energy-ecient thermal-aware task scheduling for homogeneous high-performance computing data centers: A cyber-physical approach,” IEEE Trans. Parallel Distrib. Syst., vol. 19, no. 11, pp. 1458–1472, 2008.
[Chase et al. SOSP ’01] J. Chase, D. Anderson, P. Thakar, A. Vahdat, and R. Doyle, “Managing energy and server resources in hosting centers,” in SOSP ’01: Proceedings of the eighteenth ACM symposium on Operating systems principles. New York, NY, USA: ACM, 2001, pp. 103–116.
[Chen et al. NSDI ’08] Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang, and N. Gautam, “Managing server energy and operational costs in hosting centers,” SIGMETRICS Performance Evaluation Review, vol. 33, no. 1, pp. 303–314, 2005.
[Ranganathan et al. ISCA ’06] P. Ranganathan, P. Leech, D. Irwin, and J. Chase, “Ensemble-level power management for dense blade servers,”. ISCA ’06. 33rd International Symposium in Computer Architecture, 2006, pp. 66–77.
[Kusic et al. CCJ ’09] D. Kusic, J. O. Kephart, J. E. Hanson, N. Kandasamy, and G. Jiang, “Power and performance management of virtualized computing environments via lookahead control,” Cluster Computing, vol. 12, pp. 1–15, 2009.
28
LRH weight
Experimental setup to validate TASP and TAWD
Heat recirculation Coefficient
LRHA=0.69P2
LRHB=0.36P2
LRHC=0LRHD=0
A B
CD
Experimental Results TASP Validation:
NoSP: All servers are equally utilized (25%)
TASP(sLRH): Two thermal efficient servers are utilized 50% and the other two machines are turned off
CPSP (Thermally oblivious): Two non thermal efficient ate utilized 50% and the other two are turned off
TAWD Validation LB: All servers are equally utilized such that their
utilization fluctuate over fine time slots (30 second) TAWD: Workload is skewed toward thermal efficient
servers in fine time slots, such that the total workload in any moment equals to LB scenario
30
System model and assumptions Heterogeneous Data Center
Different computing efficiency (MIPS) Different computing power efficiency (Joules/MIPS) The solutions can be applied for Homogenous data centers
Heat recirculation in the data center room Computing Racks are organized in hot aisle and cold isle Heat recirculation among computing nodes Different thermal efficiency
31
Energy consumption model of Data Center
32
Coefficient of Performance(source: HP)
= + ×
N 1 A C
R ecircu la t io n
T su p T in T o u t T A C in
N 2 N 3
1 2 1 3
2 13 1
1 1
Tin Tsup D P
inlettemperatures
supplied airtemperatures
heat distribution
powervector
Computing power Cooling
power
+Etotal=
[1] J. Moore, J. Chase, P. Ranganathan, and R. Sharma, “Making scheduling "cool": temperature-aware workload placement in data centers,” in ATEC ’05: Proceedings of the annual conference on USENIX Annual Technical conference. Berkeley, CA, USA: USENIX Association, 2005, pp. 5–5.[2] Q. Tang, S. K. S. Gupta, and G. Varsamopoulos, “Energy-efficient thermal-aware task scheduling for homogeneous high-performance computing data centers: A cyber-physical approach,” IEEE Trans. Parallel Distrib. Syst., vol. 19, no. 11, pp. 1458–1472, 2008.
tPcomp tTCoP
pcomp
)( sup = tpDTCoP
pcompii
i
red
comp
))(( maxtPcomp +
Improving cooling energy by minimizing the maximum of servers’ inlet temperature
Perquisites of the analytical formulation for TASP Energy consumption modeling
Computing Power consumption modeling Linear model with respect to utilization
Performance modeling (Response time) Posing a cap for the CPU utilization:
Workload modeling Kalman filtering to predict average traffic,
33
= tpDTCoP
pcompii
i
red
comp
))(( maxtPcomp +Computing
power Cooling power
iiicompi up
i
threshi
i c
umax
avgprev
avg
prevavgpeak
)(
Idle power
Workload arrival during fine time slots
Formulating TASP: Optimization problem Unknown variable
How many servers are required? Which servers among all servers should be chosen as active server set?
Objective: Minimizing total energy consumption:
Constraint: Meet the capacity requirement: x is a binary vector:
34
Defining a binary vector as the variable. Each element determines if a server should be chosen or not. x: 1 0 0 1 0 1 …..
peak
i
threshi
N
ii c
ux
1
Nixi ...1},1,0{
tuxuxDTCoP iiiiiiiii
i
red)(
))(((
11
max
Computing power
Heat recirculation
Formulating TAWD
Unknown Variable: Finding the workload distribution weights of :
Objective: Minimizing total energy consumption during a slot
Constraints: Performance Constraint :
Capacity Constraint:
Solutions: Using heuristic approaches (CP-sLRH) Ranking servers based on CP-sLRH Giving the maximum affordable workload to the highest ranking servers
35
TccDTCoP iiiiiiiii
i
red)(
))(((
11
max
,1ˆ
Ss
i
i
}{ ib
Ssuuc ithreshiiii
ˆ,
Evaluation
Baseline Algorithms TASP with respect to CPSP
TAWD with respect to LB
Evaluation methods Experiments (Small scale) Simulation
36
Series1
Series2
Simulation results-The performance of various TASP approaches
Saving energy over time: MinMax and CP-LRH always surpasses CPSP, sLRH may perform worse than CPSP when active server set becomes large
37
Energy consumption of thermal aware server provisioning scenario over time (intervals in epochs). MinMax always do better than CPSP.
Request arrival rate over time of SPECweb2009 where epoch-level peaks are obtained from the 1998 FIFA World Cup traces
The More overestimation The less energy saving
The more overestimation the less QoS violations
The smaller active server size the larger saving
Energy saving with respect to CPSP for different TASP schemes over ϒ. Note that higher utilization yields higher savings.
38
Simulation results -Energy saving with respect to the overestimation factor (ϒ)
CPU utilization violations with respect to ϒ over time. Violations for ϒ=1 are much higher than for
the rest values.
Energy saving of MiniMax over different number
of active server size for ϒ =6.
Simulation results -The performance of various TASP approaches
39
Total energy consumption with respect to server provisioning scenario. The energy-saving percentages are with respect to CPSP.
Simulation resultsPerformance of TAWD Saving energy through
skewing workload toward thermal efficient servers
Average data center utilization of each server (over one week), as sorted with respect to LRH. The effects of TAWD’s load skewing on the utilization are obvious.
40
System model and assumptions Virtualized Data Center
All systems are capable of running any web application
Internet traffic Short transaction-based traffic Short and long term variation
41
- Software assumptions
http://www.internetworldstats.com/stats.htm
TASP Algorithm
42
TAWD Algorithm
43
How effective are TASP and TAWD? Simulation setup according to the physical layout of ASU
HPCI data centre and the combination of SPECweb traffic profile and FIFA World CUP 1998 web trace
Evaluating TASP compared to CPSP(Computing Power based Server Provisioning) Saving energy from 4.5% to 8.4% with respect to TASP
scenario and overestimation of active server set size
Evaluating TAWD compared to LB(Load Balancing) Saving 1% more energy by combination of TASP and TAWD
44