data center energy efficiency - pnnl · data center server load 51% data center crac units 25%...
TRANSCRIPT
Data Center Energy Efficiency
SC07 Birds of a FeatherNovember, 2007Bill [email protected]
Benchmark results helped to find Benchmark results helped to find best practicesbest practices
The ratio of IT equipment power to the total The ratio of IT equipment power to the total (or its inverse) is an indicator of relative (or its inverse) is an indicator of relative overall efficiency. Examination of individual overall efficiency. Examination of individual systems and components in the centers that systems and components in the centers that performed well helped to identify best performed well helped to identify best practices.practices.
Percentage of electricity delivered Percentage of electricity delivered to IT equipmentto IT equipment
IT Power to Total Data Center Power
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
0.80
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Data Center Number
Rat
io
Average .57
Higher is better
Source: LBNL Benchmarking
Performance varies
Data Center Server Load
51%
Data Center CRAC Units
25%
Cooling Tower Plant4%
Electrical Room Cooling
4%
Office Space Conditioning
1%
Lighting2%
Other13%
Computer Loads67%
HVAC - Air Movement
7%
Lighting2%
HVAC - Chiller and
Pumps24%
The relative percentages of the energy actually doing computing varies considerably.
Typical power useTypical power use
Courtesy of Michael Patterson, Intel Corporation
Computer/SystemArchitecture
System Architecture /Power Engineering
Facility Design /Mechanical Design
Server Load/ Computing Operations
Cooling Equipment
Power Conversion & Distribution
Alternative Power
Generation
• High voltage distribution• Use of DC power• Highly efficient UPS systems• Efficient redundancy
strategies
• Load management• Server innovation
Energy efficiency opportunities are Energy efficiency opportunities are everywhereeverywhere
• Better air management• Move to liquid cooling• Optimized chilled-water plants• Use of free cooling
• On-site generation• Waste heat for cooling• Use of renewable energy/fuel cells
Best practices topics identified Best practices topics identified through benchmarkingthrough benchmarking
HVAC – Air Delivery – Water Systems
Facility Electrical Systems
IT Equipment
Cross-cutting / misc. issues
Air management
Cooling plant optimization
UPS systems
Power Supply efficiency
Motor efficiency
Air economizers
Free cooling Self generation
Sleep/standby loads
Right sizing
Humidificationcontrols alternatives
Variable speed pumping
AC-DC Distribution
IT equip fans Variable speed drives
Centralized air handlers
Variable speed Chillers
Standby generation
Lighting
Direct liquid cooling
Maintenance
Low pressure drop air distribution
Commissioning/continuous benchmarking
Fan efficiency Heat recovery Redundancies Method of charging for
space and power Building envelope
A word about appropriate A word about appropriate environmental conditionsenvironmental conditions……
ASHRAE published thermal guidelinesASHRAE published thermal guidelines–– All IT suppliers participated and agreedAll IT suppliers participated and agreed–– Guidelines allow most centers to relax setpointsGuidelines allow most centers to relax setpoints–– HPC community agreesHPC community agrees
Recommended and allowable ranges of Recommended and allowable ranges of temperature and humidity are provided temperature and humidity are provided –– at the at the inlet to the IT equipmentinlet to the IT equipment
High temperatures in the High temperatures in the ““hot aisleshot aisles”” and return to and return to air conditioners are desirable.air conditioners are desirable.
Humidity guidelines Humidity guidelines –– at the inlet to IT equipmentat the inlet to IT equipment
ASHRAE HUMIDITY GUIDELINES
0
10
20
30
40
50
60
70
80
90
100
% R
elat
ive
Hum
idity
ASHRAE Allowable Maximum
ASHRAE Allowable Minimum
ASHRAE Recommended Maximum
ASHRAE Recommended Minimum
Temperature guidelines Temperature guidelines –– at the inlet to IT equipmentat the inlet to IT equipment
ASHRAE TEMPERATURE GUIDELINES
40
50
60
70
80
90
100
Deg
rees
F
ASHRAE Allowable Maximum
ASHRAE Allowable Minimum
ASHRAE Recommended Maximum
ASHRAE Recommended Minimum
Air management best scenario Air management best scenario –– isolate cold and hotisolate cold and hot
70-75º
95-100º
Another isolation schemeAnother isolation scheme
Measured fan energy savingsMeasured fan energy savings –– 75%75%
If mixing of cold supply air If mixing of cold supply air with hot return air can be with hot return air can be eliminatedeliminated--fan speed can be reducedfan speed can be reduced
Better temperature control can Better temperature control can allow raising the temperature in allow raising the temperature in the entire data center!the entire data center!
Cold Aisle NW - PGE12813
40
45
50
55
60
65
70
75
80
85
90
6/13/2006 12:00 6/14/2006 0:00 6/14/2006 12:00 6/15/2006 0:00 6/15/2006 12:00 6/16/2006 0:00 6/16/2006 12:00
Time
Tem
pera
ture
(deg
F)
LowMedHigh
Baseline Alternate 1
Setup
Setup
Alternate 2
ASHRAE Recommended Range
Ranges during demonstration
See the problem areasSee the problem areas
Infrared thermography and CFD modeling can be used as visualization tools
Best practices Best practices –– Free cooling with Free cooling with air economizersair economizers
HVAC – Air Delivery – Water Systems
Facility Electrical Systems
IT Equipment
Cross-cutting / misc. issues
Air management
Cooling plant optimization
UPS systems
Power Supply efficiency
Motor efficiency
Air economizers
Free cooling Self generation
Sleep/standby loads
Right sizing
Humidificationcontrols alternatives
Variable speed pumping
AC-DC Distribution
IT equip fans Variable speed drives
Centralized air handlers
Variable speed Chillers
Standby generation
Lighting
Direct liquid cooling
Maintenance
Low pressure drop air distribution
Commissioning/continuous benchmarking
Fan efficiency Heat recovery Redundancies Method of charging for
space and power Building envelope
Encouraging outside air economizersEncouraging outside air economizers
Issue: Issue: –– Many are reluctant to use air economizersMany are reluctant to use air economizers–– Outdoor pollutants and humidity control considered Outdoor pollutants and humidity control considered
equipment riskequipment risk
Goal: Goal: –– Encourage use of outside air economizers where Encourage use of outside air economizers where
climate is appropriateclimate is appropriate
Strategy: Strategy: –– Address concerns: contamination/humidity controlAddress concerns: contamination/humidity control–– Quantify energy savings benefitsQuantify energy savings benefits
Outdoor MeasurmentsFine Particulate Matter
0
20
40
60
80
100
120
140
160
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
Part
icle
Con
c. (
μg/
m3 ) LBNL
NERSC
Center 3
Center 4
Center 5
Center 6
Center 7
Center 8
Outdoor measurementsOutdoor measurements
IBM Standard
EPA Annual Health Standard
EPA 24-Hour Health Standard
and ASHRAE Standard
Indoor Measurments Fine Particulate Matter
0
20
40
60
80
100
120
140
160
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
6:00AM
12:00PM
6:00PM
12:00AM
Part
icle
Con
c. (
μg/
m3 ) LBNL
NERSC
Center 3
Center 4
Center 5
Center 6
Center 7
Center 8
Measurements inside the centersMeasurements inside the centers
IBM Standard
EPA Annual Health Standard
EPA 24-Hour Health Standard
and ASHRAE Standard
Data center w/economizerData center w/economizer
Center 8w/economizer
0.3-5 Particulate Matter
0
10
20
30
40
50
60
70
80
90
100
8/18/060:00
8/18/0612:00
8/19/060:00
8/19/0612:00
8/20/060:00
8/20/0612:00
8/21/060:00
8/21/0612:00
8/22/060:00
8/22/0612:00
8/23/060:00
8/23/0612:00
8/24/060:00
8/24/0612:00
8/25/060:00
8/25/0612:00
8/26/060:00
Part
icle
Con
c. ( μ
g/m
3 )
OutsideOutside (PostFilter)PreServerRmAmb
FindingsFindings
Water soluble salts in combination with high Water soluble salts in combination with high humidity can cause failureshumidity can cause failuresNew ASHRAE particle limits drastically lower than New ASHRAE particle limits drastically lower than one manufacturerone manufacturer’’s recommendations recommendationParticle concentration typically an order of Particle concentration typically an order of magnitude lower than new ASHRAE limits (no magnitude lower than new ASHRAE limits (no economizer) economizer) Economizers, without other mitigation, can cause Economizers, without other mitigation, can cause particle concentration to approach new ASHRAE particle concentration to approach new ASHRAE limits but filtration can mitigate thislimits but filtration can mitigate thisLarge energy savingsLarge energy savings
050
100150
200250300
350400
450500
Hrs/YR
28 35 42 49 56 63 70 77 84 91 98 105 112Temperature (F)
Berkeley Weather
hr/yr0 10 0
11 20 021 30 331 40 50441 50 232551 60 315361 70 154271 80 84681 90 29791 100 75
101 110 15111 120 0
Range F
Set Inlet TemperatureTo upper end of ASHRAE
NERSC/LBNL CRT Building Conceptual DesignDRAFT: Subject to change Howard Walter
Schematic Design ConceptsSchematic Design ConceptsDRAFT: Subject to change; Do not re-distribute
Howard Walter
Best practices Best practices –– power conversionpower conversion
HVAC – Air Delivery – Water Systems
Facility Electrical Systems
IT Equipment
Cross-cutting / misc. issues
Air management
Cooling plant optimization
UPS systems
Power Supply efficiency
Motor efficiency
Air economizers
Free cooling Self generation
Sleep/standby loads
Right sizing
Humidificationcontrols alternatives
Variable speed pumping
AC-DC Distribution
IT equip fans Variable speed drives
Centralized air handlers
Variable speed Chillers
Standby generation
Lighting
Direct liquid cooling
Maintenance
Low pressure drop air distribution
Commissioning/continuous benchmarking
Fan efficiency Heat recovery Redundancies Method of charging for
space and power Building envelope
Inverter
In Out
Bypass
Battery/ChargerRectifier
Internal Drive
External Drive
I/O
Memory Controller
μ Processor
SDRAM
Graphics Controller
DC/DCAC/DC
DC/DC
AC/DC Multi output PS
Voltage Regulator Modules
5V
12V
3.3V
12V 1.5/2. 5V
1.1V- 1.85V
3.3V
3.3V
12V
PWM/PFCSwitcher
Unregulated DCTo Multi Output Regulated DC
Voltages
Data center power conversions
AC voltage conversions
Prior research illustrated large Prior research illustrated large losses in power conversionlosses in power conversion
45%
50%
55%
60%
65%
70%
75%
80%
85%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
% of Nameplate Power Output
% E
ffici
ency
Average of All Servers
Uninterruptible Power Supplies (UPS)
Power Supplies in IT equipment
Factory Measurements of UPS Efficiency
70%
75%
80%
85%
90%
95%
100%
0% 20% 40% 60% 80% 100%
Percent of Rated Active Power Load
Effic
ienc
y
Flywheel UPS
Double-Conversion UPS
Delta-Conversion UPS
(tested using linear loads)
With over 25 industry partners direct DC powering of servers was demonstrated
Typical AC distribution todayTypical AC distribution today
DC/ACAC/DC480 VACBulk Power
Supply
UPS PDU
AC/DC DC/DC VRM
VRM
VRM
VRM
VRM
VRM
12 V
Loadsusing
LegacyVoltages
Loadsusing
SiliconVoltages
12 V
5 V
3.3 V
1.2 V
1.8 V
0.8 VServer
PSU
480 Volt AC
FacilityFacility--level DC distributionlevel DC distribution
AC/DC480 VACBulk Power
SupplyDC UPS
orRectifier
DC/DC VRM
VRM
VRM
VRM
VRM
VRM
12 V
Loadsusing
LegacyVoltages
Loadsusing
SiliconVoltages
12 V
5 V
3.3 V
1.2 V
1.8 V
0.8 VServer
PSU
380 VDC380V.DC480 Volt AC
AC system loss compared to DCAC system loss compared to DC
DC/ACAC/DC480 VACBulk Power
Supply
UPS PDU
AC/DC DC/DC VRM
VRM
VRM
VRM
VRM
VRM
12 V
Loadsusing
LegacyVoltages
Loadsusing
SiliconVoltages
12 V
5 V
3.3 V
1.2 V
1.8 V
0.8 VServer
PSU
AC/DC480 VACBulk Power
SupplyDC UPS
orRectifier
DC/DC VRM
VRM
VRM
VRM
VRM
VRM
12 V
Loadsusing
LegacyVoltages
Loadsusing
SiliconVoltages
12 V
5 V
3.3 V
1.2 V
1.8 V
0.8 VServer
PSU
380 VDC
7-7.3% measured improvement
2-5% measured improvement
Rotary UPS
websites: websites:
http://http://hightech.lbl.govhightech.lbl.gov/datacenters//datacenters/
How soon will we get to liquid cooling? How soon will we get to liquid cooling? Why not doing it today?Why not doing it today?Why isnWhy isn’’t efficient distribution power more t efficient distribution power more widely utilized?widely utilized?–– High efficiency UPS or onHigh efficiency UPS or on--site generationsite generation–– High voltage distribution to RackHigh voltage distribution to Rack–– Direct DC Direct DC –– eliminating conversions eliminating conversions –– ease of ease of
incorporating renewable sources incorporating renewable sources –– available available todaytoday