best practice and infrastructure guideline for on-premise ... · best practice and infrastructure...

APC by Schneider Electric – July 2017

Best practice and Infrastructure

Guideline FOR On-Premise Data Center

Patphong Petkaewna

DC Solution Architect - International

APC by Schneider Electric – Patphong – July 2017

Explosive Growth

●… driven by the number of connected users,

& the Internet of Things (IoT)

21 Billion network

devices

TB transmitted

(flight hr.)

2

1.3Million video views

(per min.)

40

Billion Internet users


Trends

Energy Efficiency

(Do more with less)

Everything as a Service

Speed & Simplification

Internet of Things

Capital Preservation

Market

Social Technologies

Big Data

Mobile Computing

Edge Compute

Converged IT

Cloud Computing

Security

Outsourcing

Medium Voltage

Distribution

New cooling techniques

Servers w/ UPS

Prefabrication

Micro Data Centers

Lower Tier Designs

Relaxed Operation

Standards

IT Physical Infrastructure


Unprecedented Changes

Controlling Energy and Service Cost

BalancingAvailability through Hardware and Software

Leveraging Prefabrication

Integrating On-premise with Outsourced IT

Introducing Edge Computingfor Reduced Latency

Achieving Higher Energy Efficiency

Meeting Regulatory Requirements

Leveraging Software to Automate Services

Deploying Converged Infrastructure

Designing for Unknown,Future IT

Managing IT and Facility Assets

Evaluating or Adopting Open Compute Solutions

Operating a Bimodal IT Environment that is both Stable and Agile


Increasingly Complex JobIn the future, 60% of a Data Center’s assets will be cloud or collocation based.

CUSTOMERS,

DATA, &

CRITICALITYColocation

Cloud

Services

Cloud

Services

Edge

On Premise

data

Leads to the NEED for more:

OPTIMIZATION & INSIGHT,

plus MEGA DATA CENTERS &

COLOCATION FACILITIES

More

IDC FutureScape: Worldwide

Datacenter 2015 Predictions

http://www.idc.com/getdoc.jsp?containerId=253116


How will distribution change?

●By 2025, hyper-scale Data Centers will

contain 1/3 of the world’s Processing Power

Retrofitting(or investing in)

existing DCBuilding Outsourcing

Enterprises

& Government

Cloud

& ColosSMB


Power to

data centerCooling

Lights

Fire

Security

Generator

Switchgear

Physical

infrastructure

IT Load

Power

to IT

UPS

PDU

Cabling

Switches

Etc.

Power path

to IT

Power to

other support

USEFUL

Power= Power consumed

by IT loads

SUPPORT

Power= Power consumed

by support

equipment

Losses in

power system

Goal is to

reduce this

Data Center Power Consumption

In many data

centers, less than

half the electricity

makes it to the

computer loads.


Where Does the Power Go?

Power flow in a typical 2N

data center at 50% load

Power enters the

data center as

electrical energy

Virtually all power

(99.99%+) leaves the

data center as heat.

The rest is converted by IT equipment.

PUE = estimated annual PUE


Data Center Applications Affect Efficiency

Increased Efficiency

High density or dynamic loading

applications offer an opportunity for

increased efficiency (if supported by

smart row-based power and cooling).

Reduced Efficiency

Reduced operating loads and

over-capacity of power and

cooling reduces efficiency.


Point (Computer Equipment) Solutions

Virtualization

Has the potential to reduce the total

amount of IT equipment required to

perform a specific function.

This offers a means to reduce IT-

load power consumption.

IT systems operating at higher

densities with time-varying

power draw are driving down

electrical efficiency.


Best Practices for Low PUE

“If you pick the best-of-breed commercial products and you combine that

with best practices in terms of how you design and run your operation,

you’re looking at an overhead of about .24.”

~ Chris Malone, thermal technologies architect for Google

Fix cooling first

● Close-coupled cooling

● Separate hot and cold aisles

● Efficient cooling-plant designs

● Tame energy consumption

● Heat exchanger in line with chiller

http://www.datacenterdynamics.com/focus/archive/2011/05/google-low-pue-is-easy-if-you-follow-best-practices

Optimize power distribution

and backup

● Minimize the number of

conversions

● Use efficient UPS units


Scalable power and cooling

Row-based cooling

High-efficiency UPS

AC power distribution

Variable-speed drives on pumps and chillers

Proposals to Address Waste of Power


Five Key Contributors to Inefficiency

These all contribute to SUPPORT efficiency.

1 Inefficiencies of the power equipment

2 Inefficiencies of the cooling equipment

3 Power consumption of lighting

4 Over-sizing of the power and cooling systems

5 Inefficiencies due to configuration

Each of these are analyzed in detail in APC White Paper 113 – Electrical Efficiency Modeling for Data Centers,


1.) Inefficiencies of Power Equipment

When the equipment is doubled for redundancy or operated

well below its rated power, efficiency falls dramatically.

Efficiency values are misleading and cannot be used

to calculate the power wasted in real installations


2.) Inefficiencies of Cooling Equipment

The inefficiency

(waste heat) of cooling

equipment typically

greatly exceeds the

inefficiency of power

equipment.

An increase in the

efficiency of the

cooling equipment

directly benefits overall

system efficiency.


The Goal of Cooling Distribution

To create clear paths for

cooled air and hot exhaust

air

Need to reduce or

eliminate the cold and hot

air leakage

CRAH

Front

IT

Rack

Front

IT

Rack

Front

IT

Rack

Cold air

leakage

Hot air

leakage


BYPASS AIRFLOW

● A comprehensive study

performed by Upsite

Technologies Inc. concluded

that 60% of the air supplied in

traditional data centers is

wasted due to bypass airflow

● Significant efficiencies and cost

savings can be realized by

eliminating bypass airflow.


RECIRCULATION

● Hot air exhaust circulating back into its own intake can cause device thermal overload.

● Typical manufacturer inlet temperature threshold for device operations is 35°C.

● Exceeding manufacturers operating device threshold can lead to unplanned computing system outages and data loss.


TEMPERATURE STRATIFICATION

● Significant gradient of air temperatures beyond ASHRAE TC9.9 places devices at risk of thermal overload

● Maintaining inlet temperature gradients within the ASHRAE recommended range significantly saves energy


The Goal of Cooling Distribution

To create clear paths for

cooled air and hot exhaust

air

Need to reduce or

eliminate the cold and hot

air leakage

CRAH

Front

IT

Rack

Front

IT

Rack

Front

IT

Rack

Cold air

leakage

Hot air

leakage

Need to design for Thermal

ride through.


Thermal run-away challenge in high density

source: WP 105, Data Center Thermal Run away

There is a compelling reason for exploring chilled water cooling for High density

data centers.


3.) Power Consumption of Lighting

The heat generated by lighting

must be cooled by the cooling

system, which causes the air

conditioning system to

consume correspondingly

more electrical power.

Increases in the efficiency of the lighting, or controlling lighting to be present only

when and where needed, materially benefits overall system efficiency.


4.) Over-sizing

Over-sizing occurs when the

design value of the power

and cooling system exceeds

the IT load.

The average data center is

ultimately oversized by three

times in design value.

Capital cost of power/cooling systems = $5 per Watt

● 70% is wasted

Excess electricity costs are significant when data centers or network

rooms are oversized.

● Idling loss of power system + cooling costs = 10% of power rating

See APC White Paper 113, Electrical Efficiency Modeling for Data Centers.


5.) Inefficiencies Due to Configuration

The physical configuration of the IT equipment can have a dramatic effect

on the energy consumption of the cooling system. Poor configurations:

● Force the cooling system to

move more air than required.

● Cause the cooling system to

generate cooler air than

required.

● Force cooling units into a

conflict where one is

dehumidifying while another

is humidifying.

These configuration problems are present in virtually all operating

data centers today and cause needless energy waste.


Common Ineffective Hot Spot Solutions

● Decrease the cooling system set points

● Add more cooling units

● Place large circulating fans in front of the rack


Improved Data Center Architecture –

Available and Practical Today!

A “system” makeover that combines the best elements of data center design:

● Engineering design of

individual devices

● Power distribution

● Inter-component

communication and

coordination

● Cooling strategy

● System planning

● Management tools

When all of these elements are combined as an integrated system, the performance improvement can be dramatic.


Data Center Design input…

● Uptime Institute – TIER LEVEL

● PUE

●ASHRAE – TC 9.9

● Location, Density, temperature, redundancy, water

availability, building constraints


Tier Classifications


Standard / Guideline for Data Center Design area

Confidential Property of Schneider

Electric |

Tier I

Tier II

Tier III

Tier iV

http://www.google.com.hk/url?url=http://www.tiaonline.org/&rct=j&frm=1&q=&esrc=s&sa=U&ei=CgsiVaj7EMu4uASfjIHwDw&ved=0CBUQ9QEwAA&usg=AFQjCNHFy6hcGfKqHYm7DmL5uTBQAGxTYQ

http://www.google.co.th/url?sa=i&rct=j&q=&esrc=s&source=images&cd=&cad=rja&uact=8&ved=0ahUKEwiw_JbotdjLAhWjnqYKHS_vAycQjRwIBw&url=http://dmg-az.com/hvac-links/&psig=AFQjCNEmzz3SJhSpRMGKlT61byVwM__7RQ&ust=1458877879519843


Power Usage Effectiveness- PUE


ASHRAE TC 9.9 Requirements

Temperature Range:

• in the suction section of the rack (cold aisle)

the air temperature must be at a temperature

of between 18°C and 27°C;

•hot aisles situated at the discharge section of

the rack have a temperature of between 30

and 38°C depending on the active thermal

load.


ASHRAE TC 9.9 Requirements

Humidity Range:

•Humidity should be less than 60% with

the lower and upper dew point

temperatures of 5.5°C and 15°C,

increasing the traditional allowed range

for humidity control.


Optimize the Architecture

● Turn it off!

● Reduce over-sizing

● Take advantage of technology

● Pay close attention to rated capacity

● Beware of stranded capacity

Data center efficiencies can be substantially improved when an integrated system is developed based on the following principles:


Optimize the Physical ConfigurationOptimized, integrated physical configuration should be inherent within the

system, and not tied to the characteristics of the room where it resides.

Configuration should include:

● Integrated row-based cooling

● Instrumentation to identify and warn about conditions that generate sub-optimal

electrical consumption

● Installation and operation tools that maximize operating efficiency


Commercially Available Integrated DC How does this address inefficient design factors?

Batteries Scalable

UPS

Row-based

CRAC

High-efficiency

transformerless PDU

● Power equipment

● Cooling equipment

● Lighting

● Over-sizing

● Configuration


Cost Savings by Subsystem


$0 $200,000 $400,000 $600,000 $800,000 $1,000,000

UPS

PDU

Generator

Switchgear

Distribution Wiring

CRAC

Heat Rejection

Pumps

Chiller

Humidifier

Lights

Aux Devices

IT Load

$ per Year Electric Cost @ $.10/kwhr

Improved Architecture

Traditional Baseline

In terms of PUE efficiency value*:

Traditional data center = PUE of ~ 2.5

Improved architecture = PUE ~ 1.5

*Operating at 50% of rated IT load, under

the same conditions.

Eliminating dual path power redundancy or N+1 air handlers

would cause efficiencies to rise and savings to fall somewhat.

Images available at: http://www.apc.com/tt?tt=6

Traditional Improved


How Does Improved Architecture

Achieve Savings?

The key elements that give rise to the improvements of the

new architecture include:

1. Scalable power and cooling

● To avoid over-sizing

2. Row-based cooling

● To improve cooling efficiency

● Free Cooling also provides

regionally derived cost savings

3. High-efficiency UPS

● To improve power efficiency


How Does Improved Architecture

Achieve Savings?

4. 415/240 V AC power distribution

● To improve power efficiency

5. Variable-speed drives on pumps/chillers

● To improve efficiency at partial load and on

cool days

6. Capacity management tools

● To improve utilization of power, cooling,

and rack capacity

7. Room layout tools

● To optimize layout for cooling efficiency

The key elements that give rise to the

improvements of the new architecture include:


Data center efficiency as a function of IT load comparing modular vs.non-modular designs

Scalable Power and Cooling

At lower IT loads the

efficiency always

declines and is equal

to zero when there is

no IT load.

To correct the problem of reduced efficiency due to an over-sizing condition,

the power and cooling equipment could be scaled over time to meet the IT

load requirement.

Efficiency Varies with IT load


Scaled Buildout: PUE and TCO Advantage

Savings from

scaled deployment

Capex 38%

Opex 19%

Non-energy 44%opex

TCO 33%savings

TCO

Right-sizing benefits over a 10-year data center lifetime

SCALABLE

ADAPTABLE

4.0

3.5

3.0

2.5

1.5

2.0

UPFRONT buildout – day one

SCALED buildout – as needed

1 3 4 5 6 7 8 9 102

PUE

Data center lifetime (years)

See APC White Paper 143, Data Center Projects: Growth Model


Row-based Cooling to Improve

Cooling Efficiency

In-row air

conditioner

Hot aisle air enters from rear,

preventing mixingCold air is supplied to

the cold aisle

Heat captured and

rejected to chilled water

Cold Aisle

Cold Aisle

Row-based cooling is a key enabler of cost-

effective implementation of room layout and

capacity management tools, and scalable cooling


High-efficiency UPS to Improve Power

Efficiency

LBNL report on UPS efficiency: http://hightech.lbl.gov/documents/UPS/Final_UPS_Report.pdf, Figure 17, page 23.

Efficiency gain is

greatest at

lighter loads.


415/240V AC Power Distribution to

Improve Power Efficiency

Eliminates power distribution unit (PDU) transformers and their associated losses.

Efficiency reduction of transformer-based PDUs = 2% to 15%

Larger percent losses occur in data centers operating with redundant power paths and lighter IT loads.

208 V

Neutral

Typical

Load

Typical

Load

Neutral

240 V

Adoption of the European standard of 415/240V offers significant efficiency improvement for North America Data Centers.

See APC White Paper 128, Increasing Data Center Efficiency by Using Improved High-Density Power Distribution.


Variable-speed Drives on Pumps and

Chillers to Improve Cooling EfficiencyChillers and pumps with fixed-speed motors are configured for maximum expected

load and worst case (hot) conditions, and therefore spend much of their operating

time with their motors working harder than necessary.

Pumps and chillers equipped with variable-speed drives (VFDs) and

appropriate controls can reduce their speed and energy consumption to match

the current IT load and outdoor conditions. The energy improvement varies, but

can be as large as 10% or more.


Capacity Management Tools Improve

Utilization of Power, Cooling, and Rack

Capacity

● Most data centers do not fully utilize power, cooling, and rack capacity

● Modern IT equipment = 5 to 20 kW per cabinet

● Typical data center operation ≤ 3 kW per cabinet

●Efficiency gains:

● Shorter airflow paths = less fan horsepower

● Less air mixing = higher heat rejection temperatures = improved chiller

efficiency & increased CRAC capacity

● Shorter wiring lengths = less wiring and PDU losses

See APC White Paper 150, Power and Cooling Capacity Management for Data Centers.

More IT load can be powered by the same

power and cooling infrastructure


Capacity Management Tools

5% in overall infrastructure electrical efficiency

5-10% capital savings due to higher power density

Drill in to row

or rack level

Graphic representation of

data center floor layout

Visibility to average

and peak power

usage by measuring

actual consumption


Room Layout Tools Optimize Layout for

Cooling Efficiency

● Minimized airflow path

lengths (reduce fan power)

● Minimized airflow resistance

(reduce fan power)

● Maximized heat transfer - IT

equipment exhaust air returned

directly to the air conditioner

● Balanced airflow capacities

to nearby load airflow

requirements

Room layout optimized for efficiency =


Overall Infrastructure Efficiency Gains

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% IT Load

% E

ffic

ien

cy

Redundant (Dual power path, N+1 air handlers)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

% IT Load

% E

ffic

ien

cy

Non-Redundant (Single power path, N air handlers)

Traditional data centers:

• Power and cooling redundancy reduces the overall efficiency by about 5%

Improved architecture:

• Power and cooling redundancy have a negligible effect on efficiency

Phased deployment of modular power and cooling has the

most significant efficiency benefit for data centers


Other Factors that Affect Efficiency Gains

Helpful Factors

● Dropped ceiling for air return

● Full dual-path air handlers

● Full dual-path chillers

● Energy-efficient lighting

● Powering air handlers from the UPS

● Packaged chillers or DX glycol systems

Hindering Factors

● Uncoordinated room-perimeter air conditioners

● Lack of hot-aisle/cold-aisle rack layout

● Imbalanced sizing of power and cooling systems

● Shallow raised floor (0.5 m or less)

● Large auxiliary loads

● Hot and/or humid climate

● Very long coolant pipe runs


Data Center Efficiency Calculator:

TradeOff Tools

http://tools.apc.com

● High-level planning tools

● Show actual implications of various deployment decisions

● Accurately calculate potential impacts on a data center

http://tools.apc.com/


Conclusion

● There is pressure to reduce energy consumption of data centers

● Traditional power and cooling systems are inefficient, stranded

capacity rates are high, and server utilization is low

● The users of energy have an equally important role in improving

electrical efficiency

● Data centers can begin by ”rightsizing” their power and cooling

infrastructure so that the actual IT load operates closer to full load

capacity

Why a more efficient architecture?

best practice and infrastructure guideline for on-premise ... · best practice and infrastructure...

Documents