critical design elements for high power density data centers

1

Critical Design Elements for High-Power Density Data Centers

2

Critical Design Elements for High-Power Density Data Centers

When it comes to high-power density data centers, all are not created equal. As organizations increasingly focus on heavy-duty, transactional workloads like big data analytics, they’re seeking space that can support upwards of 17kW per rack. Delivering these super high-power densities while ensuring tolerable working conditions requires careful planning and significant attention to air flow management, temperature control and electricity.

#DataCenter

3

Artificial intelligence will exceed human intellectual capacity and control, thus radically changing civilization in an event called the singularity.

50th Anniversary of Moore’s Law

#DataCenter

4

•  In 1984, the number of Internet devices was 1000

•  In 2015, the number of Internet devices

exceed 30 Billion

•  By 2020, it will exceed 50 Billion

Did you know?

#DataCenter

5

There are two main issues regarding faster processors and those are:

1.  Transmission delays on the chip

2.  Heat build-up on the chip

Transmission delays As the size of the wires and transistors have gotten smaller over the years, the time required to change states has gotten smaller, too. But there is some limit -- charging and draining the wires takes time. That limit imposes a speed limit on the chip.

HEAT - Every time the transistors in a gate change state, they leak a little electricity. The faster a chip goes, the more heat it generates.

What is the issue with Faster Processing? HEAT!

6

High power density computing is here if you can design for it

Smaller and More Servers = Lots of Heat & Power!

#DataCenter

7

Courtesy of Schneider Electric Most data centers average to be low density with some high density cabinets only.

Most efficient density range 5-8kW

Enterprise kW / Rack High Density Study

8

•  A decade ago, standard densities were 3-5 kW per rack

•  Today, it is not uncommon to see 8-12 kW per rack with some deployments reaching 15-20 kW per rack

•  For this presentation, 10-20 kW is considered as high density in a colocation facility

Colocation Data Center Power Density Trends

#DataCenter

9

High Density Benefits

Lowered Total Cost of Ownership (TCO) •  Emerson Network Power Study [2] – New Data Center

•  Compare: 5 kw / rack (400 racks) vs. 20 kW / rack (100 racks) – Low Density vs. High Density

•  Floor Space: 10, 000 SF vs. 2,500 SF •  TCO difference after 5 years is $ 3, 660, 000 less for high density scenario

assuming similar maintenance cost

Categories Description 400 Racks @5 kW/rack

100 Racks @20 kW/rack

Capital Costs Building shell, rack, PDU 3,500,000 875,000 Cooling Capital Costs CRAC, high density modules, installation 830,000 1,900,000 Annual Cooling Operating Costs Cooling energy costs 946,080 525,600 Maintenance No Data No Data No Data

Estimated TCO ( End of 5th Year) 9,060,400 5,403,000

Reduced Space @ $150-200/SF Reduced # of PDUs @ $80-100K/ PDU

#DataCenter

10

High Density Benefits

Increased Scalability •  Example: Customer starts up with 6 kW / rack, and after a few years has need

for 18 kW for their equipment

Current

Future

High Density Capable

11

High Density Challenges

•  More cooling required in smaller footprint

•  Higher cooling equipment initial costs

•  Stronger weight bearing structures required for heavier racks •  Response time in the event of a cooling failure

•  Example: If the chiller pump failure or CRAH fan fails, data center operator will have less than 1 minute for 20 kW / cab regions before temperatures exceed ASHRAE recommended temperatures

•  Solution: Feed CRAH fans and chiller pumps from UPS

•  Power distribution

#DataCenter

12

Not all racks in colocation data center are high density: equipment from the past and present creates a mixed environment

•  Can have 2 kW per rack and 20 kW per cabinet in same data center space

•  These servers have different operating requirements (e.g. rack inlet temperatures and air flow rates)

•  From a cost / watt perspective, average densities of 5-8 kW per rack is optimal [3]

Stranded Capacity (Space, Power & Cooling) •  Available floor space and racks but no remaining power or cooling

•  Air handlers have remaining cooling capacity but not enough air flow rates to match IT equipment requirement

•  PDU has remaining electrical capacity but RPPs out of breaker positions, caused by low density racks

Resource Management (Reducing Stranded Capacity) •  Quantity and location of air handlers driven based on cabinet densities in each region

•  Proper planning and book-keeping for power distribution to avoid overloading PDUS and to balance phases

•  Build out in modular pods to minimize stranded resources

Mixed Environment Challenges

13

Cooling Strategies for Mixed Environments

•  Blanking Panels & Rack Skirt •  Data Center Zoning •  Vertical Exhaust Ducts (VED) •  Cold or Hot Aisle Containment •  In Row Coolers (IRC) •  Other Strategies

#DataCenter

14

Blanking Panels & Rack Skirts

Recommended for any kW / rack for proper air flow management

12 kW / rack with and without blanking panels & rack skirts

15

Data Center Zoning (Underfloor)

Similarly, owner can build out high density cabinets in contained pods (hot and/or cold aisle)

16

High Density Cooling Solutions

•  No Raised Floor – (although it could be used for routing chilled water pipes)

•  Row Cooling (Close-coupled)

•  Hot Aisle Containment •  25 kW+ •  Scalable (add cooling units

with increased rack densities)

The challenge with any high density application is how to provide that much volume of air (both supplying cold air and rejecting hot air), typically 120-160 CFM/kW

17


•  Raised Floor Supply •  Chimney Racks •  Ceiling Plenum Return •  Cooling units can be located outside •  Up to 20-25 kW

•  Raised Floor Supply •  Fully Contained Hot Aisle •  Ceiling Plenum Return •  25 kW+ •  Needs extra space (to increase volume)

18


•  Hybrid Cooling •  Raised Floor Supply •  Row Cooling •  Hot Aisle Containment •  Ceiling Plenum Return •  25 kW+

#DataCenter

19


•  Rear Door Heat Exchangers

•  Overhead Cooling Units •  Hybrid – combination of

different cooling architectures

•  Pipe routing – raised floor, ceiling plenum

•  20kW+

#DataCenter

20


•  Direct Server Liquid Cooling •  Water can absorb about 4000

times more heat than air for the same volume

•  Supercomputers (100 kW+) •  Immersion Cooling (Dielectric

Fluid)

21

•  Minimize underfloor obstruction by planning route of conduit and piping to limit the maximum obstruction in any given area

•  Strict hot aisle-cold aisle configuration with containment when necessary

•  Balance tile flow rates to match required rack flow rates: manually or use tile with automatic dampers

•  Overhead coolers: similar to in row coolers but do not occupy floor space

•  Direct Liquid Cooling – provide water based cooling directly to chip level

Other Strategies:

#DataCenter

22

The Electrical Issue of High Power Density

•  Proper management of power cables and electrical distribution system

•  Management of air flow due to # of conduits if placed under floor

•  Flexible distribution design plan is needed

•  Number of remote power panels and their location

•  Cost

#DataCenter

23

•  The very low loads are mainly rack enclosures with wiring patch panels, switches, and hubs

•  Loads in the 1 kW range are mainly sparsely populated rack enclosures •  Loads in the 2-3 kW range are mainly rack enclosures that are populated

with typical equipment but with significant unfilled rack space •  Loads in the 5 kW range are partially loaded with 1U servers, or contain a

mix of technologies •  Loads in the 7 kW + range are extremely rare but, according to customers,

are going to become more common with the recent density increases resulting from server technology advancements

•  Colocation Facilities must provide for all the above and in INAP’s case up to 20kW/ Cabinet

Power densities in an average Enterprise Data Center

#DataCenter

24


Checklist 1.  Know your equipment power requirement (Voltage, Amps, Phase,

Primary, Redundant) needs to allow electrical design planning. If you’re a collocation provider you must prepare for all types of circuits

2.  Be sure your team understands how to manage circuits and determining overloading situations before they are installed

3.  Pre-plan your electrical architecture with your rack plan layout to manage conduit and circuit locations

4.  Design your system to be flexible to avoid stranded capacity, installation cost sticker shock and risk reduction

#DataCenter

25


Infrastructure deployment needs to consider: •  Sold / reserved customer power capacity •  Actual capacity being drawn by the racks •  Ensuring enough circuit / pole positions are available

Infrastructure topology should: •  Minimize upfront / Day 1 installation costs •  Avoid prematurely installing or stranding infrastructure •  Scale without significant cost premium to accommodate more racks & more power capacity

Customer load profiles at INAP data centers are trending towards an INCREASE in kW / Rack or Power Density (w/sq ft)

•  Increased risk of inadvertently overloading the infrastructure •  Increased risk of Customer power capacity demand out-pacing the ability to

deploy infrastructure

#DataCenter

26

The Electrical Issue

We want to ensure that our power infrastructure does not have STRANDED capacity

•  The generators, switchgear, UPS are the most costly components; we want the capacity of those to be “fully COMMITTED”

The Customer power circuit demands vary throughout the Data Hall

•  Each circuit requires a circuit breaker that uses “pole positions” within an RPP panel

•  To ensure we do not strand capacity, we need enough circuits to handle “low density” racks

•  “High density” racks require fewer pole positions and fewer RPPs

#DataCenter

27

1.  There are different SIZES of Circuits (capacity) that can be provided:

•  The table below indicates a number of different breaker configurations and the resulting kW

•  The most popular circuits are 208V/30A, 2-pole (5kW) & 208V/30A, 3-pole (8.6kW)

•  Although the normal power draw across these circuits may be lower, there is nothing to prevent a customer from drawing up to the maximum circuit load at any point in time

2.  The sold or COMMITTED capacity is always significantly higher than the ACTUAL capacity

•  If COMMITTED capacity EXCEEDS the infrastructure capacity then it is OVERSUBSCRIBED

•  Even if you monitor the actual power consumption, a customer may increase their power draw without warning

•  There is no means to automatically shed load from the RPP, PDU, or UPS


28

1.  Each Data Hall can be sub-divided into 3-zones (upper / middle / lower)

•  Initially install ONE UPS and 3 PDU pairs (6 PDUs) for a data hall •  Capacity across the total datacenter is up to 1200kW

2.  As customer racks are deployed in a given zone install additional RPPs to support LOW power density

•  Each zone initially provisioned with up to 500kW of capacity •  Up to 10 RPPs (4 panel with dual 400A inputs, RPP Option 3) per zone (5

A-side, 5 B-side) means there will be sufficient circuit capacity •  RPPs to be floor mounted to reduce length of branch circuits

3.  Monitor the overall power consumption and install an additional UPS as required

4.  As the power density in a given zone increases deploy additional PDUs and RPPs to serve that increase in load

•  Additional RPPs to be wall-mounted (2 panel with single 400A input, RPP Option 1) for easier retrofit installation

The following slides indicate a phased deployment approach

#DataCenter

29

4-‐Panel RPP (A-‐side)

4-‐Panel RPP (B-‐side)

PDU Pair (Fed by UPS 1)


30

2-‐Panel RPP (A-‐side)

PDU Pair (Fed by UPS 2)


31

3 Key Takeaways

1.  The High Power Density option is available, but for the average enterprise it remains a small footprint. Internap provides this option for all customers

2.  There are various types of cooling solutions dependent upon the actual density that needs to be addressed

3.  Electrical distribution in a high power density data center requires pre-design and proper circuit management to avoid issues

#DataCenter

32

References

1.  Clark, J. (2013, October 24). Raising Data Center Power Density. Retrieved March 19, 2015, from http://www.datacenterjournal.com/it/raising-data-center-power-density/

2.  High Density = Lower Cost ( Efficiency Without Compromise E-Book Series, Chapter 2). (n.d.). Retrieved March 19, 2015, from http://www.emersonnetworkpower.com/documentation/en-us/brands/liebert/documents/white papers/emerson network power higher density equals lower cost.pdf

3.  Brown, K., Torell, W., & Avelar, V. (n.d.). Retrieved March 19, 2015, from http://www.apcmedia.com/salestools/VAVR-8B3VJQ/VAVR-8B3VJQ_R0_EN.pdf?sdirect=true

#DataCenter

critical design elements for high power density data centers

Technology