hacking the conventional data center infrastructure

Hacking the Conventional Data Center Infrastructure

Presented by: Kevin Brown

Confidential Property of Schneider Electric

Commonly Stated Benefits of Open Compute

• Simplification of servers

• Consolidation of server power supplies

• Eliminate centralized UPS and move to the rack with Li-ion

• Less downtime

• Less cost

Confidential Property of Schneider Electric |

Critical questions arise about what is upstream of the rack…

• If I adopt Open Compute, what does my power architecture look like?

• Can I get 2N (i.e. tier 3) redundancy levels like I have today?

• How do I support traditional and Open Compute IT loads in the same data center?


? Answering these questions will broaden adoption…

Important to understand the cost trade-offs

• Analyzed capital cost (material cost only)

• Compared traditional and Open Compute specific architectures

• From MV switchgear down to (and including) IT power supply

• Key assumptions:

• OCP PSU/BBU/shelf costs based on design by Schneider Electric

• Traditional server PSU costs & sizing based on various IT vendors/suppliers

• Models based on 9.6MW data center, 10 kW/rack

• Costs normalized to $/Watt

Based on the content of white paper 228, Analysis of Data Center Architectures Supporting Open Compute Project (OCP)


Traditional vs. OCP-based designs

$2.77/watt

$1.53/watt

$2.08/watt

Traditional 2N

Open Compute Specific 1N

Open Compute Specific 2N

We think most will want to maintain 2N (Tier 3) redundancy

“Tiers” in a nutshell

Tier 1: Single power path to IT load; single points of failure Tier 3: Dual (redundant) power paths to IT load, concurrently maintainable


Traditional 2N power architecture today… Typical 2N design for traditional IT loads

• 2N power paths from utility to load, Tier 3(ish)

• Adds complexities (“Belts & suspenders”):

– Load bank

– Ties

– Additional UPS output breakers

• Concurrently maintainable


Open Compute Specific 1N architecture An example of a cost-reduced architecture to support OCP loads

• Aligns with the simplicity and cost-reduction mindset of OCP

• Open Compute servers with one PSU path

• Single path to the IT load

– with rack-based battery backup

– no centralized UPS

• Minimal breaker count

Traditional 2N vs. Open Compute Specific 1N


0.87

0.22 0.15 0.34 1.53

0.03

2.77

Traditional 2N

Redundancy Rack system Open Compute

Specific 1N

Power supplies

Batteries UPS

45% capex savings… but where do these savings come from? $

/ wat

t



0.87

0.22 0.15 0.34 1.53

0.03

2.77

Traditional 2N

Open Compute

Specific 1N

Power supplies

Batteries UPS Rack system Redundancy

31% savings is from redundancy differences… $

/ wat

t



0.87

0.22 0.15 0.34 1.53

0.03

2.77

Open Compute

Specific 1N

Power supplies

Batteries UPS Rack system

Redundancy Traditional 2N

1% cost adder for rack system $

/ wat

t

0.87

0.22 0.15 0.34 1.53

0.03

2.77



Open Compute

Specific 1N

Power supplies



8% savings from eliminating upstream UPS…

$ / w

att



0.87

0.22 0.15 0.34 1.53

0.03

2.77

UPS Rack system


Open Compute

Specific 1N

Power supplies

Batteries

5% cost adder for Li-ion batteries

$ / w

att



0.87

0.22 0.15 0.34 1.53

0.03

2.77

Open Compute

Specific 1N

Power supplies



12% savings from power supplies

$ / w

att

• Redundancy • Concurrent maintainability

2N (or tier 3) is still important to many data centers


Utility A

OC

32A

400A

400A

3200A

RPP

1250 A

1250 A

Utility B

32APod level

Rack level

400A

400A

3200A

RPP

1250 A

1250 A

PDU

ServerPSUBBU

ServerPSUBBU

PDU

PSU

PSU

1600A

Cooling

1600A

Cooling

MV level

LV level

G G G G G

630 A

G G GG

Open Compute Specific 2N architecture 2N simple design to support OCP IT loads


reduces complexity by eliminating unnecessary cross-ties, additional breakers

saves cost with one battery path, one raw utility path

trusts the redundant server power supplies

Compared to traditional 2N, this architecture….

Cost difference of Traditional 2N vs. Open Compute Specific 2N


2.080.22

0.01

2.77

0.160.22

0.40

Open Compute

Specific 2N

Power supplies


Architecture simplification

Traditional 2N

$ / w

att

25% capex savings…



2.080.22

0.01

2.77

0.160.22

0.40

Power supplies

Open Compute

Specific 2N



Traditional 2N

$ / w

att A simplified 2N design costs $2.37/watt… (that accounts for more than half of the 25% savings)



2.080.22

0.01

2.77

0.150.22

0.40

Open Compute

Specific 2N

Power supplies



Traditional 2N

Rack, UPS, battery, and PSU differences represent the remaining…

$ / w

att

What if I have a mix of Open Compute and traditional IT loads?


$2.77

$1.53 $2.08 $2.15

0 to 0.23

Traditional 2N

Open Compute

Specific 1N

Open Compute

Specific 2N

Simplified 2N

Premium for traditional power supplies (PSUs) depends on mix of loads

A design that accommodates both types of loads

Assumes $0.46 traditional PSU cost per watt of IT load $

/ wat

t


• Flexible architecture that allows for mix of traditional IT loads and OCP loads

• UPS is upstream to support both traditional and OCP loads

• To minimize cost, one power path with UPS, one without

• OCP loads have dual PSUs without BBUs

Example of a Simplified 2N design that accommodates both types of loads

$2.08 $2.08 $2.08

$0.07 $0.07 $0.07

$0.12 $0.23

100% OCP loads

50% OCP loads

0% OCP loads

9% premium

Premium for traditional PSUs*

Impact of load mix on cost premium

Premium for simplified architecture that accommodates both load types

Cost of Open Compute Specific 2N design

* Assumes $0.46 traditional PSU cost per watt of IT load

3% premium

15% premium

$ / w

att

PSU cost is a highly sensitive variable...


$0.077 per PSU

watt

2 PSUs per

server X

3 PSU

oversizing X =

$0.46 per IT load

watt

Our baseline assumption for traditional PSUs:

Sensitivity analysis:

We varied the PSU oversizing down to 2 and up to 4.5, or $0.32 to $0.70 / IT load watt

$2.08 $2.08 $2.08

$0.07 $0.07 $0.07

$0.04 to

$0.25

$0.09 to

$0.47

100% OCP loads

50% OCP loads

0% OCP loads

Premium varies significantly as traditional PSU cost changes

Sensitivity analysis

3% premium

8% to 26% premium

5% to 15% premium

We assume a more right-sized PSU for OCP • 12 kW of PSU watts x 2 PSUs = 24 kW of PSU for

10kW of IT load

• PSU oversizing = 1.2

$ / w

att

Freely available resources to help with planning decisions


Reference Designs • Designs to support OCP • One-line diagrams, bill of materials, layout drawings • www.schneider-electric.com/datacenterdesigns

TradeOff Tools • OCP vs. traditional cost comparison tool • Li-ion vs. VRLA TCO tool • www.tools.apc.com

White Papers • WP228, Analysis of Data Center Architectures

Supporting OCP Designs • WP229, Battery Technology for Data Centers:

VRLA vs. Li-Ion • www.whitepapers.apc.com

Key take-aways…

1. To increase adoption, critical questions about power infrastructure must be addressed

2. Today’s traditional 2N architectures have opportunity for simplification and cost reduction

3. Simplified 2N represents a small premium (3%) over OCP-specific 2N, and gives flexibility for mixed-loads



Lithium-ion Batteries

Benefits of lithium-ion batteries over VRLA batteries for UPS applications

• Higher energy density (70-260 kWh/kg vs. 15-50 kWh/kg)

• Double the life span (10-15 years vs. 4-6 years)

• 1/3rd smaller footprint

• Weighs 2/3rds less

• Faster re-charge time (1/2–1 hour vs. 6-12 hours)

• Improved cycle life (# of charge/discharge cycles) - >1000 cycles vs. <400 cycles

• Specified to operate at higher temperatures (40C) without impact on cycle life


Li-ion batteries and Open Compute

• Li-ion technology has enabled the rack-based approach

• BUT… important to separate the discussion of battery type from where the battery goes…


Where energy storage resides determines benefits:

• Centralized storage (upstream of rack)

– Flexibility to support mix of OCP and traditional IT

– Isolation of battery problems from IT equipment

– Frees up U space in the IT rack

– Minimize stranded capacity of batteries

– Less batteries to manage

– Filtering and transient protection from upstream UPS

• Decentralized storage (at the rack)

– Fault isolation

– Incremental deployment

– Maintainability

Questions?
