mission critical facilities infrastructure consulting · mission critical facilities infrastructure...

Mission Critical Facilities Infrastructure Consulting

Designing a Green Data Center – From Conception to Delivery

By Dennis DeCoster and Michael Fluegeman

Mission Critical West www.mcwestinc.com 310-791-0734

ABOUT US

PlanNet Consulting www.datacdenters.net 888-925-5551 Mission Critical West www.mcwestinc.com 310-791-0734

2

• Dennis DeCoster

• Executive Principal, MC West

• Over 25 years Mission Critical Facilities Experience

• Design, Construction, Commissioning, Peer Review

• Renowned Mission Critical Author & Presenter

• Lead Author of 2010 Large UPS Evaluation Report

• Michael Fluegeman, P.E.

• Principal, Manager, PlanNet Consulting

• 25 years of Data Center experience

• Licensed Electrical Engineer

• Planning, Programming, Assessments and Design Concepts

• Construction , Commissioning and Operations Support

WORKSHOP OVERVIEW


3

Notes to Web Viewers/Downloaders:

• This presentation is a guide tool for the 2.5 – 3 hr Workshop. We could not include all of the detail information passed on during the workshop here due to space limitations. Feel free to contact the authors with any specific questions.

• This workshop combines critical facilities design basics with latest green initiatives to give a somewhat balanced overview in short form. To cover all of these topics comprehensively would take a full week course!

D. DeCoster / M. Fluegeman

WORKSHOP OVERVIEW


4

New Energy-Efficient Data Center Planning

• Site selection

Upgrade of Existing Data Centers for Energy Efficiency

• Assessments

Design Configurations: Cooling, Power, Aisle Containment

----------------BREAK---------------

Equipment: Cooling, Back-up Power, Power Distribution, Structured Cabling, Life-Safety, Monitoring

Procurement: Design, Equipment, Construction, Commissioning

Implementation: Design review, Construction oversight, Commissioning, Cutover, Transition to Operations

Determine growth needs – Data Center Strategy – Load history may not predict future growth needs

– Growth can skyrocket or can be reigned in

– Disaster Recovery / Business Continuity (DR/BC)

– Understand what may change from past, present to future

• 1U servers? Blades? Consolidating? Virtualizing?

1U, 2U servers Blade servers

5

YOU NEED A DATA CENTER STRATEGY!

SITE SELECTIONS


6

Priorities for new Data Center Site Selection

• Cool, moderate climates best for HVAC economizing

Avoid temperature and humidity extremes

• Near clean but cost-effective power generation

Nuclear, Wind, Hydro, Geothermal, Solar, etc

• Power capacity

• Water/water treatment capacity (if water-cooling used)

• Avoid natural & made-made risks

Seismic, hurricanes, terrorist targets

• Low Operating Costs: Power, real estate, technical help

• Connectivity: Diverse & redundant pathways & carriers

SITE SELECTIONS


7

Priorities for new Data Center Location inside Building

• Ideal: Detached single-story, single-tenant, single-use

No constraints; minimized risks

• If multi-tenant:

Get as much physical & support equipment separation as possible

• If multi-use: Make the data center the priority

Top floor is best for rooftop air handlers

Top floor is best to avoid overhead plumbing risk

High structural capacity required for floor & ceiling

• Avoid basements for data center & support equipment

Flood risk

EQUIPMENT LOCATIONS


8

Heavy equipment: Generators, Chillers, Batteries

• Slab-on-grade

Lowest installation cost

Easiest service access

HVAC equipment

• Air handlers

Rooftop above cold aisles minimized ducting

Along side of data center for cross flow

• Cooling towers, chillers, water storage

Balance real-estate cost with rooftop structural cost

DC DESIGNS TODAY


9

• Brick & Mortar / Legacy

• Containerized

• Modular

• “Chicken Coops”, etc

PREFAB CONTAINERS


10

Containerize entire data center or only support systems

• Fast deployment

• Suited to remote areas with construction challenges – ISO – road approved

• Tax advantages – legally portable, “equipment”

• Re-deployable, perhaps scalable

• Stackable

CONTAINERIZED DCs


11

• PROs: – Refresh with IT ~3-5 years, Flexible, Redeployable

– Often Cost & Tax Advantages

– Custom Build seeking High PUE

• CONs: – Vulnerable to Threats (thin container walls)

– Comfort Compromises, sometimes Vendor-Locked

DC DESIGNS TODAY

2011 DESIGN DRIVERS


12

• Free Cooling / Chiller Bypass

• High Efficiency UPS

• Evap Cooling, Ultrasonic Humid

• Medium V Dist or less Transformation

• BUT…Still need High Reliability/Uptime

PUE


13

Power Usage Effectiveness (PUE)

• Lots of claims. Some say the PR Dept has hijacked the issue.

• Really gets out of control in mixed use

• Sometimes counter to virtualization effort

• Not always annualized, as it should be. KWhr in / KWh IT

• A good tool as guideline

• Old days/legacy: 1.6 to 2.0+ PUE

• Old chillers & UPS lightly loaded, air mixing, frigid rooms

• Today: 1.2 or below is possible

• Cold outside ambients, free cooling, modern UPS, improved transformation, new ASHRAE limits

• Identify Facility areas that pose the most risk

– Failures about to happen

– Limitations for future technology

• Develop a plan to address issues

– Options

– Costs

– Implementation phases, risks

– Priorities

14

DATA CENTER ASSESSMENT

• Identify IT areas that pose the most risk

– Application Level Failures including lack of redundancy

– Discrepancies between Tier of Application and redundancy of equipment / facilities.

• Single power supply on Server

• Single PDU in a Rack

• Develop a plan to address issues

15

DATA CENTER ASSESSMENT

Assessments should begin with drawings

• Full A, S & MEP accurate and as-built in CAD

– Any full discipline set w/ architectural backgrounds

– Any as-built prints, redlines, sketches

• Piecemeal drawings

– Site plan

– Floor plans

– Single line power diagram

– Mechanical schedules

– Panel schedules

– Spot check as-built accuracy

16

DRAWINGS: FACILITY

UPGRADE ASSESSMENTS

• As-built Facility Drawings

• Inventory & equipment

• Efficient space plan

• Space & capacity for growth

• Fire/Life-Safety

• Green IT strategies

• Energy Efficient Facility


17

UPGRADE ASSESSMENTS

• Incomplete Monitoring

• Unstructured Cabling

• Power maxed out or not reliable

• Hot spots or not enough cooling

• Many single point failure items

• Service requires shutdown

• Telecomm inadequate

• Poor Maintenance

• Operations lacking Capabilities

• Unmanaged change


18

UPGRADE ASSESSMENTS

• Drawings out-of-date or missing

• Equipment Inventory out-of-date

• Space plan not efficient

• Growth constraints

• Inadequate fire protection

• Water leak risk

• High risk of false EPO shutdown

• IT approach not updated to be Green

• Facility not updated to be Green


19

RELIABILITY NEEDS

• Redundancy – What level of backup, and backup to the backup do

we need?

– Redundancy lowers energy efficiency

– What component or system is more likely to fail?

– What failures can we not tolerate?

• Availability Levels – Classifications of reliability, availability & resiliency

• Change Management – Documented processes to deal with changes


20

RELIABILITY NEEDS


21

• N: the number of units or capacity to handle the load

• N+1: N units, plus one extra unit in case of unit failure

• N+2: has two extra redundant units

• N+1, or N+2: may be one system with overall failure potential (Single Point Failure - SPF)

• 2N: two complete systems, or 100% redundancy

• 2(N+1): two complete systems, with each system N+1 redundant

AVAILABILITY

With higher availability levels, cost & complexity generally increase, and energy efficiency decreases

Higher reliability and availability can be achieved at all levels with more diligent maintenance & operations


22

AVAILABILITY


23 WWW.MCWESTINC.COM

AVERAGE UTILITY DISTURBANCESEPRI DPQ Survey

Understand YOUR Utility Availability - Site Availability Depends on it

AVAILABILITY LEVELS


24

• 99% Availability -Two 9s, 85-90 hours annual downtime

• 99.9% Availability -Three 9s, 8-10 hours downtime

• 99.99% Availability -Four 9s, ~1 hour downtime

• 99.999% Availability -Five 9s, few minutes downtime

BUT…Five 2 sec downs are ALWAYS worse for IT than one 5 min down, so “9’s” are not the only answer. Still, a good guideline.

5 year Probability of Fail Analysis more Targeted

Assumes N+1 CRAC units, Generators but otherwise non-redundant. Availability includes maintenance windows/shutdowns.

25

EXAMPLE: ~99% OVERALL AVAILABILITY

26

REDUNDANCY ENHANCEMENTS

27

REDUNDANCY: HIGHER AVAILABILITY

LOAD DENSITY


28

Load density (Watts/SF or Watts/rack)

• Related to available power & cooling capacity, access floor height, ceiling height, hot/cold aisle configuration and containment, or other cooling approaches.

– Too low: <30 W/SF

– Very Low: 50-75W/SF or 1kW/cabinet

– Low: 75-125W/SF or 2kW/cabinet

– Medium: 125-200W/SF or 4kW/cabinet

– High: 200-500W/SF or 6-15kW/cabinet

– Very high: >500W/SF or 15+kW/cabinet

29

MEDIUM DENSITY

STRUCTURAL


30

Floor loading capacity

• Too Low <75 lbs/SF - typical upper office floor

• Medium 125-150 lbs/SF - minimum data center target

• High 250 lbs/SF - desirable for data center

Ceiling hanging capacity

• 50 lbs/SF

Roof loading capacity • As needed for specific equipment

EFFICIENCY, GENERAL


31

Poor: Very old data centers with lightly loaded power and cooling equipment, significant hot/cold air mixing, frigid cold Moderate: Newer and right-sized power and cooling equipment, hot/cold aisle separation with little short-circuiting, newer, efficient water-cooled precision cooling systems High: Extensive use of air handlers with air-side economizing, water side economizers or thermal storage, ducted hot and cold air (aisle containment) with no short-circuiting, high efficiency UPS technology, higher voltage power distribution.

32

FLOOR PLAN – NON OPTIMIZED

33

HOT AISLE COLD AISLE

34

FLOOR PLAN – OPTIMIZED/ HIGH DENSITY

CONFIGURATIONS

Cooling

Air-side economizing for high efficiency •Open: Use filtered outside air

• BUT: Not good in corrosive, conductive contaminant, or certain other ambients

•Keep compressors & humidity control off as much as possible

•Locate data center in cooler climates

•Locate AHUs to minimize ducting


35

CONFIGURATIONS

Cooling

Water-side economizing with or W/O thermal

storage

Evaporative cooling

Direct or indirect evaporative cooling

Chiller bypass

Water storage or redundant source may be required


36

Bigger opportunities

Relaxed ASHRAE standards…run at 80oF+ No longer necessary to maintain tight

temperature & humidity control

Water side free cooling (no chillers)

Allows for more efficient chilled water operation at higher temperature and away

from dew point

Allows for many more days free cooling/bypass

Free cooling may require use of external air handlers in lieu of CRAC units in the room

37

ENERGY EFFICIENCY: COOLING

COOLING TECHNOLOGY

Cooling

Chillers, CRACs, AHUs, CRACs w/ air-side, Legacy

In Row or rack-based cooling possible

Rack liquid cooling possible

Air side or water side economizers

Evaporative/Adiabatic

Thermal storage

Lots of improvements


38

39

CRAC, CRAH UNIT

COOLING TECHNOLOGY

Humidification

Steam – least efficient

Ultrasonic

Evaporative


40

All power into the data center turns into:

41

HEAT!

WHERE DOES THE POWER GO?

Cooling needs & growth

Cooling capacity based on power capacity

Additional heat gain from UPS, transformers, etc

Additional heat from outside, sun, air leaks

Humidity control based on outside air intake

Cooling needs for concentrated high density

Where are the hot spots?

42

CAPACITIES, CONDITIONS, NEEDS & GROWTH

Incremental opportunities

Access floor

Look for supply air blockage under floor

Perf tile location, number, % opening

Blocking of unwanted floor openings

43

ENERGY EFFICIENCY: COOLING

44

HOT SPOTS - THERMAL IMAGING

CONFIGURATIONS

Cooling Power & Controls •Redundant cooling equipment

•Reduce/eliminate power Single Point Failure risk

•Avoid single, centralized control system

•VFDs should fail to default “on” mode

•2N cooling: may not be energy efficient or cost-effective

•3N/2 “3 to make 2” using swing ATSs

•“Ladder loop” sectionalized piping to isolate failures

•In-the-pipe or tank thermal storage on UPS only


45

46

LADDER LOOP SECTIONALIZED PIPING

– Q&A

– Please be back in 10 minutes…

47

BREAK!!!

CONFIGURATIONS

Power: Long-term backup

Standby diesel generators typically required

•Paralleling Vs Independent

Avoid natural gas generators for standby?

•$$, requires over-sizing for UPS, seismic, etc.

•But good for many applications

Co-generation – Combined Heat & Power

•May/may not be cost-effective

•Prime power generation (not standby)

•Gas turbines, micro-turbines, fuel cells PlanNet Consulting www.datacdenters.net 888-925-5551 Mission Critical West www.mcwestinc.com 310-791-0734

48

• Single, parallel, System + System

• May not handle block load…test!

• Tier 3-4 EPA/AQMD affects performance

• Gas much worse than diesel on dynamic performance

• Freeze threats, bad fuel, bad batteries, etc

• Just because it passes 2 hour burn-in does NOT necessarily mean it will run for days in an ice storm

49

STANDBY DIESEL GENERATOR

50

STANDBY DIESEL GENERATOR

Many electric utilities have cash available

• Many rebate programs for Green

• Virtualization payback

• Be sure to research before beginning upgrades

– Utility typically requires before & after survey!

– Utility wants to see less efficient equipment permanently removed from location

– More difficult to earn rebates for new construction

– Challenge: combine efficiency improvements & growth in small – medium DCs

51

ENERGY EFFICIENCY: UTILITY REBATES

52

ONSITE PRIME POWER GENERATION

CONFIGURATIONS

Power: Short-term backup

Centralized UPS Vs rack mount >10kW

•Locate UPS/batteries outside data center

Remove fire / hazmat risk from data center

Remove UPS heat generation from data center

Keep UPS service commotion outside data center

Non-redundant UPS

•Provide a complete isolation maintenance bypass

•Provision for direct support of data center on generator


53

CONFIGURATIONS

Redundant UPS Configurations

Full 2N, 2(N+1) most reliable, costly; least efficient

N+1 cost-effective, efficient; single point failures

•Single input breaker/switchboard

•Common parallel point

•Single output breaker/switchboard

3N/2 or “3 to make 2”

•Provides nearly full 2N reliability for some

•May be more cost effective than 2N

•Can be energy efficient

•More complex – load balancing challenge


54

55

3-TO-MAKE-2 REDUNDANT CONFIGURATION

CONFIGURATIONS

Power Distribution

Energy efficiency

•Trending towards higher voltages: 240V 1-phase

•415/240V UPS output (Euro standard) now in US

Direct Current (DC Power)

•Dubious efficiency improvement

•Safety concerns with higher voltages

•Provides nearly full 2N reliability

Redundancy: Work backwards from server/device

•Dual-corded redundant power supplies

•High-speed point-of-use transfer switch for single cord

•A/B power feeds from redundant panels/PDUs PlanNet Consulting www.datacdenters.net 888-925-5551 Mission Critical West www.mcwestinc.com 310-791-0734

56

2N REDUNDANT UPS/PDU


57

CONFIGURATIONS-PDUS

Rack Distribution Units (RDUs)

•3-phase RDUs efficient and cost-effective but difficult to use and balance phases

•240V 1-phase can be used at 208V now

•Load monitoring with remote network monitoring

Remote Power Panels (RPPs)

•Locate in rows; feed from outside or in data center

•Branch circuit load monitoring

Trending away from traditional PDUs

•Feed RPPs directly from UPS

•May use transformer & switchboard in UPS room

•Avoid K-factor transformers – not needed & 2X cost


58

59

PDU – UPS POWER DIST

60

UNSTRUCTURED CABLING

61

STRUCTURED CABLING

• Rack Elevations

62

DIAGRAMS

• Where are your check points (policies/procedures)?

– Environmental • Guards, Bollards, Enclosures, Fences, Landscaping, Architectural

– Building • Guards, Man Trap, Sign in Procedures, Access control, CCTV (DVR),

intrusion detection (also applies to building perimeter)

– Room • CCTV (DVR), Access Control,

Rack Access Control

– Server • Firewall Intrusion Detection

63

PHYSICAL SECURITY

UPS TECHNOLOGY

•On rack, on server, on floor, or centralized •Traditional transformer design vs. transformer-less •Double conversion •Line Interactive / Delta •Offline (or “ECO”, “ESS”, “ECO-Boost”, etc) •Flywheels or UCs or Li/NiMH in place of LA batteries •12VDC to 12KV •UPS or CPS •75% Efficiency to 99% •On server energy storage – UCs, battery


64

UPS-CPS: LOTS of Choices!

UPS TECHNOLOGY

Double conversion UPS w/ wet batteries most reliable

•Efficiency now 91 - 97% at varying loads at 40 KW+

•Recent innovations allow offline modes 99% efficient • Not “double conversion” in this mode

•Batteries continue to pose reliability challenges • Reserve time SECONDARY to string redundancy

• If VRLA, always try to use N+1 redundant strings

• Go flooded if room

• If VRLA, monitor temp, redundant strings, change-out

• Flywheels, Lithium, Nickel an option for niche apps


65

UPS TECHNOLOGY

Carrier Stored Trench Bipolar Transistor (CSTBT)

•3-level rectification for improved energy efficiency

•Improves low - high load efficiency to 97.1%, double conversion


66

DOUBLE CONVERSION TOPOLOGY

AC/DC

Conversion:

Rectifier

Battery/DC

Storage

CONNECTED

CRITICAL

LOAD

DC/AC

Conversion:

Inverter

UTILITY or

GENERATOR

INPUT

Transformer(Optional)

EATON 9395 w/ESS MITS 9900 99% Eff in Offline mode 97% Eff in Doub Conv mode

67

UPS - Very Hi Eff examples

http://powerquality.eaton.com/images/product-images/more-images/9395.jpg

UPS Types – Line Interactive

Over $1 Billion now Installed*

Accepted for High Criticality Service

Lack of Freq Reg a non-issue for most

Higher Efficiency due to Power Bypass

Most rotaries/Flywheel UPS Line Interactive BUT…watch out for sharp low loading efficiency drops with some

Other issues: bearing change-outs, vacuum fail, extreme speeds, loss of magnetic air baring control, etc

Which one is right? Varies sharply with app

Flywheels themselves may or may not be efficient/green

* 2010 Large UPS Evaluation Report, includes Delta Conversion, CPS Line Interactives

69

LINE INTERACTIVE TOPOLOGY

AC/DC Conversion:

Inverter/Battery

Charger

Battery/DC

Storage

UTILITY or

GENERATOR

INPUT

CRITICAL

LOAD

FILTER / MAGNETICS

UPS TYPES

70

WWW.MCWESTINC.COM

UPS DC AlternativesFLYWHEELS

High Speed Flywheel courtesy VYCON Med Speed Flywheel courtesy Active Power

Low Speed Flywheels courtesy HITEC, Piller

UPS TYPES

Medium V UPS

•In yard instead of in building

•May save on HVAC

•Rotary/CPS, Offline Types

•82 – 96%+ Efficient, Many Bearings


71

UPS TECHNOLOGY

Energy Saving Load Testing without Load Banks

•Option with some vendors

•Avoid cost & disruption of load bank connection

Power circulates from inverter back-fed thru static bypass to rectifier with load on external bypass

•Load nearly to 100%

•Requires only ~5% additional input power for losses

•Ability to load UPS with data center load on bypass without overloading source capacity.

•Long burn-ins without wasting energy

•Battery discharge test capability with limitations


72

EASY CAPACITY TEST


73

FIRE SUPPRESSION

Pre-action “dry-pipe” sprinklers

Gaseous suppression

•Do you need it? Yes if data center not staffed 24/7

•Expensive to purchase, maintain & refill if discharged

•Innergen environmentally friendly but costly

•ECARO, Novec 1230, others

•Trending away from FM-200/HFC-227

Very Early Smoke Detection/Annunciation (VESDA)

•Inexpensive

•With VESDA, some data centers forgo gas

•Only recommended for 24/7 staffed data centers


74

75

FIRE SUPPRESSION

UPS Flameout

FM200 PUTTING OUT THE FIRE

EPO

Emergency Power Off may be required by code

•Allow for emergency power & cooling shutdown at IT equipment at exits

•Erroneous activation a major cause of downtime

•An EPO system typically is a Single Point Failure risk

Reduce risk of false EPO shutdown

•Use hinged covers over EPO buttons

•Make sure EPO circuitry is “open circuit”

•Consider adding strobes & horns to EPO cover

Consider Dual EPO system if power is A/B

•2 independent EPO circuits with dual buttons at exits; each shuts down only redundant power & cooling


78

Bad! Better

79

EPO

Best!

80

AVAILABILITY AND RELIABILITY: DUAL EPO

• Local & Remote Monitoring

– Temperature & humidity

– Power continuity from utility input thru to load

– Load levels vs. capacity & redundant failover

– Minor and major alarms, notification, escalation

– Year-round power trending for PUE analysis

• UPS KVA vs. KW

– UPS rated in kVA but typically overloads on kW!

81

MONITORING

82

KVA & KW: GRAPHIC DISPLAY

CAPACITIES, CONDITION, NEEDS & GROWTH

What, no load data available?

• Utility bills may provide utility loads

• Generator service records may provide load data

• UPS service records may provide load data

• Meter loads

– Snapshot

– 7 day load survey w/ peak demand

– 30 day load survey w/ peak demand

– Summer afternoon loads typically the highest

83

CAPACITIES, CONDITION, NEEDS & GROWTH

• Many BMS/NMS systems out there

– ETHERNET, BACnet, MODBUS, Web, etc, etc

– Get power usage/demand or PQ or both

– Custom GUIs = $$

84

MONITORING

• Circuit Loading

– Monitor at circuit breaker or plug strip

– Prevent overload and breakers tripping

– Track and trend loads

85

MONITORING

86

PLUG STRIP MONITORING

• UPS battery monitors

– May predict failures (batteries need to be exercised to get meaningful data)

– Expensive. 2nd string of batteries same cost?

– May provide false sense of security

– Monitors don’t replace service & load testing!

• Power monitoring

– Key to energy efficiency improvement

• SNMP interface in each piece of equipment

– Auto shut down software for critical systems

87

MONITORING

PROCURE DESIGN

Project Manager •Experienced in-house PM or owner’s rep (consultant) needed…or both!

•Develop performance criteria/basis of design/budget

Architect & MEP Designer Sourcing

•Qualifications, Scope, RFP, Interviews & Selection

•Traditional Design/Bid/Build

•Design-Assist

•Close oversight by PM/owner’s rep

Design Review •30%/70%/100% drawing review stages

•Major equipment purchase spec review PlanNet Consulting www.datacdenters.net 888-925-5551 Mission Critical West www.mcwestinc.com 310-791-0734

88

PROCUREMENT

Procurement /

Project Delivery


89

PROCURE CONSTRUCTION

Constructability & Cost Estimating •Constructability review at 60% design or earlier!

•Construction budget at 60% design or earlier!

Traditional Design/Bid/Build •Separate contracts for design & construction

•Engineers lead and perform very detailed design

•Contractors construct

Design-Assist •Separate contracts for design & construction

•Contractors assist engineers with design

•Better collaboration, constructability


90


Design/Build or “Turnkey” or “Single-Source”

•One contract for design, equipment & construction

•No finger pointing

•Can reduce cost and project duration, - or -, may lead to disaster with un-vetted contractor or poor choice of contractor!

•Owner may not have visibility to engineers’ concerns, peer review worth considering


91


Major Equipment Procurement •Allow contractor to purchase (& markup) with oversight

Contractor owns equipment thru construction

•Owner direct purchase

Owner assumes more risk during transit, construction, but can save significantly

PM, Owner’s rep and/or engineer must review major equipment vendor proposals/submittals &

approve prior to purchase


92

CONSTRUCTION

Construction Oversight •RFIs

•Change orders

•Punch list, resolution

•PM, owner’s rep oversight

Construction close out package •As-built drawings

•Vendor O&M manuals, warranties

•PM or owner’s rep to review & approve


93

COMMISSIONING

Develop commissioning scope; insert budget & schedule early into project and keep sacred!

•Commissioning Program Manager

Can be sourced with RFP & bidding

•Commissioning Plan

Team input & review

•Hands-on test technicians

Bring, connect & operate test equipment

Load banks & cables, PQ meters, IR thermal camera, breaker test kit, ground test kit

•Detailed step-by-step test procedures, timeline


94

COMMISSIONING

What is cost-effective Cx for YOUR Data Center?

•Factory Witness Testing?

•Independent Site Cx or In House/Vendor?

•2-4-8-12 Hr Burn-Ins?

•CB Inj Testing, FLIR, Dynamic Testing, etc

•Very Inexpensive to very expensive

•Varies with Availability target

•Only certain way to confirm it will work

•Plan on 1.5 – 3.5% of Capital Budget for Cx


95

LOAD BANK TESTING IN A LIVE DC


96

COMMISSIONING

Manage onsite commissioning •Schedule participants

•Manage safety & risk to live data center

•Work around problems

•Manage punch list to resolution

Manage critical load cutover (live DCs / loads) •(MOP/MAPs), scripts, rehearsal, multiple sign-offs

Develop detailed commissioning report •Concise executive summary

•As-tested procedures with results

•Recorded test data, analysis


97

TRANSITION TO OPERATIONS

Customized, site-specific operating manuals

•Standard Operating Procedures (SOP)

•Emergency Operating Procedures (EOP)

•Best if developed during design

•Validate during commissioning (it’s much riskier to validate procedures after the data center is live!)

Operator Training

•Customized to site-specifics

•Vendor site training limited to their equipment

•Train operators with customized procedures during commissioning!


98

TRANSITION TO OPERATIONS

Warranties

•Construction

•Major equipment

Service agreements

•Negotiate prior to warranty expiration

•Factory direct vs. 3rd party service

Avoid mom & pop service shops for critical gear

3rd party may not have access to firmware, revs

Plan periodic / annual / triennial performance testing, re-commissioning


99

• Look for potential Single Point Failures (SPF)

– “single bullet” theory

– Failures that are more likely to happen

• UPS battery failure when utility fails

• Circuit breaker nuisance tripping

– Power supplies on single corded servers

– Any failure from NIC to switch for single attached interfaces

– Application failures (without monitoring)

100

AVAILABILITY AND RELIABILITY

• ATS, even with bypass isolation

• UPS, even parallel redundant

• Single input UPS

• UPS batteries, especially VRLA

• Cooling tower make-up water

• EPO button, even w/ cover, strobe, siren, time delay

• Pressurized water pipe above data center

• Utility transformer & generator next to each other next to street or in parking lot w/o bollards

• Many more….

101

SINGLE POINT FAILURE EXAMPLES

102


103


UPS in bypass

(condition

ignored)

104

FAILURE EXAMPLES

105

condition

ignored

FAILURE EXAMPLES: GENERATOR NOT IN AUTO

Standby generator battery charger off or failed

(condition ignored)

106

FAILURE EXAMPLES

• It’s not always facilities fault

• Does the facility availability plan match my application uptime requirements

– Business continuity

• Does Facilities work with change management process?

• Is Facilities represented on the CAB?

107

IF I’M AN IT GUY WHAT DO I NEED TO KNOW ABOUT FACILITIES?

• Structured Cabling

• What are NOCC’s for?

• What is important to IT?

• Do I have any maintenance windows?

• How much advanced notice do I have to give for planned shutdown or risk of unplanned shutdown during maintenance?

• What is the cost of unscheduled downtime?

• How long does it to recover from a power failure?

• Will you inform me of new equipment?

108

IF I’M A FACILITIES GUY WHAT DO I NEED TO KNOW ABOUT IT?

Contact us:


109

PlanNet Consulting (888.925.5551) Web Sites/email: 2850 Saturn Street www.datacenters.net Brea, CA 92821 www.plannet.net Speaker: Michael Fluegeman – (714) 982-5836 [email protected] ---------------------------------------------------------------------------------------------------------- Mission Critical West Web Sites/email 6010 Pacific Coast Hwy., Suite 11 www.mcwestinc.com Redondo Beach, CA 90277 Speaker: Dennis DeCoster – (310) 791-0734 [email protected] [email protected]

Questions? Comments?

CONTACT INFO

http://www.datacenters.net/

http://www.plannet.net/

mailto:[email protected]

http://www.mcwestinc.com/



mission critical facilities infrastructure consulting · mission critical facilities infrastructure...

Documents