engineering operations ratio - infrastructure masons ... · engineering operations ratio ... 3rd...

ENGINEERING OPERATIONS RATIO

EOR is a proposed methodology to show a systems operational performance in relation

to its design intent.

Distribution: Public

Released: December 12, 2016

Version: 1.0

©2016 Thought Authors & Infrastructure Masons, Inc. All rights reserved. No part of this publication may be used, reproduced, photocopied, transmitted,

or stored in any retrieval system of any nature without proper credits given to the copyright owner.

ABOUT INFRASTRUCTURE MASONS

Infrastructure Masons (IM) is a group of industry professionals who design, build and operate the

infrastructure of the digital age. Membership as of December, 2016 represented over $100Bn in

Infrastructure projects in over 130 countries. http://imasons.org

ABOUT IM THOUGHTS

IM Thoughts are papers written by members to share problems, concepts and ideas. Our End User and

Partner Members have unique perspectives and insight built from their extensive experience. The

intent of these papers is to stimulate discussion and debate to help advance the industry.

Content is owned by the author(s) who assumes all liability from opinions expressed within the papers.

http://imasons.org/pubs/thoughts

CREDITS

AUTHOR:

Oz Morales, 2nd Degree Master

CONTRIBUTORS:

Dean Nelson, Founder & Chairman, iMasons, 3rd Degree Master

Mark Monroe, Executive Director, iMasons, 1st Degree Master

http://imasons.org/

http://imasons.org/pubs/thoughts

https://www.linkedin.com/in/osmorales

http://linkedin.com/in/deannelson

https://www.linkedin.com/in/markmonroecolorado



Table of Contents

ABOUT INFRASTRUCTURE MASONS............................................................................................................................. 2

ABOUT IM THOUGHTS ................................................................................................................................................... 2

CREDITS ......................................................................................................................................................................... 2

INTRODUCTION .............................................................................................................................................................. 4

ANALYSIS ....................................................................................................................................................................... 6

CONCLUSION: .............................................................................................................................................................. 12

Appendix A – Examples of Variables to be Measured .............................................................................................. 13

Appendix B ................................................................................................................................................................... 13

ABOUT THE AUTHOR.................................................................................................................................................... 14

REFERENCES ............................................................................................................................................................... 15



INTRODUCTION

I am an avid car fan. I love cars and their mechanical and electrical components. While watching an

F1 race, I wondered if there was a metric or ratio the teams used to see how closely the cars

performed to their engineered design. I wondered how the engineering performance (thus design

Horse Power) vs. the actual performance of the cars was calculated and if there was a ratio to identify

any system degradation, how did the racing teams mathematically calculate what the performance of

the vehicles was determined and if the degradation in performance was properly vetted and

measured. I realized that through rules and design limits the cars were meant to ensure a

homogenous design regardless of manufacturer, thus the design criteria was unity or one.

I wondered how this would apply to one of my own vehicles, an aging BMW 135i, so I started thinking

about performing my own evaluation. For example, BMW specifies that the top speed of my car to be

one value, but that didn’t always match the top speed I could actually attain. Of course in Seattle, the

temperature was cool and the air density was high (this could pose drag on the car and it could

provide additional oxygen for combustion – thus good and bad). As I tracked the car, I did not attain

the engineered top speed. Yes, it could have been an underperforming driver (me) or it could be the

age and miles of the car, it’s a 2008 with 110K miles. The car did reach top RPM without attaining

top speed, thus the vehicle itself was not performing up to its engineered specifications for some

reason. It dawned on me that my car is a system made up of many components, so is data center

building. Both are composed of subsystems that make up the whole. Underperforming subsystems

could be compensated by over-performing systems and vice versa.

During my career in the data center design and operations business, I have been in a position to hear

many reasons why a building or data center was “under-performing” from the design engineers, the

operations staff, the weather, the alignment of the planets, etc. However, my teams could not provide

a metric that could show how the overall system was underperforming. The closest metric I found was

the Power Factor (PF) in electrical distribution systems, which is a clear ratio that informs how

efficiently your electrical systems are performing. The PF is often inconsequential to executive



management as the distribution losses are simply not that interesting (unless you are generating heat

and outages).

My goal with this Thoughts paper is to start a conversation on how to measure a system’s design

performance vs its operational performance. Let’s call this the Engineering Operations Ratio (EOR).

The EOR does not have a comparative analysis between components, thus It has no unit of

measurement, thus, when executing your calculation, you will need to ensure that the mathematical

units are properly managed and removed. This will ensure proper management of your equations

yielding a result that has no units of measurement. An EOR < 1 will indicate a system is under-

performing, and conversely, an EOR > 1 will indicate a system is over-performing (that may also be

worth watching). An EOR equal to 1, will be unity or performing as designed. This is not meant to

create a blame game, it is a simple calculation that can take your systems design to the next level.

EOR is a high level calculation that provides a performance comparison at any point in time. The

calculation can be a tops down view that shows the over-all system performance. For example, if a car

is designed to achieve a top speed stated of 100 mph and you are only reaching 90 mph, it is yielding

an EOR of 90 mph ÷ 100 mph = 0.9 or 90%. This is less than 1, thus the system is deemed to be

underperforming.

Over the last 15 years I have built Data centers all over the world. This has become a passion of

mine. This paper concentrates on data centers and uses examples in mechanical designs like pumps,

HVAC, etc. The hope is that this simple calculation can take your systems design to the next level.



ANALYSIS

My former bosses loved to manage with data, but I often found there was a substantial knowledge gap

between the designers, operators, decision makers, and executives. Years ago, a couple of my peers

in the data center industry, Christian Belady and Chris Malone,1 created a very useful metric called

PUE, or Power Usage Effectivenessi. This “unit-less” ratio is an effective, easy to understand metric

that requires no additional data gathering or telemetry. In my opinion, PUE is the single most effective

and globally understood ratio used in the industry. The implementation of PUE measures have

significantly slowed the overall energy consumption in the data center industry. This trend is

explained by John Koomey and other professionals in the United States Data Center Energy Usage

Report2, published in June of 2016 as a follow up to the 2011 report3, and the initial 2007 report4.

Bottom line, the PUE metric has positively affected the design of facilities and driven MASSIVE

reduction of waste in the data center. (This earns Christian the right to drive any gas guzzler he

wants!).

My goal with EOR is to provide a simple but effective way to drive data center operational

effectiveness. EOR should enable executives to easily track data center performance levels. It should

also be a useful tool for SMEs (subject matter experts) to deep dive and litmus test engineering

designs – system by system, helping them to identify underperforming and over-performing systems.

Having EOR data would allow us to double down on the good and fix the bad. The age old adage of “if

you can’t measure, you can’t control it” goes a step farther – if you can’t measure it, you can’t

compare it and therefore you have no datum line to change performance. I believe that we already

use this ratio at a high level, however, it is not easily explained nor formalized. By defining this metric,

we will spur meaningful discussion and debate on performance as it pertains to design and

1 A1 - Malone, C., Belady, C., “Metrics to Characterize Data Center & IT Equipment Energy Use”, Proceedings of 2006 Digital Power Forum, Richardson, TX, 2006

2 “United States Data Center Energy Usage Report”, Lawrence Berkeley Lab, June 2016 3 “Growth in Data Center Electricity Use 2005 to 2010”, August 1, 2011

4 “Report to Congress on Server and Data Center Energy Efficiency”, Public Law 109-431, August 2, 2007

https://www.linkedin.com/in/cbelady

https://www.linkedin.com/in/chrisgmalone

http://koomey.com/

https://eta.lbl.gov/sites/all/files/publications/lbnl-1005775_v2.pdf


http://www.koomey.com/post/8323374335

https://www.energystar.gov/sites/default/files/buildings/tools/Report%20to%20Congress%20on%20Server%20and%20Data%20Center%20Energy%20Efficiency.pdf



operations. I want this calculation to focus on the tenets below: these are the rules we should

consider applying:

1) We will not increase the amount of telemetry to extract this number

2) The number has to be unit-less; thus a ratio.

3) To keep reporting consistent, EOR < 1 always means under-performing. This means the

formula used for calculating EOR must be adjusted for metrics where “up is good,” like top

speed or power factor, or where “down is good,” like PUE or cooling system flowrate. The

formulas are:

𝐸𝑂𝑅 = 𝑂𝑏𝑠𝑒𝑟𝑣𝑒𝑑

𝐷𝑒𝑠𝑖𝑔𝑛 (𝐼𝑓 𝑎 ℎ𝑖𝑔ℎ𝑒𝑟 𝑣𝑎𝑙𝑢𝑒 𝑖𝑛𝑑𝑖𝑐𝑎𝑡𝑒𝑠 𝑏𝑒𝑡𝑡𝑒𝑟 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒)

𝐸𝑂𝑅 = 𝐷𝑒𝑠𝑖𝑔𝑛

𝑂𝑏𝑠𝑒𝑟𝑣𝑒𝑑 (𝐼𝑓 𝑎 𝑙𝑜𝑤𝑒𝑟 𝑣𝑎𝑙𝑢𝑒 𝑖𝑛𝑑𝑖𝑐𝑎𝑡𝑒𝑠 𝑏𝑒𝑡𝑡𝑒𝑟 𝑝𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒)

4) There must be a design intent with all performance numbers

5) The number should first be calculated at the highest level (for example, the designed PUE vs.

actual PUE), then at the subsystems level to give additional insight into the design

6) The number should not be a “projected” number. The number must be an “outcome” of

empirical data and thereby a number showing actual performance vs. projected data

7) Over-performing and under-performing systems should be given the same focus

8) The engineered values may change over time. The component performance will degrade as

the components age. This exercise will need to be part of a continuous audit program

9) The data gathering should be easily automated

10) The delivery of the metric should be easily understood and meaningful

These tenets should drive the focus on what you are trying to achieve – determine the efficiency of the

entire system as it compares from design to operation. An outcome may be that the design is off or

the operation needs tuning. Bottom line this will yield a “just-in-time” determination of an overall



system’s performance. Each measurement should start at the pre-determined “top” of the system.

For example, a car has an engineered top speed design. If it is performing below top speed it could be

due to the HP power rating (a subsystem), the torque rating (another subsystem), then transmission

performance, tire performance, etc. Please note that I am suggesting there should be a breakdown of

subsystems. The user can and should do a deeper dive to determine the system’s hierarchy. The

user’s engineering abilities and installed telemetry will define what they can do and how well they

know their systems. If you are able to only do a tops down system review, it will still be an adequate

high level metric. As the user goes deeper, it will require further thought and more sophisticated

analysis. Be careful of going too deep. It could take away from the overall focus and have diminishing

returns. Stay focused on the tenets. You should also consider what automation you could implement

as you conduct this analysis. Without it, measurement becomes a full time job and over complicates

the tasks.

Let’s apply EOR to a 20MW data center building example (please note these numbers are entirely

fictional and do not represent any location I may have built or developed).

Below is a top level PUE measure and its list of subsystems:

Designed Observed

Ratio

(Up is good: design/observed)

(Down is good: observed/design)

Over/under

performing

Power Usage Effectiveness 1.5 1.7 0.88 Under

Power Factor 0.8 0.65 0.81 Under

Cooling loop velocity 45 GPM 60 GPM 0.75 Under

Air Velocity (CFM) 150,000 125,000 1.20 Over

Loop Temperature (F) 50 44 0.88 Under

Room Temperature (F) 85 72 0.85 Under

From the top-level, we see that the numbers are not optimal. The performance of each sub-system is

dependent upon on how that system actually performs. When you design and install a system, there

is a design “intent”. The design intent is the theoretical system performance. Once the system is



installed and commissioning occurs, the telemetry will show how the system performs under live

operational conditions. The theoretical performance now becomes actual performance. The EOR can

now be measured. For example, Power Factor (PF) design will require special handling as will any

value that yields a lower number than design intent (loop temp, PF, water flushes, etc.). We will need

a methodology to track the results. The first challenge with the methodology is the divergent electrical

systems to mechanical systems efficiency point. A higher power factor indicates higher efficiency in

the distribution whereas a lower number (airflow, velocity, pump speed) in the mechanical systems will

indicate a better performing system. Thus, we will need to do a system by system review when we

perform our efficiency deep dive. At this point we can determine the datum line by using the

engineered design as a base. This will determine if the result is above or below the line.

For now, let’s work with the above example.

PUE:

Lower is better; use formula EOR = observed / design

EOR = 1.5 / 1.7 = 0.88

The system analysis should be done component by component. Base knowledge of the systems is

critical in ensuring this analysis works.

The key drivers should be a system by system efficiency vector(s). I am in no way suggesting we game

the system or manage the data so it “looks” good. This is a comparison of engineered design and

operating efficiency. The output should identify opportunities and lead to discussions on the design

and operating efficiency of the system – no more and no less. We should remember that often the

design and operations diverge from their intent. A single facility manager that decides he/she “wants”

to run a datacenter room a little cooler, can and will cost a corporation a substantial amount of

money. The operational effectiveness, like the engineered design, should be thoughtful with specific

success measurements. EOR should enable users to measure theoretical vs actual performance,

from the first day the site goes online, to any time a large component is put into service, to when a site



achieves “middle-age”, to determining when a system should be upgraded or retired. e.g. performance

drops below a pre-determined critical level and a system or component has reached its End of Life.

I believe many in the industry have veered away from the “pride” that comes from managing design

and operations together. While this is a “soft criteria” that cannot be measured, it often derails a

project. A juxtaposition of this example is a data center provider whom I’ve had the privilege of

reviewing one of their major data centers. It has achieved 100% availability through thoughtful design

and what the clear design requirements and outcomes (stated PUE, stated PF, stated air flow

capabilities). I recently discussed the operational design with the facility manager, specifically low

loads and how the facility is affected by the start-up loads. He proudly told me that he had met his

design PUE from day one. That interested me, as usually that is not the case for a new data center.

As I dug deeper, he showed me how he was managing his UPS loads, how he brought additional

modules online when necessary, using a simple yet effective cold air mix plenum, and again, how he

staged his cooling capacity based on the site load. His method had no measurement except the

resulting PUE – which was designed to operate at scale. His process was measurable and definitely

repeatable. As we continued discussing his operation, I realized that he had intrinsically put a process

in place to measure himself against the design intent.

Conversely, I had a discussion with different data center provider’s Critical Facility manager who had

developed his own design intent but had not discussed it with his engineering teams. He had

determined he was doing well running at a PUE of 1.7 since the facility was originally running at a PUE

of 1.8 when he took over the job. It’s important to note that this facility has seen a PUE of 1.20 in

winter and 1.4 in a highly negative entropic environment. I decided not to engage in an electrical

efficiency discussion with him.

In both cases an operational metric was established, and each facility manager established his goal.

The efficient data center had an engineered design, while the other data center determined their own

metric with limited data. Both thought they were doing a great job based on the data in front of them,

but with drastically different outcomes. One was focused on comparing the design performance to the



actual performance (EOR), the other on achieving a marginal gain against previous performance. The

later accepted an underperforming system and a significant increase in costs simply by not measuring

correctly. The impact of not measuring and running your data center efficiently has a huge financial

impact on the bottom line.

What we need to establish is a simple litmus test to ensure engineers and operators close the design

loop. To be more specific, the operators and engineers establish a process of continuous

improvement to achieve the optimal performance to the design. One that is realistic and repeatable.

That is why EOR is so important.

The Engineering Operations Ratio that I am suggesting can drive agreements on what performance

you should expect and what performance you are achieving. This is paramount to a customer/service

provider relationship. For example, if you agree to a PF of .9 and a start-up PUE of 1.2, then you

should agree to a ramp up in efficiency. This agreement will drive the right behavior if they are held to

this efficiency goal in the negotiated contract. This allows providers and users to agree to terms on

launch, ramp, and at-scale operational loads. Note that there is an old ops engineer I worked with who

may disagree with me. He strongly believes ramp load should not matter if you manage your load

profile correctly. Bottom line, a datacenter partner should agree to provide this design/operations

ratio from day one. The discussions should focus on how far outside unity their EOR is operating. An

example of this agreement is Chad Towner. He has specific goals he meets when he is designing for

his customer’s start up loads. He specifies his equipment to ramp up and meets the design criteria as

the load profile grows. Capital Power and Ascent are both top notch operators and both have come to

the same conclusion – efficiency is of paramount importance regardless of scale load design.

The Engineering Operations Ratio should extract the maximum output for the minimum amount of

effort. Slight changes to the Sequence of Operations with continued feedback and discussion

between engineering and operations must occur to be successful. To achieve optimal performance,

there must be a specific measurement tool that opens discussions between the teams. It will ensure

that operational hurdles that are impeding the ability to achieve design intent are being addressed.



CONCLUSION:

The EOR needs to be applied to each component. Each component should have a ratio, and each

component measure should adjust the master ratio. This ratio will need to be adjusted depending on

the component you are testing. The ratio needs consistent telemetry for ease of measurement. This

ratio should be applied every day to measure performance. The only way to optimize or improve a

design is by comparing the design intent and actual performance. To close the loop and ensure

additional value-add to any design, you must do a top down review of the process, evaluating if each

subcomponent is running effectively or not. Once each component is analyzed, you will identify which

area needs to be tuned to achieve the optimal Engineering Operations Ratio.



Appendix A – Examples of Variables to be Measured

Component Expected Good (above/below)

Which direction

is better?

Power Factor (PF) 1.0 <1.0 Higher

Cooling loop Temperature 44F >44 Higher

AirFlow 110 CFM/KVA <110CFM/KVA Lower

Room Temperature 85F >85F Higher

Water loop Flow rate 300gpm <300 gpm Lower

PUE 1.25 <1.25 Lower

Blow Down (times/min) 2 <2 Lower

Facility available power vs actual 80% >80% Higher

Appendix B

1. Initial determination of design EOR. When a Design Engineer is designing a new facility,

is the EOR projected? I would assume yes, but how accurately do you feel they could get

on paper vs reality. My question here is really, at what point does one establish the design

EOR? Say, for example, the DE team calculates it on paper as 1.04. Is it the validated at

commissioning? Let’s assume that during commissioning, all system are fully tested at

"perfect world scenario" and the EOR result is 1.06? Does that then become the "as

tested" or "real" EOR by which the facility would be measured? A – Yes, predetermination

would set the Engineered design at 1.06, I would track the deviation via a document or a

“correction of error document”.

2. Regional correction factor or standard error. In line with question 1, if the exact same

design is deployed in Northern Virginia and also in Dublin with the same "perfect world

scenario", and the EOR is over-performing, does that then become the "as tested" or

corrected EOR. A - Yes, empirical data would supersede design data, thereby establishing

the new base line (datum).

3. I've heard of a P-PUE or Partial PUE to individually measure the PUE of electrical (ePUE)

and mechanical (mPUE) efficiency. This is not a true design intent and it should be

considered in order to obtain clear telemetry and drive proper behavior. Nevertheless, do

you feel that EOR allows for valid and reliable independent sub-system measurement?

A – There is no such thing as a partial “ratio”, it moves away from the base measurement



ABOUT THE AUTHOR

Osvaldo (Oz) Morales

Oz Morales former Vice President for Data Center Global Services for Amazon Web Services, was

responsible for maintaining the life of all of the Amazon.com data centers around the world. He had

been with Amazon for over 16 years and started as an Architect building data centers in 2000. He

moved his way up to Director working to design and engineer mechanical & electrical platforms for all

of Amazon.com and AWS data centers. Oz applied new innovations and proven technology to deploy

into the data center space and has developed over 65 innovative project patents for efficiency and

cost savings. His patent awards range from developing strategies for low cost distribution of electrical

power for data centers to methods of airflow control and he currently has over 40 patents pending.

Currently, Oz is an independent consultant providing guidance to several Fortune 500 companies.

Prior to working for Amazon, Oz worked as a Senior Manager at Enbridge Technology Inc., an oil and

pipeline gas industry worldwide based in Canada. Oz led the SCADA team to automate control and

loss prevention of all of the countries hydro-carbon pipeline system. Oz's team implemented

automated control systems for PetroBras, Ecopetrol and Pemex.

Oz graduated from University of Alberta with an Honors BSc Eng in Mechanical Engineering, he

received an Honors BSc in Material Engineering from Northern Alberta Institute of Technology and

obtained his Honors MSc Eng in Electrical Engineering from the University of Delaware.

Besides building data centers, Oz has a passion for cars and motorcycles. In his spare time, Oz enjoys

playing hockey and reading books by his favorite authors Edgar Allan Poe and Pablo Neruda.

http://amazon.com/

http://amazon.com/



REFERENCES

1. Malone, C., Belady, C., “Metrics to Characterize Data Center & IT Equipment Energy Use”, Proceedings of 2006 Digital

Power Forum, Richardson, TX, 2006

2. “United States Data Center Energy Usage Report”, Lawrence Berkeley Lab, June 2016

3. “Growth in Data Center Electricity Use 2005 to 2010”, August 1, 2011

4. “Report to Congress on Server and Data Center Energy Efficiency”, Public Law 109-431, August 2, 2007


https://www.energystar.gov/sites/default/files/buildings/tools/Report%20to%20Congress%20on%20Server%20and%20Data%20Center%20Energy%20Efficiency.pdf

engineering operations ratio - infrastructure masons ... · engineering operations ratio ... 3rd...

Documents