202 tips free

8/6/2019 202 Tips Free

1/47

8/6/2019 202 Tips Free

2/47

202 tips for performance measurement

202 tips for

performancemeasurement

practical ideas to choose, create ause meaningful performance

measures with lots of buy-in!

Page 2 of 47 www.staceybarr.com

8/6/2019 202 Tips Free

3/47


202 tips for performancemeasurementIf you have a copy of this report without purchasing it from Staceys website or withoutsubscribing to Staceys free email newsletter, Measure Up , or without receiving a copy directlyfrom Stacey, then it is an illegal copy.

You can obtain a legitimate copy by subscribing to Staceys free email newsletter, Measure Up ,which provides you with even more practical performance measurement tips twice each month,at www.staceybarr.com .

http://www.staceybarr.com/http://www.staceybarr.com/

8/6/2019 202 Tips Free

4/47


202 tips for performancemeasurementpractical ideas to choose, create and use meaningfulperformance measures with lots of buy-in!

ISBN 1-921011-01-7

Written and published by Stacey Barr.

Stacey Barr, 1999-2009

For additional information about how to make your performance measurement system a lotmore usable and a lot more useful, visit www.staceybarr.com and explore a wide range of performance measurement resources.

Contact Stacey with your feedback about this book at [email protected] .

No part of this book may be reproduced, transmitted, re-posted or duplicated in any form or by any means without thedirect permission of the author and self-publisher, Stacey Barr. If you wish to quote a tip from this book, you may doso providing you acknowledge the author where you quote that tip.

Disclaimer: The information provided is for the purpose of expanding the awareness of the reader of how toeffectively design, develop and use organisational performance measures and the author accepts no responsibilityfor the subsequent use or misuse of this information.

http://www.staceybarr.com/mailto:[email protected]?subject=feedback%20about%20202%20tips%20for%20performance%20measurementmailto:[email protected]:[email protected]:[email protected]?subject=feedback%20about%20202%20tips%20for%20performance%20measurementhttp://www.staceybarr.com/

8/6/2019 202 Tips Free

5/47


thanks!A heartfelt thankyou to those who gave me the feedback to improve this book and make it more

useful and more usable for others!

Vanessa

Suleman

Philip

Paul

Michael

LeonaJohn

Andrew

And my gratitude to those who took the time to share how 202 Tips For PerformanceMeasurement have helped them, in particular:

Stacey, I have used some of the information that youre sending me in the mezhermnt ezine,and guess what - I have used the 202 KPI tips ebook to cascade KPIs to all people reporting tome. Since then we are always above target in terms of production.

Alert Thovhokale, South Africa

I just signed up to your website and downloaded your 202 tips file. Thought to tell you howinformative I found it. At 44 pages it took some focus but I read it from top to bottom. I have apersonal view that data is of no value unless you do something with it. Your tips have given meseveral ideas on how to be a little more selective on what to measure and how to get theanalysis message across... Thought to let you know I really enjoyed the read and will be backon the site searching for other items or interest.

Steve Nolan, USA


8/6/2019 202 Tips Free

6/47


contentsabout the author, Stacey Barr....................................................................................................................7

how Stacey can help you .................................................... ................................................................. .............. 7

an introduction ............................................................................................................................................9 how to use this book........................................................... ................................................................. ............... 9 a framework for performance measurement ........................................................... ........................................... 9

phase 1: select useful measures of performance .............................................................................11 consciously adopt a performance measurement framework ........................................................ .................... 11 decide which outcomes are most worth measuring.................................................. ........................................ 11 design measures that provide objective evidence of your outcomes ....................................................... ........ 13 understand the relationships between and among measures ............................................ .............................. 14 define your measures to specify how to bring them to life............. ....................................................... ............ 15

phase 2: collect data with realistic integrity ......................................................................................17

understand your data requirements thoroughly........................................................ ........................................ 18

manage the integrity of your data ...................................................... ............................................................... 18 design cost-effective data collection processes ....................................................... ........................................ 18 forms and questionnaires should facilitate easy, cost-effective data collection.. .............................................. 20

phase 3: store your data for easy access ..........................................................................................21 standardise and organise your data to make it easy to find ................................................ ............................. 22 integrate your data to get cross-functional use out of it ......................................................... ........................... 22 capture your data in a way that preserves its integrity.............................................. ........................................ 22 access your data in preparation for analysis ..................................................... ............................................... 22

phase 4: analyse performance data to create information...............................................................24 know the purpose of your data analysis ....................................................... .................................................... 25 summarising your data first helps you get to know it........... ........................................................... .................. 25 explore your data for patterns (thats where the real answers are) .................................................................. 25 test to decide which patterns you can trust ............................................. ......................................................... 26

phase 5: present your performance measures .................................................................................27 design graphs to make them useful for decision making.................................... .............................................. 28 decide the most useful structure and content for your performance reports..................................................... 29 use layout and formatting to facilitate valid and easy decision making............................................................. 30 ensure your reporting process and tools support decision making............................................... .................... 30

phase 6: interpret what your measures are saying...........................................................................32 know when a difference is really a difference....................................................... ............................................ 32 assess if you are on target (or not)................. ........................................................... ....................................... 33

phase 7:

apply your measures to improve performance..................................................................34

use measures to review your plans .................................................... .............................................................. 35 be clear about the role of measures in decision making..................................... .............................................. 35 make improvement stick by fixing root causes ........................................................ ......................................... 36 the most useful targets dont come out of thin air .................................................... ......................................... 37

overall: implementing your performance measurement system .........................................................38 engage the right people in the right ways...................................................... ................................................... 38 integrate with other management processes............................................................ ........................................ 38 know what success will look like ..................................................... ................................................................. 39 plan it like a project................................................... .................................................................... .................... 39

references for further discovery..............................................................................................................41 in a bookstore..................................... ........................................................ ...................................................... 41 on the internet ...................................................... ....................................................... ..................................... 45

ideas for where to next .............................................................................................................................46


8/6/2019 202 Tips Free

7/47


about the author, Stacey Barr Stacey Barr is the leader of one of the worlds largest performancemeasurement practitioner networks , and a teacher and mentor for corporate planners, business analysts, corporate performance managers,and others who guide the development of meaningful, results-orientedperformance measures that focus their organisation on executing strategyand achieving its purpose.

Since 1999, Stacey has been a freelance specialist in businessperformance measurement - helping people to develop and usemeaningful performance measures to move their business or organisationfrom where it is, to where they want it to be. And now, she sees her

primary role as giving this capability to others.

Shes the creator of PuMP - the methodology that gives people the detailed practicalsteps to develop performance measures which compliments whichever framework theyvechosen, be it Balanced Scorecard, Performance PRISM, or their own.

Stacey also publishes a free twice-monthly email newsletter, Measure Up , to share simplebut powerful tips to make measurement more meaningful.

With her consulting programs, public workshops, performance measurement practitioner network and information products, Stacey has helped many organisations develop meaningfulperformance measures more easily and with more buy-in than ever before.

Her goal is to help performance measurement practitioners build their performance

measurement capability, because its one of the most critical and foundational systems anyorganisation has.

Staceys clients know her for her passion and practicality. They include many federal andstate government agencies, local government authorities, corporations, non-profit organisationsand small to medium enterprises throughout Australia and New Zealand. She also has agrowing customer base internationally, particularly in the USA, UK and South Africa.

how Stacey can help you

Staceys website, www.staceybarr.com , is renowned as a very practical and approachableresource centre for information about how to meaningfully measure performance. In addition todozens of free articles she has written, youll find information about her wide range of consultingprograms, public workshops and information products that can assist you in choosing, creatingand using meaningful performance measures: the Measures & More Mastermind Program a very affordable monthly membership

program giving you access to real-life performance measurement and management casestudies, Q&A sessions with Stacey, a print newsletter filled with practical how-to informationand a spotlight on a specific performance measure, plus more(www.measuresandmore.com )

the Performance Measure Blueprint Workshop a 2-day hands-on learning experience todiscover the 8 steps to implementing performance measures in your organisation or team(www.performancemeasureblueprint.com )

the Performance Measure Blueprint Self-paced Audio Program an audio CD learningexperience, supported by fantastic reference materials and templates, to discover the 8 steps

http://www.staceybarr.com/http://www.measuresandmore.com/http://www.performancemeasureblueprint.com/http://www.performancemeasureblueprint.com/http://www.measuresandmore.com/http://www.staceybarr.com/

8/6/2019 202 Tips Free

8/47


to implementing performance measures in your organisation or team(www.staceybarr.com/PMBAudio.html )

PuMP How-to Kits DIY step by step instructions, examples and templates to help youeasily and quickly master each step in the performance measurement process(www.staceybarr.com/productsandservices.html )

How to Make Your Strategy Measurable details the Results Mapping technique thattransforms the way your strategy is articulated and cascaded and aligned to measures(www.staceybarr.com/resultsmapping.html )

How to Design Meaningful Performance Measures details of the 5-step technique todesign great measures, even for those intangible or seemingly immeasurable results(www.staceybarr.com/measuredesign.html )

How to Define Your Performance Measures details of the technique to define thespecifications for how each of your measures will be implemented, so you have theaction plan to bring each one to life ( www.staceybarr.com/measuredefinitions.html )

How to Design Useful and Usable Performance Reports details of the technique for presenting your performance measures and performance information so it focuses itsusers on performance improvement ( www.staceybarr.com/reportdesign.html )

plus more such as personal coaching, in-house consulting and more, atwww.staceybarr.com/productsandservices.html

http://www.staceybarr.com/PMBAudio.htmlhttp://www.staceybarr.com/productsandservices.htmlhttp://www.staceybarr.com/resultsmapping.htmlhttp://www.staceybarr.com/measuredesign.htmlhttp://www.staceybarr.com/measuredefinitions.htmlhttp://www.staceybarr.com/reportdesign.htmlhttp://www.staceybarr.com/productsandservices.htmlhttp://www.staceybarr.com/productsandservices.htmlhttp://www.staceybarr.com/reportdesign.htmlhttp://www.staceybarr.com/measuredefinitions.htmlhttp://www.staceybarr.com/measuredesign.htmlhttp://www.staceybarr.com/resultsmapping.htmlhttp://www.staceybarr.com/productsandservices.htmlhttp://www.staceybarr.com/PMBAudio.html

8/6/2019 202 Tips Free

9/47


an introduction

how to use this book

If you are looking for some extra ideas , for a way to reflect on what you have been doingwith performance measurement or for a quick reference to keep at the tip of your fingers,then this little book will be worth keeping on your desk.

This book is not designed as an implementation guide for organisational performancemeasurement, but as a supplement to what you already have in place to design, build andimplement your organisations performance measurement system. If you want more specificguidelines and how to information for organisational performance measurement, let me know([email protected]) and Ill do my best to help you find what you need.

Here are some ideas for how to use this book:

If you are working on a specific aspect of performance measurement, such as designing areport, working out data requirements, or trying to decide what to measure, read thecontents page to find the most relevant topic and read the tips in that section to helpexpand or reinforce the approach you take.

Bit by bit, build up your knowledge about what excellent performance measurement is, byreading and reflecting on one tip per day or randomly flick through the pages and readwhat jumps out at you or scan your eyes over the highlighted summary words in eachtip or open a page and read a tip while you are waiting on hold on the telephone, for atrain or bus, or while you eat lunch.Share the ideas with others involved in performance measurement activities, by taking thebook to your performance measurement development/implementation team meetings or share a tip with someone you know who reports performance measures or someone whouses them.

Make yourself a coffee or hot chocolate or herbal tea and visit one of the internet sites listed in the references.

Read through the list of references in the back and take the book to your favouritebookshop and have a look at the books that sound most useful or interesting to you.

Its all about gradually growing your knowledge and wisdom about how to create fabulousfeedback that helps you and your business improve until you really shine! I hope you have funand really benefit from using this book. I had a great time writing it it always feels good toreflect on your experiences and list out all the things you learned from them!

a framework for performance measurement

This book is a collection of the many mistakes and innovations and discoveries and learningsthat I have had over 10 years as an organisational performance measurement consultant.Because there are so many of them (1001 to be exact), I have organised them into sevenchunks, each chunk being a phase in the Performance Measurement Process, or

PuMP as it is more commonly known.PuMP comprises 7 phases that collectively encompass every activity you can possiblyassociate with performance measurement:


8/6/2019 202 Tips Free

10/47

8/6/2019 202 Tips Free

11/47


phase 1: select useful measures of

performanceSelecting what to measure means being centred on the outcomes that matter most to you and your business. Define your measures by carefully considering what form the evidence of these outcomes takes.

When it comes to selecting performance measures, many people have simply brainstormedmeasures they could potentially use, looked at the data they have available to see what theycould measure, adopted measures from other organisations in their industry or measured whatothers told them to measure. None of these approaches has ever really lead to tremendouslyuseful performance measures that could claim to have enhanced organisational learning andimprovement.

The SELECT phase of PuMP is about making the selection of performance measures moreconscious and deliberate with a focus on strategic direction or the outcomes that really matter.Its topics cover:

the fundamental framework that helps to identify the types of outcomes worth measuring

designing measures so they are linked to (and provide objective evidence of) importantoutcomes

defining measures to remove ambiguity and fluff, and detailing how they will be brought tolife

consciously adopt a performance measurement framework

tip 1: When you select a performance measurement framework, check that its underlyingpremises and assumptions align with the culture and vision of your organisation . Makesure that what you measure is motivating and meaningful for the stakeholders of your organisation.

tip 2: The Balanced Scorecard by Kaplan and Norton [ref 5] doesnt tell you HOW to measure, ithelps you decide WHAT to measure . It is only ONE of many different frameworks for decidingwhat to measure. Have a look around to see what else you have to choose from: The PerformancePrism [ref 21], OPM [ref 31], Triple Bottom Line, EFQM [ref 32], ABEF [ref 33], Malcolm BaldrigeAward [ref 34], what else can you find?

decide which outcomes are most worth measuring

tip 3: Many organisations believe performance measurement is about measuring people.Performance management, performance appraisal and other terms are used for the same thing. If you want a high performing organisation, then dont measure people , measure processes andoutcomes instead. The organisation works for the people, not the other way around, as many believeand have never questioned. Dare to test your assumptions about this by reading books like Peter Senges The Fifth Discipline [ ref 8], Tom Coens and Mary Jenkins Abolishing PerformanceAppraisals [ref 1], Margaret J. Wheatleys Leadership and the New Science [ref 16] and MargotCairnes Approaching the Corporate Heart [ref 4].

tip 4: Try to pay attention when people seem to be hesitant to measure something you might even hear comments like if we measure this, it might show how bad things are or lets


8/6/2019 202 Tips Free

12/47


measure the things we are doing well. These kinds of comments are symptomatic of a few things, inparticular a lack of appreciation that measurement is not about proving, its about improving.

tip 5: Peoples reluctance to measure things can also be symptomatic of an unhealthyperformance culture, where blame and scape-goats dominate over curiosity and innovation. Do whatyou can to reinforce that performance measures are feedback about the systems and processes andbehaviours, performance measures ARE NOT feedback about people .

tip 6: If you are stuck for what to measure, try starting with listing all the stakeholder groups of your organisation (or department or process), then find out for each stakeholder groupthe 2 or 3 attributes that are currently most important to them about your organisation (or departmentor process) and are currently not performed very well by your organisation (or department or process). Measure and improve those attributes.

tip 7: A very clear, concise strategic direction is essential before you can expect to develop reallyuseful performance measures. Make sure your vision, mission, strategies and goals are all worded inrich, descriptive, sensory based language [ref 14] that evokes clear mental experiences of what the future will be like when you have achieved them. These descriptions will make it easier toselect measures.

tip 8: Kaplan and Norton introduced the term strategy mapping [ref 27] which is a method of laying out the various components of a strategic direction and examining the relationships betweenthem. The Balanced Scorecard [ref 5] is their template for identifying and categorising and linkingcomponents of strategy, but the same concept is useful even if you dont subscribe to the BalancedScorecard. The basic idea is to visually map all the specific outcomes needed to bringyour strategy to life , linking them together in cause-effect and companion relationships. Readmore about a transformational and engaging method, called Results Mapping, which makes it easyto focus on the results that matter and cascade them throughout your business or organisation, athttp://www.staceybarr.com/resultsmapping.html

tip 9: Define each outcome or result or entity before designing the measure or expectto spend a long time debating the measure and to end up with a measure that is not going to giveyou the feedback you really need. Begin with the end in mind, and Stephen Covey [ref 23] would say.

tip 10: Stocktake your existing performance measures by placing them in a matrix basedon the level of decision making they best serve (strategic, tactical or operational) and the stakeholder group they best respresent (shareholders/owners, customers, employees, partners, communities).Where are the gaps? Where is the oversaturation?

tip 11: What are your organisations values? Would improving performance based on your current performance measures mean you would be living your values , or working in conflictagainst them?

tip 12: If you want to measure the effectiveness of your projects or other changeinitiatives, and realise that on-time and on-budget measures do nothing to help (they onlymeasure a couple of dimensions of efficiency), then use tip 16: to help you develop some measuresfor how the project will impact on the business, how the project is tracking against its objectives andhow well the projects strategies are working.

tip 13: If you can help people understand processes and process thinking , they willfind it much easier to decide what to measure for feedback into the operational level of decisionmaking. They just have to define their process and assess how it impacts on the organisationscurrent strategic direction.

tip 14: Help people understand how their work impacts on others such as internal or external customers, internal or external suppliers and other stakeholders as clues for what might beimportant to measure.

http://www.staceybarr.com/resultsmapping.htmlhttp://www.staceybarr.com/resultsmapping.html

8/6/2019 202 Tips Free

13/47


design measures that provide objective evidence of your outcomes

tip 15: Remember that performance measures are objective evidence of the degree to which an

outcome is occurring. How much objectivity or subjectivity do your measures have? Whatkind of evidence forms that basis of the measures values?

tip 16: If you want performance measures to be owned and used, then the people to ownthem MUST be involved in the processes of choosing the measures and defining them.

tip 17: If you have identified a really useful measure, make sure you also identify which personor people will be responsible for using the measure. Measures have to be used if they are goingto lead to improvement.

tip 18: What do you do if you believe something cant be measured ? Well dont give upstraight away most people dont realise you can just follow a simple process to come up with somepotential measures fairly quickly. First, describe the outcome to measure. Secondly, make it sensoryspecific detail how it would look, feel, sound if it were happening. Thirdly, brainstorm all the differentthings you could count (the evidence) that would prove the outcome was occurring. These counts areyour potential measures. Choose the most direct, objective and feasible. You can read more about avery logical, powerful and engaging Measure Design technique athttp://www.staceybarr.com/measuredesign.html

tip 19: Often people will say we cant measure that because we dont have the data for it. Be warned if you dont start trying to measure important things, you will NEVER have the datayour business needs. Performance measures are one of the ways that decision makers articulatetheir data needs to the IT people who are trying to design information systems that align withbusiness decision making.

tip 20: Are you trying to measure too much? Any one person can really only effectively focus onat most 7 2 (that is, between 5 and 9) [ref 14] performance measure at any one time. More thanthat and no one measure will likely be managed well at all.

tip 21: If you like TOC (Theory of Constraints) thinking, then you might connect with the approachto measurement that Eli Goldratt offers. It is simply based on using three measures of businessperformance, Throughput, Operating Expense and Inventory , as the basic building blocksof management information.

tip 22: A lead indicator is a performance measure that gives you forewarning of an impendingchange in another performance measure (the lag indicator). Usually they are measures of results or outcomes of activities or steps that occur in the early stages of a process (or chain of events) thatproduces the result or outcome tracked by the lag indicator. To find useful lead indicators , youneed to flowchart or map or diagram the chain of events that lead to your lag indicator and test whichsteps in this chain are the most influential on your lag result.

tip 23: To be useful, you need to track your lead indicators with greater frequency thanthe lag indicator they predict.

tip 24: Dont aim for a perfectly complete set of performance measures before you start bringingthem to life. Instead, adopt an action learning process whereby you focus on a small number of measures first, bring them to life (which will iron them out a little more) and then use them to see if they add the value you anticipated. Learn from this entire process and use this learning to bring to lifeand use another set of measures.

tip 25: Its better to have an imperfect measure that captures the essence of what you aretrying to measure than not measuring it at all. Using the action learning approach ( tip 24:) so you canbuild some momentum and accelerate your learning.

http://www.staceybarr.com/measuredesign.htmlhttp://www.staceybarr.com/measuredesign.html

8/6/2019 202 Tips Free

14/47


tip 26: To get a head start on selecting measures, do a little industry research to find out other best practice organisations are measuring things. But DO NOT adopt any measure unlessit directly supports your strategic direction! It actually helps if you have formulated your strategic goalsor objectives before doing this kind of research.

tip 27: Avoid so-called measures such as completion of plan by June 06 or new IT system inplace. These are milestones or events and not measures at all (nothing is being counted over timeand there is no real comparison to guage change). They are not really outcomes but more thechosen means toward the outcome. Many an IT system has been put in place and not delivered itsintended outcomes! Always measure outcomes, not events or milestones .

tip 28: Whatever you do, dont rely on brainstorming as a method for coming up withmeasures. Its too easy to go off track or end up listing potential measures that are no better than anymeasure you have ever used in the past. The best measures are measures that provide direct andobjective evidence of the outcomes you want to track, so use tip 16:.

tip 29: Involve some people from outside the area of the business you are trying to measure tohelp you get a fresh perspective . Encourage them to ask dumb questions that challenge your current thinking and reframe the ways you look at the purpose and outcomes of that area of thebusiness.

tip 30: Criteria for deciding whether or not to take up a measure can really help clarify theselection process. Criteria might include: strength of alignment to strategic direction, cost-benefit of data collection, degree of influence you have over it, level of complexity or understandability, etc

tip 31: When you choose a measure to track a particular outcome, ask the question whatkinds of behaviour could this encourage people to choose ? as a way to check thatmeasuring the outcome wont create any other performance problems (you know, you measure cycletime and everyone does their best to work faster at the expense of reliability of their work or quality of their relationships with customers or other stakeholders).

tip 32: Its becoming common knowledge now, but check that you have a balance betweenmeasures of finanacial performance and measures of non-financial performance .Financial measures are very lag, and make it hard for you to anticipate problems before theyeventuate.

tip 33: It is possible that you might end up with a set of measures that track your current strategicdirection AND a set of measures that track business as usual , which are outcomes you want tomaintain or keep an eye on even though you have no current intention to improve them as part of your strategic goals.

tip 34: Do people in your organisation have a tendency to measure the easy stuff ? Youknow, the number of inquiries, the number of transactions processed, the number of calls answered,

and so on? These are activity measures, and while somewhat useful for workload management, theyare not outcome measures and thus do not drive performance improvement. Often you have littlecontrol over them anyway.

understand the relationships between and among measures

tip 35: Map the linkages between measures used at different levels of decision making inyour organisation, to show the line of sight of operational measures up to tactical measures andthen to strategic measures. This measurement map will be useful to see at glance what is measuredthroughout the organisation and how integrated the set of measures is. Its very similar to ResultsMapping, where you link and cascade performance results throughout your business or organisation,and you can read more at http://www.staceybarr.com/resultsmapping.html

tip 36: Your measurement map ( tip 35:) can act as a road map for cause analysis andstrategy implementation . If you are responsible for a small collection of measures, then look at

http://www.staceybarr.com/resultsmapping.htmlhttp://www.staceybarr.com/resultsmapping.html

8/6/2019 202 Tips Free

15/47


the relationships your measures have to others. Which measures are counting on you to improveyour measures before they can improve? Which measures would you look to for leverage to improveyour measures? Which measures should you be wary of inadvertantly affecting through improvingyour own?

tip 37: Your measurement map ( tip 35:) can help everyone in the organisationunderstand how they influence organisational success and sustainability . Byunderstanding which measures they directly affect, they can follow the linkages through to thestrategic level to understand why their improvement actions matter.

define your measures to specify how to bring them to life

tip 38: Define each of your measures so that it is clear what they really mean. Consider writing a description, their intent, who owns them, how they are calculated, what data items arerequired to calculate them, how they will be charted and reported, and what kinds of responsesshould be taken depending on which signal the measure gives.

tip 39: It helps to have a single, standard, organisation-wide template for definingperformance measures. It will make it easier to manage all the measures and more efficient to reportthem (see phase 5:) . Read more about the step by step framework for defining your performancemeasure details, so each measure has an action plan to bring it life and report it, athttp://www.staceybarr.com/measuredefinitions.html

tip 40: When you define the intent of each measure , think about what business questionyou are trying to answer with it, or what action you cant take unless you have this information.

tip 41: When you define the calculation formula for each measure , try to be asmathematical as possible to identify each data item that will be needed. For example, [sum(satisfaction_rating) / count (respondents)] is better than average customer satisfaction. This makesthe sourcing of the data items less ambiguous and thus less likely for your measures integrity to becompromised.

tip 42: When you define the owners of each measure , consider different types of ownership, such as the owners of the data items, the owners of the defintion itself and the owners of the performance area being measured (usually the one who uses the measure).

tip 43: One way to check whether to invest in a particular performance measure is to describeexactly how it will be used and interpreted if you cant see any action or decision beingtaken as a result of using the measure, then theres probably not much point bringing it to life. Thetime to do this is when you are defining the measure, after you have defined its intent ( tip 40:) ,calculation ( tip 41:) and owners ( tip 42:) .

tip 44: When everyone has an opinion about the "shoulds" of a measure as you try todefine it, there's a good chance that you haven't converged on the specific entity you need tomeasure.

tip 45: Keep your performance measure definitions in a dictionary so people cankeep up to date on the meaning of each measure, keep track of changes made to measures, quicklyidentify which measures are currently most important and have a specification for how to consistentlycalculate and report the measures. Make sure your dictionary is based on a clear and completemethod for defining each performance measure, like the one described athttp://www.staceybarr.com/measuredefinitions.html

tip 46: I use a Microsoft Access database for performance measure defintiondictionaries which I share with clients. It makes it easier to search for specific measures, track

them using a unique identifier and produce action plans for data collection and sourcing from the dataitems section in the definition.

http://www.staceybarr.com/measuredefinitions.htmlhttp://www.staceybarr.com/measuredefinitions.htmlhttp://www.staceybarr.com/measuredefinitions.htmlhttp://www.staceybarr.com/measuredefinitions.html

8/6/2019 202 Tips Free

16/47


tip 47: Make a very conscious decision about where you write your performance targets :in your performance measure definitions, in your performance reports or in your business plans? Aconscious decision means you have a clear rationale or reason.


8/6/2019 202 Tips Free

17/47


phase 2: collect data with realistic

integrityThe process of collecting data is critical to its integrity and can be very resource intensive. Its worth giving serious consideration to how you will go about it, so that your data can be fit for purpose.

Its too easy to limit your choice of performance measures just to the data you already haveavailable. Its a quandary, because we all know that the only way we can get the data we reallyneed is to ask for it. Thats why the COLLECT phase of PuMP is so important. It provides themechanisms for how to collect the data we dont already have, but really, really need.

Using performance measure definitions as the starting point, the COLLECT phase focuses ontopics including:

defining very specifically and clearly the data requirements for each measure

understanding and managing data integrity

designing and implementing cost-effective processes for collecting data

designing forms and questionnaires that support data collection and preserve integrity


8/6/2019 202 Tips Free

18/47


understand your data requirements thoroughly

tip 48: Check your performance measure definitions to see how frequently the data itemsfor each measure are required . Are the measures calculated daily, weekly, monthly, quarterly or annually? Can your data collection processes deliver data at the required frequency, or do you needto revise the measure? If you dont have clear and complete definitions for your measures, then learnhow to at http://www.staceybarr.com/measuredefinitions.html

tip 49: Tabulate a summary of your data item requirements by collating them from your performance measure definitions (see tip 38:) . List the data item names, descriptions, whichmeasure(s) they are required for, the source system (such as a database, if one exists) and whether or not each data item is currently available and has sufficient integrity.

tip 50: Where your data item requirement summary ( tip 49:) shows you need data that is notcurrently available , you will probably want to consider designing a data collection process to makeit available. Read on for more help!

tip 51: Where your data item requirement summary ( tip 49:) shows you need data that doesnot have sufficient integrity , you will probably want to consider improving its data collectionprocess to make build in more integrity. Read on for more help!

manage the integrity of your data

tip 52: It might help to ask users of your performance measures how much integritythey require of the measures (not how much they would like!). What percentage or amount of error can the measure have before it becomes dangerous or useless? For example, is 5% too much error,or is 10% still okay? This will depend on the importance of the decision being made and howsensitive it is to small changes in performance levels.

tip 53: Dont rely on data from volunteer surveys, like mail surveys. Response rates of 9% (for example) can not be trusted to give you objective information.

tip 54: Just because you got a response rate that was unusually high for this type of survey doesnt mean you got reliable data!

tip 55: Good survey data can only come from a randomly selected sample . Otherwise your data is biased. What happens if you make a decision based on biased data? You usually get differentresults to what you need or expect, and that almost always means wasted money, time and effort.

tip 56: A good sample size IS NOT a particular percentage of a population. If you have 1000employees and you want to survey their satisfaction, a 10% sample size is a purely arbitrarydecision. Your sample SHOULD depend more on how much variation in responses you are

expecting (the more variation, the bigger the sample size) and how reliable you want the estimates tobe (the more reliability you want, the bigger the sample size).

tip 57: Survey statisticians and market researchers will have a formula that will help youcalculate your ideal sample size based on what you are trying to measure through the survey.

tip 58: You can actually measure the amount of integrity your existing data has , by wayof a data audit. This usually involves taking a sample of records or data items from your databasesystems and checking them against the original sources (such as people, forms or objects) for suchthings as completeness (e.g. missing values), accuracy (e.g. typing errors) and precision (e.g. theamount of detail). Ask your business or quality auditors for advice.

design cost-effective data collection processes

http://www.staceybarr.com/measuredefinitions.htmlhttp://www.staceybarr.com/measuredefinitions.html

8/6/2019 202 Tips Free

19/47

8/6/2019 202 Tips Free

20/47


forms and questionnaires should facilitate easy, cost-effective data collection

tip 72: Have you looked at the design of forms used to collect data in your organisation

lately? Look for any opportunities to make them less ambiguous, simpler, laid out in a way that iseasier to navigate and use. The design of your forms affects the integrity of your data like youwouldnt believe!

tip 73: Make sure that each field or question on your forms or questionnaires focuses on a singleconstruct. For example, instead of asking Are our products exciting and useful?, ask Are our products exciting? and Are our products useful?. This will reduce the ambiguity in your formand thus improve the integrity of your data.


8/6/2019 202 Tips Free

21/47


phase 3: store your data for easy access

Where and how you store your data directly determines what data you can access, when and how quickly you can access it, how easy or difficult it is to access and how much cross-functional use you can get out it.

PuMPs STORE phase is about how your organisation captures and retains and manages andmakes available the raw data that it captures. For performance measurement to work well,timely access to a diverse selection of data is essential, so data management systems need tobe designed in a way that makes it easy to find the data you need, and easy to extract it to.

The STORE phase of PuMP addresses the topics of:

standardising data items across the whole organisation

organising data into logical tables or repositories

being able to link data from different tables or repositories together, to answer those crossfunctional questions

accessing data in preparation for analysis


8/6/2019 202 Tips Free

22/47


standardise and organise your data to make it easy to find

tip 74: If you are collecting new data, consult with your IT person or team before buying or building a new database system. Try to integrate with the database systems you already have.

tip 75: A data dictionary can be very useful when designing new measures . Like amenu, it lists all the data items available in the organisation, where they can be found, what formattheir values take, how regularly they are collected, and so on. Very useful if you want to avoidduplicating your data collection efforts.

tip 76: Avoid keeping your organisations data stored in informal, independent systems likespreadsheets or applications custom made for specific projects. You get a much higher returnon your data when it can be shared and used around the organisation.

integrate your data to get cross-functional use out of it

tip 77: Many organisations have trouble linking data across organisationalboundaries . This is often the case when there are no standard conventions for uniquely identifyingdata entities like assets, customer orders, employees or cost items. If you wanted to know the cycletime of processing customer orders, youd have to be sure that the code used to name each order bythe sales people who receive the order was the same coding convention used by the people whofulfil the order. If not, you cant match up the received date with the completion date of each order without a LOT of manual work.

tip 78: Getting access to data is equally important as capturing it. The way that data iscaptured in a database system or other format (such as a spreadsheet), limits how easy it will be topull it out again so you can analyse it and create the values of your performance measures.

capture your data in a way that preserves its integritytip 79: Data entry is one point where data integrity can be compromised . Are thereways that you can get data entry software to automatically detect some of the data entry errors thatoccur, such as values out of range or values not in a predefined list?

tip 80: What are your organisations protocols for protecting confidential data ? Do theyprevent you from sourcing data for a particular performance measure, or are you able to still calculatethe values of your performance measure without compromising confidentiality?

tip 81: Who has access to data and who doesnt is an important to decision to make, inorder to prevent the datas integrity from being comprised and to manage confidentiality.

access your data in preparation for analysis

tip 82: How much historic data is kept before being archived is one factor that can really thelimit the quality of analysis you can do on your performance data. Most analyses of time series data(the majority of performance measures are of this form) require a minimum of 20 to 30 consecutiveperformance values. This means for a typical monthly measure, you will only get a good qualityanalysis if you have 2 to 3 years worth of data.

tip 83: Sometimes the process to actually extract or source the data for your performance measures can be too difficult or technical without the help of an IT expert.Consider working with the experts to create some simple macros or queries that you can easily runeach time you need to calculate the latest value for your performance measure. Better still, how canthe calculation of your performance measures be automated entirely?


8/6/2019 202 Tips Free

23/47


tip 84: Data analysis (to create your performance measure values) almost always requires thatyou take (copy, really) a chunk of data out of its source system and put it into the analysis packageyou will use. Think very carefully to understand the format you need the data in to analyseit before you extract or pull that data out of its source systems. For example, do you want transactionlevel data, or summary level data such as weekly totals.


8/6/2019 202 Tips Free

24/47


phase 4: analyse performance data to

create informationAnalysis is the process of turning raw data into information. Make sure it is the most appropriate information by adopting the simplest analysis approach that can produce the information in the form required to answer your driving questions.

Raw data is virtually never useful for assessing the performance of a process, system or organisation its just too detailed. Because of natural variation, more information comes from

patterns in datasets than from individual points of data alone. Thats why the ANALYSE phaseof PuMP is so important it is the process that turns the raw data into the information our performance measures are intended to provide.

There are a few important topics associated with the ANALYSE phase of PuMP:

being clear about the questions you want the measures and their analysis to answer

understanding the different processes of analysis (summarising, exploring, explaining andpredicting) and when to use them


8/6/2019 202 Tips Free

25/47


know the purpose of your data analysis

tip 85: Write down the questions your performance measures and information are supposedto answer before deciding what kind of statistical or analysis technique will be most appropriate.

tip 86: Find out who will be the users of your performance measures and ask themwhat questions they are wanting the measures to answer.

tip 87: Find out the questions that users of your performance measures have and areexpecting the measures to answer for them, and then choose analysis and graphical methods thatbest answer these questions.

summarising your data first helps you get to know it

tip 88: Before you analyse your performance data, have a closer look at it. Check for whackyvalues that might be errors or typos, check that the amount of data is typical for that period, and if youfind anything weird, check it out with the people that collect the data. This is called cleaning your data and will help you maintain the integrity of your performance measure.

tip 89: One of the first things to do in data analysis is to get to know your data by summarising itby calculating simple statistics like its average, minimum and maximum values, median, mode, 25 th and 75 th percentiles and standard deviation. These simple statistics help you understand theshape of your data, so you are less likely to be mislead by the mean.

explore your data for patterns (thats where the real answersare)

tip 90: A mean or average doesnt really tell you enough about the level of performance.Accompany a mean with other statistics that give you a feel for the shape of variation in the level of performance, such as the standard deviation, the range, the mode or median, or the quartiles (the25 th and 75 th percentiles). You can find out more about these statistics in any good business statisticsbook or by looking for statistics pages on the internet.

tip 91: Avoid using straight counts or tallies as the values of your performance measures(such as the number of transactions processed). Performance is almost always a relative thing, andthus you get more information when you use rates or percentages or averages (such as the number of transactions processed as a percentage of the total number of transactions received, the number of transactions processed per employee).

tip 92: Remember that performance measures are most useful when you are tracking a

specific result over time in a regular way. This will help you understand the dynamics of performance levels, that is, how much the level naturally varies over time, when the level is changing,and when something abnormal happens.

tip 93: Analysis of performance data is more than just computing the values of your performancemeasures its also about providing a context or explanation around your performancemeasures . Consider supplementary information like a Pareto analysis of the potential causes of achange in performance, or a breakdown of your performance measure by some relevant classifyingfactor such as region, customer segment or business process. These supplementary analyses mayrequire you collect a little more data (see tip 70:) .

tip 94: If you want to explore causal relationships between measures, then try techniqueslike scatter plots or correlation or regression analysis. These analyses test the strength of therelationship between variables, but beware: just because there is a relationship, doesnt mean it is acausal relationship! Use your common sense and curiosity to inquire further.


8/6/2019 202 Tips Free

26/47


tip 95: Did you know that you need a bare minimum of 20 consecutive performancevalues before you can even begin to do a reliable trend analysis? In fact, youll often need evenmore than 20 before you can see the big picture pattern of variation in your performance values.

tip 96: If you are tracking your performance measures over time, then statistical process

control charts will be really useful for you they make it really straightforward to decide howmuch the level of performance naturally varies over time, when the level is changing, and whensomething abnormal happens. A little more information about SPC charts are provided in tip 97: and .

tip 97: When you have your minimum of 20 consecutive performance values, and they show areasonable amount of stability over time (i.e. no big changes or shifts), then calculate the mean(average) of these values and put a straight, horizontal line for this mean through a time series chart(line chart) of your performance values. This mean line is a good benchmark for deciding whenperformance really is changing or not . Dont recalculate this mean line until you see asignal. The following tips tell you about the signals.

tip 98: If you have implemented tip 97: where you added a mean line to the time series chart of your performance measure values, then you might like to add some additional, useful benchmarkinformation, called control limits. The control limits show you how much variation you canexpect from your current level of performance , unless something fundamentally changes(like you change the process producing that performance). The control limits are simply calculated as3 standard deviations above and below your mean line.

tip 99: After getting to know your performance data ( tip 89:) , looking for patterns is often whathappens next. To explore your data for patterns , try analysis methods like scatter plots, linecharts, bar charts, box plots and correlation coefficients.

test to decide which patterns you can trust

tip 100: When you have found some patterns in your performance data ( tip 99:) that you would liketo draw some conclusions from, you can test the significance of patterns using techniques likeregression analysis, analysis of variance, and other tests of comparions like t-tests.

tip 101: If you just cant live life without linear trend lines (I suggest you should find a way), thenalways report the R 2 (r-squared) statistic with the trend line . This statistic takes a valuebetween 0 and 1 (or 0% and 100%) and effectively tells you just how good your linear trend line is atconveying the message in the data. The lower the R 2 value, the less confidence or trust you shouldplace in your trend line as any kind of useful conclusion about what performance is doing.

tip 102: If you have a theory about a potential lead indicator for a performance measure (such ascycle time as a lead indicator of customer satisfaction, or customer satisfaction as a lead indicator for profit), find out more about how correlation analysis can help you test the lead indicator strength in predicting what the performance measure will likely do in the future.

tip 103: Be wary of data mining applications. They dont replace the need for performancemeasurement (well, not yet anyway). Performance measurement needs to be a very consciousprocess, where you deliberately seek the answers to business questions around the achievement of goals. Data mining is a more exploratory method of analysis to surface potential trends or patternsyou were otherwise unaware of. These trends or patterns may not necessarily mean anything, either.


8/6/2019 202 Tips Free

27/47


phase 5: present your performance

measuresIn communicating performance information, you are influencing which messages the audience focuses on. Take care to present performance measures in ways that provide simple, relevant, trustworthy and visual answers to their priority questions.

This phase of PuMP is about how you design graphs and visual reports that presentperformance measures to their audiences. Its important to be aware of how they will use the

measures, how to make the measures easy to interpret and how to not overload the audienceswith too much detail!

The topics relevant to the PRESENT phase of PuMP include:

choosing the right graph type

formatting graphs to ensure integrity, simplicity and a focus on patterns, not points

designing performance reports that are easy to access, easy to use and targeted at theneeds of those that use them


8/6/2019 202 Tips Free

28/47


design graphs to make them useful for decision making

tip 104: Dont take your graph design for granted ! The design of your graphs has a HUGEimpact on how well they are used, IF they are used, and the validity of the conclusions users drawfrom the data they present.

tip 105: The purpose of graphs are to make huge amounts of quantitative information more easyto digest. So, keep your graphs as simple as possible or they wont be adding anything usefulto your ability to use your performance measures wisely and effectively.

tip 106: To keep your graphs simple, try getting rid of what Edward R. Tufte [ref 21], a visualinformation expert, refers to as chartjunk all the stuff on your graph that uses ink but doesnt adduseful information. Typical chartjunk includes grid lines, extra tick marks, long titles and explanations,self-explanatory legends, borders, kooky fill patterns (like checkers or stripes).

tip 107: If you use gridlines in your graphs, ask yourself the question, what purpose are do thegridlines serve ? If it is to encourage users to focus on the value of individual points of data, then

why do you have the graph? Graphs are about highlighting patterns and signals and are notsupposed to be another form of a table.

tip 108: If you must use them in your graphs, make sure your gridlines are very fine and lightin colour so they can serve their purpose without dominating the data.

tip 109: Do you need that many tick marks ? Are they really helping, or are they just cluttering upthe graph?

tip 110: Graphs are not a canvas for artwork , they are a canvas for patterns and signals inquantitative information. Little piles of coins dont make useful bars in your barchart of revenue or expenses.

tip 111: Avoid using 3-dimensional charts as they have a tendency to confuse the eye withthe addition of sloped lines to produce the 3-D effect. Nice straight horizontal and vertical lines are thebest for visual comparisons.

tip 112: There are some useful things to include in your graphs to help users interpret andunderstand what they are looking at, like a title or axis labels that clearly explain the measure beingcharted, a legend if you are reporting more than one variable, a footnote that clarifies the scope or exclusions of your measure and sometimes even the source of the information or data is helpful.

tip 113: Do you ever chop out the middle or chop off the bottom of your scale on thevertical axis? So instead of starting at 0, you have your axis starting from something like 350 (for example)? This practice over exaggerates any patterns or signals in your data, and may lead usersto draw invalid conclusions.

tip 114: Avoid using pie charts to report performance . Pie charts are very good for prettying up your annual reports or marketing material, and sort of good for showing how big a singlepart is relative to its logical whole, but are pretty much awful at helping you make performanceimprovement decisions.

tip 115: Bar charts work best with between 5 and 9 bars and for the purpose of comparingthe size of elements to one another. Avoid using bar charts for time series data they just look likemountain range sillouettes, not trend information.

tip 116: If you have charts that have stacked bars, grouped bars, area charts or line charts withmore than 3 lines on them, you have information overload. It will take users quite a while to figure outwhat the chart is trying to achieve, and they will be forced to look at individual numbers to figure out

trends or patterns. One chart per performance measure .tip 117: To make your statistical process control charts look a little less cluttered , trymaking your mean line light grey and solid, and your control limit lines light grey and dashed, both


8/6/2019 202 Tips Free

29/47


without symbols. The line for your performance measure values can then take pride of place by usingcolour and symbols to your liking.

tip 118: Consider designing a standard graph template for each graph type you will use todisplay your measures. A standard look and feel helps the stengthen the focus on the information,and minimise the distraction of formatting differences. Standards like these can be appliedcorporately. Read more about how to make your graphs useful for decision making, and get atempalte of sample graphs that work, at http://www.staceybarr.com/mgu.html

decide the most useful structure and content for your performance reports

tip 119: How do you report your performance measures? Have you taken a conscious approach todesigning how performance measures are collated and arranged and narrated in performancereports? What is your rationale for how performance measures should be reported or displayed to their users?

tip 120: Design the contents page of your performance reports before you designanything else this will help you set up a useful and usable structure for the report before you get lostin the detail. Did you know that the contents page of a performance report can become an executivesummary? You can read more at http://www.staceybarr.com/reportdesign.html about my how-to kitfor designing useful and usable performance reports, which includes templates of great reports youcan use as your starting point.

tip 121: Identify all the users of your performance measures before designing your performance report you may need different formats of the report depending on the type of user.

tip 122: Interview users of your performance measures to find out what needs theperformance report has to satisfy , such as how frequently they need it, what they need it for,

how much time they want to commit to using it, what specific questions they are wanting it to answer,what format best suits them (see tip 135:) and so on.

tip 123: Dont succumb to all the needs of the users of your performance measures reportdesign is not all art, its a science as well . If you want it to still be useful, avoid includingsomeones idea just to avoid hurting them or being politically correct. The best thing you can do for such people is help them make the best use of their information.

tip 124: Decide the theme of your performance report. Dont just lump a whole lot of measurestogether without thinking about the decision process that it should serve. Consciously decide your rationale for which measures to include and which to exclude.

tip 125: If the commentaries in your performance reports consist mainly of explanations about why

targets werent met or why this month is worse or better than last month, then they are not too far from useless. Managing performance means managing the trends and patterns, not events. Howcould you put more information about the trends and patterns in your reports , and lessof the ad hoc trivia that suggests you never need to, or are able to, do anything to improveperformance?

tip 126: Put a contact name with each measure in the performance report so users canfollow up if they want more information. This might be the owner of the measure, as documented inthe measures definition (see tip 38:) .

tip 127: Consider if it would be useful to include with each measure in the performancereport its description , as documented in the measures definition (see tip 38:) , to remind userswhat the meaning of the measure is, so they are less likely to jump to the wrong conclusions.

http://www.staceybarr.com/mgu.htmlhttp://www.staceybarr.com/reportdesign.htmlhttp://www.staceybarr.com/reportdesign.htmlhttp://www.staceybarr.com/mgu.html

8/6/2019 202 Tips Free

30/47


use layout and formatting to facilitate valid and easy decisionmaking

tip 128: Colour can really help information jump off the page in your performance

reports. Red can strongly communicate unacceptable performance, green for improving or acceptable performance and orange for early warnings about likely future results.

tip 129: Some of the more useful types of information to include in a performance reportinclude the graphs of the measures, brief comments about any patterns or signals the graph is giving,what the causes of the patterns or signals are and comments about what is being done to respond tothe patterns or signals (if a response is required).

tip 130: Carefully design the page layout to give a consistent look and feel throughoutyour performance report. Consider where you will place graphs, interpretation comments, causeanalysis, action updates and so on. This helps people navigate through the report and find theinformation they are looking for.

tip 131: At your local library or bookshop, look for books on graphic design and layout toget ideas and example for making your performance reports professional, simple and incredibleusable. A good one is The Non-Designers Design Book [ref 18] by Robin Williams (no, not the actor).

tip 132: Try designing your performance reports with one page per measure and includethe title of the measure, a graph, a simple comment interpreting the graph, and a brief causeanalysis. This way you can pull together different collections of performance measures for differentusers.

tip 133: Play around with interpretation flags , little icons that symbolise visually and rapidlywhat each performance measure is doing. For example, can mean that performance is in balanceand where it should be, can mean that performance is headed in the wrong direction and needsimmediate attention, can flag if there is an early warning of poor performance and cansymbolise improvement.

tip 134: Keep your performance reports as visual as possible, avoiding lots of text . This istime consuming and often doesnt tell the user much that they really want to know, or couldnt find outby talking to someone if they felt they needed to. Focus on graphs, interpretation, cause identificationand action progress. You can read more at http://www.staceybarr.com/reportdesign.html about myhow-to kit for designing useful and usable performance reports, which includes templates of greatreports you can use as your starting point.

ensure your reporting process and tools support decisionmaking

tip 135: However you choose to disseminate your performance reports, be conscious of howpeople will want to and be able to access them . Convenience, literacy, technology andlocation all have a bearing on this.

tip 136: Remember that performance reports are not all paper based documents ! They canbe wall charts, web pages, pdf files, PowerPoint slide shows, flipchart story boards anything thatworks for their users!

tip 137: Electronic reports, such as pdf or html based reports, enable hyperlinking. And withhyperlinking you can automate the links between your performance measures to makecause analysis a little faster. Use the linkages on your measurement map ( tip 35:) as the specificationfor how to set up the hyperlinks in your reports.

tip 138: If you use the latest in Business Intelligence software to report your measures, youstill need to apply the thinking in the above tips most BI applications I have seen are more about

http://www.staceybarr.com/reportdesign.htmlhttp://www.staceybarr.com/reportdesign.html

8/6/2019 202 Tips Free

31/47


flashy things to do with clicking and linking and drilling down, and have very limited capability toencourage really good cause analysis (see tip 168:) and valid interpretation of information ( see tip140: to tip 149:) . Design a paper-based report first, to get clear how youre going to use your measures and supporting analysis, and you can read more athttp://www.staceybarr.com/reportdesign.html about my how-to kit for designing useful and usable

performance reports, which includes templates of great reports you can use as your starting point.tip 139: Flowchart your reporting process , linking together all the steps needed to source,analyse and present the data and measures to their audiences. Define who does what, and when itneeds to happen.

http://www.staceybarr.com/reportdesign.htmlhttp://www.staceybarr.com/reportdesign.html

8/6/2019 202 Tips Free

32/47


phase 6: interpret what your measures

are sayingInterpreting your performance measures means translating messages highlighted by performance information into conclusions about whats really going on. To turn information into implication, you must discern which messages are real messages.

The INTERPRET phase of PuMP is about translating what the numbers are sayingquantitatively, into our everyday verbal language. Its a step that the vast majority of decision

makers dont even know they are doing, and thus, dont do very well. As a consequence, manydecisions have been made which lead to inappropriate or ineffective results.

The topics covered in phase 6 of PuMP are:

traditional approaches to interpreting data and their limitations

more effective approaches to interpreting data and their advantages

how the interpretation of data is important in assessing if targets have been met or not

know when a difference is really a difference

tip 140: Just because your spreadsheet or charting software can draw a trend line through your data points, doesnt always mean there is a real trend there . Often there can be too muchvariation to be sure, or another non-linear pattern might better explain the change over time (such asa step change). If your R 2 value (see analyse chapter) is less than 80% (or so), you have to questionhow what the trend line really saying to you about performance.

tip 141: Understanding variation is a very important (no, essential) skill in interpretingperformance measures. And it doesnt mean using statistics like percent variation from budgeteither. Find out about Donald Wheeler [ref 30] and his book Understanding Variation: the key tomanaging chaos to discover why this is so important and what it really means.

tip 142: Beware of rolling averages or moving averages . They were really designed to

smooth variation (such as seasonal variation) out of a time series. But because they mask thisvariation, you simply cant tell if a real change has occured (until much later) or when it actuallyoccured. Looking at overall long term trends doesnt help us respond when we need to in order tokeep performance tracking in the direction we need.

tip 143: In her book Leadership and the New Science [ref 16], Margaret Wheatley has a chapter called The Creative Energy of the Universe Information. In this chapter, she addresses the issueof interpretation of data from a very non-measurement and non-statistical perspective. She mentionsthat the greater the ability to process information, the greater the level of intelligence that theorganisation will have. Valid interpretation of measures is vitally important since it is an essential partof the ability to process information. Valid interpretation is an essential condition for organisational intelligence .

tip 144: What do you do if users are reacting to individual points of data instead of looking for real trends and real changes in performance? Ask them how do they know that last


8/6/2019 202 Tips Free

33/47


month (or the same month last year, or last year, or whatever the benchmark they are comparingagainst) was typical, and thus a reliable benchmark?

tip 145: If you have implemented tip 97: where you added a mean line to the time series chart of your performance measure values, then if you see seven or more points in a row (or 12 out of 14 in arow) above the mean line, or seven or more points in a row (or 12 out of 14 in a row) below the meanline, you have a signal that performance has suddently shifted ! Calculate a new mean linefrom the series of points that you found on the one side of the mean line. Thats your new overalllevel of performance. Dont recalculate this new mean line until you see another signal.

tip 146: If you have implemented tip 97: where you added a mean line to the time series chart of your performance measure values, then if you see seven or more points in a row (or 12 out of 14 in arow) consecutively increasign or consecutively decreasing, you have a signal that performancehas gradually shifted ! Calculate a new mean line from the first seven points you get AFTER theseries of points that you found consecutively increasing or decreasing. Thats your new overall levelof performance. Dont recalculate this new mean line until you see another signal.

tip 147: If you have implemented tip 98: where you added control limit lines to the time series chartof your performance measure values (that is, you have a proper statistical process control chart),then if you see a point fall outside the control limit lines, you have a signal that an abnormalevent occurred . This is not indicative of any change in performance, just the result of a singleevent having a temporary effect.

tip 148: If you have implemented tip 98: where you added control limit lines to the time series chartof your performance measure values (that is, you have a proper statistical process control chart),then if you see several or many points fall outside the control limit lines, you have a signal thatperformance is chaotic . This is indicative of lack of control or stability in the process or systemthat produces your performance result..

tip 149: Survey results are used by many, many organisations, but they are often not used

properly. Do you know what a confidence interval is and how to use it? Effectively it tellsyou how reliable the survey result is that it relates to. As one example, imagine that employeesatisfaction this year is 4.5 and last year it was 4.0, giving a difference of 0.5 (an improvement). Thisdifference of 0.5 would have a confidence interval, and just say its 95% confidence interval was 0.5 0.6 which is represented as [-0.1,1.1]. While 0.5 looks like a nice improvement, theres a 95% chancethat there was no improvement at all, since the confidence interval contains zero in its range.

assess if you are on target (or not)

tip 150: Knowing whether or not your performance has met a target or not is really aboutknowing whether the process or system that produces that performance result is capable of

continuing to produce that result. Dont rely on a comparison of year-to-date with an annual target, or last month with a monthly target to decide if your system or process is capable. Remember thatvariation is the key to understanding changes in performance (see tip 141:) .

tip 151: If you use statistical process control charts (see tip 96:) then decide what exactly youshould be comparing with your target . If your target is to improve the average level of performance, then use the meanline, or if your target is to set a maximum level, then use the upper control limit, or if your target is to set a minimum level, then use the lower control limit, or if your targetis for a range, then use both the control limits.

tip 152: Dont forget to keep monitoring performance even after the target has beenmet make sure the improvement you implemented to reach the target is sustainable before takingthe measure off your radar.


8/6/2019 202 Tips Free

34/47

8/6/2019 202 Tips Free

35/47


use measures to review your plans

tip 153: If your performance measures were selected in line with your business plans (see phase1:), then your measures are an indication of the degree to which you have achieved your plan. Itmight, therefore, be useful to refer to your business plans when you are reviewing your performance measures.

tip 154: When you report performance measures (see the tips for tip 103:) , also reportinformation about improvement actions that are being taken in response to the signals (seethe tips for phase 6:) those measures have given.

tip 155: Organisational learning is a process that permeates every business activity, and thatincludes performance management. Learning happens when you check to see if your decisions and improvement actions worked . In other words, as a result of your decisions andimprovement actions, did your performance measures achieve their targets without any unintendedconsequences?

tip 156: Do your measures indicate that you achieved your business plan? If so, then you maynow be ready to review your business plan and also your selection of performancemeasures . But dont forget to ensure the improvements you made are now business as usual andnot temporary.

tip 157: One very important application of performance measures is in regular plan review . Strategic performance measures should be used by senior managers to regularly reviewprogress of performance relative to strategic goals. Likewise for tactical and operational goals.Performance measures are the feedback loop between the planning process and the improvementprocess to help us monitor the gaps between as is performance and should be performance.

be clear about the role of measures in decision making

tip 158: One of the most basic capabilities any decision making team will need to effectively useperformance measures is the skill of dialogue [ref 8]. Dialogue is not just talking, its engaging in aprocess of balanced listening, enquiry, observation and reflection that results in deeper understanding, expanded mental models and more possibilities.

tip 159: Do you know who to hold accountable for performance results ? This usuallymeans that if a performance measure is trending in the wrong direction, or not achieving target, aparticular person is dragged over the coals or given the sack. Businesses are getting more andmore complex and as a consequence, performance results are being influence by so many differentthings. NO ONE person should ever be held accountable for performance results. See tip 160:.

tip 160: When it comes to performance measurement and management, the ideas things tohold people accountable for include regularly monitoring the performance measure, validlyinterpreting the measures results, identifying root causes, initiating improvement actions, evaluating if those improvement actions worked, and learning from the evaluation on the organisations behalf.

tip 161: The decision making process that uses performance measures has distinctsteps, which include: prioritising which results to respond to, describing the desired outcome or results to achieve, understanding the current capability of the system producing the outcome,exploring any unintended consequences of taking action, developing strategies, implementing thestrategies and finally, checking if it all worked.

tip 162: Benchmarking with other organisations (either in your industry or not) is a greatway to share learnings and gather ideas for improvement. Go ahead and compare your performancemeasure results but be aware that they may define their measures differently, be operating in adifferent context and put a different priority on their measures. Focus more on dialogue and sharingideas about improvement.


8/6/2019 202 Tips Free

36/47


tip 163: When you sit down to review your performance measures, and you identify a list of improvement opportunities, how do prioritise them? Or do you try to do them all? To borrow fromStephen Covey [ref 22], which are the big stones and which are the pebbles?

tip 164: Before you decide on any improvement activities, you will save heaps of time and drama if

you clearly define the results you want to create first. Use sensory-based language [ref 14]t opaint a verbal picture of the outcomes you want as a result of improvement activity.

tip 165: Newtons Third Law states that for every action, there is an equal and opposite reaction. If you make a change to your business to improve it, where is the reaction going to come from ?Is it a reaction you want or one you dont want? Is it a reaction you can manage, or one better avoided? How can you improve performance in a way that minimises the inevitable unintendedconsequences?

tip 166: Unless your organisation checks to see if decisions made and actions taken have thedesired effect on performance, your organisation will miss out on creating some very valuableknowledge about its own performance dynamics . Often the root causes of one performancearea are the same root causes of other areas if you find out how to fix them, this knowledge can beused elsewhere.

make improvement stick by fixing root causes

tip 167: If you want to find performance improvements that stick , then you will need to dosome root cause analysis. This means thinking beneath the symptoms of poor performance that areso obvious, and digging down until you find the fundamental causes of these symptoms. Try usingPeter Senges the five whys technique [ref 7]: keep asking why until to find the action that, if changed or corrected, will prevent the symptoms from returning.

tip 168: Drawing causal loop diagrams is also a useful strategy for understanding the

dynamics producing a particular (unwanted) symptoms. These diagrams consist of loops that showthe causal relationships between various results or outcomes which collectively produce thesymptoms your performance measure might be showing you. Pegasus Communications website isa great resource for tools like this, so visit http://www.pegasuscom.com/cld.html to find out how todraw causal loop diagrams.

tip 169: Process mapping and analysis is a great way to understand the factors limiting thecurrent capability of the system creating your performance results. It helps greatly to define thesystem you intend to improve, before you try to improve it. Just type in process mapping andanalysis in your web browser and go prospecting for resources!

tip 170: If you use statistical process control charts, then the current range indicated by the upper and lower control limits (the voice of the process or limits of natural variation) compared to your target range is an indicator of your systems current capability . If the current control limitsare within the target range, your system is capable, but if the current control limits are wider than thetarget range, your system is not capable. Unless something changes, you will continue to getperformance results that randomly vary within your current control limits and thus some will falloutside your target limits. You will know when capability has changed when these control limitschange.

tip 171: Always develop a range of improvement options and evaluate their feasibility without judgement before making the final choice for what improvement actions to implement. Startwide, then get narrow. Its much harder to start narrow,then get wider when you discover your solution ideas just wont cut it.

tip 172: There are many improvement methodologies available, and one in particular has provenitself: six sigma . Made famous my Motorola, six sigma is a very structured and disciplinedmethod of improvement which requires a very dedicated champion. Not for the light hearted.More for those serious about real improvement.

http://www.pegasuscom.com/cld.htmlhttp://www.pegasuscom.com/cld.html

8/6/2019 202 Tips Free

37/47


tip 173: If you involve the people who work in the process or area that you want toimprove when you are designing improvement solutions (or get them to do it themselves), you willget more ownership of the change. People hate having things done to them, we all like to beconsulted and even better, involved.

tip 174: After deciding to implement a particular improvement action in response to a performancemeasure signal, you may like to go back to phase 1 of the performance measurement process(select) and develop one or two new measures to provide regular feedback about how your improvement actions or strategies are going .

the most useful targets dont come out of thin air

tip 175: Before you set a target for a performance measure , first identify what the rootcauses are that are holding performance at the unacceptable level and evaluate the resources youhave available to manage these root causes. What amount of change

202 tips free

Documents