The Good Indicators Guide:Understanding how to use and choose indicators

DAVID PENCHEON

Page

Foreword 2

1. Introduction 3

2. Indicators: some useful background 5

3. The anatomy of an indicator 9

4. Understanding variation 13

5. Changing hearts and minds 18

6. Frequently asked questions 21

7. Criteria for good indicators and good indicator sets 23

8. Ten myths about indicators 28

9. Glossary 30

10. Further reading 33

Appendix A: Full anatomy of an indicator 35

1

Contents

2

ForewordAs leaders, we have a responsibility toknow the essential data andinformation better than anyone else.We need our teams and organisationsto be able to capture, interpret andcommunicate the essence of anysituation in order to make the rightdecisions at the right time. Theindicators we use and choose thereforeneed to be carefully designed to bepractical, complete, valid and robust sowe can concentrate on those areas that

need further investigation. In short, weneed to sort the wheat from the chaffin the information overload world wenow live in. This short guide focuses onthe key principles behind developing,understanding and using indicators. It is designed to be an essential andreadable guide to those in seniorpositions who may not always feelentirely comfortable with this importantarea in healthcare leadership.

Bernard Crump

Chief ExecutiveNHS Institute for Innovation andImprovement

3

1. Introduction

1.1. What is this guidedesigned to do?

This guide is intended to be a short,practical resource for anyone in anyhealth system who is responsible forusing indicators to monitor andimprove performance, systems oroutcomes. A guide this size can’t giveall the answers to indicators. However,it can help you ask the importantquestions.

For instance…

• if you are working with imposedmandatory indicators, you needto be sure that your “measured”performance isn’t falling short ofwhat is really true, either becauseyou don’t understand them orhow to make the best of them

• if you have a choice over whichindicator to use, or anopportunity to introduce a newindicator to your organisation –you need to be competent inchoosing the ones that willgenuinely help you measure localsystems, services and outcomes.

After reading this guide, you should beable to assess the validity of theindicators you are working with,allowing you to exert more controlover the way your organisation isproperly judged, regulated, and run.

Underlining all this is the reality thatanyone working in a health system isworking in a complex and politicalenvironment. This guide aims tobalance what is desirable in terms ofusing indicators in the most correctand most rigorous way, with what ispractical and achievable in suchsettings.

1.2. Why focus on indicators?Why devote a whole guide to justindicators? They are, after all, just onevery specific tool for measurement andimprovement (see definitions in theGlossary on page 30). The answer is that we all love indicatorswhen they seem to summarise and bringsummary/simplicity, but not when theyjudge us, or something dear to us. It is then that we realise that people and organisations can be unfairly

Indicators are a fact of life for every NHS local authority, and socialcare organisation. For the most part, you will have little or no choiceover the indicators that are used to measure performance in yourorganisation. If you don’t fully understand the indicators you have towork with; if you think they paint an inaccurate picture of howthings really are on the ground; or suspect they are fundamentallyflawed in some way – you can find yourself locked into an unhappyrelationship with a performance measurement tool that you canneither challenge nor improve.

4

judged and rewarded on the basis ofindicators that may tell a wrong orincomplete story.

Or think of it like this: of all thecomments from new non-executivedirectors that join the boards of healthorganisations, one of the mostcommon seems to be:

1.3. How to use the guideThis brief guide is NOT the last word onindicators; nor is it designed to provideoff-the-shelf indicators for anyparticular service or system.

The Good Indicators Guide should beused alongside other related resourcesoffered by the National Health ServiceInstitute for Innovation andImprovement (www.institute.nhs.uk),and the Association of Public HealthObservatories (APHO)(www.apho.org.uk), as well as real

examples from the Department ofHealth (www.dh.gov.uk), theInformation Centre (www.ic.nhs.uk),the National Centre for HealthOutcomes Development(www.nchod.nhs.uk), the HealthcareCommission(www.healthcarecommission.org.uk)and others.

In particular, you should read:

• the NHS Institute’s ImprovementLeaders’ Guides on Measurementfor Improvement and ImprovingFlow www.institute.nhs.uk/improvementleadersguides

• APHO’s Annual Health Profileshttp://www.communityhealthprofiles.info/

1.4. And if you read nothingelse…

Our hope is that the guide is shortenough and important enough to beread right through. However, there aresome key sections that will be useful bythemselves – especially as quick-reference resources you can revisit.

These are:

• section 2. Indicators – the importantprinciples (page 5) and

• section 7. Criteria for goodindicators and good indicator sets.(page 23).

If you only read two sections from thisguide, read these!

“How can we ensure that theorganisation is actually doing

what it says it is doing?”

“Because we use good indicatorsto measure the right things and

feed these with the best data wecan get.”

The answer should be;

2.1. What are indicators, and why are they soimportant?

• indicators are succinct measuresthat aim to describe as much abouta system as possible in as few pointsas possible.

• indicators help us understand asystem, compare it and improve it.

Indicators are extremely importantforms of measurement, but they canalso be controversial (see 2.3 below). Like all powerful tools, they can easilydo as much harm as good. The world is becoming a moretransparent and competitive place,where people want instant summaryinformation. Indicators appear to fitthis need and are therefore becomingan increasingly important part of howeverybody works.

2.2 The three key roles ofmeasurement

Indicators, like many other forms ofmeasurement, can be used in threebroad ways:

1. for understanding: to know how asystem works and how it might beimproved (research role)

2. for performance: monitoring if andhow a system is performing to anagreed standard (performance/managerial/improvement role)

3. for accountability: allowing us tohold ourselves up to patients, the government and taxpayers and beopenly scrutinised as individuals, teams and organisations(accountability/democratic role).

2.3. Why are people suspiciousand mistrustful towardsindicators?

Mistrust and ambivalence towardsindicators is hardly surprising whenconsidering what they are designed todo. While they play an increasinglyfundamental role in monitoring ourperformance, they can only everindicate and summarise the complexsystems in which we work.

5

2. Indicators: some useful backgroundObjectives of this section:

• to help you understand why indicators have assumed such importance

• to explore why indicators cause so much anxiety and resentment

• to discuss the different ways indicators can be used.

For instance… A particularhospital has a high death rateamongst its patients. This isimportant and needs investigating.It may be because it is a specialisthospital and admits many very illpatients from other hospitals, or itmay be because there is room forimprovement in the way thehospital treats and cares forpatients. Lesson: indicators rarelygive definitive answers but theynearly always suggest the next bestquestion to ask that ultimatelyWILL give the answer required.

Consequently, indicators very oftenmake people and organisations feelvulnerable, exposed and defensive. This feeling is not likely to changeunless more is done to help peopleunderstand and accept the strengths,as well as the limitations, of theseimportant measurement tools.

2.4. Four things to know andaccept about indicators

• Indicators only indicate:an indicator will never completelycapture the richness and complexity ofa system. This makes people nervousthat they will be judged unfairly onthe basis of only one (or a few) facts.A set of indicators will usually notimprove things much. Indicators aredesigned to give ‘slices’ of reality. Theymight provide the truth, but theyrarely give the whole truth. This leadsto people’s understandable fear thatthey are being unfairly measured andjudged. Like any reductionistapproach, an indicator must beunderstood in context

• Indicators encourageexplicitness: indicators force us tobe clear and explicit about what weare trying to do. Not everyone feelscomfortable with this as there is

often a desire to retreat to nonspecific agreements rather than faceimportant differences inunderstanding. It can be difficultattaining a true agreement andunderstanding of the work, thoughindicators can be very helpful inachieving this by asking suchquestions as “What would successlook like if we could only measurethree things?”

• Indicators usually rely on numbersand numerical techniques: A lot ofpeople – even competentprofessionals - fear numbers. Yet, inorder to be able to use indicatorsproperly or challenge them, we needa basic understanding of elementarystatistics (rates, ratios, comparisons,standardisation etc). Although manystatistical methodologies can bedifficult and counter-intuitive to learn(especially if they are only usedoccasionally), indicators don’t alwaysuse complex methods.

• Indicators should not just beassociated with fault-finding:people often assume that indicatorsare designed to find fault. In fact,they can help us understand ourperformance be it good or bad.Well-designed measurementsystems identify high performers(from whom we can learn), as wellas systems (or parts of systems),that may warrant furtherinvestigation and intervention.

2.5. Measuring to improveWhile indicators do come (deservinglyor not), with a lot of ‘baggage’, usingthem properly can yield enormousbenefits – particularly in terms ofimproving systems and outcomes forpatients, staff and taxpayers.

6

A useful analogy

Imagine a car dashboard:an indicator is a warning lightflashing on the dashboard. It is fedby one of many streams of data –maybe oil level, temperature, etc...It flashes when all is not well,suggesting we stop the car. Theindicator “alerts us to somethingworthy of further investigation.”

Improving any system (e.g. a carepathway), depends on goodmeasurement and good understanding:

• we measure to understand howthings work (e.g. a care pathway) so:

• we can understand how we might dothings better.

There is deliberate emphasis on the wordunderstand in this guide – manyapproaches to improvement assume thatsimply by measuring something, it can beproperly improved. Using indicators wellis an important part of understandingand therefore improving. Butmeasurement on its own very rarelyleads to improvement - you can’t makepigs fatter just by measuring them!

Measurement is necessary forimprovement but is not sufficient on it’sown.

Successful improvement depends onmore than good measurement – not leastunderstanding how to turn measures intothe sorts of messages that will changehearts and minds. (See section 5, p18).

2.6. Improvement starts withclarity

Before beginning the process ofimproving any system, there needs tobe clarity over what that system isaiming to do.

In fact, agreeing and articulating theobjectives of a system can often be the

most valuable part of the wholemeasurement and improvementprocess. Lack of shared understandingis very often at the root of inefficienciesin a system.

Reaching a consensus about objectiveshas to start with constructiveconversation between all the keypartners within the team, system ororganisation.

However this conversation isapproached, two fundamental questionneed to be addressed:

• “do we have a consensus about whatthis organisation is trying to do?”

• “can we agree on how to translatethese broad aims into measurableobjectives?”

The Model for improvement1 is avalued tool right across the NHS. It too starts with the key questionabout what a service or system isaiming to achieve.

7

Of all the comments from new Non-Executive Directors (NEDs) that join theboards of health organisations, one ofthe most common seems to be:“How can we ensure that theorganisation is actually doingwhat it says it is doing?”

Figure 1: The Model for improvement

1. The conversation leadingto agreed, measurableobjectives1

2

3

1 Langley G, Nolan K, Nolan T, Norman C, Provost L, (1996), The Improvement Guide: a practicalapproach to enhancing organisational performance, Jossey Bass Publishers, San Francisco.

2. Well-designed indicatorsthat measure the right data;in the right parts of thesystem; at the right time

3. Indicators that help youunderstand what part ofthe system to change andhow

8

2.7. The conversation: how it might start and what it can doAs a typical board meeting comes to a close, the conversation between a ChiefExecutive Officer (CEO) and one of the Non Executive Directors (NEDs) might golike this…

CEO: Anything else to discuss?

NED: I’m still a bit worried about the quality of our chronic pain service.

CEO: OK. What’s concerning you?

NED: I’ve heard that our nextinspection is going to be difficult.Plus, a colleague of mine has hada particularly unsatisfactory experience recently as a patient ofthe service.

CEO: That’s interesting. We haven’t gottime to debate it in length now,but I’ll make sure we find out:

- how we can measure the qualityof this service at the moment

- how the national inspectionagency will measure it

- and how we currently incorporate the views of people who use the service.

- We’ll try and get this ready forthe next meeting when we aredue to discuss our next version ofthe performance dashboard.

Even from this brief exchange, wecan see:

• the importance ofunderstanding the keyobjectives of a service

• the importance of reconcilingdifferent ways of measuring it

• the importance of linkingmeasurement to quality andimprovement, and

• the challenge of combining thevery powerful and realexperiences of one person(even if not necessarilyrepresentative or typical), withan assessment using reliabledata that summarises theexperience of all users.

Using indicators can help to meetall these challenges moreconfidently, explicitly and withbetter end results.

3 points to take away

• indicators are only one form of measurement and people mistrust them• if used properly, they will help to better understand what systems claim to

do and how they can be improved• indicators must be chosen and used in a way that relates very specifically to

the objectives of the system in question.

3.1. Understand what you’relooking for

Whether you are using indicators thathave been imposed on you, or whetheryou need to choose the best indicatorsfor your local needs, the only way to feelcomfortable, confident and competentwith these measurement tools is tounderstand their structure or ‘anatomy’.

When you understand what constitutes agood indicator – it becomes much easier to:

• select the most appropriate indicatorswhen there is a choice.

• distinguish between perceived ‘pooror excellent performance’ due to apoor or inappropriate indicator, and genuine ‘poor or excellentperformance’ due to a well designed indicator.

• address any weaknesses in themeasurement systems if the indicatoris well constructed, but the local databeing used in the indicator isinaccurate.

3.2. The basic anatomy of anindicator

In operational terms the indicator isknown as a metadata, referring to thetitle, the rationale, and the informationabout how it is actually constructed.

This is different from the information thatis fed into the indicator. Which is called the data.

For example, “Infant Mortality Rate” isoften used as a basic indicator of thehealth of a community, especially ineconomically poorer countries.

9

3. The anatomy of an indicatorObjectives of this section:

• to show the basic construction of an indicator

• to help you deconstruct an indicator and assess the appropriateness

the infant mortalityrate

the number of deaths ofchildren aged less than 1 yearfor every 1000 live births inthat community in the sameyear

56 deaths ofchildren underthe age of onein a communitywhere therehave been4963 live births

local infant mortalityrate = 56 deaths for4963 live births =approx 9 deaths per1000 live births)

the title how the indicator is defined the numbers thatare fed into it

the metadata the data

3.3. Good metadataGood decisions depend on the mostappropriate indicator populated with thebest available data. A poorly designed, orpoorly chosen indicator with reliabledata, or a well designed/chosen indicatorwith unreliable and /or untimely data,has very little value (and is sometimespositively dangerous).

It is the metadata that will help youassess if a particular indicator is:

• important and relevant for you

• able to be populated with reliabledata

• likely to have a desired effect whencommunicated well.

A detailed example of metadata isoffered in Appendix A at the end ofthis guide. It uses the ‘People withdiabetes’ indicator from the Associationof Public Health Observatories’ (APHO),Health Profiles for England 2007.

It’s worth spending some time looking atthis and getting to grips with the level ofdetail you should aspire to whenassessing and choosing indicators. If thisseems onerous at first, most of theessential metadata elements of anindicator can be clarified by asking these10 basic questions:

10

10 key questions Answers and examples

1. What is being measured? Levels of diabetes

2. Why is it being measured? It is a serious disease with serious consequences.Although it can be prevented and treated, it is stillthe leading cause of chronic disease globally andaccounts for about 10 per cent of NHS costs

3. How is this indicator From recorded levels in general practiceactually defined?

4. Who does it measure? All persons, all ages

5. When does it measure it? Which day/month/year?

6. Will it measure absolute Proportions: numbers of case per thousandnumbers or proportions? resident population

7. Where does the data actually Collection and collation from Quality Outcomecome from? Framework (QOF) data in General Practice via the

NHS Information Centre

8. How accurate and complete The data cover more than 99 per cent of GPwill the data be? registered patients in England, although not

everyone is registered with a GP (especially somegroups with particular needs - see next box)

9. Are there any caveats/ Potential for errors in collection, collation andwarnings/problems? interpretation (such as an under-sampling of

ethnic populations, young people, homelesspeople, migrants, and travellers)

10. Are particular tests needed E.g. when comparing small numbers, in small such as standardisation, populations, or to distinguish inherentsignificance tests, or statistical (common cause) variation, from special process control to test the cause variation (See section 4 for more on meaning of the data and the variation).variation they show?

3.4. Is the appropriateindicator populated withthe best available data

While the indicator needs to be asappropriate as possible, the data needto be as valid as possible. Neither willever be perfect (See section 8 p28‘Myths about indicators’).

You will always need to use yourjudgement to assess whether both theindicator and the data it uses are goodenough, when combined for yourpurposes. Remember, a trustworthyindicator won’t give you definitiveanswers itself, it will only alert you toan issue you may need to explorefurther.

How you select your indicators andthe data that feed into them will ofteninvolve making a trade-off betweenwhat is convenient (and possible), tocollect, and what you ideally want foryour measurements.

Making this judgement call won’talways be easy, mainly because youwill need to factor in how otherpeople will react to being asked tocollect the data if they see it as morework, or don’t understand therationale behind the measure (anothergood reason to pin down clear anddetailed metadata for your indicators).

3.5. Good indicators meangroundwork

Understand the pressures on others.Health organisations already collectvast amounts of data and often this isduplicated - and staff on the groundare often not entirely clear whathappens to the data or why they areneeded. Feedback to those whocollect data is important in order toengage the whole system.

You should aim to nurture some activeownership of the data and indicatorswith frontline staff. At the very least,be aware of the burden on frontlineteams and don’t add to it withoutspending time talking to them.

11

For instance… It may be quite easyto use data from routine collections forsurveillance or administrative purposes(e.g. people who come into contactwith some part of the health service).However, this data will inevitably bebiased. Some of the most importantinformation (e.g. population baseddata that relates to behaviour andlifestyle), will be impossible to judgefrom routine systems based on healthservice contacts.

12

3 points to take away

• every indicator is made up of metadata and data• the metadata helps you understand the background of an indicator and judge

whether the indicator is likely to be an appropriate and possible measure• an indicator without trustworthy data to feed it, is often worthless and

sometimes dangerous.

For example ask staff:

• how the service works

• what, if anything, they want tochange about it

• what barriers they face

• what information they alreadycollect

• what they consider the fairest

measure of their work process andits outcome.

This is all about building up trust,credibility and ownership with thepeople and teams who are not onlyresponsible for collecting the data, butwho will be pivotal to making andsustaining any subsequentimprovements in the system.

13

4. Understanding variation

4.1. Why is variationimportant?

It has already been started in Section 2that indicators only indicate. They aredeigned as a high level summary of asystem and they help to identify andprioritise the issues that are worthinvestigating further.

But measurement of any system orprocess will reveal some variation. It’s thisinevitability that makes understandingdifferent sorts of variation so important.Perhaps the most important skill whenyou are presented with data is to assesswhether the inevitable variation issignificant enough to warrant furtheraction or not.

4.2. Statistical process controlA particularly valid and useful way ofanalysing variation is using “StatisticalProcess Control” (SPC). Developed byWalter Shewhart in the 1920s, itinvolves distinguishing between:

• the normal, everyday, inevitable (andusually unimportant), variationwhich is intrinsic and natural to anysystem – ‘common causevariation’

• and the more important variationwhich is indicative of somethingspecial happening and which callsfor a fuller understanding and oftenaction – ‘special cause variation’.

Objectives of this section:

• to help appreciate how fundamental an understanding of variation is tomanaging a system

• to help answer the question: “Is this variation we have spottedimportant enough to warrant further investigation?”

• to give an example of a statistically robust and visually intuitive way ofunderstanding variation

‘Ultimately, the ability to leaddepends on one's ability to

understand variation’.

W Edwards Deming

For instance… A frequently used wayof assessing and comparingperformance is to construct a leaguetable by simply ordering the values andhighlighting those at the bottom of thepile. The most important flaw with thisprocess is that, even if everyone isdoing well, there will inevitably besomeone at the end of the spectrum.However, this does not necessarilyautomatically mean underperformance.

(ref: Adab et al, Marshall et al)

14

Statistical Process Control, like any toolto analyse variation, can be used withina single system (institution) over time, orcan be used to analyse variationbetween different institutions.

One of the most common mistakes inanalysing variation (e.g. by usingindicators), is the failure to appreciatethat common cause and special causevariation are fundamentally different.This results in one of two problems:

• wasting time, money and effortinvestigating what appears to be an‘outlier’ when, in fact, that result isactually within a range that isdeemed acceptable by the system

(i.e. treating common cause variationas if it were special cause variation)

• wasting time, money and effortinvestigating and dealing with specialcause variation as if it were commoncause variation. So, because of justone particular outlier, an organisationmakes changes right across a systemthat is already working well, whenthe focus of attention should be onthat one particular outlier.

Statistical process control techniquescan help you distinguish between thesetwo types of variation and judgewhether you have:

A: a working system where theaverage level of performance isacceptable and there are no outliers

B: a working system whose averagelevel of performance is acceptable,but with outliers e.g. lung cancermortality

C: a system whose average level ofperformance is not acceptableregardless of the variation e.g. boys’educational attainment in England

ideal

action: address the outliers

action: concentrate on thewhole system rather thanparticular individualswithin the system

15

There is an important caveat about asystem that appears to be “in control”that is often overlooked ormisunderstood.

A set of data that does not exhibitspecial cause variation does notnecessarily represent a system that isperforming well. The whole system maybe uniformly under performing. (as inboy’s educational outcome, above boxC). (see below: figures 2 and 3).

For instance, a clinical department in atrust could be performing within theirown control limits, but failing to meet

externally required target expectations,because their horizontal line may be setmuch higher or lower (i.e. the targetexpectations are determined by data orpeople other than those within the ownorganisation).

Figure 2: A statistically uniform system (compared with itself over time)

The real strength of Statistical ProcessControl, when the results are displayedas control charts, is that it combinesstatistical rigour with an attractive andpersuasive method of displaying data.

Showing the proportion of A&E attendances seen within four hours. The averageproportion is 98.4 per cent - there is one special cause variation denoted by thesquare marker at week 10.

16

Figure 3: A statistically uniform system not meeting expectations [comparedwith itself over time]

Figure 4: An non-uniform system comparing NHS Trusts with each other

Showing the proportion of A&E attendances seen with four hours. However,compared with target expectation (red line), the whole system is performing at thewrong level.

17

This display of Statistical ProcessControl, (figure 4.), shows widespreadvariation in MRSA rates between NHSTrusts in 2005/6. The type of displayshown in figure 4 is often called afunnel plot. Those Trusts that areclosest to the blue line (the average),need not be investigated further – infact, it might be wasteful to do so.The area between the upper dottedline and upper continuous line is oftentermed the “alert” area. The areaabove the continuous red line is oftencalled the “alarm” area. Such a graphalso indicates where Trusts appear to

be performing very well. Lessons needto be learnt from such Trusts (assumingthe data are correct).

There are other ways of comparingdata using indicators in pictorial form(bar charts, simple ranking, dashboards,etc…), however, when data areanalysed using Statistical ProcessControl, and well displayed andcompared using control charts and/orfunnel plots, they combine:

- statistical robustness with

- visual intuitiveness.

3 points to take away

• the most common mistake in using indicators for improvement is notunderstanding that common cause and special cause variation arefundamentally different

• Statistical Process Control (SPC) helps you distinguish between the two• SPC gives you a valid statistical approach along with an attractive and

persuasive way of displaying your data to others.

18

5.1. What really makespeople and organisationschange?

Indicators are summary measures andno matter how important or valid theyare, they will rarely, by themselves,motivate people to change.

There are institutes and universitydepartments entirely devoted to thescience of change. People rarelychange until they judge that the risk ofnot changing outweighs the risk ofchanging. Bringing them to this pointrequires much more than having validmeasures. It requires appealing topeople’s emotional side, as well as totheir technical and rational selves.

Most people involved in measurementrealise it is necessary for change tohappen; however, too few realise whenmeasurement is not sufficient.

5.2. Why is communicationimportant?

It can be simplistic and unwise toseparate the world of measurement fromthe world of improvement. Similarly,never separate the design of indicatorsfrom the design of the larger changeprocess. All these areas are connectedthrough good communication.

While good communication skills areclearly vital, they are not in the scopeof this short guide. The focus here is onthe fundamental principles that willhelp you use measurement in a waythat genuinely has the power tochange hearts and minds when wellcommunicated as part of a largerchange process.

5.3. Four principles forchanging the way peoplethink

• think about your audience: Askyourself who you want toinfluence. Information in general(and indicators in particular), canbe most influential when theyare well-received by the peoplewho have the greatest control. So there is considerable potentialfor using information that youraudience both understands, and wants to feel they can do something about.

5. Changing hearts and mindsObjectives of this section:

• to help you understand that improvement happens by appealing topeople’s rational and emotional sides

• to help you communicate what the indicators actually tell you

‘I am not interested in measurementper se. I am obsessed by

improvement and the rolemeasurement has in that process.’

Don Berwick, Institute for HealthcareImprovement

19

• make presentation matter:Many people from a numericalbackground find it difficult tounderstand why people don’t findthe research they have done orthe data they have assembledmore compelling. But noteveryone responds well to graphsand charts. There is a balance tobe struck: if you’ve spent timeand effort on selecting the mostvalid indicators and gatheringgood data:

• you don’t want to ‘dumb down’the information they give youjust to get people to take notice

• but neither do you want peopleto ignore it because it’s toocomplex or completelyimpenetrable.

There are classic texts and examplesto show how useful and unusefulquantitative information can be tohelp people change their view andunderstanding of the world. (See Tufte’s classic works in Furtherreading).

Do the simple things well:

• labels: ensure you label yourtables and graphs clearly

• text: ensure the text on them islarge enough to read

• colour: remember that red andgreen colour coding can be aproblem for some people (8% ofmen are red-green colour blind).

• test your approach: Whatevermethod you choose forpresenting your information, it is

often helpful to test out theapproach you want to use, before you present itformally. If the presentation ortechniques you are using (e.g.Statistical Process Control), arelikely to be unfamiliar to those you want to influence,spend some time beforehanddeveloping their understanding,or consider changing yourapproach.

• appeal to people’s emotions:Failing to appeal to the emotionalside of people (what they thinkand what they feel), will probablydoom any change process.

Try not just to present your data – lookfor the ‘story’ it is telling in terms youraudience will appreciate. If you havedone your groundwork at the earlierstages and started a constructiveconversation with frontline staff and service users, you’re likely to findthis much easier.

For instance… It is usually morepowerful to say that “half of allsmokers will die early because theysmoke” than “over 100,000 smokersdie prematurely each year”.

‘People change what they do lessbecause they are given analysis thatshifts their thinking, than because

they are shown a truth thatinfluences their feelings’.

Prof John Kotter, Harvard Business School

20

5.4. The importance oftiming, context andoriginal purpose

The right indicator, at the right timeand in the right context can stimulatehuge change, where it might have noeffect in other contexts and at othertimes. So if an indicator has beendeveloped for one purpose (e.g. forresearch and surveillance), and youuse it for another (e.g. performancemanagement), make sure you’vethoroughly thought through theimplications. Indicators that arethoughtlessly used will be carelesslytreated and even manipulated bythose whose behaviour it is trying toinfluence for the patient’s and public’sgood. People will even start tomanipulate the system to meet thetarget set by the indicator, forgetting

or avoiding the original intention ofthe target (“Gaming”).

The lesson is always to remember thecircumstances in which an indicatorhas been developed before using itelsewhere. It may still be useful, butthe context always needs to beconsidered and made explicit.

For instance… Research shows thatpatients in A&E departments feelhighly reassured when they are seenby a clinically qualified professionala soon as possible after they arrive.Setting this as an indicator (time tosee a clinically qualifiedprofessional), and a target (time tosee a clinically qualified professionalis within 30 minutes), may meanthat some departments merelydeploy someone to say hello – not agood use of a professional’s time.

3 points to take away

• indicators, although essential in every improvement process, will not bythemselves motivate people to change.

• good communication bridges the gaps between measurement, understanding,and improvement.

• in order to influence people, indicators need to be presented in ways that areeasy to understand and in ways that make changes to the system compellingand possible.

21

6. Frequently asked questionsQ. What role does qualitative

evidence play when usingindicators?

A. Perceptions are very real; they candrive or block necessary change.Qualitative research (about howpeople think and feel), is extremelyimportant in indicator work – yetoften (and wrongly), indicatorsconfine themselves to quantitativedata. It is important for thosepeople aiming to improve thesystem to be very aware of thepower of feelings and perceptions inunderstanding an improvement in asystem.

Q. How closely related does theissue being measured have tobe to the objectives of thesystem or organisation?

A. The two most essential features ofan indicator are that it has to beintrinsically valid (measure what itclaims to measure), and to beimportant and relevant to theobjectives of the system ororganisation it is measuring. This isespecially true in managing theperformance of a system. Focus onwhat is important, before movingon to what is interesting.

Q. What is meant by a balancedscorecard of indicators?

A. Any indicator will give only a veryspecific and limited perspective of awider situation. Think of trying towatch a football match through avery small gap in a fence. It will only

give a limited picture of the wholegame. Different indicators (likedifferent gaps in the fence), givedifferent but complementary ‘slices’of the whole situation. They need tobe added together to get a pictureof the entire pitch and anunderstanding of the whole game.That’s why indicators are often usedin baskets or sets; each measuringan important but different aspect ofthe system. This is the ‘balancedscorecard’ approach.

Q. Some indicators are composedof ‘synthetic estimates’. What does this mean?

A. When specific data are not availablelocally (e.g. because there is noroutine data collection of that subpopulation or of that particulargeographical area), it is sometimespossible to estimate or synthesisesuch data. This can be very valuableto give local decision makers anapproximate feel for the scale of anissue, although it is clearlyunsatisfactory if an evaluation of aspecial and local initiative isrequired.

Q. Can we afford to measureourselves?

A. The pressure on health organisationsis to deliver frontline services, ratherthan invest in people with clipboards, stopwatches and tapemeasures. Measurement has a cost,and this cost should never outweighthe benefits it brings. However, the

22

question should perhaps bereversed: do we have sufficientenough resources so that we canuse them without any systematic,valid information on whether weare actually delivering what wethink we are delivering, or indeedare we addressing a need we onlythink exists?

Q. How can we be sure of thequality of commerciallyavailable indicators?

A. Health-related intelligence (like allintelligence), is an increasinglycompetitive business where thereare multiple providers of numerousproducts, including measurementsystems and indicators. The samecritical appraisal process that isroutinely applied to research

findings should be applied toindicators wherever they comefrom. Commercially providedindicators, being driven by themarket, can be highly responsive touser needs. However, thedisadvantage is that they cansometime be difficult to‘deconstruct’ because the way theyare assembled may be part of theircommercial value and thus subjectto some confidentiality. This canmake it difficult to appraise theirsuitability and validity. Nonetheless,commercially-providedmeasurement systems should notbe ignored. They can make avaluable, stimulating and innovative addition to a responsivemeasurement process.

23

7. Criteria for good indicatorsand good indicator sets

Objectives of this section:

• to offer a systematic approach to designing, appraising, and choosingindicators and indicator sets (especially for Directors of Performance andChief Executives)

• to offer a systematic approach to those people who have to makedecisions on the basis of indicators and indicator sets about which theywill not be completely familiar (especially for Non executive Directors)

• to offer a systematic approach to those people who have to deal with thenext steps of measurement and monitoring of the systems (especially forPerformance Managers and Team Leaders).

7.1. Critically appraisingindicators

No indicator in the history ofmeasurement and improvement isperfect for all purposes. No indicatorwill perfectly satisfy all the criteria andquestions below. The important issue isthat these questions have beenconsidered and assessed systemically,and any compromises judgedacceptable, and made explicit.

These questions are grouped into fivesections, with comments that relate toeach question – most of the issues arementioned elsewhere in this guide.

The first two questions are the mostimportant.

A. Do the indicators address theimportant issues?

B. Are they scientifically valid?

Don’t proceed unless the indicator(s)you are assessing are both importantand valid.

A: Importance and Relevance?

If you are not clear about the mostimportant aims of the system you aremeasuring (such as its purpose,measurable objectives, etc…), then donot proceed until you have clarified thisat a senior level. Only then will youknow if the indicators you are assessingare important and relevant.

A1. Does this indicator measure asufficiently important question/service?

Comment: People often fail tomeasure and address what is mostrelevant. Your indicators must focus on,and measure key parts of the process,and/or outcome. Indicators must bechosen and used in a way that relatesvery specifically to the objectives of thesystem in question. The indicators haveto appeal to the people’s perception ofthe importance and the possibility ofimprovement (sometimes referred to as“Perceived Public Value”).

24

A2. If you are considering a set ofindicators, is it a balanced set? (i.e. areall important areas covered withoutundue emphasis on any one area?)

Comment: Measurement oftenconcentrates unduly on some parts of asystem at the expense of other equallyimportant parts of the system.Consequently, some parts are over-measured and others under-measured.

A3. Is the set of indicators you areproposing likely to clarify the consensuson the objectives of thesystem/organisation?

Comment: Using indicators thatmeasure the right data; in the rightparts of the system; at the right time, isa powerful way to develop and sharethe consensus.

B: Validity (Does this indicatoractually measure what it is claimingto measure?).

If you are not sure if the indicators youare proposing actually measure whatthey claim to measure, then do notproceed until the proposed indicatorshave been tested (i.e. validated).

B1. Does this indicator really measurethe issue?

Comment: Many indicators claim tomeasure important parts of a system ora service but often fail to do so. There are many reasons for this e.g. theissue being studied might be anoutcome and the indicator being usedis a measure of infrastructure orprocess. This can be furthercomplicated when there is a poor or illunderstood relationship between theinfrastructure or process of a systemand its outcome.

Do not proceed until you haveclarified A and B. Then ask:

C. Is it actually possible to populatethe indicator with meaningful data?

D. What is the meaning? What isthe indicator telling you and howmuch precision is there in that?

C: Possibility (Is it actually possibleto populate the indicator withmeaningful data?).

If you are not sure if it is even possibleto construct and populate theindicators, then do not proceed untilthis has been checked. This isparticularly important for those peoplewho are tasked to actually design(rather than just choose), indicators.

C1. Are sufficiently reliable dataavailable at the right time, for the rightorganisations with the appropriatecomparators?

Comment: You need the right databoth for the issue in question and forappropriate comparators (place,people, organisation, time…).However, do not be seduced solely inthe direction of data availability. The right indicators are based on theobjectives of the organisation and thenot just the availability of data.

C2. If not, is the extra effort and costjustifiable?

Comment: It may be desirable to knoweven more than you do, but is it worthit? Is it likely to change your decision?

25

D: Meaning (What is the indicatortelling you and how much precisionis there in that?).

If you have no idea what the results ofany indicators you design or selectactually mean when they have beenpopulated and displayed, then do notproceed.

D1. Will the indicator be able to detectand display a variation that is importantenough to warrant further investigation?

Comment: A key question! The indicator should ideally be capable ofidentifying all the issues (and only thoseissues), that need further investigation. It should not identify random variationsthat should be ignored i.e. “noise”.

D2. If the indicator is high or low, whatdoes it actually tell you, and does it giveenough accurate and precise informationfor you to be able to investigate furtherand take any necessary action?

Comment: A surprising number ofindicators give you very little additionalinformation because you may not be sureif an apparent outlier really meansanything. You need to be confident youcan judge the acceptable limits of thevalue of an indicator before pursuing it.

D3. Can the indicators be understood(and deconstructed), in order tounderstand the particular reasons for theresults?

Comment: Often, people scratch theirheads and guess the reasons for aparticular result because they do notknow enough about how an indicator isconstructed in order to work backwardsand understand the issues that may becausing a particular value.

D4. Can the implications of the indicatorresults be communicated to, andbelieved/appreciated by the rightaudience?

Comment: The compelling way in which

results are displayed, interpreted, andcommunicated are key to subsequentaction and any improvement.

Finally ask what are the implications? What are you going to do about it?

E: Implications (What are you goingto do about them?).

If you do not understand the implicationsof an indicator in terms of whether theresult suggests further investigation, thendo not proceed until you do. Ensure youunderstand what actions might bepossible if such investigation confirms theneed to act.

E1. Is there sufficient understanding ofthe system so that any issues identifiedcan be investigated further andaddressed effectively?

Comment: Once you are clear there is anissue where further investigation may bewarranted, you must be clear there isgood evidence on how to act (orsufficient investment and research to findthis out). Otherwise you will merelyidentify issues you do not know how toaddress, or worse, address wrongly.

E2. Are the results likely to induceperverse incentives and unintentionalconsequences?

Comment: You need to be able to actin a way that gets to the root of theissue, not by gaming or by manipulating the data.

E3. Can the indicator monitor the issueregularly enough so that furtherinvestigation and action can be takenbefore the issue is revisited?

Comment: The system must besufficiently responsive so that problemsare addressed early but not measured sooften that the action has not had thechance to have had effect.

26

Example 1:Indicator title:

Mental health indicator in HealthProfiles 2006 (subsequentlydropped)

Defined as: the ‘practice register ofpeople with severe long-term mentalhealth problems who require andhave agreed to regular follow-up’.

Why was it problematic? Theindicator was difficult to interpret: ahigh number could either mean thatthe system is treating high numbers,or alternatively, a high prevalence, oreven both. Conversely, low numberssuggested either a low prevalence ofmental health problems, or onlysmall numbers of people accessingtreatment, or both. Consequently,the results of the indicator werelargely unhelpful to most people.

The indicator was dropped from theHealth Profiles indicator set thefollowing year.

Example 2:Indicator title:

Life expectancy at birth (usedwidely as a summary measure of population health)

Defined as: the average number ofyears a newborn baby would survive,if he or she were to experience thatarea’s current death rates as measuredthroughout his or her life.

Why is it problematic? Although itis based on a somewhat hypotheticalcalculation, this indicator has huge‘face validity’ and is also very intuitive.It is a broad, summary measure andas such is influenced by everythingthat influences the length of life lived.

However, the consequences of thisare that it can be difficult todeconstruct why a specific populationhas a particular life expectancy. Theremay be very many reasons; and thereasons for a low life expectancy inone population may be completelydifferent from the reasons for a lowlife expectancy in another.

7.2. Here are some examples of how these questions can help youhighlight some of the practical shortcomings of indicators, eventhose still widely in use.

Examples 1 and 2 have a public health focus and examples 3 and 4 have a moreclinical focus.

27

Example 3:Indicator title:

Call to needle time in suspectedmyocardial infarction

Defined as: % of patients whoreceived thrombolytic treatmentwithin 60 minutes of calling forprofessional help. MINAP clinicalaudit.

Why is it problematic? There aremany people who call for helpcomplaining of chest pain and whodo not meet a final audit definitionof myocardial infarction.

Why is it used? It remains a goodindicator of how often peoplereceive a time-dependent element oftreatment for myocardial infarction.

Example 4:Indicator title:

BMI (Body Mass Index)measurement in people withdiabetes

Defined as: Quality and OutcomesFramework DM 2. The percentage ofpatients with diabetes whose notesrecord BMI in the previous 15 months.

Why is it problematic? Measurementalone may give the illusion of somethingbeing done.

Why is it used? Getting a measure ofthe problem is really important, anddeveloping more appropriate measuresof change is hard.

28

The only useful indicator is onethat covers the complete work ofan organisation.This is highly unlikely to be even possible.Indicators merely indicate. You areaiming for the most useful slices ofreality that give the most comprehensivepicture possible, with the least amountof unnecessary detail.

It is possible to design a measurethat captures all that is mostimportant about a system.Again, this is highly unlikely. Healthcare involves large and highly complexsystems. For instance, coronary arterybypass grafting mortality only covers atiny fraction of acute activity, but isoften used as an indicator for wholeareas of the acute sector.

Homemade indicators are best:the only indicators that are of anyuse are those that you designyourself.Rarely do you need to design newindicators. Resist the temptation. If youthink it will be a better indicator thanany other existing indicator, ask why no-one else has used it before. In addition,indicators that you have designedyourself are unlikely to have usefulcomparators, except with itself over time.

You only need very few indicatorsto understand how a system isworking.You would be very lucky if this weretrue. Most indicators (and sets ofindicators), give you selected slices ofreality. You need many slices to gain areasonably valid understanding of the

whole system. Moreover, you need tounderstand the relationships betweenthe indicators you use, and the issuesthey are intended to measure.

Measurement eliminatesuncertainty and argument.Indicators exist to prompt usefulquestions, not to offer certain answers.Promoting a healthy uncertainty andstimulating the right degree ofunbiased, informed debate, are whatindicators are all about.

Unless the data are perfect, theindicator is useless.There is a temptation to throw out thebaby with the bath water (especiallywhen the result does not conform toyour prior assumptions). There are nosuch things as perfect data or perfectindicators. The skilled manager knowshow to appraise the quality of the dataand the subsequent indicator incontext, and judge how important theindicator is to the whole decisionmaking process. The real question is:are the data good enough for thepurpose in hand?

It is possible to justify the resultof any indicator.Being too defensive and not sufficientlyopen minded often prompts people toexplain away the result of any indicator(especially the indicators that suggestpoor performance). Simply finding aplausible excuse for every indicator istactically short sighted, and strategicallydangerous. Be open and honest;otherwise you will be rumbled.

8. Ten myths about indicators

29

It is acceptable to improve theindicator rather than the system.This eventually becomes gaming, andleads people to do anything to improvethe result of the indicator, exceptaddress the problems it is designed toaddress. Again, you will be rightlyaccused of not engaging in the truespirit of improvement.

It needs to be a common event tomake a useful indicator.Events (e.g. deaths, infections ormistakes), need to be sufficientlycommon when trying to makestatistically significant comparisonsbetween different organisations, areas,or time periods. However, qualitativedata (what people think or feel), can bevery powerful from even small numbersof people; focus groups often tap intoimportant issues that surveys oradministrative data collections maynever reveal.

Secondly, rare and significant events(e.g. adverse outcomes), can be veryimportant indicators; certainly as astarting point to a more completestudy to understand the underlyingsituation. Remember, indicators onlyindicate; they are no more diagnosticthan a screening test.

Only local indicators are relevantfor local people.There is no doubt that indicators thatare relevant to local people orpractitioners are more likely to bebelieved. But you will find it difficult toknow what the results mean if youhave no consistent comparators orbenchmarks from elsewhere. Indicatorsmandated and designed by otherpeople or higher authorities may notalways be perfectly relevant locally, butthe little you lose in relevance will bemore than compensated by validity andcomparability.

30

Basket (of indicators): A group ofindicators that aims to cover a widerange of issues as validly as possible. A common way of using a basket is forpeople to choose which indicators theyneed locally. The idea is to construct(and often populate), a validated set ofindicators that promotes consistencybut still gives local users some controlin selecting the indicators they want to use.

Balanced (set of indicators): A set ofindicators which, taken together, reflectas much of a system as possiblewithout duplication, overlap or gaps.

Benchmark: An externally-agreedcomparator to compare performancebetween similar organisations or systems.

Control charts: A method ofdisplaying the results of StatisticalProcess Control (See Section 4 (p13)‘Understanding variation’).

Composite indicator: An aggregationof numerous indicators which aims togive a one-figure indicator in order tosummarise measures even further. (ref:Jacobs, Goddard, et al; 2006).

Criterion: An area or issue which,according to good evidence, is relatedto the overall objectives and outcomesof the system or organisation beingmeasured. It needs at some point to be

made quantifiable as a standard,indicator or target.

Dashboard: A visualisation of the mostrelevant indicators in one place.

Data: Information that is fed intoindicators. Without context andcomparators, data rarely have significant meaning.

Funnel plot: A method of displayingthe results of Statistical Process Control.

Gaming: To improve the result of theindicator directly, rather thanaddressing the underlying system thatthe indicator is seeking to address.

Indicator: A summary measure thataims to describe, in a few numbers asmuch detail as possible about a system,to help understand, compare, predict,improve, and innovate.

Information/Knowledge/Intelligence:Processed and accurate data; collated,linked, contextualised, interpreted,presented and disseminated in a timelymanner to enable a decision-maker tobe better informed.

Metadata: (literally: data about data) Adescription of the data that go into anindicator and how and why the indicatoris constructed. The information on thespine of a book is the metadata for the

9. GlossaryA guide to some of the common terms used when working with indicators

31

information inside the book. (Title,author, publisher…) (See Section 3 p9‘The anatomy of an indicator’).

Metrics: Any set of data. An indicatoris a particular sort of metric thatidentifies issues that may be worthy offurther investigation.

Monitoring: The process of regularfollow up for specific indicators, with aview to action when a particularthreshold is crossed.

Outcome: An outcome is a measurablechange in health status, sometimesattributable to a risk factor or an earlierintervention.

Outlier: A result outside the desirablerange.

Performance: The degree to which asystem delivers, as measured againstspecific and agreed standards and criteria.

Quality: A broad term to describe themultiple dimensions of anorganisation’s function. Dimensionsmight include effectiveness, efficiency,responsiveness and accessibility.

Repeatable/Reproducible/Reliable:Although some would argue these termshave slightly different meanings, formost purposes they refer to the degreeto which a measurement is the sameeach time it is repeated (see ‘validity’).

Standard: The level at which acriterion is set (sometimes called abenchmark or a minimum standard).

Statistical Process Control (SPC):

Statistical analysis and display (e.g.control charts), which helps distinguishnormal, everyday, inevitable variationfrom the (much more important),special variation. The latter indicatessomething special is happening whichwarrants a fuller understanding andoften action.

Surveillance: Regular and systematiccollection, collation and analysis ofdata. It can be used to spot emergingproblems (such as important changes indisease rates), or monitor the importantoutcomes of e.g. a health care system.

Synthetic data: By using researchfrom elsewhere and the characteristicsof the local population, it is possible to‘synthesise’ a locally useful estimate ofdata that would otherwise be missing.

Target: Whereas indicators imply onlydirection, targets imply direction,speed, and destination. However,targets are sometimes plucked out ofthe air with little evidence to suggestwhether it will be achieved anywaywithout any effort, or whether it willbe impossible to meet despite any levelof effort.

Many indicators used in the NHS aregiven in the form of targets. The advantage of a specific target isthat it can increase the clarity of thetimescale. The disadvantage is thatspecific targets are frequently wildguesses about what is achievable.

32

Trajectory: A predicted future directionand rate of change of the value of anindicator, as predicted by previouschanges.

Trend: A comparison of the sameindicator over time.

Validity: Accuracy; the degree towhich a measurement truly measurethe issue of interest.

Variation – common cause:Common cause variation is theinevitable variation that is observed in

any process. Such variation ischaracteristic of the whole system. (See section 4, (p13) ‘Understandingvariation’)

Variation – special cause:A departure from the norm due to aparticular characteristic of the outlier,rather than to a characteristic of thewhole system. To address such anoutlier, the focus should be on theindividual case/event/outlier, and notthe whole system. (See section 4, (p13)‘Understanding variation’).

33

Background to indicatorsFlowers J, Hall P, and Pencheon D.Public Health Indicators. Pub Health119:239-245, 2005.

Thomson R. Appropriate use of data:the example of indicators. Chapter 9,Clinical Governance in Primary Care,2nd Ed. Van Zwanenberg T, Harrison J.(eds). Radcliffe, Oxford 1998.

How to present performance datain compelling and meaningfulways:INphoRM 4: Presenting performanceindicators: alternative approaches.Battersby J, Flowers J. erpho. Dec 2004www.erpho.org.uk/viewResource.aspx?id=7518 (accessed: 17 September 2007)

Tufte E. The Visual display ofquantitative information. 2nd ed.www.edwardtufte.com/tufte(accessed: 17 September 2007)

Tufte E. Envisioning informationwww.edwardtufte.com/tufte(accessed: 17 September 2007)

Tufte E. Visual explanations: imagesand quantities, evidence and narrative.www.edwardtufte.com/tufte(accessed: 17 September 2007)

Smarter Reporting. NHS ConfederationNo 92. December 2003.

How to link measurement usingindicators with improvement:NHSI: Improvement Leaders’ GuideProcess and systems thinkings,Measurement for improvement, andImproving flow. 2005.

How to appeal to the emotionalside of those responsible forchangeKotter JP, The heart of change. HarvardBusiness School Press; Cambridge;Mass. (2002)

Distinguishing between thedifferent ways of usingperformance indicators toimprove public services:Performance Indicators: Good, bad andugly RSS Working Party onPerformance Monitoring in the PublicServices. Bird S. (Chair) (2003)

Examples of how balanced setoffindicators can be used to give anoverview of the health and needsof a populationAPHO Health Profileswww.communityhealthprofiles.info/(accessed: 17 September 2007)

Kelley E, Hurst J, OECD Expert Groupon the Health Care Quality IndicatorProject - Criteria for selectingindicators (2006)http://intqhc.oxfordjournals.org/cgi/reprint/18/suppl_1/21?ijkey=BT8Q5QCoHyR1w&keytype=ref&siteid=intqhc(accessed: 18 September 2007)

Using indicators onlineKarolinska Institute Innovative methodsof displaying indicators online –www.gapminder.org (accessed: 17September 2007)

Examples of how indicators canbe selected from a large databaseAPHO Indicator Database:www.yhpho.org.uk [Go to indicatordatabase] (accessed: 17 September2007)

10. Further reading

34

The pros and cons of compositemeasures:Jacobs R, Goddard M, Smith PC. PublicServices: Are composite measures arobust reflection of performance in thepublic sector. Centre for HealthEconomics, York. CHE Research Paper 16June 2006.

The shortcomings of leaguetables…Adab P, Rouse AM, Mohammed MA,and Marshall T. Performance leaguetables: the NHS deserves better. BMJ324 (7329):95, 2002.

…and the advantages ofstatistical process controlMohammed MA, Cheng KK, Rouse A,and Marshall T. Bristol, Shipman, andclinical governance: Shewhart'sforgotten lessons. Lancet 357(9254):463-467, 2001.

Marshall C, Spiegelhalter DJ,Sanderson C, and McKee M. Reliabilityof league tables of in vitro fertilisationclinics. BMJ 316 (7146):1701-1705,1998.

Edwards Deming W Critical Evaluationsin Business and Management By WoodJC, Wood MC, (eds), Routledge, 2005.

Further technical briefings……are available from organisations suchas the Association of Public HealthObservatories, athttp://www.apho.org.uk

and include:

• Statistical Process Control methodsin public health intelligence

• confidence intervals

• a practical guide to target setting ina multi-agency environment.

International comparators.It is sometimes worth thinking aboutinternational comparisons of healthdata. This is increasingly of interest tothose working in economicdevelopment in the regions wherehaving comparable data and indicatorswith partner regions in Europe isimportant. Data are available on healthin Europe though mostly this is at anational level. However some data areavailable at a sub national data and isworth reviewing:

OECD: Statistics Directoratehttp://www.oecd.org/std

Health for all database of the WHO:http://www.euro.who.int/hfadb

Eurostat:http://ec.europa.eu/eurostat

The EU website via the EU health portal(http://ec.europa.eu/health-eu/index_en.htm) is a useful entry pointto a range of relevant public healthmaterial from the European Union.

35

The ‘people with diabetes’ indicator is takenfrom the 2007 Association of Public HealthObservatories (APHO) Health Profiles forEngland.

This is just one example of several indicatorsfeatured in the Health Profiles. Like all theother indicators in this resource, it has beenchosen specifically for its potential positive.

impact on the health of the population, butalso because it is:

• valid – it measures what it says it measures

• meaningful – it allows comparison oftime, place and or people

• possible to communicate – to a wideaudience.

Indicator name Prevalence of recordeddiabetes (“People with diabetes”).

Indicator definition Diabetes prevalence(from Quality and Outcomes Framework),persons, all ages, June 2006, per 100 residentpopulation.

Geography England, Region, Local Authority:Counties, County Districts, MetropolitanCounty Districts, Unitary Authorities, LondonBoroughs.

Timeliness Data is extracted from the QMASsystem annually in June and published in QPID(Quality and Prevalence Indicators Database) inSeptember-October each year.

What this indicator purports to measurePrevalence of diabetes.

Why this indicator is important (rationale)Diabetes is a serious disease with seriousconsequences. It is the fifth leading cause ofdeath globally and accounts for about 10% ofNHS costs. The burden falls disproportionatelyon elderly and ethnic populations. We use theindicator in this context as a proxy forhealthcare need and demand (a highprevalence of diabetes can indicate a lesshealthy population with higher serviceutilisation). The sequelae of diabetes includeblindness, amputation, neuropathy, renaldisease, heart disease and other complicationssuch as amputation. It is treatable andpreventable. Important modifiable risk factorsare obesity, diet and lack of physical activity.

Reason to include this particular indicatorTo encourage better collection of the primarydata to give more accurate estimates of diseaseprevalence. To monitor diabetes prevalence. Toemphasise the burden of disease. To encouragepreventative action.

Policy relevance Diabetes National ServiceFramework.

Interpretation: What a high / low level ofthe indicator value means A high value canindicate genuinely high prevalence and/orbetter detection and recording. Conversely alow value may indicate genuinely lowprevalence or poor detection and recording.There is some evidence (by comparing QOFdata between 2004-5 and 2005-6), of all thesei.e. there were large increases in prevalence insome practice and slight falls in others, thoughthe national average increased marginally andmost practices had reasonably stable estimatessuggesting that by and large recording rateshad stabilised. In many areas the levels ofrecorded diabetes were close to thosepredicted by the PBS model i.e. we believe theindicator to be a good estimate of actualprevalence. (See the PBS diabetes prevalencemodelhttp://www.yhpho.org.uk/viewResource.aspx?id=7).

Interpretation: Potential for error due tothe measurement method See above. Alsobecause recording is rewarded through QOFpoints there may be potential for “gaming”.There are a large number of codes used torecord diabetes on GP systems which may leadto counting errors depending on how the datais extracted (see the QOF definitions for thecodes used). There may also be potential biasesin the attribution of practice populations tolocal authority areas but these are probablysmall.

Interpretation: Potential for error due tobias and confounding There may be under-sampling of young people, ethnic populationsand other vulnerable groups e.g. the homelessand travellers in the numerator.

Appendix A: Full anatomy of an indicator

36

Project Team: Jean Penny, Clare Allen, Sandra McNerney, Catherine Hannaway,Mark Lambert, David Pencheon.

The project team would like to extend sincere thanks to everyone who contributedtheir time, expertise and personal experiences to make this guide more relevantand readable.

Special thanks go to:

Julian Flowers, Jake Abbas, Bobbie Jacobson, John Wilkinson, John Kemm, MaryMorrissey, Paula Whitty, Mohammed Mohammed, Ruairidh Milne, Martin Bardsley,Padmananhan Badrinath, Richard Thomson, Veena Raleigh, Bernard Crump, David Spiegelhalter, Maria Goddard, Vern Farewell, Sheila Bird, Tom Ling.

Wider reference group:

Clare Allen, Padmananhan Badrinath, Martin Bardsley, Sian Bolton, Sue Braysher,Susan Cannon, Phil Cowley, Bernard Crump, Hannah Dobrowolska, Mark Emerton,Julian Flowers, Maria Goddard, Catherine Hannaway, Jane Hayward, Martin Heaven, Stuart Jeffrey, Mark Lambert, Sandra McNerney, Ruairidh Milne,Mohammed Mohammed, Mary Morrissey, Rachel O'Brien, David Pencheon, Jean Penny, Liz Robin, Emma Sanford, Julia Sharman, Clive Slater, Gary Thompson,Trish Thompson, Nick Tyson, Simon Watson, Paula Whitty, Daisy Wild.

Confidence intervals: Definition andpurpose A confidence interval is a range ofvalues that is normally used to describe theuncertainty around a point estimate of aquantity, for example, a mortality rate. Thisuncertainty arises as factors influencing theindicator which are subject to chanceoccurrences that are inherent in the worldaround us. These occurrences result in randomfluctuations in the indicator value betweendifferent areas and time periods. In the case ofindicators based on a sample of thepopulation, uncertainty also arises fromrandom differences between the sample andthe population itself. The stated value shouldtherefore be considered as only an estimate ofthe true or ‘underlying’ value. Confidenceintervals quantify the uncertainty in thisestimate and, generally speaking, describehow different the point estimate could havebeen if the underlying conditions stayed thesame, but chance had led to a different set ofdata. The wider the confidence interval thegreater the uncertainty in the estimate.

Confidence intervals are given with a statedprobability level. In Health Profiles 2007 this is95%, and so it is said that there is a 95%probability that the interval covers the truevalue. The use of 95% is arbitrary but isconventional practice in medicine and publichealth. The confidence intervals have also beenused to make comparisons against thenational value. For this purpose the nationalvalue has been treated as an exact referencevalue rather than as an estimate and, underthese conditions, the interval can be used totest whether the value is statisticallysignificantly different to the national. If theinterval includes the national value, thedifference is not statistically significant and thevalue is shown on the health summary chartwith a white symbol. If the interval does notinclude the national value, the difference isstatistically significant and the value is shownon the health summary chart with a red oramber symbol depending on whether it isworse or better than the national valuerespectively.

The mission of the NHS Institute for Innovation and Improvement is to support the NHSand its workforce in accelerating the delivery of world-class health and healthcare for patientsand public by encouraging innovation and developing capability at the frontline.

NHS Institute for Innovation and ImprovementCoventry HouseUniversity of Warwick CampusCoventryCV4 7AL

Tel: 0800 555 550Email: enquiries@institute.nhs.ukwww.institute.nhs.uk

The Association of Public Health Observatories - Better information, better decisions, better health.

The Association of Public Health Observatories (APHO) represents and co-ordinates the workof 12 Public Health Observatories (PHOs) working across the five nations of England, Scotland,Wales, Northern Ireland and the Republic of Ireland. APHO ensures effective joint working acrossthe PHOs to deliver information and intelligence on people's health and health care for practitioners,policy makers and the public. Its high quality public health intelligence informs health policy anddecision making at national level.

PHOs work in partnership with governments, health, academic and commercial organisations fromlocal to national level. They have a lead role in key policy areas to:

* develop expertise and in-depth knowledge* publicise significant work* provide a single point of contact and information source* develop training programmes for health intelligence staff and

other practitioners

Further information can be found at: www.apho.org.uk