governance by targets & terror: synecdoche, gaming & audit gwyn bevan (lse) &...

Governance by Targets & Terror: Synecdoche, Gaming & Audit

Gwyn Bevan (LSE) & Christopher Hood (All Souls College, Oxford)Westminster Economic Forum

20 April 2005

Governance by Targets & Terror

New approach to health care governance in 2000s

Decisive breakthrough in governance – or repeat of history of Soviet Union?

Conclusion

Governance by Targets & Terror New approach to health care

governance in 2000s targets & indicators linked to negative

feedback Voltaire: ‘ici on tue de temps en temps un

amiral pour encourager les autres’ Decisive breakthrough in governance –

or repeat of history of Soviet Union? Conclusion

Three phases of governance of public health care systemUK Concordat clinical autonomy & resource

constraints no real command or control from the centre 1980s: various attempts to empower managers

1990s: attempt to control public health-care professionals through quasi-markets

separating providers & purchasers

England only from 2001- new ‘concordat’ of higher spending

accompanied by P.I.s & targets monitored from centre by multiple & overlapping units (& ‘terror’?)

NHS spend as % of GDP

0123456789

10

% GDP

Sources: Office of Health Economics, HM Treasury, & official projection of 9.4 per cent by 2008

Prime Minister’s Delivery Unit22 targets

Treasury130 PSA targets– c. 10 for health

Dept of Health50 targets by trusttype

Healthcare Commission Quality regulator develops & publishes ratings

Health care ‘trusts’ (c.700)

Audit Commission & NAOAudit finance & vfm

money reporting dialogue

Institutional arrangements

Some underlying assumptions Synecdoche: a part can meaningfully

stand for the whole

What is measured is a good indicator of performance (e.g. ‘threshold effects’ at the top of the quality range either will not occur or do not matter)

Gaming effects (the ‘knights-to-knaves’ problem) are either small or unimportant

Defining priorities

Residual domain β

Domain α: government’s priorities

Measuring priorities

Residual domain β

Domain αg: good measures M[αg]: no false positives or negatives

Domain αn:

no measures

Domain αi: imperfect measures M[αi]: large numbers false positives & negatives

The synecdoche assumption: the part can represent the whole

Domain α-: government’s priorities for which good & imperfect measures exist

Residual domain β+: omitted because unimportant or cannot be measured

‘Threshold Effects’ either don’t matter or won’t Happen

Target

Success

Waiting timeIn months

Failure?

Frequency

Knights either will not turn into Knaves or Knavery can be Controlled‘Saints’ may not share mainstream goals public service ethos so high that they voluntarily disclose

shortcomings to central authorities‘Honest triers’ broadly share mainstream goals do not voluntarily draw attention to their failures but do not attempt to spin or fiddle data in their favour‘Reactive gamers’ broadly share mainstream goals, but aim to spin or fiddle data if they have a motive or

opportunity to do so‘Rational maniacs’ do not share mainstream goals aim to manipulate data to conceal their operations

Target & PI systems against 4 types of actors

‘Saints’

‘Honest triers’

‘Reactive gamers’

‘Rational maniacs’

Expected effect of targets

NO CHANGE

What is measured is a good indicator of performance?

Works for saints Problems for honest triers Vulnerable to Reactive gamers Fails for rational maniacs Gresham’s law: saints & honest

triers Reactive gamers?

What is measured is a good indicator of performance? Agent satisfied signals M[α-]

Domain α-

All is well? Domain α- but

domain β+ ? Domain α- ? Failure on M[α-]

concealed by problems of

definition & measurement?

gaming?

Domain β+

Signals M[α-]:

Governance by Targets & Terror

New approach to health care governance in 2000s

Decisive breakthrough in governance – or repeat of history of Soviet Union? Some evidence from the English NHS

Conclusion

Development of star ratings in England

2001 2002from2003

Acute (156)Specialist (20)

Ambulances (31) Mental Health (88)

PCTs (304)

Evidence of impact of key targets

Ambulances 75% category A

calls < 8 minutesHospitals total time in A&E

< 4 hours waiting times for

elective inpatient admission

Reported successes

Problems of measurement & gaming

Reported success: 75%category A calls<8 minutes

0

25

50

75

100

20000

25

50

75

100

2003

Before After

Problems of measurement & gaming: third ‘corrected’ response times

75% < 8 minutes

Source: http://www.chi.nhs.uk/eng/cgr/ambulance/index.shtml

‘Corrections’’ only 2% to 6%

75% < 8 minutes

Impact: A&E total time < 4 hours

60

80

100

Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1

2002-03 2003-04 2004-05

& a 20% increase in numbers in A&E

Source: http://www.nao.org.uk Improving Emergency Care in England

Problems of measurement & gaming in A&E Problems of measurement

Government figures: mid-2004, target met by 96% of patients

Healthcare Commission survey (55,000 patients): 77% of patients stayed < four hours in A&E

Problems of gaming Queues of ambulances outside Moving staff & cancelling operations over

period of measurement

Impact of key targets on hospital waiting times for elective inpatient admission

0 3 6 9 12 15 18

2001

2002

2003

2004

Target waiting time (months)for elective inpatient admission

Impact: hospital waiting times elective admission

0

50

100

150

200

1997 1998 1999 2000 2001 2002 2003 2004

>12 (2003)>9 (2004)

Numbers waiting elective admissions (‘000s)

Star ratings published

Source: http://www.dh.gov.uk Chief Executive’s Report to the NHS – Statistical Supplement (2004)

Impact of key target: hospital waiting times elective admission

0

50

100

150

200

1997 1998 1999 2000 2001 2002 2003 2004

>12 (2003)>9 (2004)

Numbers waiting elective admissions (‘000s)

Star ratings published

Source: http://www.dh.gov.uk Chief Executive’s Report to the NHS – Statistical Supplement (2004)

Impact of key target: hospital waiting times elective admission

0

10

20

30

2000 2001 2002 2003

England

Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

% patients waiting for hospital admission > 12 months

Impact of key target: hospital waiting times elective admission

0

10

20

30

2000 2001 2002 2003

EnglandScotland

Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

% patients waiting for hospital admission > 12 months

Impact of key target: hospital waiting times elective admission

0

10

20

30

2000 2001 2002 2003

EnglandWalesScotland

Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

% patients waiting for hospital admission > 12 months

Impact of key target: hospital waiting times elective admission

0

10

20

30

2000 2001 2002 2003

EnglandWalesScotlandNorthern Ireland

Source: http://www.statistics.gov.uk National Health Service hospital waiting lists by region: Regional Trends 35, 36, 37 & 38

% patients waiting for hospital admission > 12 months

Natural experiment in UK countries England (from

2001): annual performance (star) rating Zero to ‘naming &

shaming’ Wales, Scotland,

Northern Ireland no ranking no incentives

90

95

100

105

110

115

England Scotland Wales NorthernIreland

1998-992002-03

Spend per capita on health care (UK = 100)

Source: www.hm-treasury.gov.uk/media//B4887/pesa04_chapter08_190404.pdf

Waiting times: problems of measurement & gaming

Problems of measurement Audit Commission: reporting errors at

least one PI in 19 trusts Problems of gaming

NAO: 9 NHS trusts inappropriately adjusted their waiting lists

Audit Commission: 3 cases of deliberate misreporting of waiting list informationSources:

www.nao.gov.uk/publications/nao_reports/01-02/0102452.pdfwww.audit-commission.gov.uk/health/index.asp?catId=english^HEALTH.

‘Synecdoche’ problems over quality of care 2 failures that resulted in major public

inquiries Bristol case of paediatric cardiac surgery Shipman

All difficult to detect from outside All could plausibly have sent ‘satisfactory’

M[α-] signals under star ratings regime And M[α-] ‘failures’ might nevertheless

provide excellent quality of care

Governance by Targets & Terror New approach to health care governance

in 2000s Decisive breakthrough in governance – or

repeat of history of Soviet Union? Conclusion

How far did English NHS satisfy assumptions? Repeating Soviet history: 1939? 1969? 1989? Policy implications?

How far did English NHS satisfy assumptions? Synecdoche: a part can meaningfully

stand for the whole?

What is measured is a good indicator of performance?

Gaming effects (the ‘knights-to-knaves’ problem) are either small or unimportant?

Repeating Soviet history: 1939? 1969? 1989? 1939: waiting times? 1969: ? 1989: balanced scorecard?

Policy implications: targets retained, limiting gaming? Limits on transparency Designing & resourcing systems of

validation Sources of information other than

data reported by those being monitored