The intrepid researcher:Evaluating programmes and social

interventions in public services

Mary Dixon-WoodsDepartment of Health Sciences

University of Leicester

Why intervene Because there is a problem to be addressed

Because there is a reasonable basis for thinking that it can be addressed, for reasonable cost and with minimal damage

– There are good reasons for believing the intervention is better than current situation

– There is some theoretical or empirical basis for those beliefs

Why evaluate

To find out what the interventional programme actually was (not what it was intended to be)

To identify the effects of the programme (good and bad)

To produce evidence of costs and value for money

Why evaluate To figure out how and why the programme

produced the effects it did

To determine whether to continue with a programme/replicate it elsewhere

To make modifications and adaptations to improve the programme

To satisfy needs for accountability

Foot care in England There are over 70 amputations of feet a week in

England, of which 80% are potentially preventable.

In 2007/8, nearly a quarter of people with diabetes did not have a foot check.

You are twice as likely to have your foot amputated if you live in the Southwest compared with the Southeast.

So – improvement is needed!But how do we intervene? Should we:

• Send an email around to remind GPs to do foot checks?• Train GPs in doing foot checks?• Pay them to do the checks?• Fine them if they don’t do them?• Give them a goal they have to meet for their annual

performance review?• Give them feedback every month on how they compare

with others?• Publish a league table of amputations by GP?• Tell patients they should ask their GP for a foot check?

What tends to happen in social interventions in public services

Bright idea /recommendation from policy push/ “latest thing”

Borrowing or building on a currently fashionable or politically favoured model – e.g. “tough accountability”

Theory of change not explicit

What tends to happen in social interventions in public services

Not clear what process is being targeted No attempt to expose to systematic challenge

or understand how it works No search for unanticipated consequences or

toxic effects Poorly described or at wrong level of

specification, so impossible to reproduce

Provision of fruit and veg

a school

Lower obesity in children

Then a miracle occursccurs

Provision of fruit and veg

a school

Lower obesity in children

Programme theory

“Nothing improves research design so much as having a clear idea about what is being

investigated. An important function of theory in research design is to help researchers ensure that they are playing in the right ballpark to

begin with— that is, to help them avoid studying the wrong thing” (Lipsey, 1993)

What is programme theoryPlausible model of how the programme works

– Makes explicit the assumptions different parties hold– Identifies the desired outcomes– Identifies the activities and resources, and who will

undertake them– Identifies the mechanisms that link those activities and

resources to the desired outcomes– Identifies conditions likely to be favourable or

unfavourable to functioning of the mechanisms– Identifies possible unwanted outcomes (the side

effects)

Programme theory

Can be expressed as a narrative

Or as logic models/diagrams

Inputs Outputs Outcomes - impactSitation

WWhat we invest What we do What the results

are

Evaluative causal analysis: three components

A logical component that allows justification that the causal link is reasonable/plausible

Methodological (design) component capable of enabling reasonable inferences about the extent to which outcomes can be attributed to the program actions

Empirical component that produces evidence of process and outcomes

Programme theory

“All three of the components of causal analysis are importantly and substantially strengthened by an explicit theory about the nature and details of the change mechanism through which the cause of interest is expected to produce the effect(s) of interest”.

But 70% of studies offered either no theory or only general statements of programme strategy/principles.

Lipsey, 1993

Without a decent programme theory

Programme logic may be flawed

Programme may not be causally linked to desired outcomes

The implementers may be blamed if the problem is not solved

It’s hard to replicate the programme or design new, similar ones

Explaining Michigan’s success

Theory Program theories are “small” theories

Not big theories of social phenomena

Not doctrines

May be multiple

Subject to continual re-specification and improvement

Specifying the theory What is the problem you are trying to address?

What is the rationale that supports the way you are trying to tackle it?

Why should it work? Why should it work better than what’s already happening?

What are the distinctive features of the intervention? What makes them distinctive?

Reviewing the theory Are program goals well defined? How will we know

if they have been achieved?

Are the program goals feasible and realistic?

Is the change process plausible, with no significant conceptual and operational weaknesses?

Is there a good operational plan? Are the components, activities and functions well specified?

How to build the programme theory Off the shelf theory + previous studies provide a

point of departure

Programme personnel are working with a theory (even if they don’t know it) – which you need to elicit e.g. through interviews

Programme documents

May produce a causal diagram/logic model or a narrative that links the components

Who cares about the programme? The funders

The people providing the programme

The people implementing the programme locally

The intended beneficiaries

Others serving the same beneficiaries

Kinds of questions we might ask

If the programme is going really well, what will we see? Why?

If the programme is going just ok, what will we see? Why?

If the programme is floundering, what will we see? Why?

Programme designers’ theory as elicited

Children insufficiently exposed to fruit and veg and eat junk instead

Elicited theory

Children insufficiently exposed to fruit and veg

and eat junk instead

Once given turnips, carrots and apples they will realise how lovely fruit and veg is and want to eat it all the

time

Elicited theory

Children insufficiently

exposed to fruit and veg and eat

junk instead

Once given turnips, carrots and apples

they will realise how lovely fruit

and veg is and want to eat it all the time

Children substitute low

calorie fruit and veg for junk

Elicited theory

Children insufficiently exposed to

fruit and veg and eat junk

instead

Once given turnips, carrots and apples they will realise how lovely fruit and veg is and want to eat it all the

time

Children substitute low calorie

fruit and veg for junk

Obesity goes

down and health

improves!

Challenges of building programme theory

Elicited theory is often incomplete or not fully coherent

It sometimes reflects an attempt to please or appease various parties

It is often weak on key links– Operational links in the chain – Logical and conceptual linkages

Challenges of building programme theory

Need to think about values and interests

Whose definition of success gets to count?

What are the responsibilities of evaluators?

Programme in action may look very different from programme as intended

Key activities may not happen or look different from how expected

The mechanisms may not function in the way intended

External and internal contexts may powerfully modify what is possible

Programme designers may improvise, innovate and adapt

So part of the job of the evaluator is working out what really happens, and updating the theory

Evaluating the programme in action

• May need multiple methods (quant and qual) to figure out

– What the programme really is– How it works (mechanisms)– What effects (both good and bad) it has– Whether it operates differently in different

contexts

Evaluation designs• Need to choose design carefully, using programme theory and

nature of answer wanted as a guide

• Before and after studies

• Controlled before and after studies

• Prospectively designed experiments

• Mixed methods

• Qualitative

• Case studies

What is deemed “evaluable”?

• Perceptions of relevance– Just the goals of the program – or side effects too?– How it works as well as whether it works?

• Perceptions of credibility• Perceptions of threat– Who really wants to know the answer?

• Difficulties of accessing data on some measures• Costs or burdens of gathering data• Ethical issues

Using team ethnography• Team of observers + non-observers

• Allows multiple observations

• “Blitzing” or multiple observations over time, documentary analysis, interviews, chats

• Allows for integration of multiple perspectives, esp through team debriefing

• Ideally integrated with quantitative measurement

A programme to improve patient safety

• New system for identifying patients at risk of blood clot

• Process measures - are forms filled in?• Immediate outcomes – do higher risk patients get

preventive treatment? • Ultimate outcomes – are fewer patients harmed?• Mediators and contexts – type of ward, type of

patient, training of staff etc• Side effects – distracts staff, patients ignored once

assessed as low risk, diversion of resource

A programme to improve patient safety

• New system for identifying patients at risk of blood clot

• Process measures - are forms filled in?• Immediate outcomes – do higher risk patients get

preventive treatment? • Ultimate outcomes – are fewer patients harmed?• Mediators and contexts – type of ward, type of

patient• Side effects – distracts staff, once assessed as low risk

ignored, diversion of resource

A training programme to reduce racism

New system for training teachers

Process measures -

Immediate outcomes –

Ultimate outcomes –

Mediators and contexts –

Side effects –

Measuring

Measuring often produces changes in the activities you are monitoring, in addition to any programme effects

Risks of– Not knowing whether your programme worked– Goal displacement– Gaming

Data for performance management• Centralised performance management of public services• “Targets and terror” regime throughout 2000s• Organisations and professionals have become adept at working out

what they need to do to survive

Measuring – unwanted consequences

Incentives to respond to the measures rather than the intentions behind them

Tunnel vision Suboptimisation Myopia Measure fixation Gaming Failure to capture subtle changes

Goodhart’s law

“Any observed statistical regularity will tend to collapse once pressure is placed on it for control purposes”

Interpreting an evaluation

Important to distinguish how the interventions worked from how the programme worked

It is possible for the interventions to be effective but not the programme

Challenges with progamme-theory guided evaluations

• Evaluators get in the way of programme

• May try to recover sunk cost of the programme theory they developed, and be unwilling to consider other possibilities

• Theory may be completely wrong – programme may work but for reasons other than those hypothesised by evaluators

Completing an evaluation Very often the results of programmes are more modest

than hoped for; sometimes a null result or negative

Sometimes the findings are uncomfortable or threatening for various stakeholders

Evaluators are often in multiple principal-agent relationships

Need to plan well in advance for communication and dissemination

Who cares?

“Administrators’ interests in organizational stability, budget maximisation, and the promotion of a favorable image, contribute to a general desire to prefer evaluations and performance reports that do not cast programs in a bad light.”

Schwartz and Mayne, 2005

Evaluating public services

Social science evaluation in public sector organizations has an important role

But such studies may be challenging or risky for the staff being studied

Architectures of ethics control Institutionalised as ethical review by

committee, modelled on medical sciences

Organised to produce an audit trail

Distinctive for the emphasis on– immutable principles as a basis of decision-making– construction of the research participant as

vulnerable and potential victim– Valorisation of informed consent

Current principles Any participation in research should be voluntary

Participants should give consent

Confidentiality and privacy should be respected

Public interest in the results should not trump interests of the individual– Granting of important forms of control to the

“researched”

Staff in public sector organisations Should staff have the protections due to all other

research subjects?

When unit of analysis is the organisation not the individual, what forms of ethics should apply?

To what extent are public sector organisations properly regarded as a “private” sphere and those within it private individuals?

Is there an ethical obligation on staff to allow themselves to be studied?

Consent• Obtaining signed informed consent can be

deeply problematic– Undermines anonymity– Question about what it means to “inform”– Frustration of purpose of research• I’m here to study racism in your organisation• I’m here to find out why child sexual abuse is so rife in

certain institutional settings

– May erode rather than strengthen rights– May undermine moral and ethical sensibility of

researchers

Structuring of research agenda

Risk of inducing cowardice or laziness– Avoid conflict, anguish, and moral turmoil

What questions are researchers prepared to ask?

What questions can they reasonably hope to get an answer to?

What studies should they be discouraged from doing?

Conclusions Conducting evaluations in public sector organisations

poses many challenges

Importance of programme theory

Importance of critical scrutiny of what is deemed “evaluable”

New methods now appearing

Need to revisit ethics for evaluations of public services