Reflections from a realist evaluation in progress: Scaling ladders and stitching theoryMelanie Punton, Isabel Vogel and Rob Lloyd21 April 2016

2

‘The effective use of research and evidence can

play a crucial role in making policy more

successful.’

3

Session overview• What is BCURE?• What is realist evaluation?• The stages of our realist journey, and challenges we faced

along the way– Developing theory– Testing theory – Refining theory

• What can realist evaluation offer international development?

4

Building Capacity to Use Research Evidence (BCURE)

Improved capacity to use evidence in policy

making (O)

Evidence is used more (and better) in policy

making (O)

Better quality policy (O)

TrainingMentoring

NetworkingEvents

Workshops

New tools, systems and guidelines

DFID funded: £13 million, 11 countries, 6 projects, 2013-2017

Working with champions

5

The six BCURE projectsPartner Project name Countries (bold = case study

country)

Adam Smith International (ASI)

Africa Cabinet Decision-Making Programme

South Sudan, Liberia and Sierra Leone

ECORYS Building Capacity for the Use of Research Evidence

Bangladesh

Harvard Data and Evidence for Smart Policy Design

Pakistan, India

African Institute for Development Policy (AFIDEP)

SECURE-Health Kenya, Malawi

University of Johannesburg (UJ)

UJ-BCURE South Africa

VakaYiko Consortium VakaYiko Zimbabwe, South Africa, Ghana

6

What did the commissioners want from the evaluation?

1. To strengthen the global evidence base on the effectiveness of capacity building approaches to support evidence-informed policy

2. To evaluate the effectiveness and value for money of the six BCURE programmes

Three year evaluation, accompanying programme, 2013-2017

7

Why realist evaluation?

• BCURE is a pilot.

• DFID wanted to understand how and why capacity building can contribute to increased use of evidence in policy making…

• …To inform decisions within and beyond DFID about whether to fund and how to design this type of programme in future.

• Theory based approach, developed by Pawson & Tilley (1997)• Not ‘what works’ but ‘what works, for whom, in what circumstances,

and why?’• Answers this through opening up the black box: theories about how

the resources introduced by programmes in particular contexts ‘spark’ mechanisms which generate outcomes.

What is realist evaluation?

Intervention A

Outcome B????

THEORYMechanisms operating in

particular contexts to generate specific outcomes (CMOs)

… the ‘magic spark’ that leads to change

Mechanism

Self-efficacy

‘Aha moment’

Sufficient pre-existing knowledge

Participants are working on (and struggling with?) ‘live’

policy processes

Increased use of evidence in day job

Learning put straight into use to shape a

policy

BCURE training course

Peer support and encouragement

Positive group dynamics; participants on same ‘level’

Attendance linked to promotion

‘Crystalliser’ Awareness of ‘this thing called EIPM’

Context Mechanism Outcome+ =

10

Underlying theoretical assumptionsType of assumption AssumptionThe nature of reality (ontology)

The material and social worlds are real

How we can we learn about reality (epistemology)

We can acquire true knowledge about the world, but knowledge will always be partial and incomplete (as opposed to positivism: 'what you see is what you get‘, or constructivism, ‘all knowledge is subjective’)

How causality works Generative logic of causality

This is important because it has implications for the approach, tools, way of thinking and what the results look like

12

How do you generalise from realist evaluation findings?

• Causal mechanisms are real forces that exist in the (social and physical) world and cause things to happen

• The same mechanisms are present in very different situations

• Realist findings are therefore portable.

How do you do a realist evaluation?

Overall aim: refined and evidenced set of theories about what works to build capacity for EIPM, for whom, in what circumstances and why

Three broad iterative stages

14

Developing theory

Initial Common Theory of Change

15

6. Policy and practice is informed by research evidence

Individual interventions:Training

Mentoring

Secondments

Development of evidence leaders /

'champions'

Organisational interventions:

Facilitating and developing institutional processes,

procedures, and systems

High-level government policy makers:

Ministerial staff

Cabinet Secretaries

Parliamentarians

Senior civil servants

Mid-level government policy makers:

Technical and research staff in government

Departments

Mid-level civil servants

Civil society, the media, researchers, and the

public

Network interventions:Policy development pilots and demonstration cases

Policy networks and relationships strengthening

Policy dialogue with civil society / media

1.1 Improved skills, knowledge and confidence of individuals around accessing, appraising and using evidence in policy process

1.2. Improved motivation and commitment of individuals to use evidence: eg Ministerial staff seek out expert advice

2.1. Targeted leaders champion and endorse EIPM

3.1. Organisational systems and procedures are established that support and incentivise EIPM eg. Budgetary and approval incentives around EIPM for policy approval processes

2.2 Strengthened interaction between national and international individuals and institutions around the production and use of evidence

3.2. Policies and guidelines on EIPM are established and being used e.g. standards, quality assurance of policy proposals

3.3. EIPM is integrated into civil service competency frameworks, professional development and training

4.2. Civil society and the media regularly and effectively engage in / report EIPM

1.3. Individuals value the use of evidence to deliver mandates and political goals

4.1. Increased interest in and debates on the use of evidence in policy making by civil society , the media and the public

5. Increase in the demand for and use of evidence

Individual level change

Interpersonal level change

Organisational level change

Changes in the institutional context

Impact

Poverty reductionand improved quality

of life

7. The quality of policies and programmes will improve

Within this overarching CTOC, explanations of particular causal links and processes are hypothesised in the form of intervention-context-mechanism-outcome configurations (ICMOs)

16

6. Policy and practice is informed by research evidence

Individual interventions:Training

Mentoring

Secondments

Development of evidence leaders /

'champions'

Organisational interventions:

Facilitating and developing institutional processes,

procedures, and systems

High-level government policy makers:

Ministerial staff

Cabinet Secretaries

Parliamentarians

Senior civil servants

Mid-level government policy makers:

Technical and research staff in government

Departments

Mid-level civil servants

Civil society, the media, researchers, and the

public

Network interventions:Policy development pilots and demonstration cases

Policy networks and relationships strengthening

Policy dialogue with civil society / media

1.1 Improved skills, knowledge and confidence of individuals around accessing, appraising and using evidence in policy process

1.2. Improved motivation and commitment of individuals to use evidence: eg Ministerial staff seek out expert advice

2.1. Targeted leaders champion and endorse EIPM

3.1. Organisational systems and procedures are established that support and incentivise EIPM eg. Budgetary and approval incentives around EIPM for policy approval processes

2.2 Strengthened interaction between national and international individuals and institutions around the production and use of evidence

3.2. Policies and guidelines on EIPM are established and being used e.g. standards, quality assurance of policy proposals

3.3. EIPM is integrated into civil service competency frameworks, professional development and training

4.2. Civil society and the media regularly and effectively engage in / report EIPM

1.3. Individuals value the use of evidence to deliver mandates and political goals

4.1. Increased interest in and debates on the use of evidence in policy making by civil society , the media and the public

5. Increase in the demand for and use of evidence

Individual level change

Interpersonal level change

Organisational level change

Changes in the institutional context

Impact

Poverty reductionand improved quality

of life

7. The quality of policies and programmes will improve

Initial CMOs

‘What works, for who, in what circumstances and

why?’

Building on existing theories, not starting

from scratch

‘Capacity building for

EIPM’ chock full of theories and

assumptions

‘Good quality’

‘evidence’

‘Good quality’ ‘policy’

‘Capacity’

Rich literature from diverse

fields

PsychologyAdult learning

Political science

Health

Role of evidence in policy

International development

The literature review shaped how we think about ‘building capacity for evidence informed policy making’…

‘Research evidence’ just one form of evidence required to

make policy

Appropriateness of evidence as important as ‘quality’

Evidence is never neutral

Need to look beyond rational and linear models of

policy processes, to encompass the role of

power and politics, networks and interactions, cognitive

limits of rationality Capacity is multi-dimensional

19

Challenge 1: Developing, unravelling and re-stitching CMOs

“Is this really a mechanism?”

What helped:• Recognising that CMOs are heuristics, not reality• Introducing an ‘I’ into the ‘CMO’• Sentences, metaphors, catchy names

20

Challenge 2: How many ICMOs?

What helped:• Pragmatism!• Focussing on operational relevance• Constrained by resources and access• Guided by literature: where could we add most value?

21

Testing theory

22

Testing theory• Three rounds of data collection and analysis, with each

round encompassing:

– 6 country visits

– Around 30 qualitative interviews per country: project staff, intervention participants, high level government informants etc.

– Review of monitoring data collected by programmes, policy documentation that provides evidence of change in processes, etc

23

Challenge 3: Mastering the art of the realist interview

“I’ll show you my theory if you show me yours”

What about confirmation bias?

What (we hope will) help:• More training• Asking for examples• Adjudicate between rival theories

24

Challenge 4: Analysing our data in a realist way

25

Refiningtheory

26

Refining theory“How do we get from the granular findings to revised ICMOs?”

• Ladder of abstraction• Elements of meta-ethnography

How and why does capacity building lead to change?

ICMO 2: the ‘eye opener’

Based on 14 interviews from two countries: Zimbabwe and Kenya; plus

ACD regional conference

28

Challenge 5: encompassing complexity

CMOs looked like this

Reality more like this:

What helped• Thinking in terms of ‘levels of a system’• Layering ICMOs: outcome on one level may become a context

on another level; feedback loops

ICMO 15: 'reinforcement.' Where organisational tools or systems (e.g. checklists or guidelines) encompass positive or negative incentives to apply evidence in policy making (I) and where the tool or system is strategically positioned or has legislative backing (C) this reinforces evidence-informed policy making behaviours (M), leading to...

ICMO 10: 'transformational leaders.' Where informal support (I) is given to a senior 'champion', who is committed to evidence-informed policy making and possesses good interpersonal skills and political relationships, credibility and respect (C), they act as a 'transformational leader' -exercising high level influence at a senior level to promote evidence use and initiate reforms (M), leading to...

...people using evidence more and more effectively in their work (O)

...new tools, systems, or procedures for evidence-informed policy making (O)

...new and future high-level organisational champions for evidence-informed policy making (O)

ICMO 2: the 'eye opener.' When training is practical, interactive, needs focussed and targets people who can directly apply learning (I) and where participants are internally or externally motivated to use evidence in their work (C) it sparks an 'eye opener' in which trainees recognise how training principles apply to their work (M), leading to...

Organisational

Interpersonal

Individual

Process(intervention, context, mechanism)

Level of change

Outcome

30

What can realist evaluation offer international development?

• Systematic way of exploring context and complexity• Operationally relevant findings (although be careful about

your messaging!)

But:

• Tensions between RE and structures and incentives of aid industry?