The role of evidence in policy making: a Welsh

Government perspective

Richard Thurston

Head of Research for Education & Skills

Knowledge and Analytical Services, Welsh Government

Outline

1.  The role of evidence (evidence-based policy-making in theory)

2.  The role of evidence (evidence-based policy-making in practice)

3.  The challenges – what makes the job interesting?

1. The role of evidence (in theory) Governments need robust evidence to...

•  Anticipate issues/trends that governments will need to respond to

•  Monitor social and economic change and policy implementation

•  Appraise which policy/delivery options are likely to be most cost effective

•  Tell us what key stakeholders think and understand

•  Know whether or not policies are effective and having an impact (and if not, why not)

•  Help to identify and apply lessons to improve policy over time (design and implementation)

•  Ultimately – allowing more informed decisions

1. The role of evidence (in theory)

The origins of evidence-based policy-making •  Not a new idea, but a relatively new ‘paradigm’ in government

•  New Labour – out with ideological government, in with rational decision-making

•  Prominent not only in government

•  Embodied in Civil Service Code

•  As ubiquitous as ever?

2. The role of evidence (in practice)

EBPM has some conceptual difficulties •  Evidence-based, evidence-informed, evidence-aware – what’s

the difference?

•  Government is no more ‘doctor’ than society is ‘patient’

•  No linear model of policy development – not formulaic

•  Evidence mainly about the past, policy mainly about the future

•  Detractors argue there must be policy before there is evidence (value judgements play a bit part)

2. The role of evidence (in practice)

All of the below constitute evidence-based

Instrumental ConceptualLeads directly to decision Leads to changes in knowledge,

making for policy and practice understanding or attitude

Mobilisation Wider InfluenceUsed as instrument of persuasion Leads to larger scale shifts in

to legitimise action/in action thinking

(Nutley et al 2002)

2. The role of evidence (in practice)

Views of politicians vary…

2. The role of evidence (in practice)

‘Social science research evidence is central to development and evaluation of policy ….we need to be able to rely on social sciences and social scientists to tell us what works and why and what types of policy initiatives are likely to be most effective’

Views of politicians vary…

2. The role of evidence (in practice)

Views of politicians vary…

"...we, as politicians, are elected by people to make a difference, to make things happen; we look at the evidence, but there comes a time when you say, 'you use your judgment‘. There could be other things where there is evidence that something works [but] for perfectly good policy reasons, we say it is not the thing we want to do”

2. The role of evidence (in practice)

Views of politicians vary……..

“For me, real value is about being absolutely focused on the outcomes we are achieving out there in the real world…Tell us the facts and set out the evidence that underpins the options”

First Minister (presentation to

senior civil servants, 2011)

2. The role of evidence (in practice)

Source: Davies (2004:7)

2. The role of evidence (in practice)

The ROAMEF cycle: a role for evidence at every stage?

2. The role of evidence (in practice) The ROAMEF cycle: a role for evidence at every stage? Rationale – e.g. evidence that there is a problem/question to be addressed, evidence that government intervention has the potential to help;

Objectives – e.g. evidence on what aspects of the problem can be remedied, over what timescales and in what quantities;

Appraisal – e.g. comparing different options to deliver the objectives, in terms of how likely they are to work and how much they cost;

Monitoring – e.g. gathering monitoring data, and other evidence, during implementation to feed in to evaluations;

Evaluation – e.g. an impact evaluation to determine whether the intended effects of a programme have been realised, or a process evaluation to determine how that happened;

Feedback – e.g. using evaluation findings to determine the future of a programme or inform the design of a new one.

2. The role of evidence (in practice)

The Welsh Government applies five principles that should be considered at every stage of our work:

Fit with the Government’s programme and Ministerial priorities

Impact on the people of Wales and the supporting evidence

Cost of our investment

Mechanisms available to incentivise change

Management of the work

2. The role of evidence (in practice)

What is KAS? What is KAS?

•  GSR, GES, GSS, GIS, Operational Research, Library

•  Collection, analysis and presentation of data and research for policy makers and the public.

•  Professional standards (quality standards, ethics, publication, etc)

•  Analytical teams for each DG

•  Cross-cutting work (e.g. National Survey, Corporate Research)

2. The role of evidence (in practice)

The sorts of things we do (in GSR) What is KAS? •  Describe situations factually and how they change over time

•  Monitor specific aspects of policies/programmes and how they change over time

•  Appraise proposed policies/options to establish their likely efficacy.

•  Evaluate how things worked in practice to learn lessons and assess impact

•  Calculate value for money through assessing cost effectiveness, efficiency and effectiveness

•  Estimate social impacts of policies/programmes

•  Understand and measure behaviour change

17

A departmental view …

•  Education & Skills Analytical Team •  DfES Evidence Plan •  DfES Evidence Board •  Minister for Education and Skills •  Ministerial Policy Board •  Delivery Board •  Ministerial Advisory Group

18

Education Examples – data collection and analysis

•  Statistics •  Economic appraisal •  Surveys •  Secondary data analysis •  Data linking •  Qualitative Research •  Research syntheses •  Policy evaluations •  Programme trials •  Explorative Research

3. The challenges – what makes the job interesting?

The trade-off between quality vs timeliness •  Good social science can take a long time

•  But governments need to make quick decisions

•  Challenge in reconciling these conflicts – how?

•  Expediting - rapid evidence assessments, interim evaluations, accessing expertise, shifting resources

•  Giving best quality advice possible, with limitations clearly laid out

3. The challenges – what makes the job interesting?

The limits of causality and certainty •  Understanding the impacts of policy is difficult

Cha

nge

in im

pact

mea

sure

Counterfactual (no intervention)

New

initi

ativ

e

Post-new initiative trend

Impact?

3. The challenges – what makes the job interesting?

The limits of causality and certainty •  Understanding the impacts of policy is difficult

•  But policy makers need to understand in order to make good decisions and spend money wisely

•  Difficulties exacerbated by the timeliness vs quality trade-off and methods of policy implementation

•  We endeavour to:

•  carry out sound impact evaluations

•  influence the way policies are designed and rolled out

•  advise against ineffective policy options

3. The challenges – what makes the job interesting?

Cultural challenges around evidence and research •  Awareness of the importance of evidence varies (e.g. “We don’t need to evaluate that, we know it works”).

•  Objectively assessing policy does not always fit with political priorities (e.g. “We’re keen to demonstrate the success of this policy”).

•  Balanced view of effectiveness sometimes seen as a negative (e.g. “We’re reluctant to do that in case it shows the policy isn’t working as well as we thought”)

•  Sometimes the ‘wrong’ evidence can be the most powerful (e.g. “Statistics show we’re doing a lot more of that, so we’re definitely having an impact”).

3. The challenges – what makes the job interesting?

Resources, resources, resources •  Good quality social science takes time and financial commitment (often lots!)

•  Balance between commissioning and in-house work to keep skills sharp

•  Budgets are increasingly under pressure across the board – more research funding = hard sell

•  We continue to find creative ways to boost resources (like PhD Internships)

3. The challenges – what makes the job interesting?

The counter-arguments •  Financial pressure means the question of ‘what works?’ is more important than ever

•  Hard to answer this question without rigorous social science

•  Even the most costly research is less costly than ineffective policy

•  More sophisticated approaches yield better estimates of impact and cost – pursue the most rigorous approach possible especially when equipoise exists

25

Barriers to evidence based policy…?

26

Evidence enablers?

27

Thank you Any questions?

Email: Richard.thurston@wales.gsi.gov.uk Web: http://wales.gov.uk/statistics-and-research/?lang=en