Principles & Practice of Evaluation

Erica Wimbush

Head of Evaluation, NHS Health ScotlandScotPHO Training, 29th March 2011

Outline

What is it? Why do we do it? Who is it for? When do we do it? How is population health data used? What are the different types of evaluation? Examples of ‘good’ evaluations

Examples

Monitoring & Evaluation of Scotland’s Alcohol Strategy (MESAS) – outcome planning

Evaluation of Keep Well (Wave 1) – pilot phase Evaluation of WoSCAP – pre-testing Evaluation of the Smoking Ban - implementation Review of Scottish Diet Action Plan - review

Evaluation – what is it?

The making of a judgement about the amount, cost or

value of somethingOxford English Dictionary

“The ultimate goal of evaluation is social betterment to which

evaluation can contribute by assisting democratic institutions to better

select oversee, improve and make sense of social programs”

Mel Mark 2007

“The primary purpose of evaluation is to improve the human condition

… to help determine if the promised improvements of social programs

are actually delivered” (Lipsey 2001)

Ensuring that the interests of all individuals and groups

in society are served” (Hopson 2001)

Focus of Evaluation

Planned interventions that are intended to bring about change

a policy or policy mechanism an agency or organisation a service a programme or project a practice

Why do we do evaluation?

How effective/successful? To provide sound evidence of programme effects - what actually happened vs what was intended

Better understanding about how programmes work

Generate learning from programme implementation to inform decision-making and improve practice

Accountability - Assurance to funders about how (public) money has been spent

Who is evaluation for?

Evaluation stakeholders What sort of evaluation is valued?

Policy-makers Effectiveness; what works?

Funders Accountability

Planning & Performance Performance monitoring/ targets

Managers Developmental/formative evaluations

Researchers Knowledge-building; research quality and utility

Service users Service quality - access, experience, relevance to needs

Principles for evaluation Be focused – on the purpose, what you really need to

know, and what will be useful and used Be realistic – about what you can and should evaluate;

what is possible and what it is in your gift to influence Be proportionate – about how much evaluation is

appropriate Be convincing– to your evaluation audience: what will it

take to convince a reasonable person? Be honest – about why you are evaluating, what the

evaluation will be used for, and what you can claim

Types of evaluation

OUTCOME – assesses effectiveness PROCESS – understanding the processes of

implementation and change FORMATIVE – feeds directly back into

programme development SUMMATIVE – review of evidence and learning

at the end of a period of implementation or funding

Outcome evaluation designs

Experimental True - Random assignment to experimental

and control groups Quasi - Controlled design, non-random

assignment; non-equivalent control groupNon-experimental Time series analysis (single, comparative) Before and after Post-intervention (single, comparative)

Non-experimental designs – the problem of causal attribution

Theory-based evaluation Realistic Evaluation (UK, Ray Pawson & Nick Tilley) Theories of Change (US, Aspen Institute) Programme theory (Australia, Patricia Rogers) Contribution Analysis (Canada, John Mayne)

Used in initial planning stage of programme development Develops dearer thinking about the logical causal connections

between program goals, activities and outcomes Develop a series of statements (assumptions or hypotheses) about

how the program is intended to work that are testable via the evaluation

Alcohol strategy - Theory of change

Reduced alcohol-related

harms

Reduced alcohol

consumption ; safer patterns

of drinking

Economic impact on

alcohol industry

Safer environment

Changed attitudes and social norms

Reduced availability

Reduced affordability

Improved support from services •Drinkers•Children affected by parental drinking

Factors affecting delivery

substitution

External factors

affecting outcomes

OUTPUTS – Direct Control

Key terms: Outputs, Outcomes, Results, Impacts

RESULTS – Visible & Direct Influence

Expected IMPACTS – not Visible, Indirect Influence

Results chain

INPUTS

Resources

ACTIVITIES

What you do

OUTPUTS

What is produced

REACH

Who the outputs reach (target group)

OUTCOMES

Immediate results of

use/uptake (eg KASA)

OUTCOMES

Changes in behaviour, practice or

environments

OUTCOMES

Changes in population

health status

OUTCOMES

Planning

DevelopmentImplement-ation

Review

When do we do evaluation?

Planning

Development

Implement-ation

Review

Evaluation in the Programme/Policy Cycle

Performance monitoring

Impact/ outcome evaluation

Evaluation of Ban on Smoking in Public Places

Use of population health data

Use of logic model to map expected outcomes related to smoke-free legislation

Routine data on ETS health outcomes (e.g. hospital admissions for acute coronary syndrome; consultations for CHD, COPD and asthma)

Routine data from SHS on smoking behaviour and quit attempts – changing patterns before-after legislation

Routine economic data (LFS, GDP, IDBR, ABI) – changes in the economic performance of the hospitality sector before-after legislation (employment, turnover, profitability, openings/closures)

Primary research – seven linked studies

Journal of Public Health, Volume 28, Number 1, March 2006 , pp. 24-30(7) http://jpubhealth.oxfordjournals.org/cgi/content/abstract/28/1/24

Short-term Intermediate Long-term 0-2 mths > 2 –12 mths > 12 mths

Expected Outcomes

Implementation of smoke-free

legislation

Enforcement of smoke-free legislation

Increasing awareness 1 of health risks of ETS, change in attitudes towards ETS exposure

Reduction in exposure to ETS 2 Reduced ETS exposure

Increasing compliance with 3 smoke-free legislation

Sustained compliance with smoke-free legislation

Increasing support for 4 legislation and change in

smoking cultures

Reduction in smoking prevalence and tobacco consumption 5

Reduction in tobacco-related morbidity and mortality 6

Reduction in costs to health service of tobacco-related

illness

Variable economic impact on 7 hospitality sector

Reduction in 8 health inequalities

health inequalities

Sustained cultural change

Logic Model of Expected Outcomes Associated with Smoke-free legislation

Performance monitoring –

HEAT targets

Inputs

Activities

Outputs

Reach

Short-term outcomes

Intermediate outcomes

Long-term outcomes

H6 HEAT target (2008-2011)

Reduced inequal in CHD and cancer

Budget, staff, training, data infrastructure

Actions to deliver effective SC services (NRT + brief advice) in key settings

Smoking cessation services

Adult smokers who want to quit

Increased % smokers who have successfully quit at 1 month

(8% between 2008/9 – 2010/11)

Reduced adult smoking rate

Results Chain

Planning

DevelopmentImplement-ation

Review

Evaluation in the Programme/Policy Cycle

Performance monitoring

systems

Evaluation of pilot

initiatives

Pre-testing

Impact/ outcome evaluation

Evaluation of Keep Well (Wave 1)

Evaluation aimTo assess the feasibility and challenges of delivering Keep Well, and the effectiveness of different approaches to engagement and service redesign, with a view to incorporating the lessons learned from the pilots into subsequent waves of implementation

Understanding of the programme - fit between national and local pilots

Rationales for different approaches in terms of feasibility and doabilty

Track links between activities-processes-outcomes Framework for comparing approaches across pilot areas and

possible unintended effects Evaluability assessment

Evaluation of Keep Well (Wave 1)Use of population health data

Phase 1 (2007-10)Informed by Theory of Change - understanding the process of implementation of Keep Well1: Tracking theories of change at national level and local pilots2: Tracking the impact of KW on ‘anticipatory care’ in the target population using secondary data

Phase 2 (2009-10)Informed by Realistic Evaluation - deeper understanding of certain facets through use of case studiesPractice level case studies to assess the impact of aspects of Keep Well (informed by Phase 1 findings)

Patient and practice experiences (2009-10)Collection of quantitative and qualitative data at practice level, and patient level including patients recruited via Keep Well practices and community-based venues.

Pre-testing – Bowel Cancer Campaign (WoSCAP) Use of Qualitative research

5 concepts pre-tested in 6 focus groups ‘niggling worries’ addressed real

barriers to action –inertia and fear Proved compelling and intriguing due

to dialogue with man confronting his fears

It had a direct call to action –‘go to your doctor if your bowel habits change or you have blood in your motions’

It was felt to be an empathetic way of tackling the fear that surrounds the subject area

Planning

DevelopmentImplement-ation

Review

Evaluation in the Programme/Policy Cycle

Performance monitoring

Impact/ outcome evaluation

Evaluation of pilot

initiatives

Pre-testing

Health Impact Assessment

Outcome planning

Alcohol strategy – MERGA/MESAS Role of evaluation in strategic planning

Planning stage – scope out the scale and nature of the problem and potential solutions.

Role of MERGA and MESAS Developing a shared understanding of problem Mapping the interlinked outcomes and potential

pathways; identifying range of plausible measures needed and target groups (logic modelling)

Developing monitoring and evaluation plans

Outcome/Results Planning

Increased knowledge and changed

attitudes to alcohol + drinking

Reduced acceptability of

hazardous drinking and drunkenness

Reduced availability of alcohol

Individuals in need receive timely,

sensitive & appropriate support

A culture in which low alcohol consumption is valued and accepted as the norm

Less absenteeism + presenteeism the

workplace.

Less alcohol related incapacity

Reduced alcohol related injuries,

physical and psychological

morbidity + mortality

Increased educational attainment

Safer & happier

families and communities

Reduced health, social care, justice

costs

Healthier individuals

and populations

Intermediate outcomes Long term outcomes

Children in need receive timely and

appropriate support

Safer drinking + wider environments

Increased workplace

productivity

Less alcohol related violence/abuse,

offences and ASB

Less absenteeism + presenteeism in

educational establishments

Fewer children affected by parental

drinking

Reduction in Individual and

population consumption

Reduced affordability of alcohol

National outcomes

Reduce significant inequalities

Model 2:

Model 3:

Model 4:

Model 5:

Model 6:

Model 7:

Fewer children affected by maternal

drinking during pregnancy

Safer drinking patterns

Reducing alcohol related harm: strategic

logic modelStudy 2

Study 4

Study 1 Study 5

Study 6

Study 7

Study 3

Planning

DevelopmentImplement-ation

Review

Evaluation in the Programme/Policy Cycle

Performance monitoring

systems

Evaluation of pilot

initiatives

Pre-testing

Health Impact

Assessment

Review processes

Outcome planning

Impact/ outcome evaluation

Review of Scottish Diet Action PlanUse of population health data

Aim - Progress with implementation and impacts 1996-2005

Community level impactsTwenty-07 cohort study

Qualitative research

Population level impactsDietary targets

Trends in eating out, breastfeeding, food retailing

SDAP ImplementationProgramme evaluations

Evidence from implementation bodies

and food industry

International comparisonsExpert commentary on

food and health policies and implementation in 13

countries

Review Panel

Planning

DevelopmentImplement-ation

Review

Evaluation in the Programme/Policy Cycle

Performance monitoring

systems

Evaluation of pilot

initiatives

Pre-testing

Health Impact

Assessment

Review processes

REAL EFFECTS LIKELY

EFFECTS

Outcome planning

Impact/ outcome evaluation

What makes a ‘good’ evaluation?

Shows the value of what we’re doing

Influences decision making

Tells us if

it’s a good

investment

of our

funding

Contributes to the

evidence base

Helps us

understand

what

difference we

are making