How to design and deliver a successful evaluation

19th October 2015

Sarah Lynch, Senior Research ManagerNational Foundation for Educational Research (NFER), UK

Email: s.lynch@nfer.ac.ukNFER website: www.nfer.ac.uk

Are you ready to measure impact?

How is the programme used in schools?

Is it used as intended? If not, why?

What challenges are faced?

What changes should be made?

Feasibility Study/ Pilot

Focus the research questions

Link to the main aims of the programme:• Age of first drink?• Frequency of drinking?• Increased knowledge?

Most important outcomes to measure

For example, background questions (gender, ethnicity etc.), where they get alcohol from, who they drink with and so on…

Other questions/outcomes

Is the programme working as planned? What do teachers think about it? What are the practical challenges?

Process & ‘fidelity’

Strength of evidence

Level 1

Descriptive

Level 2

Correlation/ Comparative

Level 3Before and after

(Users only)

Opinion of “does it work”?Small-scaleLower costVery limited evidence

Users compared with non-usersAfter use only (no baseline)Limited/emerging evidence

One group; usersBefore and after Limited/emerging evidence

€

€ - €€

€ - €€

Strength of evidence

Level 4

Before and after(Users and Comparison)

Level 5

Level 4, with multilevel model analysis

Level 6

Randomised Control Trial

€€

€€ - €€€

€€€

Users and a matched comparison groupBefore and afterModerate evidence

Add more sophisticated analysis Most credible for peer-reviewed publication (Level 5&6) Considerable evidenceExample: Talk about Alcohol, England

Random allocation to groupsMost robustMost credible for peer-reviewed publication (Level 5&6)Considerable evidenceExample: Steps Towards Alcohol Misuse Prevention Program (STAMPP), NI

Sample size considerations • Number of schools using programme • Number of students per school • Size of effect/expected change• If a small change is expected, need larger

sample to detect it• Representative of the population

More schools or more students?Increasing number of schools is better for a school-based programme

Requirements for intervention group

Talk About Alcohol evaluation Year 1 4 lessons

1 hour on website

Year 2 2 lessons

Specify minimum use of programme

Written/online instructionsBut only what will be available

in reality to all schools

Clear guidance on programme

Implement a realistic, consistent approach

Fidelity - is the programme used as intended?

Realistic and practical

Robust, systematic approach

Maintaining balance

Avoiding school drop-out • Be clear about:

– purpose and aims of the evaluation – expectations and evaluations tasks– timescales/number of time points

• Offer incentives:– Feedback on own results compared to whole sample– Prize draws– Offer comparison schools materials later

• Maintain contact; avoid burden; make it simple

Data collectionParentsYoung people

(Age?)Teachers

AttitudesKnowledgeBehaviourCharacteristics

ProcessImpact

Attitudes, role models, perceptions of norms

Questionnaire (online or paper?) Interviews

It’s a challenge in England!Face-to-face interviewsTelephoneQuestionnaire

Before and after questionnaire

Identical questions

Comparability with other studies

Who

What

How

Analysis

• Statistical models to allow for any differences in characteristics of groups

• Need to collect data on student characteristics

• Is any difference between groups statistically significant?

• Need a larger sample to do reliable analysis

Adding to the evidence-base Publish

Other coverage

Report

Peer-reviewed academic articles

Media, newsletters, magazines

Blogs

Influence

Stakeholders

Policy

Strength of evidence Levels 5&6

Discussion

• What challenges do you face? • What solutions have you found? • How can you strengthen your evidence?