Research into research use:A ten-arm cluster RCT

10th September 2014

Dr Ben Styles, Dr Anneka Dawson (NFER), Dr Lyn Robinson and Dr Christine Merrell (CEM)

Background on evidence- based education

What research use experts said:• How are you going to measure outcomes?• Can you expect to see any difference?

Research dissemination: competing ideas

Passive methods are rarely effective

Passive methods are cheap i.e. a tiny effect is still cost effective

Measurement of attainment is cheap

(National Pupil Database)

Measurement of research use is complex

and expensive

Passive methods may not require consent

Several interventions

trying to achieve the same goal

Active methods can be expensive

Trial design

Two RCTs focused on improving Key Stage 2 literacy through teachers’ research use:

• Active trial- 780 primary schools randomised to experience one of varied activities from CPD to online communities (or control)

• Passive trial- Most other English primary schools randomised to receive one of various research-based communications (or control)

Outcomes

ACTIVE TRIAL PASSIVE TRIAL

Primary outcome Attainment in literacy at age 11

Attainment in literacy age 11

Secondary outcomes Teachers’ awareness, understanding and action in relation to research use

n/a

How many arms in the active trial?

• Four providers of research evidence, each with an active and passive approach

• Control group• CEM need three arms

Active trial

IEE

Electronic/ paper

materials

Evidence fair

Campaign for Learning/ Train

Visual

Website

Support

CEM

Paper materials

Light CPD

Advanced CPD

ResearchEd/ Nat Cen

Conference

Online community

Control group

780 schools recruited

60

60

60

60

60

60

60

60

60

240

Passive trial

IEE

Electronic/ paper

materials

Campaign for Learning/ Train

Visual

Website

CEM

Paper materials

ResearchEd/ Nat Cen

Conference

Control group

12,500 schools

2500 2500 2500 2500 2500

Timeline

Recruitment and baseline research

use measure

sAutumn

2014

Randomisatio

n

December

2014

Research

dissemination

Spring 2015

New teaching methods

and future

planning

Summer 2015

New teaching methods

Autumn 2015-spring 2016

Research use outcom

es

Spring 2016

Attainment

outcome

Summer 2016

Attainment

outcome

Summer 2017

CEM intervention

LEVEL 3: Evidence-based materials, two CPD sessions, pupil diagnostic information (InCAS), peer observations

LEVEL 2: Evidenced-based materials, one CPD session

LEVEL 1: Evidence-based materials (includes advice booklet, ‘tips for teachers’ cards and monthly classroom posters)

Challenge 1-Recruitment

Starting point- why do schools want to take part in research?:

• Having their say• Doing something good for education• The process being easy and sounding easy• Trust in the organisation approaching them• Security of data • Clear concise communication • Offer of feedback

Our recruitment approach

• Promotion at Research Ed and other provider events

• Word of mouth between schools and providers, NFER, EEF

• Website with detailed information on the trial and what is expected of schools and form to express an interest

• Letters, phoning, emails targeted response• Helping inform schools about RCTs including

our guide for schools

Recruitment (continued)

Challenge 2- Secondary outcome measurement

• Secondary outcomes are ‘awareness’, ‘understanding’ and ‘action’

• Measure using a bespoke survey • Schools involved in other research projects

using this survey cannot be in this trial

Research use outcomes

• Baseline survey developed without pilot• ‘Awareness’, ‘understanding’ and ‘action’

assessed through a series of questions in a survey of teachers

• Follow-up survey will be piloted in autumn• Questions will be retained if they discriminate

well and contribute to reliable measures

Intention-to-treat analysis

• Attainment (multilevel model, attainment at age 7 as covariate)

• Research use is also hierarchical but low numbers per cluster (research use at baseline as a covariate)

Challenge 3- Coordination of the delivery organisations

With 6 delivery partners carrying out different interventions things to consider:

• Effective, timely communication (inc one main point of contact at NFER and one at each provider)

• Funder as mediator for more basic updates• Monitoring information planned in advance

Acknowledgements

• Education Endowment Foundation• NFER Education Trials Unit

For more information

For Literacy Octopus project information:

www.nfer.ac.uk/octopus

For our guide for schools:

http://www.nfer.ac.uk/publications/RESM07/RESM07_home.cfm

For information on RCTs for educational researchers:

http://www.nfer.ac.uk/publications/RCT01/RCT01_home.cfm

For the NFER Education Trials Unit with details of our other trials

http://www.nfer.ac.uk/research/trials-unit/

NFER provides evidence for excellence

through its independence and insights, the

breadth of its work, its connections, and a

focus on outcomes.

National Foundation for Educational Research

The Mere, Upton Park

Slough, Berks SL1 2DQ

T: 01753 574123

F: 01753 691632

E: enquiries@nfer.ac.uk

www.nfer.ac.uk