challenges of undertaking rcts in education sig workshop monday 18 th march 2013 hannah ainsworth,...
TRANSCRIPT
Challenges of undertaking RCTs in Education
SIG Workshop Monday 18th March 2013
Hannah Ainsworth, Research Fellow, York Trials Unit
School Based Educational Trials
• ECC Trial 1 – one-to one numeracy intervention for Year 2 children struggling in maths
• ECC Trial 2: Pairs and Triplets – small group numeracy intervention for Year 2 children struggling in maths
• 3 Writing Trials – all looking at the impact of a writing intervention on children in the transition from Year 6 to Year 7, focus on children performing below national average
Trials with University Students
• Student Nurses Numeracy – ‘Authentic World’ computer based instruction aimed at improving maths skills needed in the workplace
• Hypnotherapy Trial – hypnotherapy to reduce exam anxiety and improve exam performance students
School Based Health Trials
• CLASS – Children Learning About Second-hand Smoke – reducing exposure to second-hand smoke (Year 5 and Year 6)
• PIP – Pre-schoolers in the Playground – increasing exercise
• ASSSIST – Autism Spectrum Social Stories in Schools Trial – improving social skills and behaviour (ages 5 – 15, primary and secondary schools)
School Recruitment and Engagement
• Recruiting schools to take part in trials can be challenging– Competing demands– Relevance to ‘their’ school– Getting past the gatekeeper
• Keeping schools engaged throughout the duration of the trial– Disappointed at allocation– Staff turnover– Competing demands
School Recruitment and Engagement
Trial Recruitment Rate
ECC Trial 1 80%
ECC Trial 2: Pairs
36%
ECC Trial 2: Triplets
47%
CLASS 16.5%
Calderdale Writing
96%75%
School Recruitment and Engagement
• Hold a recruitment conference• Visit schools• Clearly explain randomisation and the need for a
control group• Use existing contacts and networks
• Ask them to sign an agreement to participant form
• Manage expectation • Be clear about their responsibilities and how you
will help• Keep in contact
Use examples to explain
30
35
40
45
50
55
60
Summer Autumn
Booster classes
Control group
Produce Clear Evaluation Diagrams
Control GroupPrimary Schools N = 12
No intervention(Primary Schools will
receive intervention next academic year)
Primary School Randomisation
Intervention GroupPrimary Schools N = 12
Improving Writing Quality intervention in Year 6
continued intervention in Year 7 in Secondary Schools.
Children in target group n = 288 (based on average 12 children per school)
Inclusion criteria: •Yr 6 •predicted to achieve Level 3 or an insecure level 4 in English by the end of Key Stage 2 (based on teacher assessments conducted at the end of Autumn term 2012)
Primary Schools n = 24Secondary Schools n = 3
Long term follow upRoutine test results and pupil characteristics recorded in National Pupil Database
Baseline data collection Information on all Year 6 pupils including Key stage 2 English Teacher Assessments from Dec 2012
Follow up data collection Dec 2013•Progress in English 11 (long form) (Conducted in Secondary School)
School Agreement From
Clear instructions for schools
Pupil Recruitment and Parent Engagement
Notoriously difficult to recruit participants in to trials generally
•Limited access/face to face contact with parents•Schools have the contacts, they need to help•Information can be overwhelming•Parents need to see a reason ‘why’ this is good for ‘their’ child
Pupil Recruitment and Parent Engagement
Trial Recruitment Rate
CLASS 2.5%
PIP*still recruiting
42%
ECC Trials: Each school had to identify approximately 12 eligible children and gain parental consent, almost all participating schools were able to achieve this – very little problems with gaining parental consent
Student Recruitment
Trial Students Approached
StudentsConsented
Recruitment Rate
Student Nurses Numeracy
382 298 78%
Hypnotherapy Trial
34 24 71%
Pupil Recruitment and Parent Engagement
• Resources to put people on the ground• Provide simple information in creative
ways• Use existing relationships (trust)
• Consider incentives• Consider collecting anonymous data• Consider an opt-out approach to
consent
• Detailed information to meet ethical requirements
• Simple version• Use a short DVD• Use assemblies
and parents evenings
Information for Parents
Information for Children
Data Collection
• Needs to fit in with academic timetable• Delays to study set up can seriously impact
the whole trial• Data collection often needs to occur at the
beginning and end of terms – difficult time for schools
• You might not be able to be there in person – placing trust in others.
• May need to coordinate large scale testing in a short period
• Unexpected events!
Data Collection
• Allow plenty of time in funding plan for study set up
• Give schools as much notice as possible• Try to do as much of the donkey work as
possible• Keep testing and data collection burden to
a minimum• Be clear and repeat instructions• Don’t leave anything to chance • Expect the unexpected!
Relationship with intervention developer
• Can be a challenging relationship
• Developer has a lot invested in the intervention
• They may try to influence trial design and conduct
Relationship with intervention developer
• Manage expectations• Try to explain that it important you
remain impartial• Refrain from voicing your own
opinions about the intervention – let the research speak for itself
• Stand your ground
Relationship with Funder
• Political context• Bigger picture / Wider aims• High staff changeover• May be more familiar with
trials in health• May be underfunded
• Understand the question that can actually be answered
• Example ECC Trial– We really wanted to conduct long term
follow up, but the funders and developers were not supportive
– Positive short-term impact of receiving Numbers Count (NC) compared with not receiving NC
Ethics and Other Approvals
• Need to ensure you acquire the necessary approvals – can be time consuming
• CLASS trial – collecting saliva samples, had to go through NHS ethics approval process, square peg in a round whole
• Hypnotherapy for exam anxiety trial– Examination board concerned students would
claim unfair disadvantage
Outcome Mesaures
• Find a suitable outcome measure• Ideally this outcome measure needs
to be able to be conducted independently and blind to allocation
• Ideally not a treatment/intervention inherent measure (designed specifically for use with the intervention e.g. questions use same format as questions in intervention)
• ECC Example– Primary Outcome was conducted by
independent testers blind to allocation – Secondary Outcome conducted by
teachers and was a treatment inherent measure
• ASSSIST Example– Very difficult to identify an appropriate
outcome measure– Unable to find any standardised tools that
are measure the changes we hope to capture
– Each Social Story will have a different aim for each child
– Using a combination of measuring frequency of behaviour and using a goal rating scale
Intervention Fidelity• Fidelity is the extent to which the intervention, as realised, is
“faithful” to the pre-stated intervention model
• Little consensus on meaning of the term “intervention fidelity. • But Dane & Schneider (1998) identify 5 aspects:
– Adherence – program components are delivered as prescribed;
– Exposure – amount of program content received by participants;
– Quality of the delivery – theory-based ideal in terms of processes and content;
– Participant responsiveness – engagement of the participants; and
– Program differentiation – unique features of the intervention are distinguishable from other programs (including the counterfactual)
Intervention Fidelity
Student Nurses Numeracy Example•Very low compliance•Actually 2 very similar trials in 2 different universities
•Only 24% and 12% of students allocated to the intervention group spent more than 15 minutes using the programme
Intervention Fidelity
• Measure fidelity to intervention
• Always conduct ITT analysis• But can also use CACE analysis to take
into account non-compliance
• Remember if you are undertaking a ‘pragmatic trial’ then compliance/fidelity is an important outcome in itself
BUT RCTs in Education are Important!
• Robust design for answering questions of effectiveness and cost effectiveness
• Although challenging they are possible!• Historically has been a serious lack of
robust evidence from RCTs in education but things are changing
• We need to be prepared for the challenges and ready to find solutions
