panel session 3: working with schools
DESCRIPTION
Building Evidence in Education: Conference for EEF Evaluators 11 th July: Theory 12 th July: Practice www.educationendowmentfoundation.org.uk. Panel session 3: Working with schools. Creative solutions: lessons learnt from evaluating the LIT programme Sarah Haywood NatCen. “Mind the Gap”. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/1.jpg)
Building Evidence in Education:Conference for EEF Evaluators
11th July: Theory12th July: Practice
www.educationendowmentfoundation.org.uk
![Page 2: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/2.jpg)
Panel session 3: Working with schools
Creative solutions: lessons learnt from evaluating the LIT programme
Sarah HaywoodNatCen
![Page 3: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/3.jpg)
Richard Dorsett, NIESREEF Evaluators Conference
12 July 2013
“Mind the Gap”
![Page 4: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/4.jpg)
What is being tested?
• a parental engagement intervention – parents work with their children to create a short
animated film– series of facilitated sessions
• whole-school intervention – Metacognition: training teachers in the principles of
“learning to learn”• Targeted at year 4 pupils in academic year 2012/13• Predicted effect size of 0.35 - 0.45
![Page 5: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/5.jpg)
Randomisation designAll schools:
NS=50
TreatmentNS
1=25
CPD & PENC=25
CPD NC=25
ControlNS
0=25
ControlNC=25
School randomisation
Class randomisation
A B C
![Page 6: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/6.jpg)
Treatment effect 1: CPD & PEAll schools:
NS=50
TreatmentNS
1=25
CPD & PENC=25
CPD NC=25
ControlNS
0=25
ControlNC=25
School randomisation
Class randomisation
A B C
![Page 7: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/7.jpg)
Treatment effect 2: CPDAll schools:
NS=50
TreatmentNS
1=25
CPD & PENC=25
CPD NC=25
ControlNS
0=25
ControlNC=25
School randomisation
Class randomisation
A B C
![Page 8: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/8.jpg)
Treatment effect 3: CPD & PE vs. CPDAll schools:
NS=50
TreatmentNS
1=25
CPD & PENC=25
CPD NC=25
ControlNS
0=25
ControlNC=25
School randomisation
Class randomisation
A B C
![Page 9: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/9.jpg)
Recruitment and randomisation• Birmingham, Devon, Haringey, Manchester• Drop-out is a worry
– 2 controls did so before knowing treatment status– 3 controls, 1 treatment dropped out & substituted– 2 controls dropped out & were not substituted
• Substitute schools – take treatment status of dropouts they replace– excluded from the impact estimates– provide potentially useful supplementary data
• Wanted 2-form entry but not always achieved
![Page 10: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/10.jpg)
Achieved sample
NS=43
TNS
1=24
CPD/PENC=24
CPD NC=15
CNS
0=19
CNC=19
![Page 11: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/11.jpg)
Some lessons• RCT Design is relatively easy – practical issues are more
complicated• The process of inducting schools is important to secure full
engagement pre-randomisation• Having something to offer schools control schools in particular
may help with drop out• Minimising drop out is best. But some drop-out is inevitable –
need for a protocol?• Some implications for analysis
– Helpful to understand reasons behind dropout– Can consider nonexperimental techniques– NPD analysis may be unaffected by drop-out of controls
![Page 12: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/12.jpg)
EEF Conference 2013Towards a Protocol for Effective Recruitment
Mary SheardJuly 12, 2013
![Page 13: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/13.jpg)
Recruitment as a problematic and complex relationship
“Recruit schools to the evaluation not the project”
![Page 14: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/14.jpg)
Contexts
EEF projects: Project and design, challenges and solutions
EEF Protocol and Survey Outcomes:Effective recruitment; what has worked well and what have been the challenges
Non-EEF projects: Experience across a wide range of research studies and evaluations
![Page 15: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/15.jpg)
What do we mean by ‘effective recruitment’?
![Page 16: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/16.jpg)
Terminology: What is meant and understood? Programme Intervention Initiative Project Evaluation …
![Page 17: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/17.jpg)
What has worked well• Relationships: schools, LA/parent organisations, programme
developers, evaluators trainers; test providers [Ethics]• Partnerships with schools: key personnel in school; lead project
contact; teacher implementers; technical support.
• Roles: clarity, responsibilities, expectations, inclusiveness [Ethics]
![Page 18: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/18.jpg)
What has also worked well• Information: quality, clarity, conciseness, sufficiency, inclusivity;
suitability, accessibility (audiences, ethics]
• Examples of documentation; inviting initial expression of interest; school agreement form/contract; pupil data; data protection
![Page 19: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/19.jpg)
Challenges and resolutions
Identifying and linking with key personnel
Senior leadership involvement
Lines of communication
Information overload
Saturation of constituency/schools asparticipant partners
![Page 20: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/20.jpg)
More challenges and resolutions
Defining/explaining and the relationship between school, programme developer (trainer) and evaluation team
Timing
The concept of random assignment
Participation as control
Testing preparation and procedures
![Page 21: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/21.jpg)
What we have learnedNeed to systematise a comprehensive recruitmentstrategy,to establish a recruitment protocol or checklist as the prequel to a project data management plan
Need to create a recruitment database
Need to consider equity/equal opportunity and fairnessin recruitment approaches: hard to reach schools and schools that are missed out
![Page 22: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/22.jpg)
Developing a consistent recruitment strategy
![Page 23: Panel session 3: Working with schools](https://reader036.vdocuments.site/reader036/viewer/2022081507/568165d7550346895dd8e707/html5/thumbnails/23.jpg)
Creating a protocol/checklist for effective recruitmentin future large-scale evaluations