Read Write Inc. Phonics and Fresh Start American Institutes for Research Pooja Nakamura, Ph.D. and Nisha Rai, Ph.D.

Evaluation Summary

Age range

Read Write Inc. Phonics is taught to children learning to read from Reception – Year 4, but the evaluation will focus on pupils from Reception – Year 2 only Fresh Start targets eligible pupils in Years 5 – 8, but the evaluation will focus on eligible pupils from Years 5 and 6 only

Number of pupils Read Write Inc. Phonics: 4,440 with 1,320 of those being FSM Fresh Start: 1,200 with 360 of those being FSM

Number of schools 120

Design Clustered-Randomized Control Trial

Primary Outcome

Read Write Inc. Phonics: New Group Reading Test at the end of the second year (independently collected by Queens University Belfast) Fresh Start: KS2 reading fine points score for Year 6 eligible students at the end of the first year of programme implementation and Year 5 eligible students at the end of the second year of programme implementation (i.e., the end of Year 6 for students who began programme in Year 5)

Background

Programmes

Brief Name Read Write Inc. Phonics and Fresh Start

Why Literacy programmes rooted in phonics using decodable texts to teach children learning to read and write and those who need to catch up quickly

Who Read Write Inc. Phonics: children from Reception to Year 4 learning to read; Fresh Start: pupils in Years 5 – 8 below appropriate reading age

What Children are taught daily in homogeneous groups:

Phonic lessons for reading and spelling

Reading activities – from decoding to comprehension

Spelling and compositional writing

Children work with a partner and articulate their thinking throughout the lesson

What Two days training for all staff in primary schools including knowledge of the alphabetic code and training on how to teach both programmes. Assessment processes to ensure children are taught at correct level and that slowest progress children receive daily tutoring. Ongoing, regular development days – a consultant trainer works with the leaders in school to ensure successful implementation and provide continuing professional development.

Who Ruth Miskin trainers

How Training sessions for teachers, and teachers then implement in the classroom

Where At schools, and in the classrooms

When and how much Read Write Inc. Phonics: Starts with 20 minutes daily in term 1 of Reception,

2

building to 40 minutes by the end of the year. One hour a day for children in Year 1 and above. Many children complete programme by the end of Year 1 or beginning of Year 2. Fresh Start: Up to 33 weeks for eligible pupils in Years 5 and 6 in place of regular English lessons

Tailoring Slower progress children are identified immediately and given daily one-to-to tuition to ensure they keep up with their peers

How well (planned) Two days training and regular development days to maximise staff knowledge and children’s progress. The headteacher and reading leader attend training before in-school training. The reading leader ideally attends three leadership trainings in the first year

Read Write Inc. Phonics and Fresh Start are phonics-based comprehensive literacy programmes.

Read Write Inc. Phonics is a complete literacy programme targeted at 4 to 6 year-olds learning to

read and write and 7 to 8 year-olds needing to catch up. Fresh Start is a 33-week catch up literacy

programme delivered to target children in Years 5 – 8 identified as having fallen behind in expected

standards in reading and writing; it aims to support pupils with reading difficulties at the end of

primary school, using systematic phonics approaches embedded in content that is tailored for older

pupils. The programme recommends a Fresh Start session every day instead of the usual English

lessons and each Fresh Start session should be one hour long.

Both programmes involve the same training (depicted in Figure 1), which is usually delivered to the

whole school. The headteacher and reading leader attend a regional two-day training before whole

school training. The reading leader assesses the children on sound and word reading and fluency

using the programmes’ assessment materials. The assessment lasts for approximately five minutes

and assesses pupils’ phonic knowledge; it is focused on decoding and fluency and does not include a

comprehension component. The first day of training is for all teaching staff including the

headteacher and focuses on the phonic element of the programmes. Three or four weeks later, the

trainer returns to support staff on a normal teaching day. She provides feedback, further training

and coaching with a focus on the slowest progress children. The second day of training is

approximately four to six weeks after the first training day and focuses on teaching reading and

writing. In the first year of implementation the reading leader and headteacher attend three

regional leadership training days. The trainer continues to visit the school termly to support and

coach staff and work closely with the reading leader.

The primary components include:

Training and ongoing support for all staff teaching the programmes

Rigorous leadership and management by the reading leader

Daily lessons that teach children to read accurately and fluently with good comprehension

Teaching in homogeneous groups

Multiple readings of age- appropriate decodable stories and non-fiction. Children read each story three times. On the first read, children focus on accurate word reading; the second, on developing fluency; and the third, on comprehension. Fluency and comprehension increase with each repeated read. An introduction, prompts for thinking out loud, and discussion help teachers ensure that children comprehend what they are reading.

Daily one-to-one tuition for the slowest progress children in addition to their daily group lesson

Teaching letter formation and spelling alongside sound and word reading

3

Teaching writing – from simple words and sentences to independent compositional writing

Opportunities for partner work throughout the lessons

Figure 1

Significance

According to the National Curriculum Assessment (2015) about 20% of pupils leave primary schools

without achieving the expected attainment level in reading, and there are nearly 676 state-funded

mainstream primary schools (5% of the total number of schools in the UK) that are below expected

primary school floor standards (DfE 2015). In 2015, 77% of pupils met the expected phonics

screening check in Year 1, which was a 3 percentage point increase from 2014. By the end of Year 2,

approximately 90% of pupils met the standards, a 1 percentage point increase from 2014. Key stage

1 reading scores remained at 90%, which reflected no change from the 2014. Attainment in reading

has been an important focus for the Department for Education in England, and there have been

various policies implemented in order to achieve the national targets of attainment in reading at Key

Stage 2. The government introduced pupil-premium funding for schools to raise the attainment level

of disadvantaged pupils. Schools now have the opportunity to use these funds to purchase resources

or teaching approaches that can support pupils who are at risk of falling behind. An essential step is

to identify reading approaches that ensure all pupils meet National Curriculum expectations,

including those pupils who are at risk of falling behind.

Large-scale reviews of the effectiveness of reading interventions in the United Kingdom, the United

States, and Australia have shown that phonics instruction is significantly more effective than non-

phonics approaches for teaching reading (Chambers, Cheung, & Slavin, 2015; Torgesen et al., 2006;

NICHD, 2000; Australian Government, Department of Education Science and Training, 2005). In the

United Kingdom, the Rose Report underscored specifically that the most effective kind of systematic

phonics approach is synthetic phonics (Rose, 2006).

According to one of the most well-established theories of reading development, it is clear that both

good word reading and language comprehension are required for reading comprehension

development; neither is sufficient on their own (Gough & Tunmer, 1986; Hoover & Gough, 1990).

4

While there is some evidence for the effectiveness of phonics approaches for older pupils with

reading difficulties (Ehri, Nunes, Stahl, & Willows, 2001), the overwhelming majority of the studies

examine the effectiveness of phonics on younger pupils, especially at the Foundation Stage and Key

Stage 1. Furthermore, there is little evidence on its effectiveness on spelling and reading

comprehension outcomes (as opposed to word reading). More evidence is necessary to understand

what aspects of reading development are supported by a phonics-focused approach for older pupils.

Fresh Start was evaluated in an aggregated efficacy trial, which showed promising results as a catch

up literacy intervention for Year 7 pupils (Gorard, Siddiqui, & See, 2015). Given that this evaluation

was conducted independently by schools, there is the potential for selection bias (although the

independent evaluators coordinating the efficacy trial do not see any evidence of such bias), and

there is also inconclusive evidence about the programme’s effectiveness under diverse conditions.

We will address these concerns by conducting our own randomisation and examining the degree to

which the programme works with differing levels of support from trainers, heterogeneity in the

groups of pupils, and the duration of the programme.

Methods

Research questions

The evaluation is designed to answer the following research questions:

1. Do Read Write Inc. Phonics and Fresh Start impact pupil reading outcomes? Improving

reading outcomes is the ultimate goal of the programme. We will assess the impact of the

programmes on reading levels or progress—as defined by the KS2 reading points scores and

KS2 writing scores for combined Years 5 and 6 pupils and the New Group Reading Test

results for Year 2.

2. How are the programmes being implemented? In order to get a better picture of the

effectiveness of the programmes on learning outcomes, we will conduct a “light touch”

examination of whether the programmes are being implemented with fidelity, described in

more detail in the Implementation and Process Evaluation Methods section. Are the

teachers teaching the programmes as envisioned by the developers? Are teachers receiving

support from the reading leader throughout implementation? How does training help

support the ability of the reading leader to implement the programme with fidelity? How

contingent is the success of the programme on this continuous professional development?

How has the programme impacted teacher knowledge on how to teach reading? Does the

programme change teacher practices and behaviours in the classroom?

Design

By eliminating selection bias and bias from confounding variables, a randomised controlled trial

(RCT) is the optimal design for making causal claims about programme impacts. Building on the

aggregated efficacy trial, which showed promising results of Fresh Start as a catch-up literacy

intervention for Year 7 pupils (Gorard, Siddiqui, & See, 2015), we will conduct a cluster randomized

controlled trial to determine the impact of the programme on various pertinent reading outcomes.

To control for potential bias that may have occurred when schools conducted internal randomisation as part of the aggregated trial, we will conduct our own independent randomisation. So that we might mimic real-life implementation the study will use random assignment at the level of the

5

school. This approach also limits spillover effects and ensures that pupils end up in more homogeneous groups. The two-arm RCT will compare the reading achievements of the programmes and a two-year wait-list control, who will continue ‘business-as-usual’ for the teaching of phonics, reading and writing. The rigorous cluster-RCT would randomise schools into two groups in equal proportion: 1) those schools who receive Read Write Inc. Phonics and Fresh Start programmes, and 2) a control group of schools who would not initially receive either programme.

Randomisation

We will use stratification as part of the randomisation process. Stratification, or blocking the sample, helps improve the precision of the estimates by helping to ensure that the treatment indicator is orthogonal to the other covariates (Cox and Reid 2000). The process involves dividing the sample into groups that possess similar values of certain observable characteristics. While we would expect the simple randomisation process to produce similar groups on average, stratification helps ensure that the groups are similar in reality. Research suggests that when conducting stratified randomisation, it is optimal to include a maximum of two variables that are highly correlated with the intended outcome to achieve balance; the findings suggest that any more than two highly correlated variables does little to improve balance while resulting in a reduction of degrees of freedom (Zeneli et al., 2016). Thus the stratification will use the school-level historical KS1 reading score provided by schools as part of their registration form and the government assigned Ofsted grade, a measure of school quality that determines what kind of training they receive. The schools will be assigned to two groupings within the Ofsted grade, those with a grade of 1 or 2 and those with a grade of 3 or 4. A block will be constituted of all the schools that share the same Ofsted rating and fall in the same “bin” of KS1 reading scores. This requirement helps achieve balance along the dimensions of Ofsted rating and helps ensure that the schools have similar pre-tests results. In each block, half of the schools will be assigned to the treatment condition and the other half will be assigned to the control. By assigning the same proportion within each block, we reduce the variance of the treatment effect estimate as compared to non-stratified randomised designs (Duflo et al., 2007). The randomisation will be done in using Stata®. The combination of the set seed command before using Stata®’s random number function ensures that the results are replicable. Upon completion of the randomisation the associated do files, which contain the programming commands, and log files, which record the session, will be provided. Participants

Schools: Queens University, Belfast is in the process of recruiting schools through their networks.

The aim will be to recruit a sample of schools with some over-recruitment of EEF target

underperforming, disadvantaged schools. All schools are eligible, but the mean FSMEver of the

sample should be greater than the national mean for England. The schools must be state funded,

and a large (but no more than 50%) share of schools must come from the North East region of the

U.K. The initial communication that QUB provides to the school includes a flyer from Ruth Miskin

Training and an information form on the trial, included as Appendix A.

6

Once schools have expressed interest QUB requests that the schools complete a registration form.

The form includes information on school characteristics such as percentage of FSM, geographic

region, school size, class size, gender ratios, and Ofsted category.

The information provided on the registration form will be used in the random assignment of the

schools. Before being informed of their random assignment, Queens University Belfast will collect

the following information from the schools:

1) List of pupils UPNs in the current year 4 and 5, and reception and year 1 as well as their SEN, FSMEver, Gender, EAL and ethnicity plus a flag for those year 4 and 5s identified as eligible for Fresh Start in the following year.

2) Confirmation that consent letters have been completed and any opt-outs removed. Upon being informed of their random assignment, all of the schools (both treatment and control) will complete a Memorandum of Understanding (MOU). This MOU will include brief questions on their existing phonics and reading programmes, and what they do for struggling readers in KS2.

Pupils: While Read Write Inc. Phonics is delivered to all children in Reception and Year 1 and those

who are assessed as needing it in Years 2 – 4, the evaluation will focus on pupils from Reception –

Year 2 only. While Fresh Start targets eligible pupils in Years 5 – 8 who are below appropriate

reading age, the evaluation will focus on eligible pupils from Years 5 and 6 only. The eligibility criteria

for the Fresh Start programme is a school progress measure, through which the children are selected

according to their reading levels. Children reading below a reading age of 9.5 years are taught Fresh

Start. The programme will be targeted to those children who are not on track to meet national

expectations in reading at the end of KS2. These are students whose reading age is 9 years or below,

which is 1 to 2 years below expectations. While the school knows the reading age compared to the

actual age of the students, the pupil’s progress towards 4b achievement is assessed by the teacher.

Outcome Measures Read Write Inc., Phonics: Primary Outcome: The primary outcome will be whole class Year 2 New Group Reading Test results at the end of the second year of the programme. The New Group Reading Test is an adaptive test that comprises three sections: phonics, sentence completion, and passage comprehension As such, it targets both the decoding and language comprehension sub-skills of reading. Since this is an adaptive test, pupils will begin with the sentence completion tasks and then move up to passage comprehension if they “pass” sentence completion or move down to phonics if they do not “pass” sentence completion. QUB will independently collect this data and ensure that the assessors are unaware of the treatment status. Secondary outcomes: The secondary outcomes include the teacher assessed KS1 writing at Year 2 and the teacher assessed Year 1 phonics screening check for both years of the programme aggregated. If the sample associated with the Year 2 retakes of the phonics screening check provides enough power, we will potentially examine the phonics retake as a secondary outcome. However, since the Year 2 retakes of the phonics screening check are only compulsory in Academies and free schools where participation forms part of their funding agreement, we cannot determine at this time whether the retakes will be a viable secondary outcome. Because the teachers will have knowledge of the treatment status when they conduct the Year 2 KS1 Writing and Year 1 Phonics screening check, these outcomes are likely to be more biased than the New Group Reading Test measure. The Department of Education does provide standardized guidelines for the implementation of these tests, which will help minimize bias.

7

Fresh Start: Primary Outcome: The primary outcome will be KS2 reading fine points score for Year 6 eligible students at the end of the first year of programme implementation and Year 5 eligible students at the end of the second year of programme implementation (i.e., the end of Year 6 for students who began programme in Year 5) Secondary outcome: The secondary outcome will be the combined KS2 writing results for Years 5 and 6. Because these tests are nationally administered and high stakes, there is little concern over bias with the test administration. For Fresh Start we will be using the KS1 results as a baseline control. Since the KS1 tests are teacher administered they could contain some bias; however since the KS1 tests will have been conducted prior to randomisation, the risk of bias is decreased. Sample size calculations

The evaluation will be carried out as a 120-school RCT with an even split between the treatment

group and the control group.

Read Write Inc., Phonics:

Assuming this sample of 120 schools with 4,400 Year 2 pupils (37 pupils per school), we

conservatively estimate the minimum detectable effect size (MDES) for intent-to-treat estimates of

Read Write Inc. Phonics to be 0.184 for all eligible pupils and 0.222 for free school meal (FSM) pupils,

assuming 11 FSM pupils per school.1

Fresh Start: Assuming this sample of 120 schools with 1,200 Year 5-6 pupils (10 pupils per school with 5 from

each grade), we conservatively estimate the minimum detectable effect size (MDES) for intent-to-

treat estimates of Fresh Start to be 0.215 for all eligible pupils and 0.267 for free school meal (FSM)

pupils, assuming 3 FSM pupils per school.2 Because of the small number of eligible pupils per school

we will be unlikely to detect a separate effect for FSM pupils, although the analysis will be

conducted.

Analysis plan

We will analyse the impact of the programmes using the difference-in-means specification:

𝑌𝑖𝑗𝑘 = 𝛼 + 𝛽𝑅𝑘 + 𝜃𝑋𝑖𝑗𝑘 + 휀𝑖𝑗𝑘 (1)

Yijk is the outcome measure for pupil i in classroom j in school k. That is, when analysing the effect of

Fresh Start for pupil i in classroom j in school k, Yijk represents (1) the combined Year 5 and Year 6

KS2 reading results; and (2) the combined Year 5 and Year 6 KS2 writing results. For analysing the

effect of Read Write Inc. Phonics for pupil i in classroom j in school k, Yijk represents (1) the New

Group Reading Test at the end of the second year; (2) the New Group Reading Test writing results at

the end of the second year; and (3) Year 1 phonics screening check for both years of the programme

1 Using data from similar EEF studies on literacy outcomes, we settled on the following power parameters:

ICC=0.10; Alpha=0.05; Power=80%. 2 Using data from similar EEF studies on literacy outcomes, we settled on the following power parameters:

proportion of variance in Level 1 (pupil level) outcomes explained by Level 1 covariates, R12=0.53; ICC=0.133;

Alpha=0.05; Power=80%.

8

aggregated. Given that the NGRT and NPD are standardized, we will utilize raw scores for these

tests. For the teacher-assessed KS1 results (that we will use as a baseline control for Fresh Start), we

will first normalize the scores to bring all scores into proportion with each other.

Rk is a variable that equals one if the pupil’s school is part of the treatment (which includes both

Fresh Start and Read Write Inc. Phonics) and equals zero if not. Xijk is a vector of controls that

includes the groups created for the stratified randomisation based on the school-level historical KS1

reading score and the Ofsted grades and the ever FSM status of the pupils from the NPD. Although

there are too few areas in the analysis to reliably estimate variation in effects across areas, Xijk will

also control for the different areas of the schools. In the analysis for Fresh Start, this vector of

controls will include KS1 results as a baseline measures of the outcome variable so that we can

perform ANCOVA analysis, which will improve our power. ANCOVA analysis is not possible with the

analysis for Read Write Inc. Phonics since no baseline measure of the outcome variable exists. We

will use cluster-robust standard errors to account for a lack of independence across observations

due to clustering of pupils at the level of school. Doing so ensures that standard errors are properly

estimated. 𝛽 is our coefficient of interest, representing the marginal effect of being part of the Read

Write Inc. Phonics or Fresh Start programmes, as compared to the control group. Results will be

reported as Hedge’s g effect sizes that standardize the estimated impact using the unadjusted

pooled within-group standard deviations of the outcome.3

We will also conduct subgroup analysis for the population of FSM pupils by estimating equation (1)

only for the sample of FSM pupils. Additionally, we will examine heterogeneous effects to see if

there is a difference between FSM and non-FSM pupil with the equation:

𝑌𝑖𝑗𝑘 = 𝛼 + 𝛽1𝑅𝑘 + 𝛽2𝑅𝑘 ∗ 𝐹𝑖𝑗𝑘 + 𝜃𝑋𝑖𝑗𝑘 + 𝛿𝐹𝑖𝑗𝑘 + 휀𝑖𝑗𝑘 (2)

Fijk is an indicator for whether the pupil ever received free school meals. Here the vector of controls

Xijk includes the groups created for the stratified randomisation, the area of the school, and for the

Fresh Start analysis the KS1 results as a baseline measures of the outcome variable. With the

heterogeneous effects regression, 𝛽1 represents the marginal effect of being part of the Read Write

Inc. Phonics or Fresh Start programmes, as compared to the control group, for pupils who do not

receive free school meals. 𝛽2 is the marginal effect of being part of the Read Write Inc. Phonics or

Fresh Start programmes for pupils who do receive free school meals. This heterogeneous effects

analysis will clarify if there is a difference in the programme effect by FSM status.

Implementation and process evaluation methods

We will conduct a relatively light-touch process snapshot to check that the programmes were

delivered as intended, to understand the contrast from usual practice and identify any barriers to

implementation and necessary conditions for success.

We will use a variety of data sources for this process snapshot. First, from the MOU we will obtain

information related to schools’ existing phonics and reading programmes, and what they do for

struggling readers in KS2. At the end of the evaluation, QUB will collect follow up information from

the schools related to these topics. Second, we will use Ruth Miskin Training’s soon-to-be-developed

3 Per both EEF and IES guidance, we will not use the level-1 within-group variance. (For EEF guidance see

https://educationendowmentfoundation.org.uk/public/files/Evaluation/Analysis_for_EEF_evaluations_REVISED_Dec_2015.pdf; for IES guidance see http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_procedures_v3_0_standards_handbook.pdf)

9

online system that will collect information on delivery from teachers. The portal will focus on

delivery as it relates to training and ongoing support, organisation, and assessment. The information

that will be included under these headings relates to factors that they believe are most crucial to the

success of the programme. Third, we will analyse the results of a survey that will be administered by

QUB to teachers at the end of the programme. This survey, that will be delivered at the same time as

the post-test will help to clarify teachers’ perceptions of the programme and learn what teachers in

control schools did instead. Finally, we will also conduct focus group interviews with 2-3 school

administrators and teachers, and visit 2-3 schools (at least one high-performing school, and one low-

performing school) to triangulate the information obtained from the learners.

AIR will contribute to the design and content of the various questionnaires that will be implemented

by Ruth Miskin and QUB. To reduce concerns associated with response bias, whereby teachers

answer with what they think the researchers want, we will use questions about actual day-to-day

teaching practices. These types of questions are applicable to teachers in both treatment and

control schools. These questionnaires as well as the focus group discussions will help us determine

the following questions:

How are the programmes being implemented?

Are the teachers using the programmes as envisioned by the developers?

Are teachers receiving support from the reading leader?

How does training quality help support the quality of the reading leader to implement the

programme with fidelity?

How contingent is the success of the programme on this continuous professional

development?

How has the programme impacted teacher knowledge on how to teach reading?

Does the programme change teacher practices and behaviours in the classroom?

Costs A formal cost-benefit analysis is not recommended due to the challenges of monetizing the benefits

(e.g., the numerous assumptions required about the returns to education). However, we will

combine the benefits estimated in our impact analyses with the costs obtained from our

implementation research to carry out a cost-effectiveness analysis. Specifically, we will assess the

costs of the programmes using the ingredients method. For this purpose, we will need to specify all

the ingredients that are necessary to replicate the program and then collect data on the unit costs of

all these ingredients (Dhaliwal, Duflo, Glennerster, & Tulloch, 2011). Information on the five major

categories of ingredients—(i) personnel and training, (ii) facilities, (iii) equipment and programme

materials, (iv) other inputs, and (v) client inputs—need to be obtained. These form the starting point

for considering specific ingredients and their costs and are disaggregated into individual cost items

(Levin and McEwan, 2001). The approach enables a clear view of how costs are distributed across all

types of expenditures. The cost can be in opportunity cost or capital cost, but the ultimate goal is to

determine the value of any ingredient.

We will obtain these costs through focus group discussions with school administrators and through

the collection of data on actual costs from Ruth Miskin Training. We will then estimate the costs of

the programmes for the average beneficiary and divide these costs by the expected gain in outcome

derived from the impact analysis to serve as the cost-effectiveness measure of the programmes.

10

Ethics and registration

Describe the process for ethical approval

What level of consent is needed from pupils (refer to current EEF guidance on ensuring

appropriate consent for linking to NPD and data archiving)

International Standard Randomised Controlled Trial Number (ISRCTN) (register the trial at:

www.controlled-trials.com)

QUB has submitted a request for study approval through their ethics board for all the data

collection. QUB will apply to the NPD for access to pupil outcome and background data. They will

conduct an initial application in September in order that any issues can be ironed out before the final

application at the end of the evaluation. QUB will ensure that appropriate parent opt-out consent

for data processing has been collected from schools.

QUB and AIR will work together to develop a data sharing agreement that accommodates the

restriction that the NPD data cannot leave the UK. One possibility is that the evaluators become

visiting fellows of QUB and stay for two weeks on campus to analyse the data and start writing the

report. If this method is chosen, QUB can include an additional request to their ethics board for the

data analysis. Alternatively, QUB could provide access to a remote portal through which the data

could be analysed from the U.S. while ensuring that the data does not leave the U.K. If this method is

chosen, AIR will submit a request to their Institutional Review Board to obtain approval for the

analysis.

QUB will obtain opt-out consent for the evaluation.

The study has been registered and the International Standard Randomised Controlled Trial Number (ISRCTN) is ISRCTN97725862.

Personnel

Pooja Nakamura, Ph.D., a Senior Researcher at AIR, will serve as a co-principal investigator. She will oversee the quality of the study, provide literacy content knowledge, provide guidance and oversight on instrument development, oversee the “light touch” process snapshot, and participate in report writing.

Nisha Rai, Ph.D., an Economic Researcher at AIR, will serve as a co-principal investigator and project director. She will oversee the day-to-day activities of the study, serve as the primary contact, conduct the data analysis, ensure high quality work, manage the budget, and participate in report writing.

Hans Bos, Ph.D., Senior Vice President, will provide oversight as an impact evaluation expert.

Andrea Coombes, Qualitative Researcher at AIR, will develop materials for and will conduct focus group discussions; she will also participate in analysis and report writing.

Paul Sirma will participate in data analysis and report writing.

Rebecca Stone, a Senior Researcher at AIR, will serve as quality assurance for the study.

11

Anticipated roles of assessment partner

For this evaluation study, Queens University, Belfast will lead the recruitment of schools into the study, provide information on which schools were recruited, provide background information necessary for the randomisation, work with schools in informing them of treatment/control status and distributing consent documents, lead the data collection of the New Group Reading Test and National Pupil Database data, and support the process evaluation as requested by the evaluation team.

Risks

One risk to the evaluation is recruitment of sites and students. QUB will need to recruit at least 120

schools to power the study as originally planned. While it may be difficult to recruit such a large

number of schools, QUB is mitigating this by working with staff who are familiar with school systems

and have connections in the schools. Additionally, the recruitment and randomisation are being

conducted in waves (approximately 19th April, 19th May and 19th June) to facilitate the timeline.

QUB will obtain opt-out consent for the evaluation. They will ensure that the consent documents

fully describe the potential risks.

Data Protection

AIR takes the protection of data seriously and has implemented numerous companywide policies to promote the security of data. Staff are required to participate in yearly trainings on data security. All AIR computers are encrypted and password protected with stringent requirements for passwords, including specification on password length, character usage, and password expiration dates. Any login into AIR’s servers and systems (including accessing email or shared network) outside of AIR offices require duo authentication.

In accordance with the Data Protection Act, for this evaluation, no data will be stored on AIR’s computers. The evaluators will either become visiting fellows of QUB and will analyse the data from the UK or QUB will provide access to a remote portal through which the data could be analysed from the U.S. while ensuring that the data does not leave the U.K.

Timeline

Date Activity

April 2016 AIR randomises first batch of schools

May 2016 AIR randomises second batch of schools

June 2016 AIR randomises final batch of schools

September 2016 QUB to submit initial request of NPD data

April 2018 QUB administer New Group Reading Test

April 2018 QUB administer teacher survey

May 2018 QUB to submit request of NPD data

May 2018 AIR to conduct focus group discussions

July 2018 AIR to conduct analyses

August 2018 AIR write up report and submit to EEF

September 2018 AIR make edits per outside reviewer, submit final report to EEF

12

REFERENCES

Australian Government, Department of Education Science and Training. (2005). Teaching Reading: Report and Recommendations. Commonwealth of Australia.

Cox, D. R., & Reid, N. (2000). The theory of the design of experiments. CRC Press.

Chambers, Bette, Robert E. Slavin, and Alan CK Cheung. Literacy and Language Outcomes of Balanced and Developmental-Constructivist Approaches to Early Childhood Education: A Systematic Review. Best Evidence Encyclopedia, 2015.

Department for Education, United Kingdom. (2015). National Curriculum, National Statistics.

Dhaliwal, I., Duflo, E., Glennerster, R., & Tulloch, C. (2011). Comparative cost-effectiveness analysis to inform policy in developing countries: a general framework with applications for education. August.

Duflo, E., Glennerster, R., & Kremer, M. (2007). Using randomization in development economics research: A toolkit. Handbook of development economics, 4, 389.

Ehri, L. C., Nunes, S. R., Stahl, S. A., & Willows, D. M. (2001). Systematic phonics instruction helps students learn to read: Evidence from the National Reading Panel’s meta-analysis. Review of educational research, 71(3), 393-447.

Gorard, S., Siddiqui, N., & See, B. H. (2015). Fresh Start. Education Endowment Foundation Efficacy Trial.

Gough, P. B., & Tunmer, W. E. (1986). Decoding, reading, and reading disability. Remedial and special education, 7(1), 6-10.

Hoover, W. A., & Gough, P. B. (1990). The simple view of reading. Reading and writing, 2(2), 127-160.

Levin, H. M., and McEwan, P. J. (2001). “Cost-effectiveness Analysis: Methods and Applications” Sage Publications, Thousand Oaks, CA.

National Institute of Child Health and Human Development. (2000). Report of the National Reading Panel. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction (NIH Publication No. 00-4769). Washington, DC: U.S. Government Printing Office.

Rose, J. (2006). Independent review of the teaching of early reading: Final report. Department for education and skills. London, UK: TSO

Torgerson, C., Brooks, G., & Hall, J. (2006). A systematic review of the research literature on the use of phonics in the teaching of reading and spelling. Nottingham: DfES Publications.

Zeneli, M., Thurston, A., & Roseth, C. (2016). The influence of experimental design on the magnitude of the effect size-peer tutoring for elementary, middle and high school settings: A meta-analysis. International Journal of Educational Research.

13

Appendix A: School Information

14

15

16

17

18

Appendix B: Memorandum of Understanding – Programme School

19

20

21

22

23

Appendix C: Memorandum of Understanding – Control School

24

25

26

27

28

Appendix D: Opt-Out Consent Form

29

30