liberia teacher training program (lttp) midterm assessment

Liberia Teacher Training Program (LTTP)

Midterm Assessment of the Impact of Early Grade Reading and Math Interventions

This publication was produced for review by the United States Agency for International

Development and by FHI360. It was prepared by RTI International for the Liberia Teacher

Training Program.

Midterm Assessment of the Impact

of Early Grade Reading and Math

Interventions

Prepared for

USAID/Liberia

RTI International (Internal Document)

and

FHI360

1825 Connecticut Avenue NW

Washington, DC 20009 USA

October 2013

Prepared by

Joseph DeStefano, Timothy Slade, and Medina Korda RTI International 3040 Cornwallis Road Post Office Box 12194 Research Triangle Park, NC 27709-2194

RTI International is a trade name of Research Triangle Institute.

This development of this document was made possible by the support of the American people through the United States Agency for International Development. The contents are the sole responsibility of RTI International and FHI 360, and do not necessarily reflect the views of USAID or the United States Government.

Midterm Assessment of the Impact of Reading and Math Interventions iii

Table of Contents

Page

List of Tables ............................................................................................................................ iv

Acknowledgments..................................................................................................................... vi

Executive Summary ................................................................................................................. vii

1. Introduction ......................................................................................................................... 1

2. Measuring Impact: The LTTP Research Design ................................................................ 2

Research Questions ...................................................................................................... 2 Research Design ........................................................................................................... 3 Data .............................................................................................................................. 4

3. Implementation of the Reading and Math Interventions in Cohort 1 Schools ................... 5

Scripted Lesson Plans ................................................................................................... 5

Teacher and Student Materials ..................................................................................... 6 Training ........................................................................................................................ 6 Coaching ....................................................................................................................... 6 Assessment ................................................................................................................... 7

4. Midterm Impact of LTTP Interventions in Cohort 1 Schools ............................................ 7

Reading ......................................................................................................................... 7

Gender Disaggregated Reading Performance ............................................................ 10 Math ............................................................................................................................ 12

5. Cohort 1 Schools Compared to Control Schools .............................................................. 17

Reading ....................................................................................................................... 18 Math ............................................................................................................................ 20

6. Characteristics of Cohort 1, External Control, and Cohort 2 Students, Teachers,

Principals, and Schools ..................................................................................................... 23

7. Conclusions....................................................................................................................... 25

Annex A: Details of the Study Methodology ............................................................................ 1

Annex B: About the Assessment Instruments ........................................................................... 1

Annex C: Comparison of the Average Performance in Cohort 1, Cohort 2, and

External Control Schools .................................................................................................... 1

Annex D: Additional Data on Characteristics of Students, Teachers, Principals, and

Schools ................................................................................................................................ 1

iv Midterm Assessment of the Impact of Reading and Math Interventions

List of Figures

Figure 1: Improvements in Oral Reading Fluency ............................................................... vii

Figure 2: Percentages of Grade 1 Students in Cohort 1 Schools Scoring Zero in

Reading Skill Areas, Baseline and Midterm ........................................................... 9


Reading Skill Areas, Baseline and Midterm ........................................................... 9


Reading Skill Areas, Baseline and Midterm ......................................................... 10

Figure 5: Gender-Disaggregated Performance in Reading at Baseline and Midterm .......... 11

Figure 6. Performance of Cohort 1 Students Across Grades in Each Reading Skill

Area ....................................................................................................................... 15

Figure 7: Performance of Cohort 1 Students Across Grades in Each Math Skill Area ....... 16

Figure 8: Trends in Average Oral Reading Fluency for Students in Cohort 1 and

External Control Students ..................................................................................... 20

List of Tables

Table 1: Implementation of Reading and Math Programs and Systematic

Assessments ............................................................................................................ 3

Table 2: Cohort 1 Average Reading Results for Baseline and Midterm .............................. 7

Table 3: Grade 1 Cohort 1 Results on Math Subtests ......................................................... 12



Table 6: Reading Performance in Control Schools ............................................................. 18

Table 7: Reading Performance in Cohort 1 Schools ........................................................... 19

Table 8: Difference of Differences for Cohort 1 Compared to Control Schools ................ 19

Table 9: Math Performance in Control Schools ................................................................. 21

Table 10: Math Performance in Cohort 1 Schools ................................................................ 21

Table 11: Difference of Differences Analysis for Cohort 1 Compared to Control

Schools .................................................................................................................. 22

Table 12: Student Characteristics at Midterm ...................................................................... 23

Early Grade Reading and Mathematics Methodology: From Pilot to Scale v

Abbreviations

EdData II USAID Education Data for Decision Making project

EGMA Early Grade Mathematics Assessment

EGRA Early Grade Reading Assessment

ID identification

LTTP USAID Liberia Teacher Training Program

MOE Ministry of Education

ORF oral reading fluency

PTA parent-teacher association

RTI RTI International (trade name of Research Triangle Institute)

SY school year

USAID United States Agency for International Development

vi Midterm Assessment of the Impact of Reading and Math Interventions

Acknowledgments

The authors wish to acknowledge with appreciation and respect the contributions and support

of education staff from USAID/Liberia, the Liberian Ministry of Education (MOE), and

many other counterparts in the implementation of the LTTP Mid-Term Early Grade Reading

and Math Assessment (EGR/MA) in Grades 1, 2, and 3. At USAID/Liberia we thank Julia

Richards for her leadership and skillful coordination of the efforts of MOE and the program

staff; Casey McHugh for her advice and insights into the complexities of data collection in

Liberia; and Miriam White for long-term support and supervision of early grade learning

efforts in Liberia. Rev. Tokunboh Lawrence, Assistant Minister for Basic and Secondary

Education; Dr. Hawa Gall Kolchi, Deputy Minister for Instruction; Ms. Felicia Doe-Sumah,

Assistant Minister for Basic Education; and their colleagues at MOE were instrumental to

EGRA’s successful implementation and to mobilizing ministry support, from start to finish.

The success of this assessment owes a debt to more than a dozen MOE staff who participated

in assessor training and directly implemented the assessment as trained assessors and quality

assurance officers. We are thankful for the more than two dozen Liberian colleagues who,

having served as assessors on all prior assessments going back to the EGRA Plus project,

brought valuable field experience to the midterm data collection.

At LTTP, Delwlebo Tuowal, Director of Monitoring and Evaluation Unit, provided critical

support to the assessment's overall quality assurance efforts. Assisted by his colleague, Oscar

Goyee, he managed and supported the training and deployment of 9 quality assurance

officers. We sincerely thank the LTTP Chief of Party, Michael Blundell, for swiftly

mobilizing resources in support of this assessment and for timely decision making. Special

thanks are due to the finance, operations, and human resource staff who worked tirelessly to

ensure the assessment teams were deployed for data collection as scheduled.

Timothy Slade, RTI’s project manager, in partnership with Geanjay Roberts, LTTP's

Assessment Coordinator, led the implementation of this assessment. They are deeply

indebted to their colleagues from the Reading and Math team who trained assessors, and

worked long hours to ensure that the data collection commenced on time. The project was

backstopped by Medina Korda, RTI’s LTTP home office technical manager. The reading

assessment tools were developed by RTI’s reading experts, Drs. Peggy Dubeck and Yasmin

Sitabkhan; Dr. Sitabkhan developed the math assessment tools and provided critical support

during the training of assessors. RTI statisticians Cynthia Augustine, Michael Costello, and

Simon King provided tireless support in cleaning and analyzing the data.

Special thanks go to Josh Anderson and RTI's ICT team, who supported the development and

adaptation of the electronic instruments, and the final compilation of the database. oe

DeStefano was the lead author of this report, and he was supported by Timothy Slade,

Medina Korda, Trokon Wayne, and Patience Suah. Michael Richards of USAID/Liberia

Liberia Monitoring and Evaluation Program (LMEP) provided valuable insight into ways

external factors that may affect student learning outcomes might be further explored by

combining this dataset with others.

Finally, we thank the principals, teachers and students who participated in this study. To all

those who have not been mentioned here, we sincerely thank them all for their contribution to

the success of this assessment.

Midterm Assessment of the Impact of Reading and Math Interventions vii

Executive Summary

The Liberia Teacher Training Program (LTTP) has been leading a reading and math intervention

in approximately 790 schools. On average, children in grades 1 through 3 in participating schools

improved their achievement in all the reading skill areas in English that were tested at the end of

the 2012–13 school year. Compared to their performance in 2011, Grade 1 students on average

almost tripled their ability to read familiar words, more than tripled their decoding skills, and

were able to read text more than one-and-a-half times more fluently. Grade 2 and 3 students on

average were 10 times better at decoding and had almost three times higher oral reading fluency

than before the program. On average, students in all grades improved their listening and reading

comprehension.

When compared to the scores of

students in schools not participating in

the reading program, the gains experi-

enced by LTTP-impacted students are

greater. Between the 2011 baseline and

the 2013 midterm assessments, average

performance of students in schools

where LTTP was not being implemen-

ted in fact declined in some reading

skill areas. Meanwhile, students in

LTTP-supported schools had

improvements above what was

occurring in control schools. For

example, Figure 1 shows LTTP-

supported students improved more than

students in control schools.

While these gains are impressive,

student performance overall remained at a level well below what is acceptable for these three

grades. On average, grade 3 students at midterm were still well below what would be considered

adequate reading fluency to ensure comprehension. Students in LTTP-supported schools did see

their comprehension scores improve, but on average grade 3 students were still answering only 1

out of 5 questions correctly; grades 1 and 2, less than that.

In contrast to the results achieved in reading, LTTP’s math program was neither as well

implemented nor did it produce comparable results. Teachers had only five months in which to

implement the math lessons during the 2012–13 school year. Math materials arrived late and

teachers received less training and less ongoing coaching related to math than they did for

reading. Average scores for students in schools participating in the math intervention improved

slightly in some skill areas, but had no change or actually slightly declined in others. A similar

pattern was evident in the math performance of students in control schools.

Given that implementation of both the reading and math programs was compromised by

difficulties in getting materials to schools and ensuring regular support visits for teachers, these

midterm results bode well for what can be expected from continued and improved

implementation of LTTP.

Figure 1: Improvements in Oral Reading Fluency

0

10

20

30

LTTP Control LTTP Control LTTP Control

Grade 1 Grade 2 Grade 3

Baseline

Midterm

Midterm Assessment of the Impact of Reading and Math Interventions 1

1. Introduction

From 2008 to 2010, RTI International—with the Liberian Education Trust—implemented the

Early Grade Reading Assessment (EGRA) Plus: Liberia pilot program as a task order within

USAID’s Education Data for Decision Making (EdData II) project. Based on the success of the

EGRA Plus pilot, USAID decided to include in its Liberia Teacher Training Program (LTTP) an

expanded implementation of the reading and math interventions.

Of interest to USAID, the Government of Liberia, FHI360, RTI International, and the broader

community of education stakeholders in Liberia

and around the world is whether the same kind

of impacts that were achievable during the

piloting of EGRA Plus could be realized if the

program were implemented on a much broader

scale. Doing so would imply a significantly

greater implementation challenge—in terms of

simple logistics, but also in terms of maintaining

quality across a larger set of actors and responding to the circumstances of a greater cross-section

of school–community contexts. This report shows that after 16 months of implementation in

almost 800 schools, the LTTP reading program was succeeding at improving student

performance across all three grades and in all skill areas. The math program, having been

implemented for only five months, thus far showed mixed results.

EGRA Plus increased the oral reading fluency in English of students in EGRA Plus full

treatment schools almost three times as much as the level of improvement experienced in control

schools. Evaluation of EGRA Plus indicated that its success relied on developing and making

available highly scripted instructional materials for teachers and accompanying books for

students, training teachers in the use of those materials, providing coaching as a critical

component of ongoing support and supervision to teachers, and making use of regular

assessments of student progress.1

The LTTP intervention is drawing on the EGRA Plus model to introduce similarly structured

reading and math programs in grades 1, 2, and 3 to roughly 1,020 in four counties (Bong, Lofa,

Montserrado, and Nimba) in a phased approach. The first cohort to receive support had 792

schools. The reading program was introduced in all three grades in these schools beginning in the

middle of the 2011–12 school year. The math program was introduced in all three grades part

way through the 2012–13 school year. It is anticipated that the second cohort, consisting of

approximately 330 schools, will begin participating in LTTP’s reading and math interventions in

the 2013–14 school year, as well as 2014-15 school year.

To fully investigate the impact of the reading and math programs, LTTP’s interventions are

structured to allow rigorous evaluation. Schools from the four target counties included in

cohort 1 served as the treatment group for the midterm evaluation that is the subject of this

report. Cohort 2 schools in the same four counties that had not yet begun participating in the

LTTP reading and math interventions served as a control to which the cohort 1 results were

1 The project methodology is explained in detail in RTI International, Early Grade Reading and Mathematics

Methodology: From Pilot to Scale (Research Triangle Park, NC: RTI, n.d.). Prepared for USAID and FHI360 under

the Liberia Teacher Training Program.

After 16 months of support provided to almost

800 schools, student performance in reading in

grades 1–3 improved in all skill areas over and

above the changes seen in control schools. But

with only 5 months of support, math

performance showed mixed results.

2 Midterm Assessment of the Impact of Reading and Math Interventions

compared. A randomly selected sample of schools outside the four target counties will serve as

another comparator, especially after cohort 2 begins receiving treatment alongside cohort 1.2 A

baseline assessment of all three groups of schools was conducted in May 2011. In this report, the

baseline data are compared to student performance on the LTTP midterm early grade reading and

math assessments (EGRA and EGMA) completed in May and June of 2013. Differences in

student performance from baseline to midterm are compared for cohort 1, cohort 2, and the

external control schools.

The details of the research design used to evaluate the impact of LTTP’s reading and math

programs are outlined in the next section (Section 2) of this report. Section 3 summarizes the

status of the implementation of the reading and math programs during the 2012–13 school year.

Section 4 presents data on the performance of students in cohort 1 schools, and Section 5

compares their performance to the students in the cohort 2 control schools. Section 6 discusses

the characteristics of the students, teachers, principals, and the three comparison groups of

schools, based on the responses from the respective questionnaires. Section 7 concludes the

report.

2. Measuring Impact: The LTTP Research Design

Research Questions

The basic research question which this report addresses is:

Are students in the first cohort of schools to participate in the LTTP reading and math

interventions achieving better results?

To answer that question, it is necessary to also pose several other questions, including:

How are students performing in similar schools that are not currently participating in

LTTP (those in cohort 2)?

How does performance in these different sets of schools compare—at baseline and at

midterm?

Are cohort 1, cohort 2, and external control schools (and students) different in any

significant ways?

The assumption underlying this study is that any improvement achieved by students in cohort 1

schools will be due to the implementation of the reading and math programs in those schools. So

an additional question of concern is:

How well have the reading and math programs been implemented in cohort 1 schools?

Lastly, when, as assuredly is the case, variations in the performance of students across schools

manifest themselves, it is useful to investigate one other question, namely:

What student, family, and school factors are associated with the variations in performance

of students at different schools?

To answer these fundamental evaluation questions, the implementation of LTTP and the baseline

and midterm assessments was structured to facilitate the necessary comparisons—of schools at

baseline and midterm, and of schools in one category to schools in another. The exact nature of

2 Primary responsibility for providing the reading and math interventions to cohort 1 schools from 2013-2015 will lie

with MOE. LTTP will continue to provide some limited support to cohort 1, but will focus its efforts on cohort 2.


the information available; the sampling, survey, and assessment methodologies used; and the

limitations confronted in this study are explained below.

Research Design

As mentioned in the introduction, LTTP is introducing the reading and math programs in the four

most populous counties in Liberia, which contain the majority of the country’s primary schools.

Schools in the four counties were randomly assigned to cohort 1 and cohort 2 groupings. The

cohort 1 schools implemented the reading program beginning partway through the 2011–12

school year, continuing it into the 2012–13 school year, but with delays experienced in receiving

materials at the start of that year. The cohort 1 schools began the math program in January 2013.

Cohort 2 schools will only begin implementing the programs in the coming school year (2013–

14) but will continue to receive treatment through the end of the 2014-2015 school year. Save for

a small number of schools associated with the Rural Teacher Training Institutes, schools outside

the four target counties will not participate in the program during the lifetime of LTTP. A

discussion of cohort assignment criteria is shared in Annex A.

For the midterm assessment of LTTP impact, cohort 2 schools were considered a useful

comparator as they were located in the same counties (and sometimes same districts) as the

cohort 1 schools. The schools located in the same counties share similar characteristics and

function within the same administrative and operational environments. Thus, they provide a

useful control for LTTP, with comparison of cohort 1 and cohort 2 performance at baseline and

midterm serving as the means to “isolate” the impact of the reading and math interventions.

To include data on all three categories of schools, random samples of schools were selected from

the cohort 1, cohort 2, and external control groups at baseline, and then again at midterm. Based

on power calculations conducted at baseline, a total of 150 schools needs to be sampled at each

phase of the project, or 50 from each category, as depicted in Table 1.

Table 1: Implementation of Reading and Math Programs and Systematic Assessments

SY 2011–12 SY 2012–13 SY 2013–14 SY 2014-15

Cohort 1 792 schools – reading

792 schools – math

Cohort 2 ~ 330 schools – reading and math

External

Assessment Baseline

May 2011

Midterm

May 2013

Final May 2015

Cohort 1 schools 50 50 50

Cohort 2 schools 50 50 50

External schools 50 50 50

SY = school year

For both baseline and midterm assessments, participating districts within each county were

randomly selected. In both instances, schools within a county were then randomly selected based

on a probability proportional to each school’s enrollment. That is, each school was weighted

based on the proportion its students represented of the total enrollment in the county. Further


details on how the three groups of schools were randomly sampled at baseline and how the

sample was re-randomized at midterm appear in Annex A.

Data

Student performance in reading and math was assessed using the EGRA and EGMA instruments

developed for Liberia and further adapted for use under LTTP. Equivalent versions of the

assessment (see Annex B for information about the equating process) were used at baseline and

midterm, with students across all three grades being evaluated using the same or equivalent math

and reading tests. Detailed descriptions of the EGRA and EGMA methodologies are available,

and the instruments used for this evaluation are available upon request. The reading assessment

evaluated students in seven skill areas: phonemic discrimination,3 letter name knowledge,

familiar word reading, nonword reading or decoding, oral reading fluency, oral reading

comprehension, and listening comprehension. The math assessment tested students on number

identification, quantity discrimination, missing number, addition, subtraction, and word

problems.

The baseline assessments were conducted in May 2011, well before the reading and math

interventions were introduced to cohort 1 schools. These data provided end-of-grade measures of

performance for students in the three grades. The baseline data presented in this report, however,

differ from those contained in the LTTP Baseline Assessment Report from October 2011. After

careful review of how schools were placed into the three groups (cohort 1, cohort 2, and external

cohort), LTTP staff and RTI analysts determined that the cohort assignments of 46 schools were

no longer accurate and therefore needed to be updated for the midterm analysis. For consistency,

the schools’ baseline designations as cohort 1, 2, or external were also changed. The

reassignment of schools to the correct group changed the composition of those groups at both

baseline and midterm, thus changing the calculations of weighted mean scores at baseline. A

fuller explanation of this is included in Annex A.

The midterm assessments were carried out in late May and early June 2013, providing end-of-

grade points of comparison. As mentioned above, this meant that the reading intervention was

implemented for a year and a half and the math intervention for roughly five months prior to the

midterm assessment.

In addition to the student assessments, the midterm data collection also surveyed students

regarding their families, their home situations, their attendance at school, and the actions of their

teachers. Teacher and head teacher questionnaires gathered additional data about each school and

its instructional environment (again, these instruments are available upon request).

The resulting set of data contributes to our understanding of the schools in each of the

comparison categories, some details and summary information about which are provided in

Section 6. In that section and throughout this report, all summaries use weighted data and thus

are representative of the entire population of schools from which the samples were drawn.

3 Several of the midterm EGRA subtests had slightly different names in the Tangerine® version than had been used

in the paper-based instruments at baseline. For simplicity, in our comparative analyses we use the subtest names

from the midterm instruments.


3. Implementation of the Reading and Math Interventions in

Cohort 1 Schools

The essential ingredients of the improved instructional programs for reading and math include

scripted lesson plans, materials for teachers and students designed around those lesson plans,

training for teachers in reading instruction and on how to follow the scripts, regular coaching and

support to help teachers improve their practice, and regular assessments throughout the school

year so teachers can monitor student progress. How each of these was implemented over the past

two years in cohort 1 schools is discussed below.

Two sector-wide challenges to LTTP implementation bear mentioning. At the school level, large

amounts of time are lost when school starts late, ends early, or is closed on the local market day

or for other—often capricious—reasons. Time is also lost due to poor teacher and student

attendance. At the national level, instability in the Ministry of Education (MOE) persists, with

numerous and ongoing changes in key technical and leadership positions during the life of the

project.

Scripted Lesson Plans

The cores of the reading and math programs are the sets of scripted lessons developed by LTTP

for each subject for grades 1, 2, and 3. The math program contains 25 weeks of daily scripted

lessons (125 lessons) for each grade, and is sequenced in a manner to follow the Liberian

curriculum and standards. The reading program consists of 30 weeks of scripted daily lessons

(150 lessons) for each grade, also aligned with the Liberian curricular expectations. The scripted

lessons provide specific instructions for the teachers regarding lesson content and conduct.

Student materials were developed so that there would be direct alignment between the content in

the teacher’s scripted lesson and in the corresponding page of the students’ books. Additional

sets of readers are also provided to each school.4

For the first two years of implementation (involving cohort 1 schools), only the grade 1 materials

were available. These materials were distributed to and used in all three grades in the cohort 1

schools. The EGRA Plus experience showed that the levels of reading skill development in

English in most Liberian elementary schools were so low that students in grades 2 and 3 could

and did benefit from content intended for grade 1 students.5

The time needed to develop and distribute the teacher and student materials was longer than

anticipated and the reading materials were not delivered to schools at the start of the 2011–12

school year as planned. Teachers and students received the materials in January and February

2012. In the second school year of support, the reading materials again reached schools later than

anticipated when heavy rains delayed distribution. Math materials were delivered to schools in

November and December of 2012 and teachers began using them in January 2013. These delays

meant that the midterm assessment was conducted after a shorter implementation period than

4 LTTP would like to thank Jennifer Cooper-Trent for allowing us to use her Fantastic Phonics Books in the first

cohort. For more information, visit: http://www.teachtheworldtoread.com/

5 Because overall the quality of instruction and the resulting student performance were poor, LTTP provided the

same grade 1 materials to all three grades. This approach permitted all the students to “catch up” and build solid

foundations of basic skills. In the 2013–14 school year, grades 2 and 3 will use their grade-appropriate materials.

The same approach will be used for both reading and mathematics.

http://www.teachtheworldtoread.com/


was originally envisaged. Reading lessons were being implemented for 11–12 months, instead of

two school years; and math lessons for only five months instead of a full school year.

Teacher and Student Materials

Per the LTTP implementation plan, each teacher received two volumes that contained the above

described scripted lesson plans. Volume 1 focuses on the first semester and volume 2 on the

second semester. For cohort 1 schools, teachers in all three grades received the same set of

manuals. In addition to the lesson plans, teachers received letter cards and pocket charts. Each

student was given a math and a reading book with content fully linked to the teacher manuals,

allowing students to easily follow along during lessons and providing practice exercises to

reinforce each day’s lesson. Additional reading materials were also provided. Each school

received around 50 titles per grade to use in their reading rooms. Even with the materials

provided by the project, schools and their communities remained for the most part print-poor

environments.

Training

To take full advantage of these materials, teachers require adequate training. LTTP provides a

two-week workshop (one week on reading, one on math) for teachers at the start of the academic

year. A follow-up refresher training is held at the beginning of the second semester of the same

year. These trainings focus on the use of the instructional materials, especially the scripted

lessons. There is heavy emphasis on practicing using the scripted lessons; teachers are able to run

through a month’s worth of lesson plans. Two weeks of initial training, however, is a bare

minimum. Regular follow-up and coaching are necessary to ensure teachers learn and internalize

the methodologies embodied in the scripted lessons. Additionally, overall, Liberian teachers

themselves have low levels of skill development, in terms of content and pedagogy. Teacher

upgrading (see below) therefore is also needed. High teacher turnover has also been a problem,

as significant numbers of teachers who have been trained by LTTP have then been transferred

out of project schools.

In addition to the training focused on the specific instructional approach of the LTTP

interventions, the project also requests that teachers who are not fully certified attend the year-

long in-service program implemented by the MOE’s Rural Teacher Training Institutes. This

involves one month of residential training, plus monthly visits. LTTP supports this initiative by

providing training to the in-service trainers who work with these teachers.

Coaching

Teachers are introduced to and get to practice using the scripted lessons during the trainings

offered at the start of the year and at the beginning of the second semester. However, ongoing

coaching plays an important role in reinforcing what they learn in those trainings and helping

them implement their lessons and improve their instruction on a day-to-day basis. When the

EGRA Plus model was taken to scale under LTTP, coaching was one area where the approach

was compromised, in the interest of lowering costs (i.e., by using fewer coaches). Under LTTP

one coach is assigned to 12 schools, as compared to one coach for 4 to 8 schools under EGRA

Plus. This means many fewer visits per month by a coach to the teachers he/she supports. LTTP

coaches visit schools once per month, following a systematized coaching model. Coaches

provide additional training, observe teachers, and assist them with any challenges that they might

have. Coaches work less frequently on math-related instruction than on reading.


The challenges faced by the LTTP coaching model involve overcoming the distances between

schools (exacerbated by one coach having to cover 12 schools); and the inaccessibility of some

areas, especially during the rainy season. Managing and supervising the coaches also poses a

problem as the scale of the project increases. With less supervision, some coaches are not

following through on the number of visits or on the prescribed amount of time to be spent at a

school during a visit.

Assessment

The last component of the interventions concerns regular assessment by teachers of their

students’ progress. Teachers are trained on how to use EGRA-like instruments to assess student

performance on letter knowledge, reading, and comprehension at four points during the year.

Teachers are supposed to track student performance from one assessment to another, producing

individual student report cards. Additionally, teachers work with their principals to develop

school report cards that are discussed with their school’s parent-teacher association (PTA).

4. Midterm Impact of LTTP Interventions in Cohort 1 Schools

When compared to the baseline6 levels of performance, are students in cohort 1 schools attaining

higher levels of reading and math proficiency following the above-described implementation of

the LTTP interventions? To answer this question, data are presented here that compare the

midterm performance of students in cohort 1 schools to the pre-intervention baseline assessment

results across all reading and math skills areas.

Reading

As mentioned earlier, during both assessments, students were evaluated in seven reading skill

areas: phonemic discrimination, letter name knowledge, familiar word reading, nonword reading

(or decoding), oral reading fluency, oral reading comprehension, and listening comprehension.

Four of these subtests were timed (letter name knowledge, word and nonword reading, and oral

reading fluency), thus providing measures not only of how well students performed in these

skills areas, but also how automatic each skill had become for them. The oral reading fluency

scores for midterm and baseline were equated to account for differences in the reading passages

used during each assessment (see Annex A). Baseline and midterm results in each of these skill

areas are presented in Table 2.

Table 2: Cohort 1 Average Reading Results for Baseline and Midterm

Cohort 1 Grade 1 Grade 2 Grade 3

Base-line

Mid-term

Effect Size

Base-line

Mid-term

Effect Size

Base-line

Mid-term

Effect Size

Phonemic discrimination (# correct out of 10 items)

1.7 3.2 .58 2.7 4.1 .45 1.6 4.8 1.17

Letter name knowledge (correct letters per min)

42.6 56.2 .47 61.0 75.2 .49 72.0 85.4 0.54

6 As mentioned earlier, the baseline mean values are different from those included in the LTTP Baseline Assessment

Report of October 2011. Please see Annex A for a full explanation of why the baseline values were recalculated.



Base-line

Mid-term

Effect Size

Base-line

Mid-term

Effect Size

Base-line

Mid-term

Effect Size

Familiar word reading (correct words per min in isolation)

2.4 8.7 .62 5.0 18.6 .72 6.6 25.1 0.96

Nonword reading (correct words per min in isolation)

0.2 0.9 .29 0.3 3.2 .44 0.3 4.5 0.56

Oral reading fluency (ORF) (correct words per min of text)

2.1 6.1 .45 4.8 14.2 .49 7.6 20.2 0.58

Oral reading comprehension (# correct out of 5 questions)

0.1 0.4 .38 0.3 0.7 .38 0.5 1.1 0.44

Listening comprehension (# correct out of 3 questions)

0.8 1.7 .79 1.0 1.9 .72 1.3 2.1 0.61

As previously stated, students at midterm performed better than students at baseline in all the

assessed areas. While the absolute levels of performance of students in each skill area, with

perhaps the exception of letter name knowledge, were still disappointingly low for these grade

levels, improvements were substantial in many areas in each grade, with effect sizes well above

the norms seen in most education interventions.7

The most impressive effect sizes emerged in the improvements in phonemic discrimination for

grade 1 and grade 3 students, in familiar word reading in all three grades, and in listening

comprehension in all three grades. It would appear that in intervention schools, teachers were

beginning to do a better job of addressing these and the other basic skill areas. However, the

levels of phonemic discrimination, though improved considerably, were still low, with students

in all grades on average identifying the first sound correctly in less than half the words presented.

Improvements in listening comprehension and oral reading comprehension are worth noting.

Listening comprehension for grade 3 students improved to the point where on average, students

answered two out of three questions correctly after being read a story in English. First grade

average listening comprehension (1.7 correct responses) improved to the point that it surpassed

the average third-grade level at baseline (1.3). Oral reading comprehension, though improved,

was still very low in all grades, with even grade 3 students on average able to respond to roughly

only one question out of five.

Some of the improvement in average scores was due to reductions in the percentages of students

scoring zero on any subtest. Figure 2, Figure 3, and Figure 4 show the percentages of students

at midterm scoring zero in each reading skill area as compared to those scoring zero at baseline,

for grades 1, 2, and 3, respectively.

7 The effect size relates the size of the change in reading scores to the pooled standard deviation for the samples of

students assessed at baseline and midterm. An effect size of .57 in grade 3 ORF, for example, can be interpreted to

mean that students in that grade had improvements equivalent to more than half of the standard deviation for typical

students tested at both points.



Reading Skill Areas, Baseline and Midterm

The biggest reductions in zero scores for grade 1 students (Figure 2) were in phonemic

discrimination, oral reading fluency (ORF), and listening comprehension. Small drops in zero

scores were evident for familiar word reading, nonword reading, and oral reading

comprehension.

For grade 2 students, like grade 1, the biggest dropoffs in zero scores from baseline to midterm

also were in oral reading fluency and phonemic discrimination (Figure 3). The reductions in zero

scores for familiar word reading, nonword reading, and reading comprehension for grade 2 were

higher than those seen in grade 1. The percentage of grade 2 students with zero comprehension

is still very high (59%).



0%

20%

40%

60%

80%

100%

PA Letters Word reading Non-wordreading

ORF Reading comp Listening comp

Baseline Midterm

0%

20%

40%

60%

80%

100%



Baseline Midterm

Reduction in zero scores

Phonemic discrimination



Grade 3 had the largest reductions in zero scores in phonemic discrimination, oral reading

fluency, and nonword reading (Figure 4). These large decreases in zero scores in the higher-order

skills of nonword reading (essentially decoding) and oral reading fluency are encouraging.



Gender Disaggregated Reading Performance

The overall pattern of improvement from baseline to midterm was essentially the same for both

boys and girls in all reading skills areas and for all grades. This is evident in the way the pattern

within each skill area’s graph in Figure 5, which shows boys and girls experiencing similar

increases in each grade and skill. The exception was for phonemic discrimination in grade 2,

where boys did not see improvement in their average scores from baseline to midterm, while

girls, starting from a lower average at baseline, did see improvement.

Differences in performance between boys and girls were statistically significant only in certain

areas. In other words, other than for the areas indicated below, boys and girls performed

essentially the same in all other skill areas and grades. The first area of statistically significant

gender differences in performance was for phonemic discrimination in grade 2 (which may

explain the one difference in the pattern of improvement pointed out above). Grade 2 girls on

average dramatically underperformed their male classmates on this skill, identifying only 1.6

sounds correctly compared to 3.6 for boys.

Another area of statistically significant gender differences was letter name knowledge in grade 1.

Grade 1 girls on average underperformed relative to boys by 10% on this skill. Additionally,

nonword reading is an area where girls scored only about half as well as boys, in all three grades.

For oral reading fluency, grade 2 girls outperformed boys at baseline, reading 7.8 words per

minute compared to 2.4 wpm for boys. Boys more than caught up at midterm, on average

achieving 16.8 wpm compared to 11.3 for girls. Girls far outperformed boys at baseline, reading

18 wpm compared to 4 wpm. But boys experienced a much greater gain than girls, so that by

midterm boys were reading 17.7 wpm compared to 21 wpm for girls.

Though listening comprehension is not shown in Figure 5, girls did have statistically significant,

albeit only slightly, lower performance at midterm on this skill.

0%

20%

40%

60%

80%

100%



Baseline Midterm



Figure 5: Gender-Disaggregated Performance in Reading at Baseline and Midterm

-

2.0

4.0

6.0

8.0

10.0

Boys Girls

Phonemic Awareness

-

20.0

40.0

60.0

80.0

100.0

Boys Girls

Letter Recognition

-

5.0

10.0

15.0

20.0

25.0

30.0

Boys Girls

Non-Words

-

5.0

10.0

15.0

20.0

25.0

30.0

Boys Girls

Familiar Words

-

5.0

10.0

15.0

20.0

25.0

30.0

Boys Girls

Oral Reading Fluency

-

1.0

2.0

3.0

4.0

5.0

Boys Girls

Reading Comprehension

baseline midterm

G1 G2 G3 G1 G2 G3 G1 G2 G3 G1 G2 G3 G1 G2 G3 G1 G2 G3

G1 G2 G3 G1 G2 G3 G1 G2 G3 G1 G2 G3 G1 G2 G3 G1 G2 G3


Math

Student achievement was evaluated in several different areas of math skill development. These

included number identification, quantity discrimination (greater than and less than), missing

number (filling in a sequence of numbers), addition, subtraction, and word problems. For all the

skill areas, student performance was assessed based on the percentage of correct responses. A

higher percentage of correct responses indicated that the student had more fully developed the

skill (because the items in each subtest covered a range of levels of difficulty, students who were

answering more items correctly were, by definition, dealing with a greater range of ability levels

for a given skill8). Three of the math subtests were also timed—number identification, addition,

and subtraction—thus permitting evaluation of how automatically students performed the skill

(as measured by the number of items correctly answered per minute).

Table 3 compares baseline and midterm math results for grade 1 students in cohort 1 schools.

Table 4 and Table 5 present the same data for grades 2 and 3 respectively. The first two data

columns in these tables show the percentages of students with zero scores on each subtest. The

second set of data shows the changes from baseline to midterm in average scores for each subtest,

including corresponding effect-size calculations for the difference between midterm and baseline

scores.

Table 3: Grade 1 Cohort 1 Results on Math Subtests

Grade 1 % Zero Scores Average Scores

Base- line

Mid- term

Base- line

Mid-term Effect Size

Number identification – accuracy 0% 1% 38% 41% 0.12

Number identification – automaticity

11.5 12.9 0.16

Quantity Discrimination 38% 12% 36% 41% 0.17

Missing number 97% 63% 1% 9% 0.51

Addition – accuracy 2% 17% 21% 21% -0.03

Addition – automaticity

6.3 6.2 -0.03

Subtraction – accuracy 5% 35% 16% 14% -0.14

Subtraction – automaticity

4.9 4.1 -0.15

Word problems 30% 17% 33% 41% 0.27

Note: Automaticity scores represent the number of correct items per minute. Zero scores are not shown for number identification, addition, or subtraction automaticity, because they are the same as those for the accuracy measures.

Only a nominal percentage of grade 1 students scored zero on number identification at baseline

and midterm, and there was modest improvement in the average percentage of correctly identified

numbers (accuracy) and in the number identified per minute (number identification automaticity).

The overall level of performance on this most simple skill area—reading numbers correctly—is

cause for concern. At midterm, on average students correctly identified numbers only 41% of the

8 However, the items on the EGMA were not arranged in order of difficulty, so we could not interpret the overall

results to discern the level of skill development students were achieving. Further item-level analysis of the EGMA

results could provide that kind of information.


time, and their responses were far from automatic—with an average of only 13 numbers identified

correctly per minute (which equates to taking more than 4 seconds per number).

The percentage of students scoring zero on quantity discrimination was reduced by two-thirds

and on word problems was almost half of the baseline percentage. However, the improvements in

the average scores for these two subtests were slight, with effect sizes of 0.18 and 0.27

respectively.

For the missing number subtest, in which students need to provide the number that correctly

completes a sequence, the percentage of zero scores was one-third lower at midterm, although

almost two-thirds (63%) of students were still unable to answer any of these items correctly. The

greatest increase in average scores was for this subtest, but 9% correct is well below what

students should be able to do.

The addition and subtraction subtests had increases in the percentage of zero scores and

essentially no change or slight decreases in average percentage correct and in the number of

items completed correctly per minute. The levels of performance in these two skill areas in

grade 1 indicated neither accuracy nor automaticity was yet being adequately developed.

Across all math subtests in grade 2 (Table 4), zero scores were fairly low except for on the

missing number subtest. Encouragingly, the percentage of students scoring zero on the missing

number subtest at midterm reduced considerably, from almost all children to just over half.

Average scores on the missing number subtest improved dramatically, largely due to the

reduction of zero scores. However, the average percentage of correct responses in this skill area

was still extremely low (14%).



Base- line

Mid- term

Base- line

Mid- term

Effect Size

Number identification 0% 0% 53% 56% 0.12


16.0 18.3 0.22


Missing number 98% 54% 1% 14% 0.61

Addition – accuracy 0% 8% 28% 29% 0.06


8.3 8.6 0.05

Subtraction – accuracy 1% 20% 19% 20% 0.06


5.6 5.9 0.06

Word problems 3% 10% 48% 47% -0.01


Discouragingly, in the addition, subtraction, and word problems subtests, the percentage of zero

scores increased from baseline to midterm. Average scores on the addition, subtraction, and word

problems subtests were essentially unchanged. Grade 2 students were still far from demon-

strating anything approaching automatic knowledge of basic addition and subtraction facts.


There was a modest increase in the grade 2 average scores at midterm for the number

identification subtest (accuracy and automaticity) and a more appreciable improvement in

quantity discrimination.

Results for grade 3 students (Table 5) were better than for grades 1 and 2, but across the board

the results indicated that students were continuing to struggle with consolidating their basic math

skills. Number identification and quantity discrimination were the only skill areas where grade 3

students were demonstrating reasonable performance—zero scores of 0% and 1% respectively

and average scores of two-thirds correct in both areas. Again there was marked improvement on

the missing number subtest, adequate improvement on the word problems subtest, but continued

low results on the addition and subtraction subtests. Grade 3 students were slightly more accurate

and more automatic on these basic math facts, but still were working at unacceptably low levels.



Base- line

Mid- term

Base- line

Mid- term

Effect Size

Number identification 0% 0% 66% 67% 0.03


20.1 22.3 0.22


Missing number 96% 38% 2% 21% 0.81

Addition – accuracy 1% 4% 38% 36% -0.09


11.3 10.8 -0.08

Subtraction – accuracy 2% 10% 25% 24% -0.02


7.4 7.3 -0.02

Word problems 8% 3% 42% 54% 0.44


To summarize the performance of cohort 1 students in both reading and math, Figure 6 and


Figure 7 on the following pages compare average scores for grades 1, 2, and 3 for both baseline

and midterm. It is evident that in reading, across all subtests, there was higher performance at

midterm in each grade (except for nonword reading in grade 1). In math, clear improvements at

midterm compared to baseline are evident only in quantity discrimination, missing number, and

word problems (except in grade 2) subtests.


Figure 6. Performance of Cohort 1 Students Across Grades in Each Reading Skill Area

Midterm

Baseline

Note that the scales

in the reading graphs

are not necessarily

the same

Benchmark value

0.0

20.0

40.0

60.0

80.0

100.0

120.0


Letter Name Knowledge

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0


Phonemic Discrimination

0.0

10.0

20.0

30.0

40.0

50.0

60.0


Familiar Word Reading

0.0

10.0

20.0

30.0

40.0

50.0

60.0


Nonword Reading

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0


Oral Reading Fluency

0.0

1.0

2.0

3.0

4.0

5.0


Oral Reading Comp.

0.0

1.0

2.0

3.0


Listening Comprehension


Figure 7: Performance of Cohort 1 Students Across Grades in Each Math Skill Area

0%

20%

40%

60%

80%


Number Identification

0%

20%

40%

60%

80%


Quantity Discrimination

0%

20%

40%

60%

80%


Missing Number

0%

20%

40%

60%

80%


Word Problems

0%

20%

40%

60%

80%


Addition

0%

20%

40%

60%

80%


Subtraction

Midterm

Baseline

For the math graphs,

all the vertical scales

are the same

No benchmarks

values are available

for math skill areas


On a common assessment instrument, grade 2 students are expected to outperform grade 1, and

grade 3 to outperform grade 2. The lines sloping upward from left to right conform to this

expectation. How steep the lines are indicates how much better students in one grade performed

than the previous. For reading, the slopes of the lines at midterm are steeper than at baseline for

phonemic discrimination, familiar word reading, nonword reading, and oral reading fluency. The

figures also show that for the phonemic discrimination, familiar word reading, and listening

comprehension subtests in grade 1 and on all subtests in grade 2, on average students at midterm

were performing better than grade 3 students performed at baseline. In math, the slopes of the

lines at midterm are steeper only for the quantity discrimination, missing number, and word

problems subtests.

How high up the vertical scale the points sit is indicative of the absolute level of performance in

each skill area. Benchmark levels of performance for letter name knowledge, phonemic

discrimination, familiar word reading, nonword reading, and oral reading fluency are shown as

red lines in each of the graphs in Figure 6. These benchmarks are taken from the EGRA Plus

final evaluation, which established them based on the scores that corresponded to the 90th

percentile of student performance in each skill area. The 90th percentile benchmarks are 120

letters correct per minute, 9 sounds correctly identified, 54 familiar words read, 28 unfamiliar

words read, 78 words for oral reading fluency, and 100% reading and listening comprehension.9

Only on the most basic skill—letter name knowledge—were students approaching the

benchmark level of performance. In math, although explicit benchmarks do not exist, we can see

that only on basic skills such as number identification and quantity discrimination did cohort 1

math scores approach levels that could be considered even barely acceptable for students in these

three grades.

5. Cohort 1 Schools Compared to Control Schools

The previous section examined the differences in performance of students in cohort 1 schools at

baseline and midterm. To fully understand the extent to which the reading and math interventions

in those approximately 800 schools are making an impact on learning outcomes, it is necessary to

compare cohort 1 improvement since baseline to the improvement that may have also been

occurring in schools where the interventions are not being implemented. The assessments of

student performance have therefore been carried out in two other groups of schools. One

comparison group was drawn from the same counties where LTTP is operating but consists of

schools that will be included in cohort 2 of LTTP beginning in the 2013–14 school year. The

second comparison group includes schools drawn randomly from outside the four counties and is

referred to as the external control group of schools. As stated earlier, cohort 2 schools, being

located in the same county, are considered the most useful control comparison. This section

therefore focuses on comparing cohort 1 to cohort 2. Section 6 and the exhibits in Annex C

compare all three groups.

To restate, then: To evaluate the net impact of the reading and math programs, it was necessary

to look at the changes in average performance from baseline to midterm in each grade for the

9 Medina Korda and Ben Piper, EGRA Plus: Liberia—Program Evaluation Report (Research Triangle Park, NC:

RTI International, 2011), pp. 61-62. Prepared for the USAID Education Data for Decision Making (EdData II)

project, Task Order No. EHC-E-06-04-00004-00. Retrieved from

https://www.eddataglobal.org/countries/index.cfm?fuseaction=pubDetail&ID=283

https://www.eddataglobal.org/countries/index.cfm?fuseaction=pubDetail&ID=283


external control group of schools. The results of this comparison are the differences (increase or

decrease) in student performance in each skill area for schools not participating in the program.

The same calculation for cohort 1 schools yielded the differences from baseline to midterm for

students in schools that are implementing the LTTP reading and math programs. Comparison of

these differences, or what is referred to as the difference of differences, indicates the net increase

(or decrease) in student performance attributable to the implementation of the reading program.

The following paragraphs present and discuss these calculations, first for reading and then for

math.

Reading

Table 6 shows the average baseline, midterm, and differences for grades 1 through 3 in each of

the reading subtests for the external control group of schools. The columns labeled “∆” (delta)

show the difference obtained when the baseline was subtracted from the midterm results. Positive

∆s indicate an increase, negative values a decrease in average scores.

Table 6: Reading Performance in Control Schools

External Control Grade 1 Grade 2 Grade 3

Baseline Midterm ∆ Baseline Midterm ∆ Baseline Midterm ∆

Phonemic Discrimination 1.5 2.8 1.2 3.2 2.8 -0.5 4.3 4.7 0.3

Letter Name Knowledge 58.0 53.6 -4.4 64.6 54.6 -10.0 65.2 79.0 13.8

Familiar Word Reading 3.3 6.8 3.5 7.0 12.4 5.4 11.2 28.5 17.3

Nonword Reading 0.5 0.7 0.3 0.9 1.2 0.3 1.4 5.0 3.7

Oral reading fluency 9.5 6.3 -3.2 6.3 9.1 2.8 18.8 24.3 5.5

Oral Reading Comprehension 0.6 0.5 -0.1 0.4 0.5 0.1 1.0 1.2 0.2

Listening Comprehension 1.1 1.4 0.3 1.1 1.5 0.5 1.4 2.2 0.8

Other than in letter name knowledge, the levels of performance at baseline in each skill area were

quite low. Only slight increases from baseline to midterm were evident in nonword reading in

grades 1 and 2, oral reading comprehension in grades 2 and 3 and phonemic discrimination in

grades 1 and 3. More significant increases occurred in oral reading fluency and listening

comprehension in grades 2 and 3, and in familiar word reading across all three grades.

Table 7 shows exactly the same information as Table 6, but for cohort 1 schools. Some of the

results are worth highlighting. In almost every skill area, average baseline performance in

cohort 1 schools was lower or about equal to the baseline scores in the external control group.

More importantly, in the columns labeled ∆’, Table 7 shows greater differences between

midterm and baseline for cohort 1 than are evident in the external control group. There were

large increases in phonemic discrimination in grade 3, in letter name knowledge and familiar

word reading in all three grades, more modest increases in nonword reading (but with a large

jump evident in grade 3), and significant improvement in oral reading fluency also in all three

grades. Listening comprehension improved in all three grades. Oral reading comprehension

doubled in grade 3. And there were no negative ∆’s for cohort 1 schools—on average there were

no reading skill areas where performance declined.


Table 7: Reading Performance in Cohort 1 Schools


Baseline Midterm ∆’ Baseline Midterm ∆’ Baseline Midterm ∆’

Phonemic Discrimination 1.7 3.2 1.5 2.7 4.1 1.4 1.6 4.8 3.2

Letter Name Knowledge 42.6 56.2 13.5 61.0 75.2 14.2 72.0 85.4 13.4

Familiar Word Reading 2.4 8.7 6.4 5.0 18.6 13.6 6.6 25.1 18.5

Nonword Reading 0.2 0.9 0.7 0.3 3.2 2.9 0.3 4.5 4.2

Oral Reading Fluency 2.1 6.1 4.0 4.8 14.2 9.3 7.6 20.2 12.6

Oral Reading Comprehension 0.1 0.4 0.3 0.3 0.7 0.5 0.5 1.1 0.6

Listening Comprehension 0.8 1.7 0.9 1.0 1.9 0.9 1.3 2.1 0.7

To appreciate the impact of the LTTP reading and math programs in cohort 1 schools, the last

table in this section, Table 8 below, shows the results of calculating the difference of differences

between cohort 1 and cohort 2 (control). For each grade and skill area, the value of ∆ꞌ minus ∆

was calculated, showing how much larger the average improvement over baseline was in

cohort 1 schools when compared to the average improvement in the control schools. These

difference of differences can be thought of as the net improvement in average reading scores in

cohort 1 schools, and therefore represent the impact on student performance plausibly

attributable to the implementation of the reading program.

Table 8: Difference of Differences for Cohort 1 Compared to Control Schools

Difference of Differences Net Improvement in Reading in Cohort 1 Schools


Phonemic Discrimination 0.3 1.9 2.9

Letter Name Knowledge 17.9 24.2 -0.5

Familiar Word Reading 2.9 8.2 1.2

Nonword Reading 0.5 2.6 0.5

Oral Reading Fluency 7.2 6.6 7.1

Oral Reading Comprehension 0.3 0.4 0.4

Listening Comprehension 0.6 0.4 -0.1

After 16 months of implementation, the LTTP reading intervention was having a net positive

impact in all reading skill areas, except letter name knowledge in grade 3. And the program was

contributing to some large net improvements in cohort 1 schools across several reading

competencies. The largest net increases were in letter name knowledge in grades 1 and 2, in

nonword reading in grade 2, and oral reading fluency in all three grades. Cohort 1 students in

grades 1 and 2 at midterm on average were recognizing about 18 to 24 more letters per minute.

Grade 2 students on average were reading 8 more familiar words per minute. And students in


cohort 1 schools in all grades on average were reading 7 more words per minute than they would

have been if the reading program were not being implemented.

The average net improvement in student performance in nonword reading in grades 2 and 3 at

midterm represented a tenfold increase over the performance at baseline. In oral reading fluency,

the amount of net improvement obtained at midterm was more than double the average scores at

baseline.

To further illustrate how the implementation of LTTP altered the learning performance of

students in reading between baseline and midterm, Figure 8 shows the baseline-to-midterm trend

lines in oral reading fluency. Note how the cohort 1 trend lines all slope up from left to right

(baseline to midterm). The control line for grade 1 slopes downward. Furthermore, the cohort 1

trend for grade 2 and grade 3 is quite steep, representing 194% and 165% increases in oral

reading fluency. LTTP’s efforts appear to have considerably accelerated students’ average rate of

improvement in oral reading fluency, and in fact, for grade 1, counteracted the deterioration of

skills that students on average experienced in control schools.

Figure 8: Trends in Average Oral Reading Fluency for Students in Cohort 1 and

External Control Students

Math

Unlike the clear, positive impact the reading intervention had had by midterm, the math

intervention, as noted in the previous section, was showing only limited or no impact. Table 9

and Table 10 mirror the presentation just made regarding reading results, first examining the

average performance of students in math in the control schools (cohort 2) at baseline and

midterm, then examining the differences between baseline and midterm in math for the cohort 1

schools, and lastly calculating the net change (or difference of differences) in cohort 1 math

scores.

0.0

5.0

10.0

15.0

20.0

25.0

30.0

baseline midterm baseline midterm baseline midterm


Average Oral Reading Fluency Control Cohort 1


Table 9 shows that on average, students in control schools at midterm had mixed performance in

math compared to baseline. On number identification, addition, and subtraction in grades 1 and

2, student performance on average declined from baseline to midterm. In all three grades,

performance on quantity discrimination, missing number, and word problems improved in all

three grades, with the larger increases on the first two skills in grade 3 and on the last in grade 1.

Table 9: Math Performance in Control Schools

External Control Grade 1 Grade 2 Grade 3

Baseline Midterm ∆ Baseline Midterm ∆ Baseline Midterm ∆

Number ID – accuracy 45% 42% -3% 51% 49% -1% 57% 70% 13%

Number ID – automaticity 13.4 12.9 -0.5 16.0 15.8 -0.2 17.3 22.4 5.0

Quantity Discrimination 39% 42% 4% 36% 44% 8% 52% 71% 20%

Missing Number 2% 7% 4% 3% 8% 5% 9% 35% 25%

Addition – accuracy 26% 21% -5% 37% 26% -12% 38% 38% 0%

Addition – automaticity 7.8 6.3 -1.5 12.1 7.7 -4.4 11.5 11.4 -0.1

Subtraction – accuracy 22% 14% -7% 27% 15% -11% 33% 26% -7%

Subtraction – automaticity 6.5 4.3 -2.2 8.2 4.6 -3.6 9.8 7.9 -2.0

Word Problems 29% 40% 11% 48% 49% 1% 61% 64% 3%

Comparing Table 9 to Table 10 reveals that in grade 1, cohort 1 students had slightly lower

average baseline and midterm performance than control school students in almost all math skills

areas and had a mixture of some higher, some lower average performance across skill areas in

grades 2 and 3.

Table 10: Math Performance in Cohort 1 Schools


Baseline Midterm ∆’ Baseline Midterm ∆’ Baseline Midterm ∆’

Number Identification – accuracy 38% 41% 3% 53% 56% 3% 66% 67% 1%

Number Identification – automaticity 11.5 12.9 1.4 16.0 18.3 2.3 20.1 22.3 2.2

Quantity Discrimination 36% 41% 5% 45% 58% 12% 54% 66% 12%

Missing Number 1% 9% 8% 1% 14% 13% 2% 21% 19%

Addition – accuracy 21% 21% -1% 28% 29% 1% 38% 36% -2%

Addition – automaticity 6.3 6.2 -0.2 8.3 8.6 0.3 11.3 10.8 -0.5

Subtraction – accuracy 16% 14% -2% 19% 20% 1% 25% 24% 0%

Subtraction – automaticity 4.9 4.1 -0.7 5.6 5.9 0.4 7.4 7.3 -0.1

Word problems 33% 41% 8% 48% 47% 0% 42% 54% 12%


As previously discussed, cohort 1 students saw improvements at midterm relative to baseline in

number identification, quantity discrimination, missing number, and word problems, but had

declines in their performance on addition and subtraction (both accuracy and automaticity) in

grades 1 and 3. The difference of differences between cohort 1 and external control schools are

presented in Table 11.

Table 11: Difference of Differences Analysis for Cohort 1 Compared to Control Schools

Difference of Differences Net Improvement in Math in Cohort 1 Schools


Number Identification – accuracy 6% 4% -13%

Number Identification - automaticity 1.9 2.5 -2.9

Quantity Discrimination 2% 4% -7%

Missing Number 3% 8% -6%

Addition – accuracy 4% 13% -1%

Addition – automaticity 1.4 4.7 -0.3

Subtraction – accuracy 5% 13% 6%

Subtraction – automaticity 1.4 3.9 1.9

Word Problems -3% -1% 9%

Cohort 1 schools on average experienced a small net improvement when compared with control

schools in grades 1 and 2 for number identification, quantity discrimination, missing number, and

addition and subtraction. In grade 3, a net positive improvement was evident only for subtraction

and word problems. Note that the net positive improvement evident for cohort 1 students in

addition and subtraction resulted from their average performance in these areas actually declining

less than that of students in control schools. While the fact that the program was introduced

partway through the year and used for only five months accounts for some of the disappointing

net effect of the math intervention, other factors may be contributing to poor implementation. For

example, teachers received less systematic training in math instruction and less frequent

coaching support for math. Overall, LTTP placed less emphasis on math improvement.

To further understand the different outcomes of students in the cohort 1 and control groups of

schools, the following section presents data on those students and their families, teachers, and

school environments.


6. Characteristics of Cohort 1, External Control, and Cohort 2

Students, Teachers, Principals, and Schools

Data from three sources provide

insight into the students, families,

and schools included in cohort 1,

cohort 2 (control), and the external

cohort groupings created by LTTP.

Students responded to a series of

questions about their families, their

home lives, and their experiences

at school. Teachers answered

questions about themselves, their

classrooms, and their instructional

practices. Lastly, principals

provided information in response

to questions about themselves and

their schools. These data as

reported at midterm are

summarized here to provide some

sense of the differences and

similarities across the three

groupings of schools. Additional

detailed tables that summarize the

findings of the student, teacher,

and principal surveys are provided

in Annex D.

As shown in Table 12, fewer

students in cohort 2 (23%) reported speaking English at home as compared to cohort 1 (28%). A

greater share reported speaking English at home in the external cohort (36%). Likewise, only

55% of cohort 1 students compared to 49% of cohort 2 reported that their parents could read and

write in English. Seventy percent of students in the external cohort said their parents were literate

in English. Cohorts 1 and 2 had similar percentages of students reporting parents who were not

literate in any language—34% and 38%. Only 17% of external cohort students indicated that

their parents were not literate. In all three groups, more than 90% of students said that their

teachers often read aloud to them. In contrast, in cohort 1 schools, 93% of students reported their

teachers practicing letter sounds with them, while 72% of cohort 2 and only 47% of the external

cohort students reported this. All three cohorts had more than 90% of students saying their

teachers made them read aloud in class, and about that many in cohorts 1 and 2 also said that

their teacher assigned reading for them to do at home. A higher percentage of cohort 1 students

reported having books to take home from school (77%) than cohort 2 (58%) or external (51%).

Similar percentages of students in all three groups reported having repeated a grade (between

66% and 70%). More cohort 2 students reported missing school (44%) than cohort 1 (37%) or

external (31%), but in all three cases those percentages would indicate high rates of absenteeism.

Table 12: Student Characteristics at Midterm

Cohort 1

Cohort 2 (control) External

Speaks English at home 28% 23% 36%

Parents read/write English 55% 49% 70%

Parents cannot read/write 34% 38% 17%

Has reading books at home 73% 63% 55%

Is read aloud to at home 59% 57% 48%

Practices reading at home 61% 61% 56%

Takes books home from school 77% 58% 51%

Has library at school 37% 31% 29%

Teacher reads aloud often 95% 93% 92%

Teacher practices letter sounds:

Never 6% 27% 52%

Often 93% 72% 47%

Teacher has students read aloud:

Never 5% 5% 7%

Often 94% 94% 92%

Teacher assigns reading for home:

Never 4% 6% 13%

Often 95% 93% 87%

Has repeated a grade 70% 69% 66%

Missed school last week 37% 44% 31%

Eats lunch at school 68% 64% 64%


Data gathered through the teacher survey are presented in Annex D, with some information

about teachers at midterm shared here. Cohorts 1 and 2 had slightly higher percentages of female

teachers than the external cohort, but all three had very few female teachers. Similarly, 25% of

cohort 1 and 30% of cohort 2 teachers were volunteers, compared to only 16% of the external

cohort. As should be expected, many more cohort 1 teachers reported having attended in-service

training, having received training on how to teach reading, and having received support visits.

Regarding receiving training related to math, 71% of cohort 1 teachers said they had; 60% of

external cohort and 47% of cohort 2 teachers did. 10 High percentages of cohort 1 (90%) and 2

(87%) teachers had teacher’s guides, but only about a quarter of all teachers reported that they

frequently used the official curriculum. Low percentages of teachers (20% or less), when asked,

could show a good example of that day’s lesson plan. Given that LTTP provides daily lessons, it

is surprising to see (and not readily explicable) that only 20% of cohort 1 teachers showed a good

example of a lesson plan.

Teachers were asked a series of questions about their instructional practices. From a quarter to a

third of cohort 1 teachers reported frequently having their students participate in activities related

to learning to read, such as repeating letters and words, sounding out unfamiliar words,

individually reading aloud, learn the meaning of new words, etc. A higher percentage of cohort 1

teachers said that they had their students do a reading assignment at home and that their students

always take books home to read. A higher percentage of teachers in cohort 1 (compared to the

other cohorts) who were teaching math indicated that they used manipulables, had students

practice math problems in partners, and called on their students to individually answer math

questions orally. Fewer cohort 1 than cohort 2 and external cohort teachers stated that they

assigned math homework frequently.

To gauge their expectations for their students, teachers were asked what students should be able

to do at the end of the year. About 60% to 67% of teachers across all three groups said their

students should be able to read grade-level stories. More cohort 1 teachers (albeit only 47% of

them) said students should be able to sound out words and 55% of cohort 1 teachers believed

students should understand stories they read, compared to less than 40% of cohort 2 and external

cohort teachers.

A higher percentage of principals in cohort 1 than in cohort 2 or external schools indicated that

this year more parents, community members, and students showed interest in math than in

previous years. However, more than 90% of all principals in all schools indicated that they were

making special efforts to improve reading in their schools. Such questions, for which

interviewees may sense that one type of answer is more desirable than another, often lead to

these kinds of conflicting results (i.e., teachers may feel they need to make their school/principal

look better by providing what they perceive to be the best answer). In other words, self-reported

answers carry with them inherent bias and some of the data from the teacher and principal

questionnaires were perhaps tainted by this. In addition, the number of respondents within some

of the different subcategories of questions asked of teachers and principals were too few to be

consistently statistically reliable.

10 Note that at the time of the baseline assessment, the math-focused classroom and coaching intervention had not

yet been envisioned or implemented. The midterm teacher questionnaire was adjusted for this program change with

an item on support visits for math instruction.


7. Conclusions

This midterm assessment of LTTP’s reading and math programs set out to determine whether

students in the first cohort of schools to participate in those programs improved. On average,

students benefiting from the reading program were achieving much better results in all the skill

areas tested at midterm when compared to the performance of students in the same grades prior

to the implementation of LTTP.

For math, the change in performance of students in LTTP-supported schools was mixed.

Students participating in the math program were seeing better performance in some areas, but not

in others. The number of students at midterm unable to compare quantities, insert a missing

number in a series, or solve word problems (i.e., those scoring zero on those skills) did decrease.

However, the percentages unable to solve any basic addition and subtraction facts actually

increased.

The midterm assessment also sought to determine whether students in the first cohort of schools

to participate in LTTP’s reading and math programs had experienced greater gains in their

learning achievement than students in the groups of schools in the same counties, but that had not

benefited from the reading and math interventions. Again, reading performance in LTTP schools

clearly increased more than in control schools. For reading, the difference between baseline and

midterm achievement was large enough in LTTP-supported schools to enable students in those

schools to perform on par with or even better than students a grade or two grades ahead of them

had performed at baseline. For example, an average cohort 1, grade 1 student at midterm could

read 8.7 familiar words per minute (Table 7), compared to an external cohort student in grade 2

at baseline who could read 7 words (Table 6).

For math on many subtests, the gains seen in cohort 1 schools were less than the improvements

in achievement experienced in external control schools.

Two other questions are worth considering as conclusions to this report.

First, for reading, are the improvements over baseline documented by this midterm assessment

attributable to the implementation of the LTTP reading intervention? Besides demonstrating

differences from baseline to midterm in all reading skill areas that surpassed the differences

evident in control schools, cohort 1 schools exhibited some characteristics that were surely

associated with the work of LTTP. For example, 93% of students in cohort 1 schools reported

that their teachers often practiced letter sounds with them, compared to only 72% of cohort 2

students (Table 12). Daily practice of letter sounds is a key feature of the LTTP-developed

reading lessons. Similarly, having students take books home and practice reading at home are

also features of the LTTP reading intervention. Fifty percent more cohort 1 students reported

doing this than cohort 2 students. Twice as many cohort 1 as external cohort teachers stated that

students should learn to sound out words (Table D2). The explicit teaching of decoding

(sounding out words based on knowing letter sounds) is also a critical feature of LTTP reading

lessons. Eighty-four percent of cohort 1 teachers reported having training on reading, compared

to only 50% of cohort 2 teachers (Table D2). And almost four times as many cohort 1 teachers

(79%) compared to cohort 2 teachers (22%) reported having received support visits related to

teaching reading (Table D2). Data such as these, gleaned from the student, teacher, and principal

surveys, while limited, do shed light on how LTTP’s reading program is changing teacher-

student interaction. So it is possible to conclude that in schools where teachers received the

training and materials and had regular coaching support, instruction is improving and that is


leading to the improvements in reading performance which the data presented in this report make

evident.

Second, for math, why was there less impact? As has already been stated throughout this report,

the implementation of the math intervention was delayed and perhaps was not as rigorously

supported as the reading intervention. The true impact of the math program cannot really be

assessed until teachers and students have had an opportunity to benefit from a full year of

instruction supported by the LTTP-provided lessons, training, and coaching.

Midterm Assessment of the Impact of Reading and Math Interventions A–1

Annex A: Details of the Study Methodology

This annex describes the methodology used to collect, analyze, and weight the data from the

midterm assessment of the LTTP reading and math interventions. The first section describes the

study’s sampling approach and procedures. The second section discusses challenges encountered

during data collection which necessitated on-the-fly modifications to those procedures. The third

section describes the cohort assignment process at baseline, the updates that were necessary at

midterm, and the impact those changes had on the data reported here, including weighting. A

discussion of the instruments used during the midterm assessment, their relation to the ones used

at baseline, and the way in which their results were linked can be found in Annex B.

Sampling Design and Procedures

The original two-stage sample design for LTTP was created such that in the end, the researchers

would be able to produce national estimates regarding (1) students’ reading and mathematics

skills, and (2) certain types of information about teachers and principals in Liberia. In this design,

four of Liberia’s 15 counties (Bong, Lofa, Nimba, and Montserrado) were designated to receive

the treatment intervention. In the four treatment counties, selected schools were assigned to

cohort 1 or cohort 2. Cohort 1 schools were those designated to receive the reading intervention

during 2011–2013; cohort 2 schools would then receive the reading intervention during 2013–

2015. Delaying the intervention in cohort 2 would allow those schools to serve as a comparison

group for cohort 1 at the time of the midterm evaluation. While a third cohort comprising schools

sampled from the remaining 11 counties was also developed, it is important to note that it cannot

serve as a true “control” for the other two. The four treatment counties form part of Liberia’s

economic development corridor, and as such benefit from greater investment and a more robust

socioeconomic environment than the other 11 counties from which external cohort schools were

drawn.11

Within counties, districts were selected with simple random sampling. Schools within selected

districts were selected with probability proportional to size (PPS). The frame was serpentine

sorted by district, location, and school type. Thus, replacement schools (which were next to

sampled schools on the frame) would be similar to sampled schools in terms of location and

school type.

The measure of size (MOS) used to select schools was the enrollment in primary 1 and primary 2

classes. The following notation was used to explicitly define the MOS and the probabilities of

selection for first- and second-stage sample units. For this discussion and the subsequent section:

h indexes the first-stage sampling stratum, district;

i indexes the target population schools within the first-stage stratum, such that i =

1,… ,Mh where Mh is the total number of school in stratum h;

mh is the total number of schools selected in the first-stage stratum h;

mh* is the total number of schools selected minus any certainty selections in the first-

stage stratum h;

11 Note that the sampling design created to select schools and counties into the LTTP treatment and control cohorts

also was used to conduct the random sampling for the LTTP baseline assessment.

A–2 Midterm Assessment of the Impact of Reading and Math Interventions

j indexes the second-stage stratum, grade, where j = 1, 2, 3;

Nhij represents the total number of primary 1 and primary 2 students listed in the

education management information system (EMIS) data files used to construct the MOS;

and

hij n represents the student sample size to be selected from the jth stratum within schools

sampled from stratum h (hith school).

We defined the desired overall sampling rate for students within the hth school sampling stratum

and the jth stratum as

𝑓ℎ𝑗 = 𝑛ℎ𝑗

𝑁ℎ𝑗 ,

where nhj is the total number of students selected across all schools and Nhj is the corresponding

population of students.

The second-stage sampling strata were defined by the cohort variable, which indicated whether

the school was being treated, would be treated in the future, or was not in the treatment plan. An

independent sample of schools was selected for each school stratum using Chromy’s sequential

PPS sampling algorithm, with minimum replacement and the MOS defined above. The

preliminary expected selection frequency for the hith school was evaluated as

𝑚ℎ𝑆ℎ𝑖

𝑆ℎ+ ,

where Sh+ is the sum of all of the size measures. After we removed the few certainty-selected

schools (e.g., schools that were to be followed via a longitudinal study throughout the time

period), the expected selection frequency for the hith sample school used to determine the design

weights was calculated as

𝜋ℎ𝑖 = 𝑚ℎ

∗ 𝑆ℎ𝑖

𝑆ℎ+ .

Before we drew the sample, we sorted the frame of study-eligible schools by location, type of

school (public or community), and MOS to form implicit strata, with the objective of ensuring

representativeness and aiding travel between selected and replacement schools. Three

replacement schools were selected for each sampled school in case a school had closed, had

moved, did not have the correct primary grades, or was otherwise not available for assessment.

Midterm Assessment of the Impact of Reading and Math Interventions A–3

Sampling Changes Due to Logistical Challenges During Data

Collection

The midterm assessment data were collected from 20 May 2013 through 14 June 2013. This

period coincided with both the beginning of the rainy season and year-end testing.12 These two

factors led to unexpectedly high rates of student absenteeism in the grades assessed. Where the

survey’s sampling protocol aimed for 10 students assessed in each of the three grades, after three

weeks of data collection, an average of only 6–7 students were being assessed in each grade in

each school. In order to compensate for the shortfall in observations that was projected to result,

the student-level sampling protocol was modified.

During prior EGRA and EGMA data collections in Liberia, students had expressed that they felt

left out if they were not chosen as part of that day’s 10-student random sample. It therefore

became standard practice to assess all students present in a classroom if that day’s attendance

numbered 14 or fewer. (If attendance that day numbered 15 or more, the standard 10-student

sampling protocol would be followed.) In an effort to make up some of the shortfall in LTTP

student observations, the assessor teams were directed to follow this precedent and start drawing

a sample of 14 students per grade as a new default. This modified sampling approach was in

effect for the final week of data collection (10–14 June 2013).

Cohort Assignment Criteria

At the time of the LTTP baseline assessment (May 2011), schools were assigned to a cohort

based on two main criteria: (1) the school’s geographic location, and (2) the period in which it

was anticipated to receive the LTTP reading and math intervention. Shortly after the baseline

assessment, however, a third criterion was added as a result of the discovery of an overlap issue:

(3) the level of intervention/treatment the school had received earlier under the USAID EGRA

Plus: Liberia project. Table A1 presents the relationships among these criteria and cohort

assignments, as well as indicating when the criteria were applied.

Table A1: Cohort Assignment Criteria at Baseline

LTTP cohort

assignment Geographic location

Anticipated

period of LTTP

treatment

EGRA Plus treatment level

Control Light Full

Cohort 1 Bong, Lofa,

Montserrado, Nimba

2011–2013 x x

Cohort 2 2013–2015 x x

External All other counties Never x

Time of application of

criterion:

Before LTTP baseline assessment After LTTP baseline

assessment (June 2011)

12 Primary students are not required or expected to complete Liberia’s year-end national exams, and during the

examination period, it is common for school personnel to excuse those students as a result. Assessment teams

reported in several instances that grade 1–3 students had been instructed by the principals and teachers to stay home

from school.

A–4 Midterm Assessment of the Impact of Reading and Math Interventions

The new third criterion was deemed to be of even greater importance than the first two in terms

of analysis of LTTP treatment effects. For instance, where schools met the geographic

considerations for inclusion in cohort 1, but had already received the “Full” treatment under

EGRA Plus, they were reassigned to the external cohort.

The addition of the third criterion caused a small number of EGRA Plus Legacy (Full) schools to

be shifted from cohorts 1 and 2 to the external cohort; the schools removed from cohort 1 were

then replaced by schools from cohort 2. A small number of external schools in Bomi County that

had been EGRA Plus (Control) schools were also shifted into cohort 1.

Separately, in the interest of leveraging LTTP’s existing efforts in the area of pre-service teacher

training, the decision was made to extend the reading intervention to those schools within the

“catchment areas” of the Rural Teacher Training Institutes. As a result, some schools affiliated

with the Kakata Rural Teacher Training Insitute in Margibi County and the Webbo Rural

Teacher Training Institute in River Gee County, which had been assigned to the external cohort,

were shifted into cohort 1. The Ministry of Education also requested that a number of would-be

cohort 2 schools receive the intervention earlier, so those were shifted into cohort 1.

At the time of the midterm assessment and analysis, schools’ cohort assignments reflected their

actual treatment status. These updated cohort assignments were then retroactively applied to the

same schools assessed at baseline, so that a single school would not appear under two treatment

statuses in the midterm data analysis results. Harmonizing cohort assignments between baseline

and midterm had an impact on the proportion of assessed schools in each. Table A2 summarizes

the cohort assignment realignments and includes descriptions of the reasons for which

reassignments occurred.

Table A2: Summary of Cohort Assignment Realignments

Original

Baseline

Assignment

Midterm

Assignment Count Reasons for reassignment type

Cohort 1

NEW Cohort

2 8

Correct a misclassification in baseline data set

NEW

External 3

Remove EGRA Plus Legacy (Full) schools

Cohort 2

NEW Cohort

1 28

Move several schools from Nimba, Upper Lofa, and

Bong County (originally assigned to cohort 2) into cohort 1, at the MOE's request

Replace EGRA Plus Legacy (Full) schools dropped from cohort 1

NEW

External 1

Remove EGRA Plus Legacy (Full)

External

NEW Cohort

1 6

Add EGRA Plus Legacy (Control) schools from Bomi

County

Add schools from RTTI catchment areas in Margibi and River Gee counties

NEW Cohort

2 —

—

Midterm Assessment of the Impact of Reading and Math Interventions B–1

Annex B: About the Assessment Instruments

This annex discusses the instruments used for data collection during the midterm assessment of

the LTTP reading and math interventions. The first section addresses the content of the

assessments’ subtasks, noting why, how, and where the assessments differed from those used for

the baseline assessment.13 The second section addresses how the results of the oral reading

fluency passage and associated reading comprehension questions were statistically equated

across time periods. The third section describes quality control measures that became available

because the data were collected electronically at midterm using Tangerine® software on tablet

devices.

EGRA and EGMA Instruments Used at Midterm

The EGRA and EGMA instruments for Liberia’s LTTP 2011 baseline assessment were

developed during workshops with the Ministry of Education (MOE) and with contributions from

reading and math experts. A pilot assessment using the instruments was conducted in a small

number of schools and the pilot results were subjected to Rasch analysis to assess the difficulty

of the questions in relation to the ability of the students who participated in the pilot test. The

goal was to ensure that the instruments contained a range of questions from easy to hard, in order

to distinguish among students of different abilities, ideally in a uniform distribution.

Although instrument modifications are always required under multiple administrations (for

purposes of testing integrity), in order to retain comparability of results between the baseline and

midterm assessments, attempts were made to minimize such changes. (Major content changes

were made to the oral reading fluency and reading comprehension subtests, however, as

discussed in the Equating section below.) Table B1 indicates the nature of any changes made to

the subtests between baseline and midterm.

Table B1: Comparison of Baseline and Midterm Instruments

Title of subtest

Modifications 2011 baseline 2013 midterm

Early Grade Reading Assessment

Orientation to Print Not applicable Subtest dropped because of time and ceiling

effects14

Letter Name

Identification

Letter Name Knowledge Items retained but re-randomized

Phonemic Awareness Phonemic Discrimination New items developed (using same parameters as

baseline) to reduce redundancy, increase

construct validity

13 The full instruments are available upon request. Please contact Timothy Slade at [email protected].

14 Piloting of the midterm instruments during assessor training revealed that overall administration time was

approaching the 40-minute mark, in contrast to the 30-minute target. It was decided that because students’

performance on the orientation to print subtest at baseline had little predictive or explanatory value, this subtest

would be dropped.

B–2 Midterm Assessment of the Impact of Reading and Math Interventions

Title of subtest

Modifications 2011 baseline 2013 midterm

Familiar Word

Identification

Familiar Word Reading Items retained but re-randomized

Simple Unfamiliar

Nonword Decoding

Nonword Reading Items retained but re-randomized

Passage Reading and

Comprehension

Oral Reading Fluency and

Oral Reading

Comprehension

New passages equated; see Equating section

below

Listening

Comprehension

Listening Comprehension Subtask retained without changes

Early Grade Mathematics Assessment

Number Identification Number Identification

20% of the items within each subtask were

retained unchanged from the baseline EGMA. The

remaining items were modified in a way that

preserved similarity to the original items (i.e.,

numbers of comparable difficulty were used).

Quantity Discrimination Quantity Discrimination

Missing Number Missing Number

Addition Addition

Subtraction Subtraction

Word Problems Word Problems

Equating

Student reading performance on an EGRA can be falsely inflated if students are given access to

and directed to memorize the story that will be used for the oral reading fluency subtest. It has

become standard practice to eliminate this potential source of bias by replacing the story used at

baseline with a new one that is of equivalent difficulty. While developing multiple passages of

exactly the same reading level is extremely challenging, a statistical analysis procedure known as

means equating can be used to ascertain the relative difficulty of a pair of passages.

Four new oral reading passages were drafted prior to the LTTP midterm assessment, and

appropriate reading comprehension questions were developed for each. (For the following

discussion, the matched pair of a story and its related comprehension questions is called a

“dyad.”) These four new dyads, along with the original dyad from the baseline assessment, were

then administered to a sample of 124 grade 3 children from five urban schools in Monrovia.

Every child was assessed on all five of the dyads. The order in which the dyads were

administered was randomized, mitigating any bias that might otherwise have been introduced by

assessment fatigue or knowledge gained from repeating similar tasks.

The students’ results were then analyzed using means equating. This procedure allows the

calculation of a mean for each passage and each set of reading comprehension question of the

dyad. These means are compared to select a passage comparable to but slightly more difficult

than the baseline passage.

Midterm Assessment of the Impact of Reading and Math Interventions B–3

In this instance, the passage selected for the midterm is being equated, and the passage that was

used at baseline is serving as the reference. In the analysis results, a factor greater than 1 would

indicate that the equated passage was more difficult than the reference passage; a factor less than

1 would indicate that the equated passage was less difficult. The equating factors used to

compare results on the LTTP midterm dyad to results on the baseline dyad were calculated to be

the following:

Midterm oral reading fluency: 1.18

Midterm oral reading comprehension: 1.36

Multiplying the scores a student earned on those two subtasks at midterm by the appropriate

equating factor provided a new equated score, which allowed for a more accurate understanding

of the dyads’ relative difficulty than raw scores.

The example in Table B2 illustrates what would happen if two different students—one sampled

at baseline and one sampled at midterm—were to earn the same scores on both the oral reading

passage and the associated reading comprehension questions.

Table B2: Example of Application of the Means Equating Method

Subtest

Student 1:

baseline

score

Student 2:

Original midterm

score

Equating

multiplier

Student 2:

Equated midterm

score

Oral Reading Fluency 30 30 1.18 35.32

Oral Reading Comprehension 2 2 1.36 2.72

While the raw scores suggest that the students’ performance was equal, in reality the midterm

student was assessed using a slightly more difficult passage. The equated midterm score thus

provides a clearer view of the midterm student’s performance relative to that of the baseline

student.

Using the same set of calculations, it is also possible to equate the means of subgroup

performance between time periods, such as the trends between children in the same grade, or of

the same gender. All scores reported here are equated scores unless otherwise mentioned.

Quality Control

The midterm assessment was conducted using RTI-developed, open-source electronic data

collection software known as Tangerine®.15 The software, which has been customized for the

EGRA and EGMA, includes built-in data validity checks, controlled timing, and conditional

logic configurations that can help drastically reduce assessor-driven error. Where mobile

connectivity allows, the data can be uploaded for nightly review of data quality and quantity.

Collecting student results directly into a digital format eliminates the need for hiring data entry

clerks to transcribe paper forms, minimizing the introduction of new errors.

15 For more about this software, see the Tangerine website, www.tangerinecentral.org.

B–4 Midterm Assessment of the Impact of Reading and Math Interventions

During the LTTP midterm assessment, approximately half of the eight teams of assessors were

able to upload their results on a regular basis (i.e., once every 2–3 days). Whenever a new batch

of results became available, project staff at RTI’s home office reviewed the data to confirm that

proper administration procedures were being followed.16 From a quality standpoint, wherever

deviations from standard protocol were noted, project staff contacted supervisors and assessors

and reminded them of the proper protocol to be followed. From a quantity standpoint, these

regular reviews of incoming data allowed the project team to anticipate a shortfall in the number

of observations collected and proactively adjust the student-level sampling procedures to

compensate.17

16 For instance, if a child had earned a zero score (first 10 items incorrect) on the letter name knowledge subtest, but

the “time remaining” record indicated that more than 30 seconds had elapsed, that would be an indication that the

assessor was not observing the 3-second rule. Project staff would then contact the assessment team by SMS or phone

and remind the assessor that a child should only be able to hesitate for a full 3 seconds per item before being directed

to move on to the next item.

17 See Annex A, Details of the Study Methodology.

Midterm Assessment of the Impact of Reading and Math Interventions C–1

Annex C: Comparison of the Average Performance in

Cohort 1, Cohort 2, and External Control Schools

How to read bar graphs below:

First, Second, and Third = Grade 1, Grade 2, and Grade 3

The bar on the left for a given cohort refers to LTTP baseline; the bar next to it refers to LTTP

mid-term.

Figure C1: Letter Name Knowledge Comparisons

Figure C2: Familiar Word Reading Comparisons

-

20.0

40.0

60.0

80.0

100.0

External Cohort 2 Cohort 1 External Cohort 2 Cohort 1 External Cohort 2 Cohort 1

First Second Third

Letter Name KnowledgeBaseline

Midterm

-

10.0

20.0

30.0

40.0


First Second Third

Word Reading

C–2 Midterm Assessment of the Impact of Reading and Math Interventions

Figure C3: Nonword Reading Comparisons

Figure C4: Oral Reading Comprehension Comparisons

-

2.0

4.0

6.0

8.0

10.0


First Second Third

Nonword Reading

-

1.0

2.0

3.0

4.0

5.0


First Second Third

Equated Reading Comprehension

Midterm Assessment of the Impact of Reading and Math Interventions C–3

Figure C5: Listening Comprehension Comparisons

Figure C6: Oral Reading Fluency Comparisons

-

1.0

2.0

3.0


First Second Third

Equated Listening Comprehension

-

10.0

20.0

30.0

40.0


First Second Third

Equated ORF

C–4 Midterm Assessment of the Impact of Reading and Math Interventions

Figure C7: Phonemic Discrimination Comparisons

0.0

2.0

4.0

6.0

8.0

10.0


First Second Third

Phonemic Discrimination

Midterm Assessment of the Impact of Reading and Math Interventions D–1

Annex D: Additional Data on Characteristics of Students,

Teachers, Principals, and Schools

Table D1: Responses to Student Survey Questions

Baseline

Midterm

Cohort 1 External Cohort 2


17% 43% 45% Speaks English at home 28% 36% 23%

62% 81% 67% Parents read/write English 55% 70% 49%

27% 13% 30% Parents cannot read/write 34% 17% 38%

58% 57% 57% Reads books at home 73% 55% 63%

37% 48% 55% Reads aloud at home 59% 48% 57%

43% 56% 55% Practices reading at home 61% 56% 61%

Teacher reads aloud:

0% 3% 4% Never 4% 8% 7%

41% 60% 56% Often 95% 92% 93%

Teacher practices letter sounds:

87% 53% 61% Never 6% 52% 27%

6% 41% 16% Often 93% 47% 72%

Teacher makes students practice reading aloud:

1% 4% 4% Never 5% 7% 5%

71% 62% 31% Often 94% 92% 94%

28% 33% 38% Always 0% 0% 1%

Teacher assigns reading for home:

1% 16% 6% Never 4% 13% 6%

71% 57% 36% Often 95% 87% 93%

28% 27% 33% Always 1% 0% 0%

48% 44% 31% Has books to take home from school 77% 51% 58%

14% 17% 6% Has library at school 37% 29% 31%

90% 82% 71% Has repeated a grade 70% 66% 69%

36% 15% 21% Missed school last week 37% 31% 44%

71% 66% 71% Eats lunch at school 68% 64% 64%

D–2 Midterm Assessment of the Impact of Reading and Math Interventions

Table D2: Responses to Teacher and Principal Survey Questions

Baseline

Midterm



1% 6% 11% Is Female 9% 6% 13%

86% 37% 45% Has C Certificate 61% 52% 47%

48% 40% 50% Is volunteer 25% 16% 30%

0% 12% 0% Will not continue teaching next year 2% 0% 11%

57% 83% 76% Has attended in-service in last year 92% 65% 78%

78% 69% 64% Has had training on reading 84% 41% 50%

50% 60% 67% Has had training on math 71% 60% 47%

17% 47% 21% Received support visits on how to teach reading 79% 21% 22%

Not availablea Received support visit on how to teach math 73% 11% 73%

92% 86% 33% Reports that principal observes classes daily 87% 70% 82%

57% 77% 76% Received an external inspection in last year 88% 64% 74%

10% 40% 25% Showed a good example of the day's lesson plan 20% 18% 11%

73% 27% 37% Uses official curriculum frequently 24% 25% 27%

62% 93% 98% Has a teacher's guide 90% 68% 87%

In last five days has frequently had students:

23% 23% 45% repeat letters/words 32% 18% 21%

76% 17% 27% sound out unfamiliar words 28% 23% 28%

17% 28% 30% read aloud 33% 38% 22%

13% 27% 31% learn meaning of new words frequently 38% 27% 23%

16% 34% 32% retell a story they read during the week 35% 18% 20%

17% 31% 33% read on their own in school 37% 30% 33%

16% 37% 56% do a reading assignment at home 47% 28% 21%

2% 10% 8% Students take books home to read never 4% 28% 31%

76% 33% 73% Students take books home to read sometimes 37% 37% 26%

15% 42% 8% Students take books home to read always 45% 26% 33%

a While students' mathematical skills were assessed at baseline, at that time a math-focused intervention designed to affect teachers' classroom practices and supported by coaching visits had not yet been envisioned. Teachers were therefore not asked this question at baseline.

Midterm Assessment of the Impact of Reading and Math Interventions D–3

Table D2: Responses to Teacher and Principal Survey Questions (continued)

Baseline Midterm

Cohort 1 External Cohort 2 Cohort 1 External Cohort 2

In last five days has:

Not availablea

used sticks, stones, flashcards to teach math 30% 15% 18%

assigned math homework 25% 34% 45%

had students complete problems at board 50% 59% 40%

had students practice problems in partners 48% 32% 32%

had students do individual seat work 38% 14% 45%

called on students to answer math questions orally 42% 25% 25%

Believes at end of year students should:

46% 35% 36% read grade-level stories 63% 67% 61%

6% 22% 29% sound out words 47% 22% 34%

44% 23% 27% understand stories they read 55% 34% 37%

4% 25% 2% know letter names 25% 18% 21%

43% 28% 11% do one-digit addition + subtraction 67% 60% 55%

11% 25% 59% know simple fractions 36% 28% 29%

5% 17% 1% do multiplication and division 52% 29% 46%

4% 25% 0% tell time 17% 10% 10%

Not available make change 15% 3% 15%

Responses drawn from Principal Questionnaire:

97% 98% 93% Has tried to improve reading. If so, because: 95% 90% 93%

36% 9% 1% saw other schools doing it 18% 3% 8%

80% 41% 35% independently thought they needed to do it 55% 61% 47%

3% 22% 8% ministry told them to do it 8% 20% 11%

2% 25% 12% teachers got training on how to teach reading 40% 8% 8%

55% 70% 93% Parents now show more interest in reading 63% 50% 54%

23% 78% 92% Whole community show more interest in reading 67% 53% 60%

26% 83% 85% Students show more interest in reading 76% 78% 76%

96% 82% 93% There is more discussion about improving reading 80% 67% 56%

a While students' mathematical skills were assessed at baseline, at that time a math-focused intervention designed to

affect teachers' classroom practices and supported by coaching visits had not yet been envisioned. Teachers were therefore not asked this question at baseline.

liberia teacher training program (lttp) midterm assessment

Documents