literacy boost belajar indonesia midline & endline report...literacy boost belajar indonesia...

Literacy Boost Midline & Endline Report

1

Literacy Boost

BELAJAR Indonesia

Midline & Endline Report

August 2014

Lauren Pisani, Ni Luh Putu Satyaning, Benny Giri, Carolyn Alesbury, Marie

de Fretes

With special thanks to Save the Children staff and our team of enumerators:

Harun Anggo, Asyikin, Maria Lakaseru, Neny Wakanno, Ignasia Aton,

Ambrosius Tanono, Robertus Acong, Hiasintus Sekundus, Januarius Bria,

Paskalis Teti, Wendelinus Inta, Wilbrodus Gasa, Yansen Bau, Yohanes Nino,

Yulius Leto, Frida Roman, Maria Seran, Melpi Marpaung, Theodora Ruing,

Viktoria Fatin, Yovita Ikun, Yovita Nahak, Susana Edon


2

Executive Summary

In 2012, Save the Children started to implement Literacy Boost methodology in Belu district, NTT

province, Indonesia. The results of the 2012-2013 pilot Literacy Boost program in Belu demonstrated

improvement in the literacy skills and behavior of children. Consequently, Save the Children expanded the scope of interventions through Better Literacy for Academic Result (BELAJAR) program. As an

evaluation for BELAJAR program, two assessments have been conducted in May-June 2014.

The first assessment evaluated the one-year BELAJAR program intervention in Belu. The sample

includes 652 children (326 boys, 326 girls) from the same 35 schools sampled in 2013, with 20 second

graders sampled per school. The result showed the increasing of BELAJAR children literacy skill in

2014, compared to the result in 2013. There was a significantly higher number of BELAJAR than

comparison children transitioning from non-readers to readers in 2014. On average, BELAJAR children

show mastery of basic skills like letter identification and common word reading, and advances in the

more advanced skills of fluency, accuracy and reading comprehension. The second assessment

evaluated the two-year ANCP program intervention in Belu. The sample includes 389 second graders

(199 boys, 190 girls) from the same 20 schools sample in 2012 and 2013. At each school where data

was collected, 20 second graders were sampled. The result demonstrated the stronger literacy skill in

second graders assessed in 2013 and 2014 than second graders assessed in 2012.

In 2014, across Literacy Boost schools in both assessments reading skill averages show no significant

differences between the children who are native Indonesian speakers and those who are not. This is an

important improvement from 2013 when the overall tendency was for second language learners to

have weaker literacy skills than native speakers of Indonesian. In addition, this trend of non-native

Indonesian speakers having weaker language skills than native speakers continues in 2014 for control

schools.

As recommendations, communities and Save the Children staff who support them should continue to

work to consistently offer out-of-school reading opportunities to children. Analyses indicate that the

amount of participation in Literacy Boost activities outside of school is directly related to stronger

reading skill gains and a decrease in the likelihood of being a struggling student (defined as scoring at

least one standard deviation below the average letter and common word scores). In addition, parents

should be encouraged and reminded to continue to support their children’s literacy development at

home as this is another critical aspect of reading skill improvement.

With more and more students beginning able to read independently program focus can shift toward

teaching and cultivating comprehension skills. Helping children grow from readers who can understand

the basic idea of a text to those who can think critically about what they’re reading is difficult but

ultimately the goal of reading, and if children can begin cultivating these skills at a young age they will be

better prepared to apply them in and outside of school as they get older.


3

Contents

Executive Summary ....................................................................................................................................................2

Introduction ................................................................................................................................................................4

2013 – 2014 BELAJAR Evaluation ..............................................................................................................................5

A. Context ...........................................................................................................................................................5

B. Methods .........................................................................................................................................................5

Results and Analysis....................................................................................................................................................8

C. Characteristics of Students .............................................................................................................................8

Home Literacy Environment (HLE) .........................................................................................................................9

Community Reading Activities................................................................................................................................9

D. Results of Reading Skills by Sub-Test ........................................................................................................... 10

E. Learning Equity and Struggling Students ..................................................................................................... 12

Literacy Boost Participation................................................................................................................................. 13

F. Low Performing Students ............................................................................................................................ 14

G. Conclusion ....................................................................................................................................................... 15

2012 – 2014 ANCP Evaluation ................................................................................................................................ 16

A. Context ........................................................................................................................................................ 16

B. Methods ...................................................................................................................................................... 16

Results and Analysis................................................................................................................................................. 20

C. Characteristics of Students .............................................................................................................................. 20

Home Literacy Environment (HLE) ...................................................................................................................... 20

Community Reading Activities............................................................................................................................. 21

D. Results of Reading Skills by Sub-Test .............................................................................................................. 22

E. Learning Equity and Struggling Students ......................................................................................................... 24

Literacy Boost Participation................................................................................................................................. 24

E. Conclusion ....................................................................................................................................................... 25

Appendix A. Background information ..................................................................................................................... 26

Appendix B. Regression Results .............................................................................................................................. 30

Appendix C. Inter-rater reliability ........................................................................................................................... 34


4

Introduction

In 2012, Save the Children began its Literacy Boost program in 20 schools in Belu district on West

Timor island, implementing both teacher-focused and community interventions. Literacy Boost teacher

trainings began several months after the baseline assessment in July, and included training sessions

featuring how to teach the five key reading skills (letter knowledge, phonemic awareness, fluency,

vocabulary and comprehension) as well as how to follow learners' progress in developing these skills

using formative assessments. Broad -based community mobilization including establishing reading

awareness, as well as Reading Camps and Book Banks in all villages surrounding the schools were

rolled out over the school year before the next data collection in July 2013.

The key research questions to be explored in this report include:

1. What can students’ literacy score gains from 2013-2014 tell us about learners’ emergent

reading skills? What does this mean for BELAJAR programming?

2. How do learners’ reading skills vary by student background, school environment, and home

literacy environment? What does this mean for targeting BELAJAR’s two strands of

intervention? 3. Who are the struggling students in 2014, and are they similar or different to those in 2013?

The 2013, students in Literacy Boost schools made significantly larger gains in concepts about print and

listening comprehension than students in comparison schools, controlling for baseline differences.

However, there was not a statistically significant impact of the program in letter identification,

individual word reading or reading accuracy. In addition, students who did not speak Indonesian as

their primary language tended to perform significantly worse than their peers who have grown up

speaking Indonesian, overall. However, the gap between Indonesian speakers and those that speak a

mother tongue at home was smaller in Literacy Boost schools. 2013 monitoring data also revealed that

students in schools that participated in more program activities scored higher than their peers in less

involved schools. Teacher training and community intervention sessions continued into the 2014

school year and this report includes a follow-up endline report with schools that have experienced the

Literacy Boost program for two years. Analyses will focus on the skills of children in 2014 and whether

differences in home language seen in 2013 persist in 2014.

Finally, the results of the 2012-2013 pilot Literacy Boost program in Belu demonstrated improvement

in the literacy skills and behavior of children. Consequently, Save the Children expanded the scope of

interventions in the Better Literacy for Academic Result (BELAJAR) program. In July 2013, a baseline

data collection was undertaken to ensure that programmatic needs were accounted for within the new

schools in the project. The 20 baseline intervention schools represent all schools where the Literacy

Boost program is being undertaken as part of BELAJAR. This report will begin by reporting on the

2013-2014 evaluation and finish with results from the 2014 follow-up study of schools that have

received Literacy Boost for two years.


5

2013 – 2014 BELAJAR Evaluation

A. Context

Belu District, along with most of NTT province, is one of the poorer areas of Indonesia. The province

had an average GDP/capita of just $527 compared to the average for Indonesia—$3,223 in 2011. As

the district is on the border of Indonesia and Timor-Leste many people in the area are refugees from

the conflict. In 2011, Belu district has total 140 primary schools across 12 sub-districts. This BELAJAR

evaluation involved 35 schools, 20 BELAJAR target schools and 15 control schools, across five sub-

districts. BELAJAR target schools are located in Lasiolat and Raihat sub-district, while the control

schools are located in Lamaknen and Lamaknen Selatan sub-district.

B. Methods

Sampling

The 2013 BELAJAR sample included 626 students (319 boys, 307 girls) who were enrolled in Grade 2

in 35 schools, 20 receiving the Literacy Boost program and 15 receiving no intervention. At each

school where data was collected, 20 children in Grade 2 were sampled. Student background

information was collected by asking children basic questions about their home and family. The 2014

sample includes 652 children (326 boys, 326 girls) from the same 35 schools sampled in 2013, with 20

grade 2 children sampled per school. The same schools were sampled in 2013 and 2014, and an

attempt was made to follow the same children from one year to the next. Therefore, this study will

report on average scores and learning gains for students present at both the baseline and endline

assessments.

Analyses

This analysis has two purposes. First, we will investigate the literacy skills of Literacy Boost learners

and the comparison learners in 2014. Special attention will be given to the skill growth between 2013

and 2014 for both groups. The second purpose is to assess what skills the students currently have, and

what areas and skills Literacy Boost should focus on.

To test the comparability of learners in the samples, this report will use comparison of means through

two-tailed t-tests, with clustered standard errors to account for the grouping of student-level data

within schools. Summary statistics will be used to analyze learners’ performance in each of the reading

sub-tests. Finally, this report will look to multilevel regression models to explore relationships between

literacy skills and student background characteristics and home literacy environment.

Comparing several different groupings of learners is possible with this data. Of particular interest are

children who do not speak Indonesian at home, girls, children with few socioeconomic resources and children with weak home literacy environments. Data in this report will be disaggregated by group only

when the aforementioned clustered t-tests indicate statistically significant differences between groups.

Instruments


6

The assessment tool used for the 2014 data collection is the same as was used for the baseline data

collection in 2013, with the exception of students’ participation in community reading activities like

Book Banks and Reading Camps. Table 1 offers an example of background indicators as well as a

detailed description of reading indicators. Information about inter-rater reliability for this assessment is

in Appendix C.

Table 1: Assessment Instruments

Student background Examples

General Sex, age, language spoken at home, work

School-related Distance to walk to school, repetition history

Socioeconomic status Type of home, household size, household amenities/possessions

Home Literacy Environment

Access to print Materials present in home, types of materials

Reading at home Presence and percentage of family members who children see read,

and who have someone read to them

Participation in community

reading activities

Participation in reading buddy, reading camp, book creation and book

bank activities.

Reading Outcome Description

Concepts about Print Familiarity with books (e.g. cover, direction of text); 11 items

Alphabet knowledge Number of letters/sounds known of 52

Fluency Number of words in a connected text read correctly in a minute

Accuracy Percentage of words in a connected text read correctly

Comprehension Ten comprehension questions answered correctly after reading text

aloud

In addition, the Literacy Boost program involves the follow main components:

1. a 9-module teacher training on explicit reading instruction;

2. provision of community-based Book Banks

3. establishing a Reading Buddy system in schools in which older children read to younger

children;

4. conducting regular community reading awareness sessions with parents

5. conducting weekly Reading Camps run by trained Reading Camp Leaders

6. conducting Story Time activities in the community

At the time of this midline assessment in May 2014, Table 2 details the activities that had been

implemented by Save the Children and the length of the intervention prior to the assessment.


7

Table 2: Literacy Boost Implementation Timeline: BELAJAR

Date of

intervention/start Intervention

Time to

assessment

(May 2014)

Year 1

21-23 August 2013 First Training of Trainers (ToT) session conducted on Highlight

curriculum 2013 (thematic learning), Active Learning and Positive

Discipline, Introduction to Reading Development and instruction for

Young Children (include print rich environment) and Letter

Knowledge, participated by 14 Master Trainers

9 months

3-18 September 2013 First Teachers Training (Cascade of ToT materials) in 20 target

schools, participated by 83 early grade teachers

8 months

11-12 September 2013 First Reading Camp Facilitator Training on 7 sessions of ‘phonemic

awareness’ activities (session 1-7) with children, participated by 73

facilitators from 40 reading camps in 20 target schools

8 months

September 2013 Book Bank provision to 40 camps and 20 schools 8 months

September-December

2013

Weekly Reading Camp Sessions in communities 5-8 months

October 2013 First In-class monitoring and mentoring by master trainers , covering

142 teachers (LB Y2 and BELAJAR Y1)

7 months

8-9 November 2013 First training for 57 Community Facilitators to run Reading Awareness

Workshop at school level with parents

6 months

November 2013-February

2014

Weekly Reading Awareness Workshop in communities 3-6 months

20 November 2013 First orientation for teachers responsible for reading buddy activity,

participated by 20 early grade teachers from 20 target schools

6 months

26-27 November 2013 Second Training of Trainers (ToT) session conducted on Phonemic

Awareness, Formative Assessment and Struggling learners, participated

by 14 Master Trainers

6 months

12 December 2013 Second Reading Camp Facilitator Training on 7 sessions of ‘letter

knowledge’ activities (session 8-14) with children, participated by 73


5 months

January-May 2014 Weekly Reading Camp Sessions in communities 0-4 months

29 January-8 February

2014

Second Teachers Training (Cascade of ToT materials) in 20 target

schools, participated by 128 early grade teachers (LB Y2 and BELAJAR

Y1)

3-4 months

13-14 February 2014 Third Reading Camp Facilitator Training on 7 sessions of ‘vocabulary’

activities (session 15-21) with children, participated by 78 facilitators

from 40 reading camps in 20 target schools

3 months

17-21 February 2014 Third Training of Trainers (ToT) session conducted on vocabulary,

reading fluency, reading comprehension and language issue, participated

by 14 Master Trainers

3 months

26-27 February 2014 Second training (refresher and evaluation) for Community Facilitators

to run Reading Awareness Workshop at school level with parents,

participated by 43 facilitators

3 months


8

Date of

intervention/start Intervention

Time to

assessment

(May 2014)

February-May 2014 Weekly Reading Awareness Workshop in communities 0-3 months

26-27 February 2014 Refresher and evaluation for teachers responsible for reading buddy

activity, participated by 29 early grade teachers from LB and BELAJAR

target schools

3 months

7-11 March 2014 Third Teachers Training (Cascade of ToT materials) in 20 target

schools, participated by 226 early grade teachers (LB Y2 and BELAJAR

Y1)

2 months

14-16 April 2014 School event (19 schools), participated by 380 children 1 month

16-17 May 2014 District Event (Reading event), participated by 180 children and

parents, facilitators and teachers

0 month

Results and Analysis

C. Characteristics of Students

Absent at Midline

The 2013 and 2014 assessments were conducted in the same schools, with the goal of assessing the

same children as baseline and midline in order to measure student learning. Overall, 81 percent of

students were located for the midline assessment, 72 percent of Literacy Boost students and 90

percent of control children (p < .001). Controlling for intervention group, children who were absent at

the midline assessment tend to have weaker common word knowledge, and are less likely to be able to

read independently. In addition, students absent at midline are more likely to be older, and report

having storybooks at home. Gender was found to mediate the relationship between Literacy Boost and

absence at midline and will be used as a control in subsequent analyses.

Student characteristics

In the midline study in 2014, 2nd grade students were 8 years old on average. Thirty-one percent had

attended some sort of ECCD program, 37 percent had repeated grade 1 and 14 percent repeated

grade 2. Forty-three percent of children reported speaking Indonesian at home. Almost all children

help with chores at home (97 percent), but only 7 percent report working outside the home. The only

observed difference between intervention and control children is that more children in Literacy Boost

schools tend to come from families who have electricity and televisions at home. More differences in

home possessions were observed in 2013, but as expected group averages remain similar from 2013 to

2014 (see Table 3). For more details about 2014 students’ background characteristics, see Appendix A.


9

Table 3. Items owned by family in 2013 and 2014, by group

2013 2014

Control

(N=235)

Literacy Boost

(N=241)

Significant

Difference

Control

(N=235)

Literacy Boost

(N=241)

Significant

Difference

Electricity 38% 65% ** 47% 73% **

Latrine 60% 73%

60% 72% *

TV 13% 37% *** 19% 40% ***

Refrigerator 3% 9% * 3% 6%

Motorcycle 21% 33% ** 23% 32%

Computer 0% 4% * 1% 2% *p < .05, **p < .01, ***p < .001

Home Literacy Environment (HLE)

Table 4 displays 2013 and 2014 averages for key measures of home literacy environment. In both 2013

and 2014 the majority of students came from households where parents were actively engaged in

literacy activities at home. In 2014, significantly more Literacy Boost students reported having more

textbooks and storybooks at home, compared to students in comparison schools. The overall number

of types of materials at home is also higher for Literacy Boost students compared to control students

in 2014.

Table 4. 2013 and 2014 average scores for students’ home literacy environments, by

group

2013 2014

Control

(N=235)

Literacy

Boost

(N=241)

Significant

Difference

Control

(N=235)

Literacy

Boost

(N=241)

Significant

Difference

Textbook 62% 69% 73% 87% **

Religious book 74% 75% 79% 82%

Magazine 3% 12% *** 6% 7%

Newspaper 13% 24% ** 22% 30%

Storybook 48% 47% 47% 66% ***

# Types of reading materials 2.0 2.3 * 2.3 2.7 **

Family member seen reading 86% 91% 86% 87%

Family member helps child study 93% 96% 92% 95%

Family member reads to child 65% 79% ** 68% 72%

Family member tells stories to

child 63% 72% 70% 71% *p < .05, **p < .01, ***p < .001

Community Reading Activities


10

As seen in Table 5, children in Literacy Boost schools report attending two community activities on

average. Children are generally attending reading camps and meeting with their reading buddies once

per week. Only 55 percent of students report using the book bank and 44 percent have created their

own reading material at a reading camp. Access to child-friendly literacy materials is a key driver of

reading skill development so program staff should consider methods for increasing students’

attendance at these activities. In addition, engaging children with book banks and creating reading

materials increases exposure to print, a key driver of literacy development.

Table 5. Frequency of attendance at Literacy Boost activities in 2014

Literacy Boost Activity

Average

(N=360)

Used book bank 52%

Named favorite story 40%

Has a reading buddy 52%

Name reading buddy 52%

Met with reading buddy last week 48%

Frequency of meeting with reading buddy (days per week) 1.2

Attended reading camp 55%

Frequency of attendance at reading camp (days per week) .98

Created book at reading camp 44%

Frequency of book creation (0=Never, 1=Rarely, 2=Sometimes, 3=Often) 0.6

Participation total (Out of 4) 2.0

In addition, the Indonesian team added questions that asked children how they think they would be

helped to read better. The most popular answer by far was having more time to practice, followed by

more interesting things to read and more help or assistance when reading. The Literacy Boost team

can take these student preferences into consideration as they plan for the next school year.

Table 6. Student reported strategies for learning to read better, Literacy Boost 2014

Activity

Average

(N=360)

More time to practice 61%

More books to read 18%

More interesting books 3%

Teachers are more exciting 7%

Help/assistance when reading 16%

D. Results of Reading Skills by Sub-Test


11

In this section we examine average scores for 2nd grade students in each literacy skill area in 2013 and

gains made in 2014. Figure 1 displays that on average the literacy skills of 2nd grade students in both

intervention and control schools have improved from 2013 to 2014. It is not clear why control

students have gained skills on the same magnitude as Literacy Boost students, but there are no

significant literacy skill gain differences between the two groups.

Figure 1. Average Gain Score for Grade 2 Students in 2013 and 2014

Enabling students to read with comprehension is the one of the primary goals of Literacy Boost. With

the expanded comprehension questions used in 2013 and 2014 it is easier to analyze which level of

comprehension students are achieving. Readers with comprehension are those students who can read

a grade-level text independently and answer at least 80 percent of comprehension questions correctly.

In 2014, 18 percent of grade 2 Literacy Boost students achieved this highest level of comprehension,

compared to 4 percent in 2013. Given the mastery that students are showing across the more basic

skills, growth in this area of reading comprehension would be an appropriate goal for future

assessments.

48%

65%

33%

8%

26%

13%

28%

58%

69%

38%

7%

27%

14%

32%

28%

20%

38%

15%

44%

20%

34%

24%

20%

40%

17%

51%

27%

42%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

CAP Letter ID Common words Fluency Accuracy Reading comprehension Reader

Control 2013 LB 2013 Control Gain LB Gain


12

Figure 2. Reading comprehension tiers, 2013 and 2014 Literacy Boost students

Note: Emergent Readers correctly answers less than 50 percent of the comprehension questions; Beginning readers

correctly answers between 50 and 80 percent of the questions correctly; Readers with Comprehension correctly answers

more that 80 percent of the comprehension questions. Tier cut-offs were set in combination with associated fluency and

accuracy levels in each language and country, vary based on the number of comprehension questions asked.

Finally, Literacy Boost students are reaching mastery of basic literacy skills and met four of the six

benchmarks set in 2013. Children in Literacy Boost schools met benchmarks set for CAP, common

word knowledge, fluency and accuracy, and came very close to meeting the 90% benchmark for letter

identification. The only area where students did not come close to meeting the baseline established

benchmark is in reading comprehension, and therefore this should be an area of focus for the next

school year. Only students who were present at baseline and endline were included in this comparison.

Table 7. Comparison of 2014 average scores with 2013 benchmarks

2013 Benchmark

(N=247)

LB Average 2014

(N=247) Meet Benchmark?

CAP 75% 81% Yes

Letter ID 90% 89% No

Common words 70% 78% Yes

Fluency 14 25 Yes

Accuracy 66% 79% Yes

Reading comprehension 70% 41% No

E. Learning Equity and Struggling Students

87%

85%

65%

57%

10%

10%

21%

26%

3%

4%

14%

18%

0% 20% 40% 60% 80% 100%

Control 2013

LB 2013

Control 2014

LB 2014

Emergent Beginner Reading with Comprehension


13

Learning Equity

This section will review the results of multivariate analyses predicting children’s reading skill in 2014,

with emphasis on equity. Overall, there are few significant differences between student’s reading skills

by gender, socioeconomic status (estimated by household items), or HLE in the overall sample or

within Literacy Boost schools. One difference observed is that across both groups and within

Literacy Boost schools only, girls are more likely to independent readers than boys. Also,

within Literacy Boost schools, children who reported doing more chores tended to have

stronger CAP and accuracy skills than children doing fewer chores at home.

We do continue to observe that similar to 2013 overall across both groups, children who do not speak

Indonesian at home having significantly weaker skills compared to children who do speak Indonesian at

home, specifically in the areas of CAP and reading comprehension. However, in Literacy Boost

schools alone there are no significant differences between the literacy skills of children

who are native Indonesian speakers and those who are not. This is an important

improvement given the overall tendency of second language learners to have weaker

literacy skills than native speakers of Indonesian.

Literacy Boost Participation

The major different between Literacy Boost students tends to be the frequency with which they attend

out of school literacy activities. Across CAP, fluency, accuracy and reading comprehension, children

who participate in more activities have stronger early reading skills (fluency not shown in Figure 3).

While this may not be surprising it is an important reminder that schools, families and Save the

Children staff should continue to find ways to offer these opportunities to children and encourage

their participation.

Figure 3. Relationship between Literacy Boost activity participation and literacy skills

77% 79% 82% 84% 86%

68% 72%

76% 81%

85%

33% 37%

41% 45%

48%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 1 2 3 4

% C

orr

ect

# LB activities attended

CAP

Accuracy

Reading comprehension


14

F. Low Performing Students

Part of the 2013 BELAJAR program included an effort to focus on the lowest performing students. The

bottom 20% of students, which we will refer to as “low beginners”, could not read a single word from

the list of 20 most common words from a 2nd grade textbook at baseline in 2013. They also knew less

than 30 of 52 upper and lowercase letters, and 3/4 of the low beginners knew less than half the

concepts about print questions.

In order to understand how to best support this group of students it is important to understand how

these students’ background and behavior differ from their peers and likely influence their poor

performance. We also need to monitor which students continue struggling after a year of Literacy

Boost programming, and this will be the first point investigate here.

Overall in 2014, being a struggling student was defined by scoring at least one standard deviation below

the mean scores of letter and common word knowledge. Using this definition, 18 percent of students

were defined as struggling, 23 percent of control students and 12 percent of Literacy Boost students.

Looking back to 2013, of the 101 students defined as ‘low beginners’ in 2013, 32 were still struggling in

2014. Within this small group of students analyses find that significantly more control students who

were low beginners in 2013 continued to struggle in 2014 compared to Literacy Boost students.

Figure 4. Struggling students in 2013 and 2014, by group

Within Literacy Boost schools, there is considerable variation in the proportion of students struggling

in each school, similar to 2013. Figure 5 displays the proportion if students scoring at least one

standard deviation below the mean letter and word knowledge scores in Literacy Boost schools.

Finally, the only variable found to predict being a struggling student was participation in

Literacy Boost activities. Students who participated in more literacy activities outside of

school were less likely to be struggling in 2014.

22% 23%

9%

17%

12%

2%

0%

10%

20%

30%

40%

50%

Low Beginner 2013 Struggling student 2014 Struggling in 2013 & 2014

Control Literacy Boost


15

Figure 5. Proportion of struggling students in Literacy Boost schools, 2014

G. Conclusion

In conclusion, Literacy Boost students in 2014 show stronger literacy skills than children in 2013 but

there are no significant differences between the skills of Literacy Boost and comparison students.

However, significantly fewer Literacy Boost students who were struggling at baseline continued to

struggle more than their peers in 2014, compared to children in control schools. This indicates that

there are fewer students in Literacy Boost schools who are not gaining the basic skills they need to

become strong independent readers.

On average, students show mastery of basic skills like letter identification and common word reading,

and advances in the more advanced skills of fluency, accuracy and reading comprehension. Children in

Literacy Boost schools met 4 of the 6 skill benchmarks set after the baseline assessment in 2013. One

important and difficult benchmark that has not yet been achieved is for reading comprehension. As

children are showing mastery of basic reading skills, more emphasis can be put on helping children find

deeper understandings of the text they read.

The previous trend of children speaking languages other than Indonesian at home having weaker

literacy skills than native speakers is less prevalent in 2014 Literacy Boost schools than in control

schools. This marks a significant improvement in equity within Literacy Boost schools. Few significant

skill differences by gender, poverty are found in 2014 Literacy Boost students, but care should still be

taken to encourage learning for all students. In addition, continuing to support students with weaker

home literacy environments will be important for making the most impact with the Literacy Boost

program. Analyses indicate that in the amount of participation in Literacy Boost activities outside of

school is directly related to stronger reading skills so communities and Save the Children staff who

support them should continue to work to consistently offer these opportunities to children.

0%

26%

30% 32%

0%

13%

5%

47%

5%

10%

20%

11% 10%

0% 0%

6%

0%

17%

25%

6%

0%

5%

10%

15%

20%

25%

30%

35%

40%

45%

50%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

School ID

% Struggling in 2014


16

2012 – 2014 ANCP Evaluation

A. Context

Belu District, along with most of NTT province, is one of the poorer areas of Indonesia. The province

had an average GDP/capita of just $527 compared to the average for Indonesia—$3,223 in 2011. As

the district is on the border of Indonesia and Timor-Leste many people in the area are refugees from

the conflict. In 2011, Belu district has total 140 primary schools across 12 sub-districts. This ANCP

evaluation involved 20 ANCP-LB target schools across seven sub-districts. These 20 schools are

located in Kakulukmesak, Tasifeto Barat, Atambua Kota, Atambua Selatan, Raimanuk, Lasiolat, and

Tasifeto Timur sub-district.

B. Methods

Sampling

The 2012 sample included 395 students (199 boys, 196 girls) who were enrolled in Grade 2 in 20

Literacy Boost schools. The 2013 sample included 334 students (158 boys, 176 girls) who were

enrolled in Grade 2 in 20 Literacy Boost schools. The 2014 sample includes 389 children (199 boys,

190 girls) from 20 Literacy Boost schools. At each school where data was collected, 20 children in

Grade 2 were sampled. The same schools were sampled in 2012, 2013 and 2014, but the same children

were not followed from one year to the next. Therefore, this study will report on average scores in

2012, 2013 and 2014 but cannot speak to learning gains made by a specific group of students.

Analyses

This analysis has two purposes. First, we will investigate the literacy skills of Literacy Boost learners

and in 2012, 2013, and 2014. The second purpose is to assess what skills the students currently have,

and what areas and skills Literacy Boost should focus on.

To test the comparability of learners in the samples, this report will use comparison of means through

two-tailed t-tests, with clustered standard errors to account for the grouping of student-level data

within schools. Summary statistics will be used to analyze learners’ performance in each of the reading

sub-tests. Finally, this report will look to multilevel regression models to explore relationships between

literacy skills and student background characteristics and home literacy environment.

Comparing several different groupings of learners is possible with this data. Of particular interest are

children who do not speak Indonesian at home, girls, children with few socioeconomic resources and

children with weak home literacy environments. Data in this report will be disaggregated by group only

when the aforementioned clustered t-tests indicate statistically significant differences between groups.

Instruments

The assessment tool used for this endline data collection is slightly different than the tool used for the

baseline data collection in 2012 and 2013. The main differences being that 2014 assessment included a


17

more difficult reading passage and additional advanced reading comprehension questions. Table87

offers examples of background and home literacy indicators and offers a detailed description of reading

indicators. Information about inter-rater reliability for this assessment is in Appendix C.

Table 8: Assessment Instruments

2012-2013 2014

Student background

General Sex, age, language spoken at home, work Sex, age, language spoken at home, work

School-related Distance to walk to school, repetition

history

Distance to walk to school, repetition

history

Socioeconomic status

Type of home, household size, household amenities/possessions

Type of home, household size, household amenities/possessions

Home and Community Literacy Environment

Access to print Materials present in home, types of

materials

Materials present in home, types of

materials

Reading at home

Presence and percentage of family

members who children see read, and

who have someone read to them

Presence and percentage of family

members who children see read, and

who have someone read to them

Participation in

community

reading activities

Participation in reading buddy,

community reading and book bank

activities.

Participation in reading buddy,

community reading and book bank

activities.

Reading Outcome

Concepts about

Print

Familiarity with books (e.g. cover,

direction of text); 11 items

Familiarity with books (e.g. cover,

direction of text); 11 items

Alphabet

knowledge Number of letters/sounds known of 52 Number of letters/sounds known of 52

Fluency Number of words in a connected text

read correctly in a minute

Number of words in a connected text

read correctly in a minute

Accuracy Percentage of words in a connected text

read correctly

Percentage of words in a connected text

read correctly

Comprehension Ten comprehension questions answered correctly after reading text aloud

Five comprehension questions answered correctly after reading text aloud

The Literacy Boost program involves the follow main components:

7. a 9-module teacher training on explicit reading instruction;

8. provision of community-based Book Banks 9. establishing a Reading Buddy system in schools in which older children read to younger

children;

10. conducting regular community reading awareness sessions with parents

11. conducting weekly Reading Camps run by trained Reading Camp Leaders

12. conducting Story Time activities in the community

At the time of this endline assessment in May 2014, Table 9 details the activities that had been

implemented by Save the Children and the length of the intervention prior to the assessment.


18

Table 9: Literacy Boost Implementation Timeline

Date of intervention/start Intervention

Time to

assessment

(May 2014)

Year 1

09-10 October 2012 First Training of Trainers (ToT) session conducted on Introduction to

Reading Development and instruction for Young Children (include

print rich environment), Letter Knowledge and Phonemic Awareness,

participated by 14 Master Trainers

1 year 7 months

17-24 October 2012 First Teachers Training (Cascade of ToT materials) in 20 target


1 year 7 months

18-19 October 2012 First Reading Camp Facilitator Training on 7 sessions of ‘phonemic

awareness’ activities (session 1-7) with children, participated by 64


1 year 7 months

October-December 2012 Weekly Reading Camp Sessions in communities I year 5 months – I

year 7 months

07-08 November 2012 First training for 34 Community Facilitators to run Reading Awareness


1 year 6 months

November 2012 Read a thon period 1 year 6 months

December 2012-January

2013

School event (5 schools), participated by 57 children 1 year 4 months –

1 year 5 months

December 2012-January

2013

Weekly Reading Awareness Workshop in communities 1 year 4 months –

1 year 5 months

November 2012 First In-class monitoring and mentoring by master trainers and 51

teachers were observed and mentored

1 year 6 months

10-11 January 2013 Second Reading Camp Facilitator Training on 7 sessions of ‘letter

knowledge’ activities (session 8-14) with children, participated by 57


1 year 4 months

January-March 2013 Weekly Reading Camp Sessions in communities 1 year 2 months –

1 year 4 months

January 2013 Book Bank Delivery to 37 reading camps and 20 schools 1 year 4 months

17-18 January 2013 Second Training of Trainers (ToT) session conducted on

Formative Assessment, Language Issue in Literacy Classroom

and Reading Fluency, participated by 14 Master Trainers

1 year 4 months

31 January 2013 First orientation for teachers responsible for reading buddy

activity, participated by 20 early grade teachers from 20 target

schools

1 year 4 months

6 February to 5 March

2013

Second Teachers Training (Cascade of ToT materials) in 20 target


1 year 2 months –

1 year 3 months

25 February 2013 Second training (refresher and evaluation) for Community Facilitators

to run Reading Awareness Workshop at school level with parents,

participated by 20 facilitators

1 year 3 months


19


Time to

assessment

(May 2014)

February 2013 Second In-class monitoring and mentoring by master trainers and 78

teachers were observed and mentored

1 year 3 months

March-April 2013 Weekly Reading Awareness Workshop in communities 1 year 1 month – 1

year 2 months

18-19 April 2013 Third Reading Camp Facilitator Training on 7 sessions of ‘vocabulary’

activities (session 15-21) with children, participated by 60 facilitators

from 37 reading camps in 20 target schools

I year 1 month

April-May 2013 Weekly Reading Camp Sessions in communities 1 year – 1 year 1

month

25-26 April 2013 Third Training of Trainers (ToT) session conducted on

Vocabulary, Reading comprehension and conclusion to sum up

the process of all session, participated by 12 Master Trainers

I year 1 month

May 2013 Third Teachers Training (Cascade of ToT materials) in 20 target


1 year

April-May 2013 School event (6 schools), participated by 299 children I year – 1 year 1

month

10-30 April, 16-17 May

2013

District Event (Child friendly classroom and Reading event),

participated by 328 children and 94 teachers and ECE tutors

1 year – 1 year 1

month

3 May 2014 Refresher and evaluation for teachers responsible for reading

buddy activity, participated by 17 early grade teachers from 17

target schools

1 year

May 2013 Additional book bank provision to 37 reading camps and 20 schools 1 year

June 2013 1 School event was conducted in 1 remaining target school,

participated by 54 children

11 months

Year 2

21-22 Agustus 2013 Fourth Training of Trainers (ToT) session conducted on Highlight

curriculum 2013 (thematic learning), Active Learning and Positive

Discipline, and refresher session on 5 literacy skills, participated by 13

Master Trainers

9 months

3-18 September 2013 Fourth Teachers Training (Cascade of ToT materials) in 20 target


8 months

13 September 2013 Fourth Reading Camp Facilitator Training (Refresher) on 21 sessions of

‘phonemic awareness, letter knowledge and vocabulary’ activities,

participated by 50 facilitators from 37 reading camps in 20 target

schools

8 months

September-December

2013

Weekly Reading Camp Sessions in communities 5 – 8 months

October 2013 Third In-class monitoring and mentoring by master trainers 7 months

9 November 2013 Fourth training for Community Facilitators to run Reading Awareness 6 months


20


Time to

assessment

(May 2014)


Weekly Reading Awareness Workshop in communities 6 months

21 November 2013 Third meeting (refresher and evaluation) for teachers

responsible for reading buddy activity, participated by 14 early

grade teachers from 14 target schools

6 months

27 November 2013 Fifth Training of Trainers (ToT) session conducted on Phonemic

Awareness, Formative Assessment (refresher) and Struggling learners,

participated by 9 Master Trainers

6 months

29 January-8 February

2014

Fifth Teachers Training (Cascade of ToT materials) in 20 target schools 3 – 4 months

17-21 February 2014 Sixth Training of Trainers (ToT) session conducted on all previous

sessions (refresher) and additional skills for vocabulary, reading fluency,

reading comprehension and language issue, participated by 25 Master

Trainers (LB and BELAJAR)

3 months

28 February 2014 Fifth training (refresher and evaluation) for Community Facilitators to

run Reading Awareness Workshop at school level with parents

3 months

February-May 2014 Weekly Reading Awareness Workshop in communities 0 – 3 months

27 February 2014 Refresher and evaluation for teachers responsible for reading

buddy activity at target communities

3 months

7-11 March 2014 Sixth Teachers Training (Cascade of ToT materials) in 20 target

schools (LB Y2 and BELAJAR Y1)

2 months

Results and Analysis

C. Characteristics of Students

Student characteristics

In the follow-up study in 2014, 2nd grade students were 8 years old on average. Seventy-two percent

had attended some sort of ECCD program, 21 percent had repeated grade 1, and 6 percent repeated

grade 2. Seventy-one percent of children reported speaking Indonesian at home. On average, most

children live in homes that have electricity and latrines (90 percent), 70 percent have a television, and

15 percent have computers. Almost all children help with chores at home (96 percent), but only 7

percent report working outside the home. For more details about students’ background characteristics

at endline, see Appendix A.

Home Literacy Environment (HLE)


21

Table 10 displays 2012, 2013 and 2014 averages for key measures of home literacy environment. Since

2012 the majority of students have been living in households where parents were actively engaged in

literacy activities at home. The proportion of children from homes where family members are engaging

them in literacy activities, and the proportion of homes with child-friendly reading materials remain

high in 2013 and 2014 and tend to be highest in 2013. Twenty additional schools were added to the

program in 2014 so it could be the case that due to increasing program delivery from 20 schools to 40

schools reduced the intensive focus that the 2013 schools received. The variation could also be

random, but program staff may want to engage a group of parents in a focus group discussion or some

other forum about how and when they engage in literacy activities with their children. Children may be

receiving more exposure to reading outside of school through Literacy Boost activities but parents and

caretakers are still key supporters of their children’s literacy development.

Table 10. Average 2012, 2013 and 2014 scores for students’ home literacy environments

2012 2013 2014

See someone at home reading 83% 89% 92%

Someone at home helps child study 95% 98% 96%

Someone at home reads to child 76% 84% 78%

Someone at home tells stories to child 69% 83% 75%

# of types of reading materials 2.9 3.3 3.0

Storybooks at home 55% 69% 61%

Community Reading Activities

As seen in Table 11, overall children report participating in about 2 out of 4 Literacy Boost activities

on average in 2013 and 2014. However, 98 percent of children in 2013 could name their reading

buddy, compared to 60 percent in 2014, and 89 percent reported meeting with their buddy in the past

week, compared to 50 percent in 2014. Similarly, 68 percent of children reported creating their own

reading materials in 2013, compared to 33 percent in 2014. This may reflect a shift in the availability of

different activities or difficulties with continuing to offer Literacy Boost activities for children. For

example, twenty additional schools were added to the program in 2014 so it could be the case that

due to increasing program delivery from 20 schools to 40 schools reduced the intensive focus that the

2013 schools received. Program staff should investigate these variations, and meet with teachers,

school groups and parents to discuss methods for continuing to support and promote Literacy Boost

activities. If new materials are needed for book banks, teachers, parents and older students could get

together to write new short stories for students. In addition, new activities like a Read-a-thon could be

introduced to maintain students’ excitement around reading outside of school.

Table 11. Frequency of attendance at Literacy Boost activities, by year

2013 2014

Used book bank 57% 52%

Named favorite story 73% 42%

Has a reading buddy 66% 61%

Name reading buddy 98% 60%


22

Met with reading buddy last week 89% 50%

Frequency of meeting with reading buddy (days per week) 1.5 1.2

Attended reading camp 55% 46%

Frequency of attendance at reading camp (days per week) 0.9 0.9

Created book at reading camp 68% 33%

Frequency of book creation (1=Rarely, 2=Sometimes, 3=Often) 1.0 0.5

Participation total (Out of 4) 1.8 1.9

In addition, the Indonesian team added questions that asked children how they think they would be

helped to read better. The most popular answer by far was having more time to practice, followed by

more interesting things to read and more help or assistance when reading. The Literacy Boost team

can take these student preferences into consideration as they plan for the next school year.

Table 12. Student reported strategies for learning to read better, Literacy Boost 2014

Activity Average

More time to practice 63.2%

More books to read 17.7%

More interesting books 2.8%

Teachers are more exciting 9.3%

Help/assistance when reading 17.7%

Other 1.3%

D. Results of Reading Skills by Sub-Test

In this section we examine average scores for 2nd grade students in each literacy skill area in 2012,

2013 and 2014. Figure 4 provide summaries of average scores on each subtest for all students in 2012,

2013 and 2014. On average, students show mastery of basic skills and strong scores for more advanced

skills as well. Reductions in reading comprehension from 2013 to 2014 should be interpreted with

caution as the number of reading comprehension questions was expanded from 5 to 10 and included

more evaluative questions in 2014.


23

Figure 6. Average Literacy Scores for Grade 2 Literacy Boost students in 2012, 2013 and

2014

Enabling students to read with comprehension is the one of the primary goals of Literacy Boost. With

the expanded comprehension questions used in 2014 it is easier to analyze which level of

comprehension students are achieving. Readers with comprehension are those students who can read

a grade-level text independently and answer at least 80 percent of comprehension questions correctly.

In 2014, 28 percent of grade 2 students fell into this highest level of comprehension. Given the mastery

that students are showing across the more basic skills, growth in this area of reading comprehension

would be an appropriate goal for future assessments.

Figure 5 displays changes in the proportion of students reading with comprehension from 2012 to

2014. However, this figure should be interpreted with caution as the reading passage as well as the

amount and difficulty of the comprehension questions changed between 2013 and 2014.


24

Figure 7. Reading comprehension tiers, 2014 Literacy Boost students

Note: Emergent Readers correctly answers less than 50 percent of the comprehension questions; Beginning readers

correctly answers between 50 and 80 percent of the questions correctly; Readers with Comprehension correctly answers

more that 80 percent of the comprehension questions. Tier cut-offs were set in combination with associated fluency and

accuracy levels in each language and country, vary based on the number of comprehension questions asked.

E. Learning Equity and Struggling Students

This section will use data from 2014 to focus on learning differences between students experiencing

different types of disadvantages. Specifically we will focus on girls, children from socioeconomically

disadvantaged homes, and those with the fewest home literacy resources.

Learning Equity

Overall, few equity related differences are found between students’ skill profiles in 2014. Previously,

children who did not speak Indonesian at home were found to have weaker advanced literacy skills,

and those differences continue to be observed for CAP and reading comprehension in this sample. The

gender differences observed are that girls’ having stronger common word knowledge and fluency skills

than boys. Additionally, children who attended an early childhood program tend to have more

advanced CAP skills.

Literacy Boost Participation

Although analyses did not find strong relationships between SES or HLE and reading skills, there is a

significant relationship between participation in Literacy Boost activities and all skills. That is, the more

Literacy Boost activities children report engaging in, the stronger their literacy skills. Figure 5 displays

this relationship for the higher order skills of fluency, accuracy and reading comprehension. These

results are particularly salient given the reduction in participation of some activities between 2013 and


25

2014. In order for Literacy Boost to continue to help children improve their early reading skills,

students need to participate in literacy activities outside of school.

Figure 8. Relationship between Participation in Literacy Boost Activities and Reading

Skills

E. Conclusion

In conclusion, grade 2 Literacy Boost students in 2013 and 2014 show stronger literacy skills than

students at baseline in 2012. Analyses indicate that in more participation in Literacy Boost activities

outside of school is directly related to stronger reading skills. To make a long-lasting impact, project

implementation should be followed by project sustainability strategies. Therefore, communities and

Save the Children staff who support them should continue to work to consistently offer these

opportunities to children. In addition, parents should be encouraged and reminded to continue to

support their children’s literacy development at home. With more and more students beginning to

read independently focus can shift toward teaching and cultivating comprehension skills. Helping

children grow from readers who can understand the basic gist of a text to those who can think critically about what they’re reading is difficult but ultimately the goal of reading, and if children can

begin cultivating these skills at a young age they will be better prepared to apply them in and outside of

school as they get older. Finally, skill differences are still seen between children who do not speak

Indonesian at home and those who do so this should continue to be a focus for teachers and

community activity facilitators.


26

Appendix A. Background information

Table A1. Background, Socioeconomic, Health, Home Learning Environment and Literacy Characteristics for

students present and baseline and midline: BELAJAR 2014

Control (N=243) Literacy Boost (N=247) Average (N=490)

Sex 50% 54% 52%

Age 8.5 7.9 8.2

Attended ECCD 31% 29% 30%

Repeat Grade 1 40% 36% 38%

Repeat Grade 2 18% 11% 15%

Speaks Indonesian at home 35% 52% 43%

Electricity 46% 76% 61%

Latrine 61% 75% 68%

TV 19% 43% 31%

Refrigerator 3% 5% 4%

Motorcycle 23% 33% 28%

Computer 1% 4% 2%

Total Household items 2.2 3.1 2.6

Textbook 73% 87% 80%

Religious book 79% 82% 81%

Magazine 6% 7% 6%

Newspaper 22% 30% 26%

Storybook 47% 66% 57%

# types of reading material 2.3 2.7 2.5

# people in home 4.3 4.3 4.3

% adults in home 53% 53% 53%

% women in home 54% 56% 55%

# people seen reading 2.1 2.3 2.2

% people seen reading 49% 54% 52%

# people help with studying 2.4 2.6 2.5

% people help with studying 58% 63% 60%

# people reading to child 1.6 1.7 1.6

% people reading to child 36% 40% 38%

# people telling stories 1.6 1.7 1.6

% people telling stories 37% 40% 39%

# people playing games 3.2 3.0 3.1

% people playing games 73% 69% 71%

Anyone at home seen reading 86% 87% 87%

Anyone at home helping study 92% 95% 93%

Anyone at home reading to child 68% 73% 70%

Anyone at home telling stories 68% 72% 70%

Anyone at home playing games 95% 96% 95%


27

Control (N=243) Literacy Boost (N=247) Average (N=490)

Does chores at home 97% 98% 97%

Morning chores 0.7 0.8 0.7

Afternoon chores 1.2 1.1 1.2

Evening chores 0.7 0.7 0.7

Weekend chores 1.4 1.5 1.4

Does work outside the home 8% 7% 7%

Morning work 0.0 0.3 0.2

Afternoon work 1.0 0.9 0.9

Evening work 0.2 0.1 0.1

Weekend work 1.2 1.1 1.2

Morning studying 0.3 0.3 0.3

Afternoon studying 0.6 0.7 0.6

Evening studying 1.3 1.3 1.3

Weekend studying 1.0 1.0 1.0

Reading frequency (0=Never, 3=Often) 2.5 2.7 2.6

What do you do to learn to read?

More time to practice 59% 61% 60%

More books to read 10% 18% 14%

More interesting books 0% 3% 2%

Teachers are more exciting 6% 7% 7%

Help/assistance when reading 20% 16% 18%

Exchange books in community 19% 25% 22%

Why miss school?

School closed 1% 1% 1%

No teacher 5% 5% 5%

Sick 5% 9% 7%

Playing 10% 8% 9%

Ate breakfast 96% 96% 96%

Breakfast size (1= small, 2=normal) 1.9 1.9 1.9

Ate lunch 100% 100% 100%

Lunch size (1=small, 2=normal) 2.0 2.0 2.0

Use mosquito net 79% 86% 83% *p<.05, **p<.01, ***p<.001


28

Table A2. Background, Socioeconomic, Health, Home Learning Environment and Literacy Characteristics for

Literacy Boost Students: ANCP 2014

Literacy Boost

Sex 48.8%

Age 7.87

Attended ECCD 72.5%

Repeat Grade 1 20.8%

Repeat Grade 2 5.9%

Speaks Indonesian at home 71.5%

Electricity 89.7%

Latrine 90.2%

TV 70.4%

Refrigerator 33.2%

Motorcycle 61.7%

Computer 14.9%

Total Household items 3.6

# people in home 4.4

% adults in home 51.6%

% women in home 57.9%

# people seen reading 2.6

% people seen reading 59.1%

# people help with studying 2.9

% people help with studying 67.6%

# people reading to child 1.9

% people reading to child 43.4%

# people telling stories 2.0

% people telling stories 45.0%

# people playing games 3.0

% people playing games 67.0%

Does chores at home 96.1%

Morning chores 0.8

Afternoon chores 1.1

Evening chores 0.7

Weekend chores 1.4

Does work outside the home 0.1

Morning work 0.0

Afternoon work 0.1

Evening work 0.0

Weekend work 0.1

Morning studying 0.6

Afternoon studying 0.9


29

Literacy Boost

Evening studying 1.4

Weekend studying 1.2

Frequency of missing school (0=Never, 3=Often) 0.3

Why miss school?

School closed 2.6%

No teacher 5.7%

Sick 4.6%

Playing 6.7%

Other 0.5%

Reading frequency (0=Never, 3=Often) 2.8

What do you do to learn to read?

More time to practice 63.2%

More books to read 17.7%

More interesting books 2.8%

Teachers are more exciting 9.3%

Help/assistance when reading 17.7%

Other 1.3%

Exchange books in community 23.9%

Ate breakfast 88.7%

Breakfast size 1.7

Ate lunch 98.2%

Lunch size 2.0

Treat water at home 94.6%

Use mosquito net 84.3%


30

Appendix B. Regression Results

Table B1. Equity regression analysis for Literacy Boost students: BELAJAR 2014

1 2 3 4 5 6 7

VARIABLES CAP Letter ID

Common

words Accuracy Fluency

Reading

Comprehension Reader

Age 0.0103 0.00412 0.00427 -0.00193 0.968 0.00994 0.06

(0.00798) (0.00395) (0.00900) (0.0165) (0.802) (0.0125) (0.097)

Sex 0.00479 -0.00119 -0.0212 0.0128 -0.536 0.0455 0.55*

(0.0175) (0.0189) (0.0339) (0.0409) (2.358) (0.0330) (0.26)

Attended ECD 0.0266 0.00865 0.0185 -0.0374 -0.450 -0.00936 -0.09

(0.0219) (0.0159) (0.0267) (0.0415) (2.794) (0.0392) (0.34)

Speak Indonesian at home 0.0362 0.00715 0.0324 0.0482 2.319 0.0595 0.34

(0.0224) (0.0197) (0.0299) (0.0387) (2.735) (0.0396) (0.30)

SES total score 0.00827 -0.00455 0.00159 0.00106 0.465 0.0292~ 0.04

(0.00800) (0.00769) (0.0163) (0.0178) (0.973) (0.0153) (0.11)

# types of reading material at home 0.000790 0.0135 0.0235 0.0368~ 2.090 0.0282 0.28

(0.00921) (0.0100) (0.0167) (0.0208) (1.335) (0.0260) (0.17)

HLE activity quintiles -0.00121 0.000701 -0.00455 -0.0238~ -1.776* 0.0107 -0.13

(0.00712) (0.00611) (0.00942) (0.0124) (0.783) (0.0161) (0.10)

Total chore time 0.00686 0.00933 0.0182~ 0.0325~ 0.573 0.00290 0.10

(0.00416) (0.00561) (0.0104) (0.0164) (0.785) (0.0165) (0.09)

Total participation in LB activities 0.0231* 0.00736 0.0160 0.0436* 1.668* 0.0375* 0.23*

(0.00837) (0.00457) (0.0111) (0.0154) (0.771) (0.0146) (0.09)

Constant 0.615*** 0.777*** 0.576*** 0.513* 9.238 0.0234 -1.19

(0.0826) (0.0673) (0.119) (0.183) (7.138) (0.167) (1.07)

Observations 287 287 287 287 287 287 287

R-squared 0.091 0.043 0.042 0.067 0.043 0.096

*p<.05, **p<.01, ***p<.001


31

Table B2. Equity regression analysis for Literacy Boost students controlling for baseline scores: BELAJAR 2014

(1) (2) (3) (4) (5) (6) (7)


Common


Reading


Age -0.00295 0.00918 0.00172 0.508 -0.00116 0.0120 0.0315

(0.00921) (0.00702) (0.0106) (0.945) (0.0279) (0.0178) (0.114)

Sex 0.0151 0.00588 -0.00265 0.863 0.00606 0.0532 0.684~

(0.0198) (0.0240) (0.0417) (2.992) (0.0432) (0.0379) (0.352)

Attended ECD 0.0207 -0.00028 0.0385 -0.247 -0.0338 0.0217 0.0333

(0.0217) (0.0196) (0.0332) (3.190) (0.0600) (0.0647) (0.448)

Speak Indonesian at home 0.0374 0.00397 0.0296 4.464 0.0596 0.0738 0.498

(0.0308) (0.0264) (0.0480) (2.862) (0.0535) (0.0499) (0.430)

SES total score 0.00978 -0.0101 0.00841 0.741 -0.00827 0.0360~ 0.122

(0.0113) (0.0110) (0.0233) (1.385) (0.0276) (0.0207) (0.154)

# types of reading material at home -0.00370 0.0206 0.0272 1.432 0.0335 0.0306 0.294

(0.00997) (0.0146) (0.0244) (1.932) (0.0322) (0.0326) (0.217)

HLE activity quintiles -0.00226 0.00101 -0.00738 -2.727** -0.0244 0.00330 -0.251*

(0.00694) (0.00709) (0.0119) (0.800) (0.0176) (0.0176) (0.115)

Total chore time 0.00907* 0.0122 0.0217 0.756 0.0371 -0.00876 0.130

(0.00430) (0.00846) (0.0157) (0.818) (0.0220) (0.0194) (0.141)

Total participation in LB activities 0.0206* 0.00776* 0.00839 1.781* 0.0402* 0.0378* 0.233*

(0.00950) (0.00355) (0.00983) (0.772) (0.0171) (0.0153) (0.106)

CAP -0.0148

(0.0499)

Letter ID

0.0420

(0.0353)

MUW

0.0351

(0.0624)

Fluency

0.289*

(0.131)

Accuracy

0.111

(0.0655)

Reading comprehension

0.111

(0.0738)

Reader

0.691

(0.430)


32

(1) (2) (3) (4) (5) (6) (7)


Common


Reading


Constant 0.736*** 0.682*** 0.553*** 11.99 0.508 0.0182 -1.261

(0.115) (0.0909) (0.126) (8.503) (0.302) (0.198) (1.288)

Observations 224 224 224 224 224 224 224

R-squared 0.091 0.083 0.046 0.083 0.082 0.123 0.100

Robust standard errors in parentheses

*** p<0.001, ** p<0.01, * p<0.05, ~ p<0.1


33

Table B2. Table B1. Equity regression analysis for Literacy Boost students: ANCP 2014

1 2 3 4 5 6 7


Common


Reading

comprehension Reader

Age 0.00500 -0.00167 -0.0204 -0.0197 -0.700 0.0132 -0.103

(0.0109) (0.00539) (0.0146) (0.0165) (1.145) (0.0163) (0.120)

Sex (Female=1) -0.0136 0.0271~ 0.0816** 0.0481 7.855* -0.0151 0.134

(0.0121) (0.0139) (0.0238) (0.0406) (3.137) (0.0331) (0.379)

Attended ECD 0.0483* 0.00979 0.0330 0.0125 2.824 0.0586 -0.365

(0.0199) (0.0108) (0.0279) (0.0612) (3.790) (0.0431) (0.505)

Speaks Indonesian at home 0.0370* 0.0227~ 0.0238 0.0482 5.799 0.0824* 0.483

(0.0173) (0.0114) (0.0235) (0.0365) (3.961) (0.0369) (0.352)

# types of reading material at home 0.0113 -0.0005 -0.00213 0.00216 1.470 0.00403 -0.102

(0.00859) (0.00577) (0.0170) (0.0201) (1.476) (0.0162) (0.163)

Quintiles of HLE activities 0.00666 -0.00039 0.00591 0.00266 -0.174 0.00860 0.006

(0.00673) (0.00413) (0.00835) (0.0135) (0.957) (0.0125) (0.180)

Quintiles of SES Index 0.00480 -0.00136 -0.0108 -0.0171 -0.415 0.00289 -0.262

(0.00842) (0.00486) (0.0136) (0.0208) (1.502) (0.0193) (0.182)

Total chore time -0.00452 -0.00488 -0.0168~ -0.0169 -1.719 -0.0126 -0.272

(0.00325) (0.00291) (0.00819) (0.0123) (1.023) (0.00887) (0.121)

Total participation in LB activities 0.0186** 0.0124** 0.0275* 0.0416** 3.078** 0.0288* 0.290

(0.00574) (0.00350) (0.00996) (0.0117) (1.030) (0.0109) (0.151)

Constant 0.660*** 0.904*** 0.935*** 0.914*** 25.37* 0.332~ 4.491

(0.105) (0.0633) (0.117) (0.131) (9.893) (0.169) (0.982)

Observations 377 377 377 377 377 377 377

R-squared 0.111 0.049 0.070 0.034 0.081 0.056

Robust standard errors in parentheses

*** p<0.001, ** p<0.01, * p<0.05, ~ p<0.1


34

Appendix C. Inter-rater reliability

To test inter-rater reliability, 9 percent of learners (61 out of 670 for BELAJAR and 39 out of 428 for ANCP) were assessed by two

enumerators simultaneously. Long one-way ANOVA techniques were used to calculate the intra-class correlation within pairs of

assessors for a measure of reliability. Table C1 presents the results below. Using Fleiss’ benchmarks for excellent (ICC>0.75), good

or fair (0.75>=ICCA>0.4), and poor (0.4>=ICC); many of the literacy outcome variables exhibited excellent inter-rater reliability.

Table 1 shows the percent of agreement between the raters.

Table C1. Inter-rater Accuracy and Reliability for BELAJAR

Literacy Skills Sub-Test Raw totals Rating

CAP 0.95467 Excellent

Letters 0.8254 Excellent

MUW 0.70337 Good

WCPM 0.6359 Good

Accuracy 0.61552 Good

Reading comprehension 0.88223 Excellent

% Reader 0.87116 Excellent

Overall, there was good or excellent inter-rater reliability on all measures. Therefore we can be confident that scores were

recorded reliably throughout the assessment.


35

Table C2. Inter-rater Accuracy and Reliability for ANCP

Literacy Skills Sub-Test Inter-rater Reliability Rating

Concepts about Print 0.93787 Excellent

Letter Knowledge 0.99134 Excellent

Most Used Words 0.96062 Excellent

Fluency 0.93683 Excellent

Accuracy 0.49171 Poor

Reading Comprehension 0.89586 Excellent

Reader 0.38388 Poor

There was excellent inter-rater reliability on all measures except accuracy and being defined as a reader which had poor reliability

scores. In places where inter-rater reliability was excellent we can be confident in the reported average scores. We should interpret

accuracy and reader scores with more caution as there was quite a bit of variability between assessors on these measures.

literacy boost belajar indonesia midline & endline report...literacy boost belajar indonesia...

Documents