let us learn (lul) formative evaluation...lul let us learn m&e monitoring & evaluation mcp...

1

LET US LEARN (LUL)

FORMATIVE EVALUATION

UNICEF BANGLADESH COUNTRY OFFICE

EVALUATION REPORT

MARCH 2015

2

LET US LEARN (LUL)

FORMATIVE EVALUATION

UNICEF BANGLADESH COUNTRY OFFICE

EVALUATION REPORT

JANUARY 2015

i

Let Us Learn (LUL) Formative Evaluation

Final Report

© United Nations Children’s Fund, New York, 2012

United Nations Children’s Fund

Three United Nations Plaza

New York, New York 10017

January 2015

The purpose of publishing evaluation reports produced is to fulfil a corporate commitment to transparency

through the publication of all completed evaluations. The reports are designed to stimulate a free

exchange of ideas among those interested in the topic and to assure those supporting the work of

UNICEF that it rigorously examines its strategies, results, and overall effectiveness.

The contents of the report do not necessarily reflect the policies or views of UNICEF.

The text has not been edited to official publication standards and UNICEF accepts no responsibility for

error. The designations in this publication do not imply an opinion on the legal status of any country or

territory, or of its authorities, or the delimitation of frontiers. The copyright for this report is held by the

United Nations Children’s Fund. Permission is required to reprint/reproduce/photocopy or in any other

way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires

a written request to be submitted. For non-commercial uses, the permission will normally be granted free

of charge. Please write to the Bangladesh Country Office at the address below to initiate a permission

request.

The report was commissioned by UNICEF Bangladesh Country Office with the use of Let Us Learn

Funding.

Evaluation conducted and submitted by:

National Evaluator: Tristi Nichols, Ph.D. ([email protected])

Tristi Nichols

Manitou, Inc.

1008 Main Street

Suite #201

Peekskill, New York, 10566

For further information, please contact:

UNICEF Bangladesh Country Office

BSL Office Complex, 3rd Floor

(Dhaka Sheraton Hotel Annex)

1, Minto Road, Ramna, Dhaka, Bangladesh

[email protected]

UNICEF Staff and Support Contact Information:

Lisa Bender, Education in Emergencies ([email protected])

Elena Matsui ([email protected])

ii

PREFACE

iii

CONTENTS

PREFACE ..................................................................................................................................................... ii

TABLES AND FIGURES ............................................................................................................................... v

ACKNOWLEDGEMENTS ............................................................................................................................ vi

ACRONYMS ................................................................................................................................................ vii

EXECUTIVE SUMMARY ............................................................................................................................. ix

1 INTRODUCTION ................................................................................................................................. 12

1.1 The Let Us Learn (LUL) Initiative ................................................................................................ 12

1.2 Challenges of Pre-Primary Education Access in Bangladesh .................................................... 13

1.3 National Efforts to Mainstream Pre-Primary Education Access .................................................. 13

1.4 Summary of the Initiative and Implementing Partners ................................................................ 15

1.5 LUL-Funded Initiatives in Bangladesh ........................................................................................ 15

1.6 Evaluation Report Structure ........................................................................................................ 16

2 EVALUATION METHODOLOGY ........................................................................................................ 18

2.1 Evaluation Framework ................................................................................................................ 18

2.2 Evaluation Questions ................................................................................................................. 19

2.3 Sampling ..................................................................................................................................... 20

2.4 Data Collection ............................................................................................................................ 20

2.5 Data Analysis .............................................................................................................................. 21

2.6 Evaluation Governance Structure and Stakeholder Participation ............................................... 22

2.7 Evaluation Limitations ................................................................................................................. 22

3 BACKGROUND AND FINDINGS........................................................................................................ 24

3.1 Evidence Used to Design the LUL-Funded Learning Centres .......................................................... 24

3.1.1 Evidence Used to Design the LUL-Funded Learning Centres ............................................ 24

3.2 Implementation Status ................................................................................................................ 26

3.3 Key Stakeholder Contributions .................................................................................................... 26

3.4 Findings ....................................................................................................................................... 27

3.4.1 Relevance ........................................................................................................................... 27

3.4.2 Effectiveness ....................................................................................................................... 29

3.4.3 Sustainability and Scalability ............................................................................................... 34

4 Conclusions, Recommendations, and Lessons Learned .................................................................... 37

4.1 Conclusions ................................................................................................................................. 37

4.2 Recommendations ...................................................................................................................... 37

4.3 Lessons Learned ......................................................................................................................... 38

iv

REFERENCES ............................................................................................................................................ 39

ANNEXES ................................................................................................................................................... 42

Annex 1 Terms of Reference............................................................................................................. 42

Annex 2 Logical Framework .............................................................................................................. 49

Annex 3 Methodology: Sampling Criteria for Identifying PPE Centres ............................................. 51

Annex 4 Interview Instruments .......................................................................................................... 54

Annex 5 Classroom Observation Guide ............................................................................................ 65

Annex 6 Application of the Tanahashi Model .................................................................................... 67

Annex 7 Stakeholder List................................................................................................................... 69

v

TABLES AND FIGURES

Tables

Table 1: Past and present student profile data for LUL-funded learning centres ........................................ 4

Table 2: Scope and scale of LUL-funded initiatives ..................................................................................... 5

Table 3: Summary of evaluation criterion for LUL ....................................................................................... 7

Table 4: Evaluation questions of the LUL evaluation ................................................................................... 8

Table 5: Evaluation questions for the relevance evaluation criteria .......................................................... 25

Table 6: Evaluation question for the effectiveness evaluation criteria ....................................................... 27

Table 7: Evaluation questions for the sustainability / scalability evaluation criteria ................................... 32

Figures

Figure 1: Numbers of LUL-funded learning centres by year established .................................................... ix

Figure 2: Map of the People’s Republic of Bangladesh ............................................................................... 1

Figure 3: The operational standards .......................................................................................................... 12

Figure 4: Governance structure for the LUL Formative Evaluation ........................................................... 19

Figure 5: LUL-funded learning centres by Upaziilla ................................................................................... 22

Figure 6: Map of the Satkhira District and targeted sub-districts ............................................................... 23

Figure 7: Chart of the LUL learning centres established by year of operation .......................................... 24

Figure 8: Image capturing BRAC’s branding of LUL learning centres ....................................................... 28

Figure 9: List of hand written child indicators ............................................................................................. 31

vi

ACKNOWLEDGEMENTS

This evaluation could not have been completed without the assistance of many important people. The

team extends sincere thanks to the staff at UNICEF’s Evaluation Office who gave their valuable time to

contribute between August and December 2014: particularly Lisa Bender, Kathleen Letshabo, and Elena

Matsui.

The team also gratefully acknowledges the guidance and feedback of UNICEF Bangladesh Country

Office staff involved in the evaluation: Isa Achoba, Anjana Mangalagiri, Mohammad Mohsin, Mohammad

Kibria, Alexandra Illmer, Shantanu Gupta, and Urs Nagel.

Above all, the team is appreciative to all of the Bangladesh central and subnational government officials,

BRAC staff at all levels, community leaders, teachers, parents, and children for their time and efforts that

they put into expressing their views and giving the opportunity to learn about and observe their work and

experiences.

I am also grateful to colleagues who contributed their expertise, time, and effort to this UNICEF evaluation

project: Ed Gaible, Kerrin Ann Barrett, Joyce Onyango Odhiambo, and Paul Randrianirina.

vii

ACRONYMS

ABAL Ability Based Accelerated Learning

ADPEO Assistant District Primary Education Officer

ALP Accelerated Learning Programme

BBS Bangladesh Bureau of Statistics

BCO Bangladesh Country Office of UNICEF

BFO Bangladesh Field Office of UNICEF

BSA Bangladesh Shishu Academy

BRAC Building Resources Across Communities

CAMPE Campaign for Popular Education

CECCD Comprehensive Early Childhood Care and Development

CEDAW Convention on the Elimination of all Forms of Discrimination against Women

C4D Communication For Development

CO Country Office

CRC Convention on the Rights of the Child

CSN Children with Special Needs

DPE Directorate of Primary Education

DPEO District Primary Education Officer

ECD Early Childhood Development

ELCDP Early Learning for Child Development Project

ELDS Early Learning for Development Standard

ECCD Early Childhood Care and Development

EMIS Education Management Information System

FGDs Focus Group Discussions

GEROS Global Evaluation Reports Oversight System

GoB Government of Bangladesh

GPS Government Primary School

HQ Headquarters

HRBAP Human Rights-Based Approach to Programming

LUL Let Us Learn

M&E Monitoring & Evaluation

MCP Master Crafts Person

MoE Ministry of Education

MICS Multi-Indicator Cluster Survey

MoWCA Ministry of Women and Children Affairs

MoPME Ministry of Primary and Mass Education

NCTB National Curriculum & Textbook Board

NER Net Enrolment Ratios

NGO Non-Governmental Organization

OECD-DAC Organisation for Economic Cooperation and Development- Development

Assistance Committee

viii

OMJs One-minute Junior (video)

OOSC Out-of-school children/youth

PPE Pre-Primary Education

WHO World Health Organisation

UEO Upazilla Education Officer

UIS UNESCO Institute for Statistics

UNDAF United Nations Development Assistance Framework

UNEG United Nations Evaluation Group

UNESCO United Nations Education and Scientific, and Cultural Organisation

UNICEF United Nations Children Fund

USAID United States Agency for International Development

ix

EXECUTIVE SUMMARY

Background

Let Us Learn (LUL) is a unique private partnership initiative that allows for flexible and innovative

programming to address inequities in education access and related education outcomes. Coordinated

from the Education Section in UNICEF, New York, LUL has been piloting solutions to address challenges

and barriers to education in five diverse country contexts - Afghanistan, Bangladesh, Liberia, Madagascar

and Nepal – over a period of three years (2011-2014). The LUL initiative was intended to sharpen the

equity focus in both programming and monitoring of results in each participating country, and to benefit

excluded and/or marginalized children and youth. The programme is focused on three equity pillars:

reaching out-of-school children, expanding girls’ education and improving quality outcomes for learners,

with each of the participating Country Offices working to address inequity through targeted approaches,

adapted to address their specific contexts.

A series of country-led evaluations for innovations on equity were completed for Afghanistan,

Bangladesh, Madagascar and Nepal in 2014, while the Liberia evaluation was discontinued due to the

Ebola crisis. The evaluation of LUL in Bangladesh was undertaken from July – December 2014. The

primary focus of evaluation efforts in all countries was learning about UNICEF’s equity approach and

drawing lessons for future programming. Another key purpose of the LUL evaluations was to establish a

baseline to increase the evaluability of the LUL initiative.

The target of the evaluation was a programme component entitled Equity in education: piloting second

chance education for out-of-school-children in disaster prone Satkhira district. Though there are four

components, the evaluation only investigates the delivery of one year pre-primary education to children

below six years of age.

Evaluation Scope

This report examines the thematic areas of: (1) pre-primary education service delivery (in terms of

availability, accessibility, and utilization); (2) the coordination among service providers; and (3) relevance,

effectiveness, and sustainability (including scalability). Findings of the evaluation are tailored for use by

the government; UNICEF at country and headquarters level; and implementing partners.

Methodology

In line with an equity-based evaluation approach, the evaluation was designed to be maximally participatory which means that direct and indirect stakeholders had the opportunity to share opinions and experiences. The evaluation’s mixed methods design included quantitative and qualitative data, drawing from primary and secondary sources. In order to follow how the LUL funds were used to establish and maintain pre-primary education learning centres at the community level, it was imperative to understand the role of each partner’s expansive network at different levels and their collaboration and coordination. Half of the time assigned to the evaluation mission, undertaken in August 2014, was therefore dedicated to understanding this institutional context, while the other half was spent in selected communities. Currently, there are approximately 400 LUL-financed pre-primary education learning centres. Due to limited time, a strategy for a maximum variation sampling was applied. The sample of learning centres for this evaluation was based on: (1) specific mutually-agreed sampling criteria between the UNICEF Country Office and the Evaluation Team; and (2) a list of 18 different sampling criteria. Of the 18, the three priority criteria considered were: the year the school was established, the village being affected by

x

natural disaster, and the learning center having easy communication accessibility. The evaluation plan included one community visit per day. Six out of seven selected communities were ultimately visited.

Limitations

Time restrictions prevented the collection of data for a comparison, or a “counterfactual” to the LUL-funded learning centres. Baseline estimates are also missing, and it is assumed that pre-primary education access for children, prior to this initiative, was either limited or none. Although the reference group members proposed that the field mission include observations of government-run pre-primary schools, this suggestion was not honored. The focus of the evaluation effort had to remain on current implementation and could not be broadened to include non-LUL funded sites. Further rationale for this decision was that there was a need to sample as many learning centers as possible with the limited time available for field data collection. The verbal permission to interview, video tape, or photograph any child was sought from the parent / guardian but had limited success being recorded on a form. Finally, the evaluator was unable to confirm emergent themes through triangulation with additional relevant stakeholders due to time constraints. Other limitations are noted in this report. With the limited number of community visits (a total of six), the themes presented should not be generalised to the entire portfolio of 400 learning centres.

Key findings and conclusions

As indicated in the figure below, the establishment of learning centres began in earnest in 2012. The highest number of learning centres established (468) took place in 2013. Since then, however, UNICEF made a strategic decision to reduce direct service delivery. The number of children who have completed the one year pre-primary education at the learning centres since 2012 totals 18,159, and the number of students currently enrolled is 11,238. The data also shows that 86% of the learning centres are considered “temporary” structures. The teachers are recruited from the community with a provision of pre-service, on the job, and monthly refresher training. The provision of early childhood development services to the hardest to reach and using a rights-based approach is consistent with international and country level commitments. Evidence from multiple stakeholders confirms that the LUL appears to be relevant to the country’s efforts to provide equitable ECD services throughout the country. While there are significant environmental challenges, the initiative nonetheless exceeded its expected accomplishments. Though all of the minimum Operational Standards for Pre-Primary Education Delivery do not appear to be fully met, indicating some room for improvements, the classroom environment combined with instruction appear to offer educational opportunities where none may have existed before.1 While the evidence indicates that there is strong potential for sustainably delivering pre-primary education, there is still a need for temporary continuation of funded activities. The hardest to reach communities, which are remote, affected by cyclones, and have limited communications access, still require support until the governments’ efforts to mainstream pre-primary education occur.

1 Due to the recent introduction of the of the Operational Standards for the Delivery of the Pre-Primary Education and their

distribution through the package in 2014, adherence still appears to be at the initial stages.

Figure 1: Numbers of LUL-funded learning

centres by year established

2 0 1 2

2 0 1 3

2 0 1 4

2012 2013 2014

Series1 134 468 400

xi

Recommendations ( in order of priority)

Evidence-based recommendations are presented in order of priority, and they are designed to strengthen the equity focus while delivering ECD services.

i. Targeting future funding into remote areas ought to be prioritised, particularly where the government is least likely to complete its mainstreaming strategy to provide ECD services. The time period when the government’s Ministry of Primary and Mass Education will effectively mainstream efforts of pre-primary education services is not yet definite. Future funds should therefore be used to support the government and non-government partners to continue extending its outreach to the hard to reach communities. This should be for a temporary time period which is through the end of PEPD3, or through the 2016-2017 school year.

ii. Plans to address how best to institute an inclusion policy for children with disabilities should be considered at an early stage. The implementing partner, BRAC, should provide training to teachers in dealing with the special needs of children with disabilities. Providing these services in the classroom would effectively support the delivery of pre-primary education services with an equity focus.

iii. The government and non-government partners and UNICEF should also work together to support the reorganisation of existing accountability systems to ensure that all of the pre-primary education teachers receive the same number of monitoring and mentoring visits per month.

iv. To ensure that wind patterns and storm cycles, which create floods, cyclones, and other physical environmental threats endemic in Satkhira, do not compromise the storage of valuable information about children’s attendance and school and their developmental progress, strategies designed to guarantee data storage ought to be prioritized. The government and non-government partners and UNICEF should work together to create a web-based and or soft copy information system which includes both children’s administrative records (attendance) and achievements (results of child indicators). The system for documenting children’s school progress from one education level to the next also appears to have gaps and requires dedicated resources. These partners should also jointly work on an information system that maintains an individual record for each child’s progress and tracks progress through higher education levels.

12

1 INTRODUCTION

1.1 The Let Us Learn (LUL) Initiative

Pursuant to Article 29 of the Convention on the Rights of the Child (CRC), all children have the right to an education that prepares them to be responsible adults (see text box). However, challenges and barriers to education, particularly among excluded and marginalized children and youth, were identified in five diverse country contexts, including Afghanistan, the People's Republic of Bangladesh, Liberia, Madagascar, and Nepal. Through the Let Us Learn (LUL) initiative (formerly known as the Basic Education and Equity Initiative), each country is targeting the hardest to reach children by sharpening the equity focus in both programming and monitoring of results.

The USD 20 million initiative is a unique, private partnership that allows for flexible and innovative approaches to address inequities in education, access, and outcomes for the period of 2011-2014. The partners involved in launching LUL agreed that a formative evaluation would be carried out by December 2014. The logic was to capture information across a broad range of goals and contexts. The global evaluation effort aims to: (1) strengthen future LUL programming; (2) contribute to the global dialog on equity; and (3) ensure accountability of LUL-funded

initiatives. Given the significant challenges of implementation and measurement among the hardest to reach communities in the low-resource contexts addressed by LUL, the primary focus of evaluation efforts in all countries is learning about UNICEF’s equity approach, and to draw lessons for future programming. Another key purpose of the LUL evaluations is to establish a baseline to

increase the evaluability of the LUL initiative. In support of this aim, a report section is dedicated to assessing the data available to help establish “retroactive baseline” data. The evaluation was prepared for use by the relevant ministries in Bangladesh2; UNICEF at country and headquarters level, implementing partners, and other relevant stakeholders.

2 These include: the Ministry of Primary and Mass Education (MoPME) and the Ministry of Women and Children Affairs (MoWCA).

A child’s “education should foster the

development of the child's personality

and talents, preparation for a

responsible adult life…”

Article 29, Convention on the Rights of the Child

(CRC)

Figure 2: Map of the People's Republic of Bangladesh

13

Evaluation Scope

This report therefore examines the following areas: (1) pre-primary education service delivery (in terms of

availability, accessibility, and utilization); (2) the coordination among service providers; and (3) relevance,

effectiveness, and sustainability (including scalability).

1.2 Challenges of Pre-Primary Education Access in Bangladesh

Though significant progress has been made over the last two decades to improve children’s access to pre-primary education, participation rates are still too low”.3 According to the 2012-2013 Bangladesh Multi-Indicator Cluster Survey (MICS),4 the national average for children aged 36-59 months (aged below 3 to 5 years old), attending an early childhood education programme, is only 13.4% (the average for the Division of Khulna specifically is 13.6%).5 Only 8.8% of children under age 5 have three or more children’s books (Khulna Division average is 10.3%). The net-intake for primary school also seems low, as only 33% of children of school-entry age go into the first grade of primary school (27.5% for Khulna Division).

1.3 National Efforts to Mainstream Pre-Primary Education Access

In response to this overall challenge of low participation,6 there were several accomplishments which took place at the policy level. The formulation of key technical inputs also guided the initial implementation of pre-primary education. Strategically, the Government of Bangladesh recognized “the benefits of early learning and its importance for preparing young children to survive and thrive in primary school settings.”7 Subsequently, several frameworks emerged to support pre-primary education access, including:

(1) The “Pre-Primary Education Operational Framework”, approved by the Ministry of Primary and Mass Education (MoPME) in 2008. This enabled the implementation of one year of pre-primary education for all children of 5 to 6 years of age;” 8

(2) The National Education Policy, approved in 2010, which recognized pre-primary education as an

integral part of primary education;”9 and

(3) The Comprehensive Early Childhood Care and Development (CECCD) policy, approved in 2013,

that outlined the profiles of children who have the right to receive ECD services and identified

multiple government and non-government partners who ought to participate in mainstreaming the

delivery of pre-primary education.

The formulation of substantive inputs also contributed to standardising the pre-primary education curriculum, including: (1) the “approval of the Operational Standards for the Delivery of the Pre-Primary Education;” (2) the development and approval of a national pre-primary education curriculum in (June 2011), spearheaded by the Directorate of Primary Education (DPE) within MoPME; and (3) the production

3 UNICEF and Centre for Policy Dialogue. (June 2011) National Budget: Are The Commitments To The Children Of Bangladesh Being Kept?, Section 2.4 under ‘Education’; and Government of the People’s Republic of Bangladesh, Ministry of Planning, Bangladesh Bureau of Statistics (BBS), and UNICEF (July 2013) Child Equity Atlas: Pockets of Social Deprivation in Bangladesh, p. 30. 4 Government of the People’s Republic of Bangladesh, Ministry of Planning, Bangladesh Bureau of Statistics (BBS), and UNICEF. (May 2014) 2012-2013 Bangladesh Multi-Indicator Cluster Survey (MICS): Key Findings. 5 Secondary data from different sources report dissimilar results for access to Pre-Primary Education (PPE) for 5 year old children. For example, the Bangladesh Bureau of Statistics, Bangladesh Institute of Development Studies and UNICEF, Child Equity Atlas: Pockets of Social Deprivation in Bangladesh (July,2013) reports a national average of 21.6%, whereas the 2012-2013 Bangladesh Multi-Indicator Cluster Survey (MICS) (May 2014) presents a figure of 13.4%. It is important to note that the MICS data shows results for children below aged below 3 up to age 5, but the age range in Bangladesh is actually 5-6 years (or below 6). 6 Government of the People's Republic of Bangladesh, Directorate of Primary Education, Power and Participation Research Centre (PPRC), and UNICEF Bangladesh. (November 2013) Bangladesh Primary Education Stipends: A Qualitative Assessment, p. 1. 7 Government of People’s Republic of Bangladesh, Directorate of Primary Education. With technical assistance from UNICEF. (December 2012) Pre Primary: Education Expansion Plan, p. 8. 8 Ibid, p. 8. 9 Government of the Peoples' Republic of Bangladesh, Ministry of Women and Children Affairs (2011) National Children Policy.

14

of a pre-primary education teaching learning package (distributed in June 2014), organised by the National Curriculum and Textbook Board (NCTB). Finally, in recognising the efforts of nongovernmental organisations providing pre-primary education, the DPE also approved the Government-NGO Collaboration Guidelines (June 2011). This agreement defines collaboration strategies for nongovernment organisations and the government during the phased approach to mainstream pre-primary education.

The Operational Standards for Delivering Pre-Primary Education

As noted earlier, the approval of the Operational Standards for the Delivery of the Pre-Primary Education, is very important, because multiple nongovernmental partners are involved. These Operational Standards were adopted under PEDP3. Due to their recent introduction and distribution through the Pre-Primary Education package in 2014, adherence still appears to be at the initial stages. The Operational Standards have nonetheless been incorporated throughout evaluation findings. Eight areas constitute Operational Standards for the Delivery of the Pre-Primary Education10, and figure 3 presents a simplified framework of these areas. A summary of these areas are: (1) Physical environment: Defines how preschools should be safe, secure, clean, spacious, attractive,

and stimulating and encouraging to children to engage them in different forms of creative activities, both individually and in small groups.

(2) Learning environment: Presents standards on classroom arrangement, teaching learning materials, daily routines, and the teaching learning process.

(3) Staffing: Provides insights for ensuring an optimum Teacher-Student ratio. (4) Monitoring and supervision: Describes what is a well-defined, structured, and needs-based qualitative and quantitative supervision and monitoring. The section also discusses monitoring personnel, work style, frequency, duration, process, tools, the reporting system, and follow up mechanisms to strengthen and ensure the quality of pre-primary education service delivery. (5) Parent and community involvement: Includes elaborated roles and responsibilities, frequencies, and processes of meaningful parent and community involvement and participation. (6) Training and professional development: Defines the parameters for training and continuous professional development of teachers.

(7) Management: Explains how the delivery of ECD services will be managed locally and centrally, by when, and the process of sharing progress from the sub-national level to central level.

(8) Administrative: Refers to the maintenance of certain categories of records. This section also describes administrative parameters and processes at school to central levels in order to support implementation.

10 Government of Bangladesh. MoPME, Directorate of Primary Education (DPE) with Technical Assistance from UNICEF (November 2012) Standards of Pre Primary Education (PPE) service delivery as a component of Pre-Primary Education Expansion Plan, p. 2.

Figure 3: The operational standards

15

1.4 Summary of the Initiative and Implementing Partners

Low participation rates in pre-primary education are due to limited access. The overall objective of the LUL-funded initiative is “to improve access to pre-primary education for school readiness for children 5 years old.”11 There are a total of 400 centres that currently comprise the overall portfolio of LUL-funded pre-primary education learning centres. All teachers, selected from the community, are females. Only 50 out of 396 teachers (or 13%) have over five years of teaching experience. The number of children who have completed the one year pre-primary education at the learning centres since 2013 totals 18,159. The number of students currently enrolled in pre-primary education centres is 11,238. The girls are the majority for both cases, as evidenced in the disaggregated data presented in table 1 below.12

Table 1: Past and present student profile data for LUL-funded learning centres

Past Student Profile Data (2011-2013) Present Student Profile Data (2014)

Cumulative Number of Students

Completing Pre-Primary Education (previous

years)

Number of Students Currently Enrolled

Girls Boys Total Girls Boys Total

10,831 7,328 18,159 6,694 4,544 11,238

Although the LUL-funded pre-primary education is executed by BRAC, the NGO has been working under the administration of the Bangladesh Shishu Academy (BSA) since 2012. Both partners have staff working in Satkhira District (Khulna Division), where all of the LUL-funded activities are currently implemented. BRAC, an international NGO and also the largest one in Bangladesh, has extensive cumulative experiences in delivering pre-primary education services to communities, as it currently delivers pre-primary education services to over 15,000 learning centres country-wide. Note that, in terms of equity, their operating standards of working with children with disabilities is overseen by the Children with Special Needs (CSN) Unit, established in 2003. The policy is “to integrate children with special needs into BRAC schools, and [to] ensure their participation in mainstream education and society in general”.13 The “classroom policies for special needs children include sitting in the front (of the classroom), studying in pairs, inclusion of CSN issues in textbooks, and awareness-building among classmates and teachers.”14

The BSA, part of the Ministry of Women and Children Affairs (MoWCA), was started in 1976 and has the aim of ensuring that Bangladeshi cultural activities are sufficiently integrated into school activities.15 As all of UNICEF’s activities include a capacity strengthening component, UNICEF has partnered with this government institution to ensure effective programmatic and financial implementation. It should be noted that UNICEF’s partnership with BSA has been long running.

1.5 LUL-Funded Initiatives in Bangladesh

Prior to evaluating the pre-primary education activities, it is necessary to review the overall UNICEF portfolio of all LUL-funded initiatives. Broadly, the funds contribute to UNICEF’s sharpened focus on educational equity, innovation, and outcomes for learners through support for pre-primary education, learning about health and hygiene, and education of adolescent girls and out of school children and youth (OOSC). The initiative, “Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira district,” addresses education challenges for children in multiple

11 UNICEF (2011) Programme Proposal: Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira district Submitted to the German National Committee. The proposal targets five year old children, but the national pre-primary education system extends the age limit to below 6 years. Hence, the target of 5 year olds means between 5-6 years of age. 12 All statistics are sourced from BRAC M&E data. 13 Sourced from: http://education.brac.net/children-with-special-needs-csn. 14 Ibid. website source. 15 For more information about their areas of expertise, please see http://www.shishuacademy.gov.bd/bn/.

16

age groups and includes three equity pillars: (1) reaching out-of-school children; (2) expanding girls’ education; and (3) improving quality outcomes for learners.16 The four interlinked components are: Component One: Improved access to pre-primary education (education system) for school readiness

for children 5 years old; Component Two: Ability-Based Accelerated Learning (ABAL); Component Three: Skills Training for OOSC of ages 8 – 14 called Supervised Informal

Apprenticeships Skills Training; and Component Four: Communication for development (or C4D) is cross-cutting. Table 2 presents an overview of the scope, scale, and expected and actual results of all of the LUL-

funded activities. A Logical Framework, located in Annex 2, is available and includes the specific

outcomes, outputs, defined unit of analysis (indicators), baseline figures, distinct targets and means of

verification. Component one was selected, because it was the furthest along in implementation

compared to the others. While the total resources is USD 3,540,000, only USD 1,645,382 is allotted to

pre-primary education.17

1.6 Evaluation Report Structure

This report includes four chapters. This chapter, which provides a brief overview of the overall challenge of low participation rates in pre-primary education and subsequent responding policy instruments and the LUL funded initiative, is followed by the chapter 2 on methodology. Chapter 3 provides the findings based on the evaluation questions. Chapter 4 concludes and also presents recommendations in order of priority as well as lessons learned. Table 2: Scope and scale of LUL-funded initiatives

LUL Initiatives

in Bangladesh

Overall

Objective(s) Expected Result(s) Actual Result(s)

Component

one:

Pre-primary

Education

(PPE)

Improve access to

pre-primary

education for school

readiness for children

5 years old.

13,500 children aged 5

years access pre-

primary education and

are better prepared for

enrolling in grade one

on time (age 6).

18,159 children (7,328

boys and

10,831 girls) have

benefitted. 11,228

additional children (boys

4,544 boys and 6,694

girls) are enrolled

through 400 LUL-funded

centres.

Component

two:

Ability-Based

Accelerated

Learning

(ABAL) for

Provide a second

chance to children

who have left school.

7,475 OOSC aged 8-14

years will access the

ABAL centres, be better

prepared for primary

school enrolment and

completion, and 4,600

7,475 adolescents

(3,637 boys and 3,838

girls) have benefited

through 250 learning

centres, and 5,993

(2,924 boys and 3,044

girls) are continuing in

16 The total project period is six years. However, the funded proposal only covered the first three years of the pilot; UNICEF BCO is in the process of raising funds for the remaining three years. 17 Note that the unit cost (range) per child of USD 35 to 40, which was reported by BRAC and UNICEF during the field mission, is not supported with a budget of USD 1,645,382. The calculation was attempted by dividing the budget by the 29,397 children (which includes 18,159 previously enrolled children and 11,238 currently enrolled children).

17

LUL Initiatives

in Bangladesh

Overall

Objective(s) Expected Result(s) Actual Result(s)

Adolescent

Girls

children will enrol into

formal primary school.

the 2nd year with Grade

V equivalent basic

education.

Component

three:

Skills Training

for OOSC of 8-

14 years old

called

Supervised

Informal

Apprenticeships

Skills Training

model.

Give an opportunity

to the OOSC to

obtain training that is

contextualized

according to the

demand of the local

labour market. This

informal

apprenticeship model

delivers livelihood

skills training and

ensures that OOSC

acquire competency

based skills with a

Master Crafts Person

(MCP)

3,000 OOSC (at least

50 percent girls) aged

14 to 17 years

completed skills training

course, become

competent in their

respective trades and

placed in decent jobs.

1,500 youth (828 boys

and 672 girls) have

benefitted.

Component

four:

Communication

for

development

(C4D) and

social

mobilisation for

OOSC of 5-14

years old.

Communities in

Satkhira district have

improved knowledge

and awareness of the

education

opportunities for

OOSC; and are

better prepared for

disasters.

100 percent of communities surrounding the learning centres will be sensitized to the needs of OOSC and understand the need for completing a full primary school cycle; and

400 targeted learning centres and 100 percent of the 331 government schools in the targeted Upazilla will be better prepared for emergencies with disaster preparedness plans in place.

Through Interactive

Popular Theater,

community dialogues,

and advocacy, 122,939

(67% female) people

have been affected by

this component.

18

2 EVALUATION METHODOLOGY

One of the purposes of this evaluation of LUL in Afghanistan, Bangladesh, Liberia, Madagascar, and Nepal is to contribute to the on-going global dialogue on equity, especially in relation to education for marginalised girls and adolescent girls. In the case of Bangladesh the definition of those hardest to reach includes children with disabilities, including learning disabilities; children located in remote geographical locations; children of lowest economic profile, children living in unstable family situations (e.g., single parent with limited income); and children living in disaster-prone areas with scarce commercial and public services delivery. The decision to focus on children in disadvantaged circumstances is justified, in that pre-primary education is being implemented from a rights-based perspective, with a focus on “children in most disadvantaged and cyclone, tornado and coastal areas”.18 Children with disabilities were not originally targeted in the Bangladesh LUL initiative. However, inclusiveness is a core value of UNICEF programming, and a strategic priority.19 UNICEF advocates for mainstreaming access to early childhood development (ECD) services for all children, including children with disabilities.20 The evaluation was undertaken from 25 July 2014 to 31 January 2014. The evaluation focuses on: (1) pre-primary education service delivery (in terms of availability, accessibility, and utilization); (2) coordination among pre-primary education service providers; and (3) relevance, effectiveness, and sustainability. This remaining part of section describes the global evaluation criteria, evaluation questions, sampling, analysis, limitations, and the evaluation governance structure.

2.1 Evaluation Framework

The Terms of Reference (ToR), located in the first annex, justify how only the most relevant of the evaluation criteria developed by the Organisation for Economic Cooperation and Development-Development Assistance Committee (OECD-DAC) (i.e., relevance, effectiveness, efficiency, impact, and sustainability) would be applied to this evaluation effort. Since LUL has not been implemented for a long time, the analysis of impact and efficiency would be seriously limited in scope and depth. With this reality, the following criteria form part of the evaluation framework: (1) relevance, (2) effectiveness, and (3) sustainability, including the sub-criterion of scalability, which assesses the likelihood that the intervention can expand its outreach. Table 3 provides brief working definitions of these criterion based on various other international sources. Table 3: Summary of evaluation criterion for LUL

Evaluation Criterion

Source (s) Working Definitions

Relevance DAC The extent to which the aid activity addresses the priorities and policies of recipients and donors

Effectiveness

DAC

LUL ToR Other documents

The extent to which the initiative achieved stated objectives

18 UNICEF BCO (2011). Programme Proposal: Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira District. Submitted to the German National Committee, p. 7. 19 UNICEF (2012) Early Childhood Development and Disability: A Discussion Paper with UNICEF and the World health Organization (WHO). 20 From http://www.unicef.org/disabilities/index_65297.html. This evaluation therefore emphasizes the use of an “inclusion” rather than an “integration” framework, when working with children with disabilities. The philosophies regarding the education of children with learning difficulties and/or disabilities have changed dramatically over the past two decades, but current approaches emphasize the need to foster inclusion rather simply integrate students into mainstream classroom settings. An additional source used comes from Norwich, B. (1994). ‘The relationship between attitudes to the integration of children with special educational needs and wider socio-political views: a US–English comparison’, European Journal of Special Needs Education, 9, p. 91–106.

19


Source (s) Working Definitions

Sustainability DAC LUL ToR Other Documents

The potential for the continuation of benefits (long) after the initiative has been completed, with attention to start-up or developmental costs (e.g., content design and development) as well as partners’ priorities, policies and demand on the part of local institutions

Scalability (sub-criterion of sustainability)21

United States Agency for International Development (USAID)

The extent to which a small-scale or pilot activity is likely to be expanded to reach larger populations

2.2 Evaluation Questions

There are 17 Evaluation Questions for this evaluation, and they are organized within the framework of the OECD-DAC criterion in table 4 below.22 Table 4: Evaluation questions of the LUL evaluation


Evaluation Questions

Relevance

(a) What are the key challenges to children’s well-being and learning?

(b) How are these or other challenges specific to girls’ and or disadvantaged children’s learning and well-being?

(c) How does the LUL initiative (as designed) address these challenges?

(d) In what ways does the design and/or implementation of the LUL initiative address challenges to student learning, their well-being, and their success in education and in life?

Effectiveness

(e) What is (or are) the objective(s) of the LUL initiative?

(f) To what extent has the LUL initiative accomplished the stated objectives?

(g) How has the LUL initiative, to the degree it has accomplished its objectives, affected the learning and well-being of students, their families, and their communities? And, similarly, what changes in terms of the education environment, educational opportunities for the hard to reach children have resulted from the LUL initiative?

(h) When challenges arose, what adaptive responses emerged?

(i) In what ways did those adaptive and/or innovative responses address the needs of the most vulnerable?

(j) What challenges to achieving change have not been addressed?

(k) How could the effectiveness of the initiative be improved?

Sustainability

and

Scalability

(l) How does the initiative relate to Ministry of Education (MoE) or other partners’ priorities?

(m) What are unit costs for the initiative and what are their ramifications for scaling?

(n) What current or potential partners are likely to have interest in scaling (or replicating) the initiative?

(o) In what ways have activities resulted in outputs (e.g., learning resources, administrative processes, etc.) that will reduce costs of scaling or replicating?

21 This criteria includes an assessment of available quantitative data is presented to support the creation of new indicators and data, which (ultimately) are intended to improve the prospects for future evaluation efforts. 22 An evaluation matrix was developed but only at the global evaluation level.

20



(p) How enduring are the changes that have resulted from the initiative? And, similarly, what are their potential longer-term effects?

(q) How could the sustainability of the initiative be improved?

The Evaluation Operating Context

A challenging factor noted within the evaluation context, and beyond the control of the evaluator, is the complexity of UNICEF’s partner networks, including their institutional structures at central, district, and sub-district levels. In order to follow how the LUL funds were used to establish and maintain learning centres at the community level, it was imperative to first understand the role of each partner’s expansive network at different levels and their collaboration and coordination. Therefore, half of the time assigned of the evaluation mission in Bangladesh was dedicated to understanding this institutional context. The other half of the time concentrated on visiting learning centres in the field. According to the project proposal, it is not evident that funding to support partners, including information sharing, was ever made available. Indeed, effective partner relationships were assumed. The implication (of this assumption) is that any data related to bottlenecks in institutional cooperation are considered beyond the scope of this evaluation.

2.3 Sampling

With 400 learning centres from which to select for this evaluation and limited time to spend in Khulna Division, a maximum variation sampling criteria was applied. The learning centres were purposefully selected using: (1) specific mutually-agreed sampling criterion between the UNICEF Bangladesh and the Evaluation Team;23 and (2) a list of 18 different sampling criterion. Annex 3 has more details about this process, including: (1) the process used for developing the sampling criteria designed to identify the selected learning centres; (2) the complete list of sampling parameters considered; and (3) the names of six out of seven centres visited. Therefore, a total of seven learning centres were selected for focus groups, semi-structured stakeholder interviews, and classroom observations. Annexes 4 and 5 contain all of the instruments used to systematically collect information from relevant stakeholders and classrooms. The sampling process also incorporated equity-focused perspectives the evaluator took proactive steps to gain a comprehensive understanding of the important contextual elements of the ECD service delivery in Satkhira District. For example, special efforts were made to sample and interview duty bearer teachers working in the most challenging (and remote communities) and with rights holder parents with limited resources, coming from long distances daily to drop off their children at the learning centres.

2.4 Data Collection

In line with equity-based evaluation guidance, the evaluation was designed to be maximally participatory, which means that multiple stakeholders had the opportunity to share opinions and experiences in an open-ended discussion format. This inclusive approach also ensures that evaluation findings are credible. As part of the mixed methods evaluation design, the data from quantitative methods consisted of:

1. UNICEF Bangladesh reports;

2. UNICEF Bangladesh targeting and monitoring and evaluation data;24

3. Monitoring data (collected directly by BRAC). Note that a discussion about the robustness of the M&E system in place is discussed in findings chapter;

23 Ethically, this implies that the sampling process transparent and honest. 24 UNICEF Bangladesh granted the evaluator permission to use their data applying the Tanahashi Model to include in the report. It is located in Annex 6.

21

4. Secondary data from documents from the relevant ministries;25 and

5. Previously undertaken surveys, including but not limited to the 2011 Population Census, MICS, and other relevant data produced in-country by the Bangladesh Bureau of Statistics (BBS). A list of secondary sources which could be used to support the establishment of a “retroactive baseline” is presented in the findings chapter.

During document review, the evaluator also paid special attention to the extent to which the initiative incorporated UNICEF's core commitments, including the CRC, Human Rights-Based Approach to programming (HRBAP), and the Convention on the Elimination of all Forms of Discrimination against Women (CEDAW).26, 27 Similarly, the qualitative data used in this report was sourced from:

(a) Focus Group Discussions with children, parents and community members at large; (b) Semi-structured interviews with key informants. 28 The list of stakeholders is in Annex 629; (c) Classroom observations; (d) Photographs; and (e) Fifteen OMJs (one-minute junior) videos, which were reviewed for content which was relevant to

responding to the Evaluation Questions. The evaluator had an interpreter for interviews and FGDs which occurred in Bangla language. Multiple steps were taken to ensure that the data collected was of strong quality. Prior to engaging in interviews with stakeholders, this person translated all written instruments, thereby providing a suitable introduction to the overall scope of the evaluation. This person also received orientation in the important techniques of interviewing with translation, including checking in with respondents to confirm key concepts during the interview / FGD processes. Prior to engaging in semi-structured interviews and observations, the evaluator employed ethical safeguards as well as notified evaluation participants that their responses were confidential. Additional ethical safeguards included the limited use of photography of the children’s faces and permission sought from the stakeholders to use video (for the OMJs).

2.5 Data Analysis

Descriptive statistics were calculated from the monitoring data and the observation data, and secondary

data was used for understanding the operating context of the learning centres and for triangulation. The

qualitative data (e.g. focus groups, observations, stakeholder interviews, and OMJs) was analysed using

coding framework.

25 Government of the People's Republic of Bangladesh. Ministry of Women and Children Affairs (MoWCA), Ministry of Primary and Mass Education (MoPME), Monitoring & Evaluation Division, Directorate of Primary Education (2012) Annual Primary School Census; and Education Watch. (2013) New Vision Old Challenges: The State of Pre-primary Education in Bangladesh. Funded by Campaign for Popular Education (CAMPE) and the European Union Delegation to Bangladesh. Also sourced at www.campebd.org 26 The use of human-rights language and gender-sensitive language has also been incorporated into this document. 27 It should be noted that data for human rights and gender monitoring processes was requested and is unavailable. 28 Interview questions were pretested to ensure that the stakeholders understood well the information that is being asked of them. 29 The following stakeholders were sampled: (a) UNICEF Country Office/Field Office personnel; (b) Reference Group Members; (c) the relevant government representatives at central, district, and sub-district levels; (d) partner-organisation personnel (BRAC) at central, district, and sub-district levels; (e) primary school and pre-primary school stakeholders; (f) parents and community members; and (g) past and current students.

22

2.6 Evaluation Governance Structure and Stakeholder Participation

The evaluation governance structure, illustrated in figure 4, includes support from: (i) UNICEF HQ Office-New York (in evaluation and education), (ii) the UNICEF Bangladesh Office, (iii) an Evaluation Reference Group, (iv) an international lead evaluator, and (v) the national evaluators. All levels of stakeholders provided substantive input to key deliverables, including, the Inception Report, instrument development, and the Final Evaluation Report. Another important aspect of the stakeholder participation occurred through the Evaluation Reference Group, which was populated by UNICEF, BRAC academia, and the government. The group’s level of involvement took place in the beginning, middle, and end of the evaluation process, making it possible to contribute to the evaluation scope, selecting learning centres visited, confirming the validity of findings, and providing critical feedback on recommendations. At the field level (in Khulna), a feedback session was also held where, UNICEF and BRAC were in attendance, and this activity enabled the evaluator to “member check” and confirm all preliminary findings. An Evaluation Data Review, held in in New York from 7-9 October 2014, contributed to a common understanding of: (1) the information sought; and (2) core expectations for the evaluation.30

2.7 Evaluation Limitations

Though there were limitations, the evaluator employed several strategies in efforts to minimize their effects on data quality. (1) Limited time to comprehensively address the emerging issues. The time for the field mission

was limited, which hindered the evaluator from confirming emergent themes through triangulation with additional relevant stakeholders. The implication is that the potential promising practices that may have begun to develop between the learning centres and government-run schools could not be fully explored.31 Time restrictions also prevented the collection of data for a comparison, or a “counterfactual” to the LUL-funded learning centres. These might have included learning centres run by other NGOs such as Save the Children or by community based Qur’anic schools. Similarly, a comparable cost analysis from other interventions was not feasible. It is recognised, however, that relevant comparative perspectives would shed light on cost-effectiveness and sustainability. Relatedly, children with disabilities were reported in every village visited. The evaluator was provided with few details about these disabilities.32 With some additional time, however, it could have

30 UNICEF (2014). Agenda: LUL evaluation data-review, p.1. 31 For example, when three mothers and a teacher from a rural community mentioned that there was an established connection with the Head Teacher in the adjacent primary school, the evaluator did not have sufficient time to interview the Head Teacher of a GPS, confirm the connection, or even present follow on questions about challenges and or strengthening the LUL-funded centre-government primary school connections. 32 For example, through the translation, it was mentioned that these “children could not speak clearly”. However, this description could include several disparate but related speech impediments, including stuttering, stammering, mispronouncing certain sounds, and unfocused speech. Further, these potential speech impediments could have been long term or short term.

Figure 4: Governance structure for the LUL Formative Evaluation

UNICEF HQ

Team Leader

National Evaluator -Bangladesh

National Evaluator -Afghanistan

National Evaluator -

Liberia


Nepal


Madagascar

UNICEF

Country

Office

Evaluation

Reference

Group

23

been possible to uncover information about the nature of the described speech disabilities, which in turn could have provided insight on the supplemental needs/aides that teachers could use to accommodate these children.

(2) Baseline estimates are missing, and it is assumed that pre-primary education access for children, prior to this initiative, was either limited or none. UNICEF managers confirmed that the learning centres were new interventions and part of the innovation of LUL. It could therefore be assumed that the observed rights holder children’s access would not have otherwise occurred without the LUL funding.

(3) Limitations with the Bangla–English language translations. Frequently, interviewees would give an answer but not answer the specific questions presented. To overcome this obstacle, the evaluator simplified the type of information that was being requested. Even then, if the evaluator and interpreter asked the same question in two different ways and did not get a direct answer, then that question was skipped.

(4) Limitations with other competing variables. Although the reference group members proposed that the field mission include observations of government-run pre-primary schools, this suggestion was not honored. The UNICEF headquarters stakeholders concluded that this evaluation effort ought to remain focused on its current implementation and would not be broadened to include non-LUL funded sites. Further rationale for this decision was that there was a need to sample as many learning centers as possible with the limited time available for field data collection. However, follow-on evaluation efforts that include a comparison of LUL-funded and non-LUL-funded learning centres ought to be considered in the future.33 Even the with the limited number of community visits (a total of six), the themes presented should not be generalised to the entire portfolio of 400 learning centres.

(5) Limitations with seeking permission from the guardians of children. The verbal permission to interview, video tape, or photograph any child was sought from the parent / guardian but had limited success being recorded on a form. As a consequence, all photos taken of children during the evaluation process will not be included in the report to protect their privacy.

(6) As an African-American female evaluator, there were many steps undertaken to minimise cultural differences and to make respondents more comfortable and forthcoming during interviews and FGDs.34 It is recognised, however, that the evaluator, as a cultural outsider, would be unable to discern certain cultural nuances the way a Bangladeshi evaluator would.

33 It should be known, however, that understanding how the government-run pre primary centres operate, their challenges, and how stakeholders value their education services is very important. Documents and interviews confirm that MoPME is in the process of mainstreaming nationwide pre-primary education. It is within this context that it should be considered that future evaluation efforts concentrate on supporting the identification of existing gaps to strengthen the delivery of quality pre-primary education. 34 For example, the evaluator respected the cultural and dress codes and learned key words (for greetings and numbers) in Bangla language.

24

3 BACKGROUND AND FINDINGS

3.1 Evidence Used to Design the LUL-Funded Learning Centres

This section describes exactly what evidence was used in programming the pre-primary education intervention; presents the implementation status; and identifies all of the key stakeholders and their respective contributions to the initiative. The findings are also presented according to the OECD-DAC evaluation criteria.

3.1.1 Evidence Used to Design the LUL-Funded Learning Centres

The results from the 2012-2016 United Nations Development Assistance Framework (UNDAF) for Bangladesh included five different needs-based maps which exposed 20 “common districts which inform the UNDAF priorities”, and this evidence was used to first identify the targeted locations where the LUL-funded learning centres would be established.35 The district of Satkhira, located in the South Western part of the country (in Khulna Division), was deemed as a vulnerable district, as it is “one of the most disaster prone districts in Bangladesh. Specifically, over the last several years, Satkhira was hit by cyclones, prolonged flood(s), and water logging.”36 Satkhira’s three sub-districts, (1) Assassuni, (2) Shyamnagar, and (3) Satkhira Sadar, were then ranked as the worst performing. Figure 6 shows the location of the Khulna Division within the country as well where these low performing sub-districts are positioned. According to the second progress report, Assassuni and Shyamnagar sub-districts “are the two most disadvantaged sub-districts of Satkhira due their vulnerability to natural disaster. Also they are the low performing sub-districts in primary education survival rates. The two consecutive super cyclones worsened the socio-economic conditions particularly of the poorest quintile population. Education facilities were severely damaged and discontinued for a long time.”37 In addition, the UNICEF Bangladesh’s Education Section conducted a bottleneck analysis, derived from applying the Tanahashi model. This exercise was undertaken in Satkhira, among several other districts throughout the country. The results from this exercise, for Shyamnagar (Munshiganj sub-district) and Budjhata Union (Ashashuni sub-district), provided invaluable contributions about the barriers preventing: (1) availability; (2) access; (3) utilization; (4) adequate coverage; and (5) effective coverage. Annex 7 presents the two charts that emerged from significant data collection efforts. As further needs were identified, additional learning centres were then supported throughout all seven sub-districts in Satkhira. Figure 5 confirms that there is a fairly even proportion of the LUL-funded learning centres across all sub-districts.

35 United Nations Resident Coordinators Office. (2001) 2012-2016 United Nations Development Assistance Framework (UNDAF), p. 11. 36 UNICEF (2012) Basic Education and Equity Progress Report (Year 2), p. 2. 37 Ibid., p. 3-4.

Figure 5: LUL-funded learning centres in Satkhira

district by Upazilla

Assasuni12%

Debhata8%

Satkhira Sadar17%

Kaligonj13%

Kalaroa16%

Shyamnagar18%

Tala16%

25

Figure 6: Map of Satkhira District and targeted sub-districts

26

3.2 Implementation Status

The establishment of learning centres began in earnest, and as indicated in figure 7, the highest number of learning centres supported (468) took place in 2013. Since then, however, UNICEF made a strategic decision to reduce direct service delivery. This reduction was in response to the government’s plans to expand its outreach and gradually integrate the LUL-funded learning centres into government-run primary schools. Staff interviews confirm that the number of learning centers will be further reduced to 300 in by the end of 2015. BRAC M&E data also indicates that 86% of the LUL-funded school structures are considered “temporary”. The majority are not affected by a natural disaster like cyclones, but all learning centres are affected by erosion of some type, the saline air, and water logging (to some extent). There is, however, a critical margin of learning centres (37%) which are remote, vulnerable to natural disasters, have limited communication access, and fall into a category labeled as “hard to reach”. The family selection for pre-primary services includes all children, but BRAC stakeholders note that “the ‘ultra poor’ are prioritised”.38 The classes, ranging from 25-35 children, take place two and half hours per day and six days per week. The teachers are recruited from the community with a provision of pre-service, on the job, and monthly “refresher training”. The number of children who have completed one year of pre-primary education at the learning centres since 2013 totals 18,159, and the number of students currently enrolled in learning centres is 11,238.

3.3 Key Stakeholder Contributions

This section discusses the most important contributions of each stakeholder group to the LUL-funded learning centres. Essentially, there are two levels of stakeholders: (1) the institutional partner engaged in coordinating teacher training and other school-level activities (BRAC), relevant ministries (MoPME and MoWCA and BSA), and UNICEF;39 and (2) the local stakeholders, including the children enrolled, their parents, and teachers. With respect to UNICEF (Bangladesh Country Office), the primary role is programming oversight. However, UNICEF’s Education Section staff have the aim of strengthening Bangladesh’s overall Primary Education Sub-Sector, and the Early Childhood Education sector is part of this overall objective. Field evidence indicates that the UNICEF’s Education professionals and managers support government professionals within different education units (early childhood education, primary education, vocational or technical education, among other relevant units dedicated to certain functions such as monitoring and evaluation, budgeting, and policy and curriculum development). The UNICEF contributions include: (1) technical assistance, (2) mentorship, (3) training, (4) the strategic leadership in engaging stakeholders in dialogue to outline future work, and (5) informal communications to promote a culture of learning. This list is not meant to be exhaustive.

38 This “ultra poor” criteria includes a family with: (1) with less than 50 decimals of land, (2) both parents selling their labor for 100 days per year or less, and (3) children aged from 5 to 6. 39 Note that BRAC, MoPME, and MoWCA (BSA) institutional stakeholders have already been described.

Figure 7: Chart of the LUL learning centres established

by year of operation

0 200 400 600

2012

2013

2014

2012, 134

2013, 468

2014, 400

27

The learning centres include teachers selected from the community, and instruction is provided in temporary facilities which also have a range of classroom materials designed to stimulate the children. With the contributions (e.g., training and supervision) from BRAC, teachers, as duty bearers, are in a position to demonstrate attitudes, knowledge, and practices which, in turn, create a positive and supportive learning environment for the children. Teacher duty bearers also foster parental interest and involvement in their children’s progress and in school matters and serve as a trusted member of the community.40 All children, as rights-holders who are active agents in their own development, not only have the right to an education, but they have the right to play. Their play ought to be “directed to the same purposes as education.” 41 Parents (and family caregivers) as duty bearers, are expected to engage in an active adult-child relationship, help with work (academic, physical, emotional, social) at home as needed, manage children’s education, and ensure school attendance.

3.4 Findings

Each sub-section briefly defines the evaluation criterion which follows the OECD framework, including relevance, effectiveness, and sustainability with a sub-criterion of scalability. All findings, based on evidence gathered through mixed methods, are then presented by taking into account the Evaluation Questions.

3.4.1 Relevance

This section discusses the extent to which the aid activity addresses the priorities and policies of recipients and donors. Four Evaluation Questions, listed in table 5, guide the analysis of this section. Table 5: Evaluation questions for the relevance evaluation criteria



Relevance

1. What are the key challenges to children’s well-being and learning?

2. How are these or other challenges specific to girls’ and or hard to reach children’s learning and well-being?42

3. How does the LUL initiative (as designed) address these challenges?

4. In what ways does the design and/or implementation of the LUL initiative address challenges to student learning43, their well-being, and their success in education and in life?44

Relevance in Addressing Limited Access to Pre-Primary Education

The LUL initiative, dedicated to improving access to pre-primary education for school readiness45 for rights holder 5 year old children, has a design that is consistent with internationally agreed

40 In this case, teachers can also be rights holders, where they would have the right to training / coaching opportunities focused on working with children, in the event that their educational backgrounds do not include issues related to early childhood development. 41 United Nations. (1959) Declaration of the Rights Of The Child. Adopted by UN General Assembly Resolution 1386 (XIV) of 10 December 1959, paragraph 7. 42 This question is related to children who are residing in challenging areas and considered vulnerable due to poverty. 43 Only this part of the Evaluation Question is addressed in this section. 44 The questions related to how the initiative’s design and implementation addresses challenges to student well-being and student success in education and in life demand impact-level data; this also falls outside of the scope of the evaluation. 45 The definition of “school readiness”, developed by the National Curriculum & Textbook Board (NCTB), has not yet been made available in an English Language document. Briefly, however, according to UNICEF Bangladesh, the curriculum for “school readiness” is based on the core principles of: (a) child centeredness; (b) children as active learner; (c) family involvement; (d) school as responsive social institute; (e) inclusiveness; (f) local culture and heritage; (g) importance of relationships; (h) the immediate environment; and (i) environment friendliness. The school readiness curriculum has been designed based on the draft Early Learning for Development Standard (ELDS) outlined for Bangladeshi children. The learning domains are aligned with the grade I curriculum in the primary education. Notably, there are fifteen areas and aspects identified to monitor the gradual progress of individual child and to measure competency.

28

commitments and current country-level efforts to address the challenges of increasing the delivery of equitable pre-primary education services. Due to environmental disasters and flood conditions, there were extended school closures which, in turn, led to a less than expected competency achievement. Increases in dropout rates then followed. It was within this context of low participation rates that the LUL-funded intervention offered access to pre-primary education for the hardest to reach children. The intent was to reduce inequities caused by the physical environment as well as by poverty. It should also be noted that at the time of formulating the proposal, natural disasters, particularly cyclones had disrupted the education of more than 1.5 million children from 2009-2011. A staggering 9,000 of Bangladesh’s 82,000 primary schools had been severely affected by disasters. In the targeted area, Cyclone Sidr alone damaged or destroyed almost 4,500 schools when it hit the South West coast of Bangladesh in November 2007. In fact, in 2009, Cyclone Aila disrupted education to over 237,000 primary school-going children in the coastal regions of Bangladesh.46

Relevance of the Initiative at the Policy Level

The LUL initiative was designed to equitably deliver pre-primary education to children and families residing in environments which were challenging due to severe climate and poverty conditions. Indeed, the evaluation evidence confirms that the operating context presents strong weather-related challenges. For example, the BRAC M&E data shows that the majority (86%) of the school structures are considered “temporary”. All six classrooms visited were exposed to harsh physical conditions, including: (1) the saline air; and (2) strong winds and rain that require frequently changing the roof.

The budgeted inputs addressed gaps impeding access to pre-primary education. The gaps included the need for: (a) learning centres (structures); (b) trained teachers; and (c) teaching and learning materials. There is also interview data highlighting gaps related to improving the quality of pre-primary education. The support included: (1) assistance to government-level stakeholders in order to finalize a curriculum package; (2) the provision of technical assistance to train trainers; (3) training to teachers; (4) operational support to BRAC; and (5) some limited tracking of children entering the government-run schools.

The delivery of pre-primary education is also consistent with both international and country level commitments to ECD. For example, the LUL-funded initiative is supportive of key elements within the CRC, the General Comment Number Seven, CEDAW, HRBAP, and other and related equity concerns to promote inclusion of the most vulnerable children and communities.47 Similarly, the learning centres are reflections of the implementation the government’s policies, including the PEDP-II and PEDP-III.

Stakeholder Views on the Relevance of LUL-Funded Learning Centres

The duty bearer government and implementation stakeholders (MoPME, MoWCA, BSA, and BRAC) at every level expressed positive views about the LUL initiative and how the intent to focus on those in the hardest to reach areas was very important and therefore appreciated.48 Interview data highlights that the pre-primary education financed by LUL is consistent with current priorities to “mainstream” ECD services throughout the country. All of the data from teachers interviewed showed that they believed in the importance of delivering ECD services to children, and they all expressed strong job satisfaction and “good relations with the children”. Parents also had positive reflections. The children interviewed also said that the “loved” school and never wanted to “leave the carpet”. These statements from interviews

46 January to December 2009 UNICEF Consolidated Emergency Report. All of the statistics and conclusive remarks noted in this paragraph of the report are derived from the original Programme Proposal. 47 UNICEF and UNESCO. (2007) A Human Rights-Based Approach to Education for All, Chapter 1; Bernard van Leer Foundation. (2006) General Comment No. 7, ‘Implementing Child Rights in Early Childhood’, Committee on the Rights of the Child, Forty-first session held in Geneva; The Hague, The Netherlands, paragraph 30 on page 47. CEDAW. (1979), Article 10. 48 In order to honor the confidentiality of respondents, more specific identification of the different levels cannot be disclosed.

“I believe that there is a good learning

environment for my child, and I am glad

that my child is learning lots of songs.”

- Parent from School #3 Visited

29

from children confirm that their experience has been positive and that they are not looking forward to moving onto primary school. In summary, delivering pre-primary education is consistent with international and country level commitments to providing ECD to the hardest to reach, including children with disabilities, in remote geographical locations and/or disaster-prone areas, and those with the lowest economic profile. Moreover, stakeholder interviews from the policy to school to community levels revealed positive evidence that confirms that the LUL-funded initiative appears to be relevant to the country’s efforts to provide equitable ECD services throughout the country.

3.4.2 Effectiveness

The analysis of this section concentrates on the extent to which the initiative achieved stated objectives and expected results. Seven Evaluation Questions, presented in table 6, guide the analysis of this section. Table 6: Evaluation questions for the effectiveness evaluation criteria



Effectiveness

1. What is (or are) the objective(s) of the LUL initiative?

2. To what extent has the LUL initiative accomplished the stated objectives?

3. How has the LUL initiative, to the degree it has accomplished its objectives, affected the learning and well-being of students, their families, and their communities?49 And, similarly, what changes in terms of the education environment, educational opportunities for the hard to reach children have resulted from the LUL initiative?50

4. When challenges arose, what adaptive responses emerged?

5. In what ways did those adaptive and/or innovative responses address the needs of the children?

6. What challenges to achieving change have not been addressed?

7. How could the effectiveness of the initiative be improved?

Effectiveness at the School Level

The LUL initiative has met or exceeded its core expected result of creating access to ECD services to over 13,500 children.51 Programme monitoring data confirms that the LUL met or exceeded its expected accomplishments by: (1) providing 18,159 children aged 5 years with access pre-primary education with an attendance rate

49 The questions related to how the initiative affects student learning and student well-being, their families and their communities was not addressed, as these questions demand impact-related data which was not available in the six community visits undertaken while on mission to Bangladesh. 50 This question can only be partially addressed with anecdotal information. 51 The Bangladesh Country Office review raised the issue of including in this evaluation an assessment of pre-primary education quality. Considerations of education quality, however, fall outside the scope of this evaluation, given that the stated objective is “to improve access to pre-primary education for school readiness for children 5 years old.” Based on observations, interviews with seven teachers and two FGDs with teachers, two FGDs with children and their parents, teaching and learning at the LUL-funded learning centres appear to be effectively implemented. However, determining the quality of pre-primary education and its impact on educational attainment likely requires competently designed and implemented randomised control trials (RCTs).

30

between 92% and 95% (target was 13,500 with a 90% attendance rate)52; (2) equipping 400 learning centers with age appropriate teaching and learning materials functional in all Upazilas of Satkhira district (target was 1,078) and (3) training 1,002 teachers (target was 150 teachers).53 All three of these achievements have resulted in: (1) strengthened pre-primary education access; and (2) created educational opportunities for children and teachers. Further, while providing pre-primary educational opportunities, almost all of the minimum Operational Standards for the Delivery of Pre-Primary Education appear to have been met.54 However, the learning centres visited lacked sufficient ventilation; Only two out of six LUL-funded learning centres had shutters to block out the rain and protect the carpet inside. Only one out of six centres visited met the criteria for sanitary toilet facilities with running water. Nevertheless, all six classrooms observed were, although limited in space, safe, secure, clean, inviting and even attractive. In every classroom, there was work on display that was designed and created by children. With the absence of baseline data, it is assumed that there were a limited number of or no learning centres before this LUL funding. It is therefore possible that the following results, based on classroom observations and the teacher’s instruction, would reflect positive changes. All classrooms included the same core teaching - learning materials, and the daily routine included small and large group activities focused on fine and gross motor skill development. The observed average child-teacher ratio for was 27:1, which is on par with standards. In addition, the organisation of instruction highlighted leadership and communication skills, where there were four small groups which had one child serving as “the leader”. Each group’s leader was responsible for ensuring that the group’s work was on track and for communicating messages to the rest of the class. Observations and stakeholder interviews also indicate that the children are exposed to math, writing of Bangla and English Language letters, the names of fruits, animals, and plants. The Pre-Primary Education curriculum had just been introduced during the evaluation mission, and so its implementation was slowly being integrated into instruction. In terms of equity, there was no distinction observed in the treatment of boys and girls. Instruction took place over 2 ½ hours, as per standards. Finally, the results from observations highlight that the children not only have opportunities to interact with each other, but also with the teacher in a structured way. It was also noted that there was very little variation in the delivery of pre-primary education. An issue raised concerning BRAC’s programme implementation is that some learning centres have been labelled as “BRAC schools” (see figure 8). While this branding could be indicative of their ownership over certain learning centres, it can also be misleading or confusing as the centre has been funded by LUL through UNICEF. Therefore, to summarise, there are significant environmental challenges challenging effectiveness. The initiative has nonetheless exceeded its expected accomplishments. Though all of the minimum Operational Standards for the Delivery of Pre-Primary Education do not appear to be fully met55,

52 One target was not fully met: 100% of attending children complete the full year. However, the achievement has been that approximately 99% completing the education programme. 53 Statement is based on UNICEF progress reports, BRAC secondary data, and interview data with (BRAC) partners and teachers. The target for learning centers established was later revised with the donor from 1,078 to 400. Therefore, the target was met. 54 The minimum Operational Standards for the Delivery of Pre-Primary Education for the following categories are used: (1) the physical environment; (2) the learning environment; (3) staffing, and to a limited extent (4) training and professional development. 55 A recommendation may be warranted, however, the data derived from six learning centres is insufficient and therefore precludes the formulation of a robust finding with a justified recommendation. There is room for improvement for addressing the challenges caused by the physical environment and climatic conditions. UNICEF and BRAC may wish to further explore such specific issues.

Figure 8: Image capturing BRAC’s branding

of LUL learning centres

31

indicating some room for improvements, the classroom environment combined with instruction appear to offer educational opportunities where none may have existed before. BRAC has strong ownership over the delivery of pre-primary education services which is positive, but their branding of certain learning centres established or maintained with LUL funding can be misconstrued by current funders.

Effectiveness of Pre-Primary Education for the Children in the hardest to reach Communities

Data collected in the field suggests that there may be high prevalence of children with disabilities in the hardest to reach communities. It is not known what the prevalence of these vulnerable children is; it is equally unclear if there is a gap in the teacher training offered which focuses specifically on dealing with children with disabilities. Given that inclusiveness is a core value of UNICEF programming and a strategic priority, this section focuses on children with disabilities. There is limited systematically collected data available to determine the full scope of the population of children with disabilities at the pre-primary education level. In the most recent Annual Primary School Census 2012, the indicator of “enrolment of Special Needs Children” is available, but statistics are only collected starting at the primary school level and not at the pre-primary education level.56 The government duty bearers are aware of this obligation, and interviews confirm that revised data collection instruments have been instituted to accommodate this obligation.

BRAC maintains records of children with disabilities, and the 2014 UNICEF Final Project Report submitted to the German National Committee indicates that a cumulative figure of 1,320 “CSN have received” pre-primary education thus far.57 This information could not be verified, however, and so this figure ought to be interpreted broadly.

Nonetheless, during the field mission, there were children with disabilities reported in all six learning centres visited. With an equity-focus used during the data collection phase (e.g., additional interviews with teachers, BRAC and UNICEF staff), the evaluator took many steps to understand the potential influence that children with disabilities may have on the learning environment, including how such children affect the delivery of pre-primary education. She also observed teaching for children with disabilities.

Currently, a systematic standard for supporting children with disabilities in the learning environment exists, but there are gaps in supporting children with disabilities which ought to be addressed in order to render the delivery of pre-primary education more equity-focused.58 Recall that BRAC has: (i) a policy “to integrate children with special needs into BRAC schools;59 and (ii) their own operating standards of working with children with disabilities, administered by its CSN Unit, which states that for special needs children, they are to sit in the front of the classroom, study in pairs, include CSN issues in textbooks, and build awareness among classmates and teachers.60 Further, the minimum Operational Standards for the Delivery of Pre-Primary Education under Teaching and Pedagogical standards indicate that “Teachers (should) occasionally show flexibility in the process towards children with disabilities as per direction.” Information from school-level stakeholders, BRAC, and UNICEF and field observations confirm that some of those strategies are taking place. For example, it was observed that children with disabilities sat in the circle in the front of the classroom and that the group leader occasionally supported these children. In addition, one teacher interview highlighted that she would work

56 Government of the Peoples' Republic of Bangladesh, Ministry of Primary and Mass Education (MoWCA), Monitoring & Evaluation Division, Directorate of Primary Education (2012) Annual Primary School Census, p. 37. 57 Information was compiled from 148 CSN from the first year of implementation, 639 CSN from year 2, and 533 CSN from children currently enrolled. 58 It is recognised that providing support to children with disabilities specifically falls outside of the purview of LUL funding in Bangladesh. 59 Sourced from: http://education.brac.net/children-with-special-needs-csn 60 Ibid. This was paraphrased from the website source.

32

with one child with a speech disability after class if needed. Such efforts are recognised, but the gaps still remain in teacher use of inclusive practices.

Evidence about the consistent use of inclusive practices was absent in classroom observations and in duty bearer teacher and parent interviews. It is recognised that open discussions about this particular subject may still be premature, even with the intent of being helpful and supportive. Nonetheless, children with disabilities have rights to an inclusive pre-primary education. As part of supporting children with disabilities, one main element to consider would be providing training to teachers in the areas of: (i) acceptance of the policy of inclusion; and then (ii) dealing with children with disabilities.61 Indeed without training in being supportive of children with disabilities and on having a positive view about accommodating such children, and then on providing teachers with inclusive practices, the specific needs (e.g., physical and sensory, cognitive and behavioral-emotional) of such children will continue to be ineffectively addressed.

The efforts to foster the inclusion of children with disabilities into the classroom are therefore warranted even though it is not confirmed how prevalent children with disabilities is in the LUL-funded learning centres.

Recommendation #1: It is uncertain what the proportion or number of children with disabilities in the learning centres is for children with disabilities. However, BRAC should provide teacher training in accepting an inclusion policy and then dealing with the special needs of children with disabilities. Teachers providing these services in the classroom would effectively support the delivery of pre-primary education with an equity focus.

Sustaining a Supportive and Effective Learning Environment

The evidence shows that the system for Monitoring and Supervision (Operational Standard for the Delivery of Pre-Primary Education # 3) may be uneven in the learning centres. In particular, those centres located in the hardest to reach and most remote areas received comparably less monitoring and mentoring support than those located in or close to urban/peri-urban areas. All learning centres should receive the same number of monitoring and mentoring visits from BRAC, but they do not. The Operational Standards for the Delivery of Pre-Primary Education were used as the framework to examine the extent to which the delivery of pre-primary education was effective. Operational Standard #3 embodies the key issue of supervision and Monitoring personnel conducting visits to ensure that quality services are delivered. All six learning centres visited met this minimum Operational Standards.62 However, the hardest to reach learning centres, situated in physically challenging locations, received less support from BRAC compared to those in more centralized areas. Based on the data from a teacher focus group and teacher interviews from two of the hardest to reach learning centres, the monitoring visits appear to be uneven. For example, at the two remote learning centres, it was reported that one weekly visit from the implementing partner (the BRAC Branch Manager or/and Monitor) took place. On the contrary, at the three learning centres, located in more comparably centralized areas, it was reported that 15 visits occurred per month, or more than two or three visits per week. It is important that the teachers in the remote areas receive BRAC’s monitoring and mentoring support, because the parental and community involvement in such communities appears challenged. It was observed that in these remote areas, the livelihood is based on selling labor daily or seasonally which is very unpredictable. Parent interviews confirm that given this uncertainty, they rely on

61 Norwich, B. (1994). ‘The relationship between attitudes to the integration of children with special educational needs and wider socio-political views: a US–English comparison’, European Journal of Special Needs Education, 9, p. 91–106. 62 The standard includes some of the following parameters: Each learning centre is receiving at least one structured supervisory visit once in a month by an assigned and trained professional; Fifty percent of total supervision is on pedagogical (quality) issues; Maintaining minimum record of supervision & monitoring; Checklist and format for monitoring that captures observations of child-teacher interactions; two-way reporting; and reporting mechanism for quarterly follow-up. (note that this Operational Standard for the Delivery of Pre-Primary Education has been paraphrased).

33

other family members (e.g., older siblings and grandparents) to fulfill their duty of being actively involved in their children’s education. For instance, parents rely on others (grandparents and older siblings) for dropping off and or picking up their children from preschool. This effectively disrupts the teacher’s ability to discuss the child’s progress with these busy parents. While parent interview data also highlighted that some parents attend a “monthly meeting,” where multiple issues are regularly discussed (e.g., support from parents, collecting/arranging local play materials, transportation for children, child attendance), this meeting would also have limited effectiveness given the unstable livelihood options for parents. BRAC, as a duty bearer, has the responsibility of supporting teachers equally, irrespective the remoteness of the community, and this is a challenge that still needs to be addressed. Having more than one BRAC staff member responsible for visiting the remote learning centres could effectively remedy the observed unevenness in monitoring visits. For instance, two or more BRAC staff members visiting these learning centres would, in turn, support the teacher, as she would then have more than one staff person with whom to debrief and “trouble shoot” about emerging issues. Areas of concern mentioned included motivating parents to become more involved in: (1) their children’s education; (2) classroom upkeep; and (3) making classroom teaching materials. With additional resources (e.g., staff, time, efforts) dedicated to the remote learning centres, the overall equity focus would gradually become stronger.

Recommendation #2: BRAC, BSA, and UNICEF should work together to support the reorganisation of BRAC field staff’s time and existing accountability systems to ensure that all teachers receive the same number of monitoring and mentoring visits per month. The provision of the same number of monitoring visits, especially to learning centres in the remote areas, will effectively strengthen the equity focus of the programme. The system for documenting children’s progress and data storage (Operational Standard for the Delivery of Pre-Primary Education # 8) appears inconsistent and is subject to risks as it currently functions. It is recognised that BRAC collects all information systematically, and then it is manually compiled. However, there is still a potential risk observed, as the information is not stored in a “soft-copy” or web-based format inside a type of computer database. It was observed that teachers record child attendance as well as a “grade” or assessment on different socio-emotional indicators on a regular basis. Figure 9 shows a picture taken of the list of indicators in Bangla Language used, in particular. With the prevalence of floods and cyclones and other physical threats, the preservation of valuable information about children’s attendance and school and developmental progress ought to have priority. This issue of “improper data maintenance” has also been raised in a UNICEF progress Report as a challenge.63 The system for documenting children’s school progress from one education level to the next seems to have gaps and requires dedicated resources in order to serve its intended purpose as outlined in the Operational Standards for the Delivery of Pre-Primary Education.

63 UNICEF BCO (2013). Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira District. Progress Report for the German National Committee for UNICEF, p. 7.

Figure 9: List of hand written child indicators

34

The cumulative progress report for each child’s socio-emotional indicators does not “follow” the children into class 1 (or government primary school). It is recognised that BRAC currently has reports on the overall number of children who move onto primary school from the LUL-funded learning centres; this information will continue to be collected for all children up to class 3. In spite of these efforts however, the tracking of child progress and achievements is not being carried out as it was planned or envisioned in the Operational Standards for the Delivery of Pre-Primary Education, which encourages children to be assessed comprehensively on different domains through a continuous process by maintaining individual records.

There is limited evidence indicating that the progress of the children who have experienced the pre-primary education is tracked through higher education levels. The records for each individual child which is routinely collected, do not appear to be stored in a “soft-copy” form (e.g., computer database). It is even less likely that each child’s socio-emotional indicators is tracked for progress.

The records of each child’s achievements and progress is very important and, when systematically collected and stored properly, this data can potentially offer precise estimates that answer analytical and strategic questions regarding the impact that pre-primary education may have on children’s future educational achievements. Inappropriate storage of such data in an environmentally hazardous context also increases the risk of losing such important data and diminishes the possibility of being able to generate strong, precise impact statistics about the effects of pre-primary education up to class 3.

Recommendation #3a: BRAC, BSA, and UNICEF should work together to create a web-based and or soft copy information system which includes both children’s administrative records (attendance) and achievements (results of child indicators). The current data collected at the level of the learning centres is not resilient (or in soft-copy form) against all the threats of the environment of the programme area (i.e., saline air, floods, severe rain), and this challenge should be addressed. Recommendation #3b: BRAC, BSA, and UNICEF should jointly work on an information system that also maintains an individual record for each child’s progress and comprehensive assessment(s) in a web based and or soft copy system. The information system should enable the information for each child’s progress to follow him/her into primary school. Each individual child’s progress should be tracked, as the she/he progresses to higher education levels or drops out until class 3. Without a robust data storage system, the potential longer-term effects of ECD services on children in Bangladesh may continue to go unexamined.

3.4.3 Sustainability and Scalability

This section discusses the degree to which there is potential for the continuation of benefits (long) after the LUL initiative has been completed, with attention to start-up or developmental costs (e.g., curriculum design and development) as well as to partners’ priorities, policies and demand on the part of local institutions. Scalability also refers to the extent to which a small-scale or pilot activities are likely to be expanded to reach larger populations (if applicable). Unless otherwise presented earlier, the remaining six Evaluation Questions in table 7 will be addressed. Table 7: Evaluation questions for the sustainability / scalability evaluation criteria



Sustainability and Scalability

• How does the initiative relate to MoE or other partners’ priorities?64

• What are unit costs for the initiative and what are their ramifications for scaling?

• What current or potential partners are likely to have interest in scaling (or replicating) the initiative?

64 The response to this question is presented under the relevance section of this report.

35



• In what ways have activities resulted in outputs (e.g., learning resources, administrative processes, etc.) that will reduce costs of scaling or replicating?65

• How enduring are the changes that have resulted from the initiative? And, similarly, what are their potential longer-term effects?

• How could the sustainability of the initiative be improved?

Toward a Sustainable Delivery of Pre-Primary Education

The government (MoPME and MoWCA) has the duty of providing pre-primary education. This obligation will be fulfilled when pre-primary education is fully “mainstreamed”, thereby contributing to the future sustainability of an ECD service delivery system. Policy implementation will, however, take place, but the timeline is uncertain. The evidence suggests that the LUL-funded learning centres are likely to be sustainable in the future. As the government is in the process of ‘mainstreaming” ECD services but has not but not yet covered the country in full, the benefits observed from the LUL intervention will likely be sustained. As noted earlier, the GO-NGO collaboration guidelines (2011) are very important. These coordination proposals provide the overall scope for the institutionalisation of the pre-primary education system, including but not limited to: (a) further policy development; (b) networking and committee work; (c) capacity strengthening for teachers; (d) service delivery standards; (e) administration; (f) supervision/monitoring, evaluation, and reporting; (g) and resource mobilisation; and (h) any new areas which could be addressed through effective consultation among all relevant partners. Essentially, all government and nongovernment partners are intended to work jointly in order to effectively deliver ECD services, so that all Bangladeshi children aged five years or older are prepared for formal primary education. Similarly, stakeholder verbal confirmation serves as a complementary signal of the government’s intention to support ECD service delivery. For example, government (MoPME-Central and Sub-district levels, MoWCA-Central level, and BSA-Central and Sub-District levels), BRAC-Central level, and UNICEF (HQ, Central and Field levels) noted that: they (1) have intention to follow the guidelines and (2) understand that efforts are to be undertaken in a phased manner. Interviews with BRAC-Head office partners also affirm that there is not only a strong commitment to mainstreaming access to ECD services, but there is also clear enthusiasm for improving the quality of ECD services.66

Unit Costs for the LUL-Funded Learning Centres

The temporary continuation of supporting the rights holder children who face barriers to accessing pre-primary education will still be needed until the government’s pre-primary education system is fully established. The USD 35 to 40 per child/year unit cost is a modest price for providing rights holder children with some school-preparedness. The ramifications for temporarily continuing support would likely be minimal. The scale of communities considered “hard to reach”, or 37% of the 400 LUL-funded learning centres (or less than 150 learning centres), is relatively small. When calculating costs, it would be prudent to take into account that there may be a difference in costs for BRAC to service the LUL-funded learning centres in remote areas compared to those located in communities in non-remote areas. Overall, however, it is recognised that the strategic support extended through LUL is not intended to serve as a substitute for the government’s efforts. It is expected that the effective

65 This question is not relevant in the case of Bangladesh, because the service that has been delivered is part of a larger education system which is already in the process of being institutionalised. The main innovative aspect of the funds is that pre-primary education is being delivered to hard to reach, rural communities. 66 Education Watch. (2013) New Vision Old Challenges: The State of Pre-primary Education in Bangladesh. Funded by Campaign for Popular Education (CAMPE) and the European Union Delegation to Bangladesh.

36

delivery of the pre-primary education system will eventually be fully mainstreamed by the government in Satkhira District, including in the hard to reach areas. The support would therefore be temporary.

Recommendation #4: The time period when the government (MoPME) will effectively complete its mainstreaming efforts is not definite. Given that there will be a phased manner in which the government (MoPME) will deliver equitable ECD services to children, LUL-funds should be used to support the government (MoPME) and BRAC to extend its outreach to the hard to reach communities for a temporary time period through the end of PEPD3 or through the 2016-2017 school year.

Data Availability for a “Retroactive Baseline”

One of the key functions of future evaluation efforts for the LUL initiative will be to provide strong evidence on a full evaluation scope (i.e., effectiveness, efficiency, and impact). This exercise should also draw from reliable data sources. The duration of implementation did not allow for an in-depth examination of the longer term impact that access to pre-primary education may have had on rights holder children in Khulna. The lack of: (1) baseline information; and (2) supportive data for a counterfactual also contributed to an imprecise indication of the situation before the LUL initiative was implemented. At this stage in the evaluation, there are, however, several information sources that can serve as baseline information and support future evaluation efforts. These resources include:

1. Government of the Peoples' Republic of Bangladesh, Ministry of Primary and Mass Education (MoPME), Monitoring & Evaluation Division, Directorate of Primary Education (2012) Annual Primary School Census.

2. BRAC Monitoring & Evaluation Data; and 3. UNICEF MICS Data.

37

4 CONCLUSIONS, RECOMMENDATIONS, AND

LESSONS LEARNED

4.1 Conclusions

School readiness for rights holder children aged 5 years or older is paramount, and the tireless efforts of the Government of the People's Republic of Bangladesh, UNICEF’s Bangladesh offices, and BRAC to increase access to pre-primary education in disaster-prone Satkhira District have been evidenced. The strength of this initiative is that it targets and creates access to rights holders where none may have existed before. The learning centres were established to address physical, environmental challenges which presented multiple barriers to access. Through the collaboration of the implementing partners, the evidence shows that the expected results of this project have been exceeded by far. Relevance was found to be strong as well as effectiveness, and there is a clear indication that the government has plans to mainstream access to ECD services for all Bangladeshi children; however, it is understood that such a vision could not occur in a short period, and therefore a phased approach has been initiated. A total of six major findings, based on multiple information sources, have been presented and discussed; the five action-oriented and evidence-based recommendations are, in turn, designed to continue promoting access to and to strengthen the equity focus of the LUL initiative.

4.2 Recommendations

The following evidence-based recommendations are presented in order of priority.

• Strategic Targeting

Targeting future funding into remote areas ought to be taken into consideration, particularly where the government is least likely to complete its mainstreaming strategy on time to provide ECD services.

The time period when the government (MoPME) will effectively complete its mainstreaming efforts is not definite. Given that there will be a phased manner in which the government (MoPME) will deliver equitable ECD services to children, LUL-funds should be used to support the government (MoPME) and BRAC to extend its outreach to the hard to reach communities for a temporary time period through the end of PEPD-III or through the 2016-2017 school year [recommendation #4 in report].

• Equity-focused Programming

Plans to address how best to institute an inclusion policy for children with disabilities should be considered at an early stage, consequently strengthening the equity focus. Given that the primary objective is the provision of ECD services in the hard to reach and remote areas, consideration for delivering the monitoring-mentoring services in an equitable manner ought to be made.

It is uncertain what the proportion or number of children with disabilities in the learning centres is for children with disabilities. However, BRAC should provide teachers with training in accepting an inclusion policy and then dealing with the special needs of children with disabilities. Teachers providing these services in the classroom would effectively support the delivery of pre-primary education with an equity focus [recommendation #1 in report]. BRAC, BSA, and UNICEF should work together to support the reorganisation of BRAC field staff’s time and existing accountability systems to ensure that all teachers receive the same number of monitoring and mentoring visits per month. The provision of the same number of monitoring

38

visits, especially to learning centres in the remote areas, will effectively strengthen the equity focus [recommendation #2 in report].

• Data Collection for Child Assessment

To ensure that wind patterns and storm cycles, which create floods, cyclones, and other physical environmental threats endemic in Satkhira, do not compromise the storage of valuable information about children’s attendance and school and their developmental progress, the strategies designed to guarantee reliable data storage ought to be prioritised. Having information from individual children’s learning tends to contribute to the production of impact statistics linking the effects of pre-primary education to lifelong opportunities for growth. BRAC, BSA, and UNICEF should work together to create a web-based and or soft copy information system which includes both children’s administrative records (attendance) and achievements (results of child indicators). The current data collected at the level of the learning centres is not resilient (or in soft-copy form) against all the threats of the environment of the programme area (i.e., saline air, floods, severe rain), and this challenge should be addressed [recommendation #3a in report]. BRAC, BSA, and UNICEF should jointly work on an information system that also maintains an individual record for each child’s progress and comprehensive assessment(s) in a web based and or soft copy system. The information system should enable the information for each child’s progress to follow him/her into primary school. Each individual child’s progress should be tracked, as the she/he progresses to higher education levels or drops out until class 3. Without a robust data storage system, the potential longer-term effects of ECD services on children in Bangladesh may continue to go unexamined [recommendation #3b in report].

4.3 Lessons Learned

The Lessons Learned for the evaluation of the LUL initiative in Bangladesh are: 67

• A targeted approach to the delivery of ECD services with an equity focus takes time, and this challenge is particularly compounded when UNICEF’s partnerships involved in the delivery of ECD are complex. The fact that expected results were exceeded in this project is a reflection of a monumental efforts undertaken by all partners. However, sustained efforts will take time. Time is also required in order for a coherent approach to implementing ECD services to take place; and

• The communities delivering pre-primary education, which are located in remote areas, are considerably more vulnerable compared to the communities are located in centralised areas. Strengthening the equity-focus in pre-primary education becomes even more critical, because the hardest to reach communities are most likely to require considerably more efforts, time, investments, and resources.

67 Definition from the Global Evaluation Reports Oversight System (GEROS) indicators which highlight that the Lessons Learned “are contributions to general knowledge. They may refine or add to commonly accepted understanding, but should not be merely a repetition of common knowledge.” GEROS. (2013), page 25.

39

REFERENCES

ALNAP (2006). Evaluating humanitarian action using the OECD-DAC criteria: An ALNAP guide for humanitarian agencies. London: Overseas Development Institute. Accessed at: http://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CB8QFjAA&url=http%3A%2F%2Fwww.alnap.org%2Fmaterial%2F78.aspx&ei=w4-zU_3HGYi6oQSAsoHgBQ&usg=AFQjCNHapoTYmS8TMyLbjRxuCfxo3IN2pg&bvm=bv.70138588,d.cGU.

Bamberger M and Segone M (2011) How to design and manage Equity-focused evaluations, UNICEF Evaluation Office. Accessed at: http://mymande.org/sites/default/files/EWP5_Equity_focused_evaluations.pdf

Bernard van Leer Foundation. (2006) General Comment No. 7, ‘Implementing Child Rights in Early Childhood’, Committee on the Rights of the Child, Forty-first session held in Geneva; The Hague, The Netherlands.

Better Evaluation, (2014) “DAC criteria.” Accessed at: http://betterevaluation.org/evaluation-options/dac_criteria

BRAC. Accessed at: http://education.brac.net/pre-primary-schools.

Education Watch. (2013) New Vision Old Challenges: The State of Pre-primary Education in Bangladesh. Funded by Campaign for Popular Education (CAMPE) and the European Union Delegation to Bangladesh. Accessed at www.campebd.org

Chianca, Thomaz (2008). “The OECD/DAC Criteria for International Development Evaluations: An Assessment and Ideas for Improvement”, Journal of Multidisciplinary Evaluation, Vol. 5, No. 9 (2008), pp. 41-51.

Convention on the Elimination of all Forms of Discrimination against Women-CEDAW (1979).

Convention on the Rights of the Child-CRC (1989).

Organisation for Economic Cooperation and Development- Development Assistance Committee (1991). Principles for evaluation of development assistance. Paris: OEDC.

Folke, C., Carpenter, S., Elmqvist, T., Gunderson, L., Holling, C.S., and Walker, B. (2002). Resilience and sustainable development: Building adaptive capacity in a world of transformations. AMBIO: a journal of the human environment 31(5):437-440.

Institute of Child & Human Development (ICHD) (2013). EFA Promising Practices in the Asia-Pacific Region: A case study of initiatives to accelerate progress in Bangladesh. Pre-Primary (PPE) and School Learning Improvement Plan (SLIP).

Government of the People’s Republic of Bangladesh, Ministry of Planning, Bangladesh Bureau of Statistics (BBS), and UNICEF (2014). 2012-2013 Bangladesh Multi-Indicator Cluster Survey (MICS): Key Findings.

Government of the People’s Republic of Bangladesh, Ministry of Planning, Bangladesh Bureau of Statistics (BBS), and UNICEF (2013). Child Equity Atlas: Pockets of Social Deprivation in Bangladesh.

Government of the People's Republic of Bangladesh, Directorate of Primary Education (2013) Guideline on the GO-NGO collaboration for universal Pre-Primary Education (PPE) Bangladesh.

Government of the People's Republic of Bangladesh, Directorate of Primary Education, Power and Participation Research Centre (PPRC), and UNICEF Bangladesh (2013). Bangladesh Primary Education Stipends: A Qualitative Assessment.

Government of People’s Republic of Bangladesh, Directorate of Primary Education. With technical assistance from UNICEF (2012). Pre Primary: Education Expansion Plan.

40

Government of the Peoples' Republic of Bangladesh, Ministry of Primary and Mass Education (MoWCA), Monitoring & Evaluation Division, Directorate of Primary Education (2012) Annual Primary School Census.

Government of the Peoples' Republic of Bangladesh, Ministry of Women and Children Affairs (2011) National Children Policy.

Government of People’s Republic of Bangladesh, Ministry of Primary and Mass Education (2008). Operational Framework for Pre-Primary Education.

Independent Evaluation Group and DAC (2006). “Impact evaluation: An overview and some issues for discussion” (Working-group discussion document). Washington, DC: World Bank. (http://www.cgdev.org/doc/eval%20gap/HowardWhiteDAC.pdf, accessed 29 June, 2014)

Norwich, B. (1994). ‘The relationship between attitudes to the integration of children with special educational needs and wider socio-political views: a US–English comparison’, European Journal of Special Needs Education, 9, 91–106.

Rogers, E.M. (1983). The diffusion of innovation. New York: Free Press.

UNEG (2011). Integrating human rights and gender equality in evaluation—Towards UNEG guidance.

UNESCO (2009) Overcoming inequality: Why governance matters. Education For All Global Monitoring Report 2009. Oxford: Oxford University Press.

UNESCO (2008) Education for all by 2015 - Will we make it? Education for All Global Monitoring Report 2008. Paris: UNESCO.UNICEF BCO (2013). Basic Education and Equity Progress Report (Year 2).

UNESCO (2007) Strong foundations: early childhood care and education. Education for All Global Monitoring Report 2008 (Paris: UNESCO).

UNESCO (2004) Enrolment gaps in pre-primary education: The impact of a compulsory attendance policy.

UNESCO Policy Brief on Early Childhood 21. Paris: UNESCO.

UNESCO (2000) The Dakar Framework for Action: Education for all: Meeting our collective commitments. Paper given at World Education Forum, Dakar, Senegal, 26-28 April 2000.

UNICEF (2014). Agenda: LUL evaluation data-review.

UNICEF BCO (2013). Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira District. Progress Report for the German National Committee for UNICEF, p. 7.

UNICEF BCO (2012). Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira District. (SC/2011/0395) Progress Report for the German National Committee for UNICEF.

UNICEF BCO (2012) Education Equity for Out-of-School-Children. A Pilot Initiative on Second Chance Education in Shatkhira. Proposal for Project Cooperation Agreement between UNICEF and Jagoroni Chakra Foundation (JCF).

UNICEF (2012). Early Childhood Development and Disability: A Discussion Paper with UNICEF and the World health Organisation (WHO). Accessed at: http://www.ecdgroup.com/pdfs/ECD-Disability-UNICEF-WHO-2012.pdf

UNICEF BCO (2011). Programme Proposal: Equity in education: piloting second chance education for out-of-school-children in disaster prone Satkhira District. Submitted to the German National Committee.

UNICEF and Centre for Policy Dialogue (2011). National Budget: Are The Commitments To The Children Of Bangladesh Being Kept?

UNICEF and UNESCO. (2007) A Human Rights-Based Approach to Education for All.

41

UNICEF (2010) Getting Ready for School: A Child-to-Child Approach. Programme Evaluation: Year One. New York: UNICEF.UNICEF BCO (2013). Livelihood Skills Training for Out-of-School Working Children in Satkhira. Proposal of UNICEF and BRAC Partnership, period: January 2014 – February 2015.

UNICEF (2009) SITUATION ASSESSMENT AND ANALYSIS OF CHILDREN AND WOMEN IN BANGLADESH 2009.

USAID (2012). USAID evaluation policy: Evaluation, learning from experience. Washington, DC: USAID. New York: UNEG.

42

ANNEXES

Annex 1 Terms of Reference

Title Terms of reference for the evaluation of Let Us Learn Initiatives (LUL) in Bangladesh, and scope of work (SOW) for (National Evaluator)

Purpose To provide technical support for the evaluation of Let Us Learn Initiatives in Bangladesh and contribute to the global evaluation

Contract Evaluation Timeline July 2014 through 26 December 2014

A. Background The Let Us Learn initiative (formerly known as the Basic Education & Equity initiative) is a unique private partnership that allows for flexible and innovative approaches to addressing inequities in education access and outcomes. Challenges and barriers to education, particularly amongst excluded and marginalized children and youth have been identified in five diverse country contexts, Afghanistan, Bangladesh, Bangladesh, Madagascar and Nepal. Each country is targeting the hardest to reach children by sharpening the equity focus in both programming and monitoring of results. 20 million USD have been designated for the period 2011-2014. The programme is focused on three equity pillars: reaching out-of-school children, expanding girls’ education and improving quality outcomes for learners. Each of the participating Country Offices is working to address inequity through targeted approaches, adapted to address their specific contexts. Each programme is conducting a country-level evaluation that will also contribute and inform a global-level evaluation. Bangladesh: The project is targeting three key areas, namely (1) improved access to Pre- Primary Education (PPE) for school readiness for children 5 years old. The other two interlinked components are: (2) Ability-Based Accelerated Learning for OOSC children 8-14 years old; and (3) Communication for development and social mobilization for OOSC 5-14 years old. Vocational skills training for OoSC has been started since February 2014 as a new component of the project. BRAC is implementing the PPE component in Satkhira district since 2012. Currently, about 12,000 children are receiving PPE service through 400 learning centres at Satkhira funded by LUL. Overall, this evaluation will focus on the following areas of the PPE component:

� PPE service delivery in terms of availability, accessibility, utilization, adequate and effective coverage through comprehensive district coverage;

� Coordination among PPE service providers including the Government to ensure the optimum district coverage of PPE service

� Narrowing down the PPE service gaps in the context of equity � PPE as a foundation course for life-long learning in terms of: � relevance – curriculum, teaching learning package and process for the targeted children of 5

years � effectiveness – learning achievements, smooth transition to Grade 1, coping with Grade 1

curricula and teaching learning process at primary school B. Rationale for the evaluation The partners involved in launching Let Us Learn agreed that an evaluation would be carried out at the end of the programme. The evaluation effort is intended to ensure accountability, strengthen any future

43

programming and contribute to the global equity dialogue. In order to capture information across a broad range of goals, a series of evaluation activities are being proposed. Evaluative activity is broadly defined to include evidence gathering, performance monitoring and measuring results. Outside of the technical assessment, consideration will also be given to improving organisational effectiveness by examining development and implementation processes, as well as programme supervision. One primary focus of evaluation will be to engage all COs and HQ in a learning process around the Let Us Learn (LUL) initiative, documenting new efforts in monitoring for equity, innovations, partnership and good practice. A series of 5 discrete evaluations and a synthesis/consolidation are planned, along with participatory products such as testimonials and ‘most significant change’ stories involving beneficiaries. We are aware that some of these analyses may be be challenged by a lack of baseline data and varied programme approaches across all of the participating COs. Activities should build evidence to fill information gaps and inform any next phase of the LUL, thus retroactive baselines may need to be established to measure change. Mixed methodologies will be utilized given the diverse range of activities taking place across COs but should fall within a global methodological framework. The initial audience for the country-level evaluation products will be internal with discussions held on what elements best contribute to the global evaluation. Each country office will have a dedicated evaluator to review their programmes. These evaluators will work closely with the lead evaluator at the global level to integrate efforts into the global evaluation. The country level evaluations should be complete in September and the final evaluation product needs to be completed by December 2014. C. Evaluation approach and scope Activities included and excluded from evaluation. The self-evaluation study will examine only activities/results that are agreed upon with the respective country offices. Processes related to planning, coordination and monitoring will be included. Selected programme activities along each of the three pillars of the programme will be included as will thematic analysis related to equity and innovation. Attribution and/or contribution. While evaluating outcomes also often determines if a programme/project has added value to country-level results as stated in the Country Programme, it is often difficult to attribute results to only one source of inputs, actions, or actors in a field where there are multiple actors and inputs. Therefore the evaluation will seek to outline, to the extent possible, LUL’s contribution to overall results through a contribution analysis. From a country perspective, the objectives of the evaluation are threefold;

• To examine the extent to which LUL projects are achieving intended outcomes at the country level; and,

• To systematically document new learning, and to document new efforts in monitoring for equity, in particular.

• To contribute to the global evaluation and analysis of cumulative impact.

The proposed approach is to conduct a self-evaluation68 exercise in Bangladesh with a primary focus on learning and a secondary focus on accountability, utilizing a common evaluation/learning framework. Methodologies and scope will be determined in consultation with the Country Office and with technical inputs from the lead evaluator. Evaluation Criteria and questions: Since this is an evaluation of innovation (equity innovations), the OECD/DAC criteria should be used only to the extent that they are relevant. However, we expect the

68 This is an evaluation that it is intended only partly for accountability, and more for the purpose of learning. Secondly, the evaluation management arrangements do not aim for a high level of independence at the country level. Ideally, the M&E Officer will be the primary manager, but the situation in the country may be such that the programme officer (Education) is the only person available to manage the consultant.

44

evaluator to propose criteria that is more appropriately suited to evaluation of innovations. Similarly, the evaluation criteria will be the basis for developing evaluation questions. Attribution and/or contribution. While evaluating outcomes also often determines if a programme/ project has added value to country-level results as stated in the Country Programme, it is often difficult to attribute results to only one source of inputs, actions, or actors in a field where there are multiple actors and inputs. Therefore the evaluation will seek to outline, to the extent possible, LUL’s contribution to overall results through a contribution analysis.

- Methodology

Sample: A draft stakeholder analysis has been conducted as part of preliminary planning for the evaluation, hence a sampling framework that reflects a range of activities, stakeholders and will be developed for each country. Detailed information on sampling will be included in the inception report. Design and data collection methods: Data will be collected from multiple sources, using both qualitative and quantitative methods. A menu of methods and sources include a review of programme documents, interviews of key/elite informants, interviews and/or focus groups of important groups of beneficiaries, testimonials involving beneficiaries (One Minute Jrs.), and direct observations of activities with beneficiaries wherever possible. A retroactive baseline will be established in consultation with programme staff and relevant education authorities to assess to extent possible contribution to quality of education (leaning outcomes, reduction in violence, etc. as per the Child Friendly Schools framework).

• Review of documents and procedure: A desk-based review will be used as an important tool to examine the theory of change, restate and/or amend it to reflect what happened in practice (empirical ToC); assemble evidence of activities or outputs. LUL focal points in each country will avail key documents for a systematic desk review. These will include LUL programme documents (proposals, monitoring frameworks, reports, human interest stories, etc.), sector plans, other education evaluations and assessments as relevant.

• Data Collection: Primary data – to be collected through interviews and focus groups, self-administered surveys, and possibly small scale assessments – will add to the existing knowledge about the activities, and will be used to gauge perceptions of stakeholders on the utility and impact of interventions.

- Evaluation responsibilities and management The LUL Bangladesh evaluation will be supported by:

• A national evaluator

• An in-country manager and reference group, responsible for technical supervision of the evaluator, and for clearance of all evaluation products, respectively;

• An international Lead evaluator (ILE), who will provide technical guidance on the overall approach to enable learning across countries; and,

• The LUL global coordinator and Evaluation Office in HQ, on administrative matters, supervision of the international lead Evaluator, and overall quality assurance.

The National evaluator (NE) in Bangladesh will be responsible for the professional conduct of the evaluation in accordance with the terms of reference, the United Nations Evaluation Group (UNEG) Norms and Standards for Evaluation69 and the UNEG Code of Conduct for Evaluation70. He/she will be expected to perform the following tasks:

• review and finalize the evaluation approach and work plan;

• customize, review and validate data collection tools, (the self-evaluation module, which may include questionnaires, interview questions and protocols);

69 http://www.uneval.org/papersandpubs/documentdetail.jsp?doc_id=22

70 http://www.unevaluation.org/documentdownload?doc_id=100&file_id=547

45

• supervise data collection and data processing;

• design and facilitate consultative workshops and meetings;

• prepare all country reports – inception, draft and final reports, and develop PowerPoint presentations;

• convene a forum to validate the findings of the evaluation;

• attend and make a presentation at the global reporting workshop; and,

• ensure that the International Lead Evaluator and reference group is regularly informed of the progress of the evaluation, any possible causes of delays and issues to resolve.

The Evaluation Manager (ideally the Monitoring and Evaluation Officer) will provide overall guidance and contribute directly to its quality assurance activities. The Bangladesh LUL country team will support the self-evaluation by providing support to arrange for meetings, write letters of introduction and/or accompany the evaluator to meetings where necessary. Reference Group: Our recommendation is to establish a reference group in each participating country. Membership will be determined within country for each of the five country offices. The Reference Group Chair is responsible for oversight of evaluation activities, while members will advise on the following:

• Consultant’s terms of reference and work plan, including any adjustments required at inception or other phases

• Approval of all evaluation products, including the final report

The international lead evaluator (ILE), contracted by the UNICEF Evaluation Office, will be responsible for leading the evaluation efforts from a global perspective. Specifically, he/she will be responsible for the following:

• develop the evaluation approach and methodology, including the evaluation/learning framework;

• develop a self-evaluation module/guide and other materials as he/she sees fit;

• facilitate a virtual planning workshop for national consultants and focal points in each participating country;

• pilot the evaluation methodology in one country, and effect the necessary revisions;

• review data collection tools and protocols used at the country level;

• undertaking quality assurance missions during data collection in up to two participating countries;

• review country reports - draft and final reports;

• ensure that the evaluation managers (Evaluation Specialist and Education Specialist in New York) are regularly informed of the progress of the evaluation, possible causes of delay and issues to resolve.

The Evaluation Specialist and Education Specialist (in New York) will manage the work of the international lead evaluator, provide overall guidance to the evaluation, and contribute directly to its quality assurance activities. - Evaluation processes and products Evaluation Planning Workshop (via Webinar): The International Lead Evaluator will develop initial draft of the evaluation approach, and convene an orientation and planning webinar for the national evaluators, to finalize the approach to the evaluation. The planning workshop is expected to agree on the methodology, including the sampling frame for each country, a set of evaluation tools with necessary customization to each country, and a tentative work plan that reflects all evaluation activities for country teams and for the international lead evaluator. Inception Reports: An inception report will be produced at the global level by the International Lead Evaluator, and adapted/customized for each country by the National Evaluator. In addition to presenting a short summary of the programme context, these reports will be used to confirm a common understanding of the description of what is being evaluated, and the logic or theory of change for each programme. The report will include, inter alia,

46

Evaluation purpose and scope – a clear statement of the objectives of the evaluation and the main aspects and limitations of the evaluation Evaluation criteria and questions – question to be answered by the evaluation, and the criteria which will be used to assess performance, including questions that address critical human rights and gender equality issues Evaluation methodology – expanding on the methodology section in the TOR, a sampling strategy, a description of data collection methods and data sources (including a rationale for their selection), draft data collection instruments, a discussion on reliability and validity of the evaluation, and a discussion on the limitations of the methodology. This section should include instruments to assess relevant human rights and gender equality aspects. Evaluation matrix and analysis plan – a mapping that identifies evaluation questions, how they will be answered through the selected methods, and a data analysis plan; Evaluation work plan and timeline – a revised work and travel plan Structure for the final report will be proposed in the inception report, and may be revised later in the evaluation Resources requirements – detailed budget for the evaluation, tied to evaluation activities, work plan, deliverables. The inception report will be 10-15 pages, including the revised work plan, and will be presented at a formal meeting of the in-country reference group, and LUL extended management team (HQ Manager plus Chiefs of Education/LUL focal points in participating countries). Evaluation Report: The final evaluation report should include, but not limited to the following 1. An analysis of concepts, trends and critical themes in innovating solutions for equitable provision of

education in respective contexts; 2. An analysis of overarching programme goals and themes (OOSCI, equity, innovation, etc.), nothing

those with most demonstrable impact; 3. An overview of each of the country evaluations, analyzing trends, areas of divergence and any

lessons learned; 4. An assessment of UNICEF’s mandate, strengths and weaknesses relating to UNICEF’s strategic and

programmatic choices for equity, against a set of agreed evaluation criteria; 5. An analytical framework which UNICEF can apply to improve the effectiveness of its support for

equity interventions, at the country level, and an assessment of the role of the regional and global offices in support of equity in education;

6. Derived from the findings of the evaluation, recommendations for improving coordination, and successful scale up of equity focused programming.

The evaluation report should not exceed 30 pages (excluding the executive summary and annexes). Annexes will include the TOR, description of methodology (including evaluation/learning framework), list of background materials used, list of people interviewed, PowerPoint presentations, and workshop materials. - Desired Competencies for Evaluator Evaluator must offer the following demonstrated experience, knowledge and competencies:

• Significant knowledge and experience of evaluation concepts and approaches;

• Good knowledge in gender in the Education Sector, and experience in evaluating equity and education innovations;

• Facilitation skills, particularly design of stakeholder consultation exercises as well participatory methods;

• Strong quantitative and qualitative data collection skills;

• Strong quantitative and qualitative data analysis skills;

• Excellent language and communication skills in English;

• Demonstrated report writing skills, in English; and

• Computer literacy in Word, Excel and PowerPoint;

47

Evaluator must remain in strict adherence with UNEG ethical guidelines and code of conduct. - Proposed resource commitments It is anticipated that the resource commitments would as follows:

• 50-60 person days for National Evaluators in each country; DSA for international and/or in-country travel

• Travel costs for one international mission and 4-6 days DSA for national consultant (reporting workshop)

• Travel costs for one international mission and 4-6 days DSA for UNICEF/LUL Bangladesh focal point or UNICEF M&E Special (reporting workshop)

- Tasks for the National Evaluator (Bangladesh) The National Evaluator will be responsible for leading the evaluation for the LUL Bangladesh Phase 1. Specifically, she will be responsible for the following:

• Adapt and customize the evaluation approach and methodology, and learning framework to the Bangladesh context, and develop and work plan;

• Develop draft evaluation tools aligning them with the overall evaluation methodology;

• Participate in the piloting of the evaluation methodology in Bangladesh, and update the tools accordingly;

• Execute the evaluation in Bangladesh; collect and analyse data as per the agreed methodology;

• Participate in reference group meetings, including a dissemination meeting where s/he will present the evaluation findings.

• Draft country reports and/or updates - draft and final reports;

• Ensure that the evaluation manager and lead evaluator is regularly informed of the progress of the evaluation, possible causes of delay and issues to resolve.

- Timeline, time allocation and deliverables

Task Output/ deliverables

Person days

Deadline

Participate in conceptualization and development of evaluation design and approach (learning framework, methodology, work planning, country-evaluation module, etc.)

Bangladesh inception report (first draft)

4 days July, 2014

Mission to Bangladesh: Inception meeting for consultations with the LUL team, KII with team

Bangladesh inception report, including trip report as appendix (second draft)

2 days August, 2014

Training of additional data collectors; pilot testing of the evaluation instruments in Bangladesh, and data collection, KII with partners and stakeholders as possible

Bangladesh evaluation instruments

12 days August, 2014

Participate in the webinar sessions to standardize evaluation approach across LUL participating countries; revision of inception report

Final Inception report, including Bangladesh instruments and work plan

1 day July, 2014

Technical backstopping with country office on outstanding data collection activities; data analysis and report writing

Bangladesh evaluation report (zero draft)

10 days Sept, 2014

48

Task Output/ deliverables

Person days

Deadline

Mission to New York: Quality review and reporting meeting:

Bangladesh evaluation report – quality reviewed (1st draft)

6 days Sept 2014

Mission to Bangladesh: FGD and KIIs during OMJ workshop; more data analysis and report writing *as needed and requested by BCO

Bangladesh evaluation report (first draft); PPT presentation (first draft)

10 days Sept/Oct 2014

Revision of reports and final reporting Bangladesh evaluation report (final draft); Final PPT presentation

10 days Oct 30, 2014

TOTAL 55 days

In summary, the national evaluator for Bangladesh is responsible for the following deliverables:

1. Inception report: This report will include, among others, the revised evaluation approach and methodology, revised evaluation work plan, country-evaluation module and other materials for the orientation and planning webinar and facilitating the workshop;

2. Bangladesh evaluation reports: These will be developed by the national evaluator - the ILE will provide quality control/review;

3. Evaluation synthesis report: First, second and final drafts, according to the UNICEF House Style and UNICEF standards for evaluation reports. These will be developed by the national evaluator - the ILE will provide quality control/review; and,

4. PowerPoint presentation for evaluation synthesis report: The national evaluator will be expected to present at 1-2 reporting/dissemination events.

49

Annex 2 Logical Framework

Level Indicators Baselines 3 Year Targets 6 Year Targets Means of

verification

OVERALL PROGRAMME RESULT: Out of school children 5-14 years old in the disaster prone Assunia and Shyamnagar Upazillas, Satkhira District have

improved access to quality pre-primary and primary education, and are better prepared for disasters ensuring their right to education is realized.

Program Component ONE: Improved access to pre-primary education for school readiness for children 5 years old.

OUTCOME:

1. Children aged 5 years

have improved access to

PPE education, are better

prepared for, and enroll in

grade one on time (age 6).

i. % of pre-primary

age children (5 years)

attending pre-primary

increased in the targeted

Upazillas.

ii. % of children (5

years) in the targeted

Upazillas enrolled in

grade one on time (age

6) increased.

i. xx% of children (5 years)

attending pre-primary in the

targeted Upazillas.

ii. xx% of children (5 years) in

the targeted Upazillas

enrolled in grade one on

time (age 6).

i. 50% of children

(5 years)

attending pre-

primary in the

targeted

Upazillas.

ii. 60% of children

(5 years) in the

targeted Upazillas

enrolled in grade

one on time (age

6).

i. 90% of children (5

years) attending

pre-primary in the

targeted Upazillas.

ii. 90% of children (5

years) in the

targeted Upazillas

enrolled in grade

one on time (age 6).

i. Programme

monitoring

reports and

baseline

survey.

ii. BBS MICS

iii. BBS CRM

iv. PEDP3 school

census

OUTPUTS:

1.1 Programme PPE

Learning Centers

(LCs) functional in

targeted Upazillas.

i. # of PPE LC functional

in targeted Upazilla.

i. 0 (zero) PPE LC functional

in targeted Upazillas.

i. 150 PPE LC

functional in

targeted Upazillas

i. 150 PPE LC

functional in

targeted Upazillas.

i. Programme

monitoring

reports.

1.2 PPE LCs equipped

with age appropriate

teaching and learning

materials.

i. % of PPE LC

equipped with age

appropriate teaching

and learning materials.

i. 0 (zero) PPE LC equipped

with age appropriate

teaching and learning

materials. (no PPE LC

open).

i. 150 PPE LC

equipped with age

appropriate

teaching and

learning materials.

i. 150 PPE equipped

with age

appropriate

teaching and

learning materials.

i. Programme

monitoring

reports.

1.3 Children aged 5 years

enrolled in PPE LCs

i. # of children enrolled

PPE LC.

i. 0 (zero) enrolled PPE LC. i. 13,500 children

enrolled in the

i. 27,000 children

attending the pre-

i. Programme

monitoring

50

Level Indicators Baselines 3 Year Targets 6 Year Targets Means of

verification

in targeted Upazillas. pre-primary LC. primary LC. reports.

1.4 Teachers of PPE LC

received training and

practice interactive

teaching learning

methods (drawing,

storytelling, reciting

rhymes, singing songs,

playing games and

showing pictorials)

with children in each

class room session.

i. # of trainers trained

ii. % of teachers trained

iii. % of teacher received

refresher training

iv. % of trained teachers

practising interactive

teaching learning

methods

i. 0 (zero) in PPE LC

ii. 0 (zero) in PPE LC

iii. 0 (zero) in PPE LC

ii. 0 (zero) teachers teaching.

(no PPE LC open).

i. 5 new trainers

trained

ii. 100% new

teachers trained

iii. 100% teachers

receive refresher

training (70 each

year starting from

Y2)

iv. 100% teachers in

PPE LC

practising

interactive

teaching learning

method with

children in class

room session.

i. 25 new trainers

trained

ii. 100% new teachers

trained

iii. 100% teachers

receive refresher

training (70 each

year starting from

Y2)

iv. 100% teachers in

PPE LC practising

interactive teaching

learning method

with children in

class room session.

i. Programme

monitoring

reports

1.5 Children completed

one year pre-primary

course from PPE LCs

enrolled in primary

schools at age 6.

i. % attendance in pre-

primary class.

ii. Pre-primary completion

rate with minimum

competencies increased.

iii. % of children from

ELCs enrolled in

primary schools

iii. 0% attendance (no PPE LC

open).

iv. 0% completion rate (no PPE

LC open).

v. 0% children from PPE

enrolled in primary (no PPE

LC open).

i. 90% attendance

in the PPE LC

ii. 100% of children

attending PPE LC

complete the full

year with

minimum

competencies.

iii. 95% graduated

children enrolled

in primary school

iv. 90% attendance in

the PPE LC

v. 100% of children

attending PPE LC

complete the full

year with

minimum

competencies.

vi. 95% graduated

children enrolled

in primary school.

i. Programme

monitoring

reports

51

Annex 3 Methodology: Sampling Criteria for Identifying PPE

Centres

Given the scope of existing PPE activities financed by LUL and the limited time in the Khulna Division, a maximum variation sampling criteria is being applied. Suggested potential criteria71 were sent to the CO for consideration, and subsequently a tool was developed with the final mutually-agreed sampling criteria. Figure 1 provides a partial view of this tool, including some the sampling criteria and actual PPE centres. The full list of sampling criteria includes:

1. Year of (school) establishment; 2. Union; 3. Upazilla (Assasuni/Debhata/Sadar/Kaligong/Kolaroa/ Shyamnagar/Tala) 4. Number of Students Enrolled (disaggregated by gender); 5. Number of Dropouts (disaggregated by gender); 6. Number of Existing Students (disaggregated by gender); 7. Number of Students Completed PPE (in previous years) (disaggregated by gender); 8. Number of Students Enrolled in mainstream school currently (disaggregated by gender); 9. Distance of Centre (in kilometers) from Upazilla; 10. Infrastructure of Centre (Semi-permanent/temporary); 11. Distance of targeted students (Catchment area within 1.5 KM/More than 1.5 KM); 12. Close to a primary school (yes or no); 13. Teacher gender (Female/Male) 14. Teacher Education (SSC/HSC/Graduate/Masters) 15. Teacher Experience in PPE (in years) 16. Teacher received Basic Training (Yes or No) 17. The union/village affected by natural disaster like cyclone/erosion/salty water logging etc? (Yes or

No); and 18. The PPE centre has easy communication accessibility with the union centre (road

condition/vehicle/travel time) (Yes or No). 71 The potential criteria included: (1) Characteristics of the targeted communities; (2) Characteristics of the target centres (infrastructure-related); and (3) Characteristics of the target centres (programme-related).

Figure 1: Partial Image of the Sampling Tool

52

PPE Centre Sample Size and Final Selection

From the 18 abovementioned sampling parameters, the three priority parameters considered were: 1 The year the school was established (2012, 2013, or 2014); 2 The union/village affected by natural disaster like cyclone/erosion/salty water logging etc? (Yes or

No); and 3 The PPE centre has easy communication accessibility with the union centre (road

condition/vehicle/travel time) (Yes or No). In addition, other key criteria considered include: 4 The highest number of centres in the different Upazilla (e.g., Assasuni, Debhata, Sadar, Kaligong,

Kolaroa, Shyamnagar, Tala) (to gauge representativeness); and 5 Centre performance which is based on the following criteria:

a. Number of Students Enrolled; b. Number of Dropouts; c. Number of Existing Students; d. Number of Students Completed PPE (in previous years); and e. Number of Students Enrolled in mainstream school currently.

Additional priority was given to teacher education levels and experience (measured in the number of years) and condition of school structure (temporary or semi-permanent). It should be noted that the sample of schools selected is in no way representative of the total number of LUL-funded PPE centres (almost 400 centres). From the priority sampling criteria, a total of nine PPE centres were initially selected. The selection of schools for evaluation activities is heavily biased toward the centres which are hard to reach, susceptible to flood, prone to disruptions in communication, and temporary building structures. The names of these PPE centres selected are presented in table 2. Further, a list of seven alternate centres to visit were also considered with similar criteria as noted in table 1. These alternate centres could be used, depending upon if: (a) additional time was available; or (b) a selected centre could not be visited for reasons beyond the evaluator’s control. This approach ensured that the limited time dedicated for data collection was maximized. Table 1: List of alternate LUL-funded PPE centres for community interviews

Year Established

Name of the school

Upazilla

Affected by

natural disaster?

Easy communication?

Infrastructure of Centre

2012 Buria Assasuni Yes No Temporary

2012 Hetailbunia Assasuni Yes No Temporary

2013 Tangrakhali

S.para Shyamnagar Yes No Temporary

2014 Jabakhali Shyamnagar Yes No Temporary

2014 Paranpur East Shyamnagar Yes No Temporary

2012 Basjaria Satkhira Yes Yes Semi

permanent

2012 Khagradagna Satkhira No Yes Semi

permanent

53

Table 2: List of selected LUL-funded PPE centres for community interviews

Year Established

Name of the school

Upazilla Affected by

natural disaster? Easy

communication? Infrastructure

of Centre Remarks

2012 Jamalnagar Assasuni Yes No Temporary

School is affected by environmental

elements with limited communication,

older, and has high number of enrolled

students

2013 Tangrakhali

Master para Shyamnagar Yes No Temporary

School is middle range, struggling

2013 Middle

Ataroi Tala Yes Yes Temporary

School is struggling and 3 girls dropped

out

2014 Paranpur

East Shyamnagar Yes No Temporary

Newer and struggling

2013 Sara

Laxmikhali Shyamnagar Yes No Temporary

Schools performing in spite of Isolation

2013 Gabura Shyamnagar Yes No Temporary Schools performing in spite of Isolation

2013 Kholsebonia Shyamnagar Yes No Temporary Schools performing in spite of Isolation

2012 Raipur Satkhira

Sadar No Yes

Semi-

Permanent

Highly educated teachers with high

experience in a semi-permanent

structure with strong enrolment

(excellent conditions) and great track-

record and centres with Excellent

Ratings (Older)

2012 Tongipur Kaligonj No No Temporary

54

Annex 4 Interview Instruments

INTERVIEW Guide: UNICEF CO / DO Staff

Criterion: Topic or Theme

Questions

Relevance

1 Relevance: Getting

started….

Please tell me about your Country Office’s activities in relation to the LUL funded PPE. Please describe your activities in relation to: Planning; Programme Design, Implementation, and Monitoring (and Evaluation) Please describe the connections /synergies (if any) to the other on-going programs also implemented by the BCO which may contribute to the delivery of LUL funded activities. This would include BCO activities supporting the implementation of the Ability Based Accelerated Learning (ABL) and skills training for out of school children and youth.

2 Relevance: Getting started

Please tell me about your own involvement with the PPE. What is your specific role?

3 Relevance: Challenges

In your opinion, what are the biggest challenges that students face in relation to education? How do these challenges affect student learning?

4 Relevance: Challenges &

equity

Are these challenges different for the most vulnerable, poorer students? If so, please describe these differences. (e.g., differences could stem from impact, intensity, frequency, etc.)


Equity

How does the LUL-funded PPE address challenges to education (isolation, children with disabilities)? Which of these challenges does the LUL-funded PPE address most successfully? (For each challenge named) In your opinion, is this challenge important? Please tell me why or why not.

6 Effectiveness: General

In your opinion, what are the most important changes brought about by the LUL-funded PPE? Why are these changes important?

7 Effectiveness: Equity

In what ways does the LUL-funded PPE specifically change conditions for the most vulnerable children in school? Why are these changes important?


What important challenges to the education of the most vulnerable does the LUL-funded PPE not address? How could the programme be changed to address these challenges?

9 Effectiveness: Learning & Innovation

The LUL-funded PPE may have been implemented differently in different schools. What are these variations (if any). Please describe the various effects and the importance that these variations may have. In your opinion, which of these are most important and most effective? [if applicable]

10 Effectiveness: Learning &

Innovation 2

What obstacles has the LUL-funded PPE encountered in relation to its launch and implementation? How have these obstacles affected the programme?

55

Criterion: Topic or Theme

Questions

Has the programme been changed to respond to these obstacles? If it has, please tell me how. How did the UNICEF CO/DO view or contribute to these responses? In what ways did other organizations contribute? How effective were these responses? And why (in your opinion)?

11 Effectiveness: Learning

Are there potential changes to the programme that might make it more effective, but that we haven’t discussed? If so, please describe these potential changes.

12 Sustainability What is the relationship of the LUL-funded PPE to MOPME/DPE’s policies and current priorities? Please describe UNICEF’s relationship with the both the MOPME/DPE. [Note to Tristi: Ask for any additional documents that may be helpful here] Please describe the role of the MOPME/DPE in supporting the LUL-funded PPE. How can the MOPME/DPE/ MoWCA best support the LUL-funded PPE in the next 3 years?

13 Sustainability Please describe UNICEF’s relationships with partners participating in the LUL-funded PPE. In what other programs and activities are you considered a partner in Khulna? Are there past partnerships that are important? If so, please describe these. Please describe the role of these partners in the LUL-funded PPE. In your perspective, in what ways has the partnership been effective in achieving the objectives of the LUL-funded PPE? In what ways has the partnership been ineffective (Probe: weaknesses in partnership(s), external influences beyond partners’ control, other thoughts)? Do you have any specific recommendations regarding partnerships in relation to the LUL-funded PPE?

14 Sustainability: Scalability

In your opinion, what are next steps for the LUL-funded PPE? Do you believe that the LUL-funded PPE should be scaled up in Bangladesh? If so, please describe the measures that you think would be necessary. Please describe UNICEF’s most-effective potential role, in your opinion, in scaling up the LUL-funded PPE. You have noted that the model that LUL-funded PPE in Satkhira demonstrates “coordinated planning and coverage by all stakeholders.” You note that the gap has been filled in by LUL initiative. Please elaborate more about this role. You also highlighted the coordination aspects. Please explain this as well.

15 Effectiveness: Wrapping up

Do you have specific recommendations with regard to the LUL-funded PPE? Do you have recommendations with regard to the LUL-funded PPE overall? Are there other aspects of the the LUL-funded PPE that are important, but that we haven’t talked about? If so, please describe these.

56

INTERVIEW Guide: Central & District Level MOPME and MoWCA Personnel

Criterion: Topic or theme

Questions


Please tell me about the MOPME and MoWCA’s relationship to the LUL-funded PPE. How were you (or other personnel) involved? Please tell me how does the LUL-funded PPE fit into the government of Bangladesh’s overall education strategy? Does the LUL-funded PPE connect to MoPME’s PPE initiative in the country. Please elaborate on how you see the connections to the government priorities? (Are these priorities formalized in any documents or otherwise adopted?)


In your opinion, what are the biggest challenges that students face in relation to pre-primary and primary education? How do these challenges affect student learning?


equity

Are these challenges different for the poorest and most vulnerable students? If so, please describe these differences. (Differences could stem from different impact, intensity, frequency, etc.)


Equity

How does the LUL-funded PPE address challenges the poorest and most vulnerable students? In your perspective, which of these challenges does the LUL-funded PPE address most successfully? (For each challenge named) In your opinion, is this challenge important? Please tell me why or why not.


What are the most important changes brought about by the LUL-funded PPE? Why are these changes important?


In what ways does the LUL-funded PPE specifically change conditions for the poorest and most isolated students currently attending pre-primary school? Why are these changes important?


What important challenges affecting the poorest and most vulnerable students does the LUL-funded PPE not address? How could the programme be changed to address these challenges?


What obstacles has the LUL-funded PPE encountered in relation to its launch and implementation? How have these obstacles affected the programme? Has the programme been changed to respond to these obstacles? If it has, please tell me how. How did your institution/office/ministry view or contribute to these responses? In what ways did others contribute? In your opinion, how effective were these responses? And why?

9 Effectiveness: Learning

Are there potential changes to the LUL-funded PPE that might make it more effective, but that we haven’t discussed? For example, please tell me your thoughts about the current coordination structure among GO/NGO partners for the LUL-funded PPE implementation. In your view, how can the coordination be changed to improve coverage and pre-primary education quality? Please tell me about what the ideal structure and partners would be and what should be the roles and responsibilities of partners?

57


Questions

Please describe any other areas where you see potential changes could take place.

10 Sustainability What is the relationship of the LUL-funded PPE to the MOPME and MoWCA policies and priorities? Please describe the MOPME and MoWCA’s education initiatives (especially in Khulna).

11 Sustainability Please describe MOPME and MoWCA’s relationship with UNICEF within the frame of the LUL-funded PPE. Is it a partnership? Describe to me how it works. In what ways could it be improved? In your view, how can UNICEF best support the LUL-funded PPE in the next 3 years?

12 Sustainability In what ways has UNICEF been effective? In what ways has UNICEF been ineffective? Do you have any specific recommendations for UNICEF in relation to the LUL programme?

13 Sustainability: Scalability

In your opinion, what are next steps for the LUL-funded PPE? Do you believe that the LUL-funded PPE should be scaled up in Bangladesh? If so, please describe the measures that you think would be necessary.


Do you have other specific recommendations with regard to the LUL-funded PPE? Do you have recommendations with regard to the LUL programme overall? Are there other aspects of the LUL-funded PPE that are important, but that we haven’t talked about? If so, please describe these.

58

INTERVIEW Guide: Central & District Level Partners (Non- MOPME and MoWCA)

Criterion: Topic or theme Questions

� Relevance: Getting started

Please tell me about your organisation’s relationship to the LUL-funded PPE; ABL, or skills training for OOSC (youth). How were you or other personnel involved in these LUL-funded activities? [Depending upon what is said, the evaluator will focus on the other LUL-funded activities. Otherwise, the interview will continue to focus on the LUL-funded PPE.] How does the LUL-funded PPE fit into your organisation’s overall strategy? Is this strategy reflected in policy or other documents?

� Relevance: Challenges

In your opinion, what are the biggest challenges that students face in relation to pre-primary and primary education? How do these challenges affect student learning?

� Relevance: Challenges &

equity

Are these challenges different for vulnerable students? If so, please describe these differences. (e.g., differences could stem from differences in impact, intensity, frequency, etc.)

� Relevance: Challenges &

Equity

How does the LUL-funded PPE address challenges to vulnerable student’s education? Which of these challenges does the LUL-funded PPE address most successfully? (For each challenge named) - In your opinion, is this challenge important? Please tell me why or why not.

� Effectiveness: General

What are the most important changes brought about by the LUL-funded PPE? Why are these changes important?

� Effectiveness: Equity

In what ways does the LUL-funded PPE specifically change conditions/outcomes (if examples are available) for vulnerable children in school? Why are these changes important?

� Effectiveness: Equity

What important challenges to the most vulnerable children receiving a quality pre-primary education does the LUL-funded PPE not address? How could the programme be changed to address these challenges?

� Effectiveness: Learning & Innovation

What obstacles has the LUL-funded PPE encountered in relation to its launch and implementation? How have these obstacles affected the programme? Has the programme been changed to respond to these obstacles? If it has, please tell me how. How did your organisation view or contribute to these responses? In what ways did other organizations contribute? How effective were these responses? And why?

� Effectiveness: Learning

Are there potential changes to the LUL-funded PPE that might make it more effective, but that we haven’t discussed? If so, please describe these potential changes.

� Sustainability Please describe your organisation’s relationship with the MOPME and MoWCA. Please describe how your organisation and the MOPME and MoWCA work in support of the LUL-funded PPE. In your view, how can the MOPME and MoWCA best support the LUL-funded PPE in the next 3 years?

59

Criterion: Topic or theme Questions

� Sustainability Please describe the relationship of your organisation to UNICEF. In what ways has UNICEF been effective? In what ways has UNICEF been ineffective? Do you have any specific recommendations for UNICEF in relation to LUL? In what other programs and activities are you partners? (this is the case for Khulna specifically) Are there past partnerships that are important? If so, please describe these.

� Sustainability: Scalability

In your opinion, what are next steps for the LUL-funded PPE? Do you believe that the LUL-funded PPE should be scaled up in Bangladesh? If so, please describe the measures that you think would be necessary. Without committing to anything in particular, do you believe that your organisation would be able to support scaling up of the LUL-funded PPE? If so, please elaborate.

� Effectiveness: Wrapping up

Do you have specific recommendations with regard to the LUL-funded PPE? Do you have recommendations with regard to the LUL-funded programs overall? Are there other aspects of the LUL-funded PPE that are important, but that we haven’t talked about? If so, please describe these.

60

INTERVIEW Guide: School Teachers72


Questions

Relevance


Please tell me about your experience and education as a teacher. How long have you been teacher at this pre-primary school? Were you always a teacher here at this school? Or were you a teacher at another school? Please tell me about your education and training.


Please tell me about some of the most significant problems that your school faces [as a Head Teacher versus as a teacher]. I’ve spoken with teachers in other schools who have told me that they face X & Y problems. Which of these problems are most important? Do you also feel that these problems are relevant? How do they affect you directly? Do these problems affect student learning? If so, please tell me how.

3 Relevance: Challenges 2

What other challenges do your students face in relation to their education? Which of these challenges are most important? How does your school address these challenges? How do students and their families address these challenges?

4 Relevance: Equity

Do any of these challenges affect the most vulnerable children (those who are poorer or come from far away or have disabilities)? In what ways? Are there other challenges you can think of that primarily affect those children who are poorer or who have disabilities?


Equity

Please tell me about the LUL-funded PPE in your school. What are the different features of the programme? (If relevant) In your opinion, which of these features are most important?

6 Relevance: Relevance

How does the PPE address the challenges to the education of those vulnerable children we talked about earlier? How well does it help with these challenges?


Perceptions of the LUL-

funded PPE

What (if any) changes have you seen in your student’s behavior in class? What about working with your parents and community? Tell me about any challenges you have from your parents and community, working with disabled, and your students’ transitions to primary school. Did you participate in training? Tell me about this training? Helpful? Effective?

8 Effectiveness: Sustainability

& Impact

How do family members react to these changes? How are these changes received by other members of the community?

9 Effectiveness: Challenges 2

What keeps more vulnerable children from participating?


Has the LUL-funded PPE changed in your school? If so, please describe these? Give examples. Are these changes important? Why (or why not)? Since the LUL-funded PPE started in your school, what problems has it encountered?

72 This instrument has a few questions related to Head Teachers, but most are for the teachers.

61


Questions

(Problems might include low attendance, parental disapproval, lack of a room in bad weather, etc.) How have these problems affected the LUL-funded PPE overall? How have these problems affected the most vulnerable and poor children who participate in the LUL-funded PPE? What have you done to respond to these problems? (Evaluator will note the degree to which these changes have been initiated by the school or programme-wide.) Have these responses been successful, in your perspective?


Can you think of other changes that might be implemented to improve the LUL-funded PPE?

12 Effectiveness: Impact

Has the LUL-funded PPE changed attitudes or behaviors among the families of the most vulnerable (hardest to reach, poorest)? If so, please give me an example.

13 Sustainability What other support do you give to help the children in the pre-primary education programme? Maybe with your time, by encouraging vulnerable or children who live far away from school in other ways? Please give me examples. If not, please describe the factors keep you from supporting the the LUL-funded PPE.


Are there other aspects of the LUL-funded that are important, but that we haven’t talked about? If so, please describe these.

15 For Head Teachers -

Only

Now I would like to talk about the students who came for the LUL-funded PPE. When those students first came to primary school grade 1, how prepared were they to do the work [numeracy-reading-writing] for that grade? What about self-expression? Cooperative play with others? Problem solving? What (if any) changes did you in those student’s behavior in class? Please discuss any changes you may have seen in the most vulnerable, poor, disabled, coming from far away.

Overview: Focus Group Discussion (FGD) Guides

The following FGD guides address: I. Field visits: FGD guide for communities and families II. Field visits: FGD guide for participating girl students III. Field visits: FGD guide for participating teachers (potential) It is anticipated that the evaluator might also require having discussions with: Community implementers (e.g., mentors, facilitators, champions, etc.) Non-participating children (including the most vulnerable children), perhaps in the Government Schools

62

FOCUS GROUP DISCUSSION Guide: Communities and Families

Criterion: Theme Question

Relevance


Please tell me about your trip to this meeting. How did you travel? How did you learn about the meeting? How long did it take to get here?


Please tell me about the importance of pre-primary education. Is it important that your children complete school? If so, please tell me why. Tell me some of the why your children may not complete school? What are the problems or obstacles? Do you have a plan to help your child complete? If so, please describe them.

3 Relevance: Challenges 2

Does your child’s school also help address these problems? If so, please tell me how.

4 Relevance: Equity

What are differences between girls’ education and boys’ education? Is it more important for boys to be educated than for girls? If so, please tell me why. What about poorer children and wealthier children’s education? What about children with disabilities and children with no disabilities? Please tell me about your hopes for your children, in relation to education. Do you believe that your child will finish primary school? Secondary school? (if possible…)

5 Relevance: Awareness

Please tell me about the LUL-funded PPE (Maybe it has another name in Khulna). Is the LUL-funded PPE compulsory like PE? How many here have children who attend PPE? Is the PPE compulsory like PE? Do you know or heard the stories/games and plays in the curriculum of LUL-funded PPE? Are these interesting? In your opinion, will these be useful? Do you give time to your son/daughter after she /he comes home from school? How often do you talk to your son’s /daughter’s teacher? When do you discuss your child’s progress in school? How long does this take?


How has your child changed since (s)he has been going to school? Are these changes important? Why (or why not)?

7 Effectiveness: Challenges

What factors do you think keep more vulnerable children from participating in school? Other groups have mentioned that X and Y obstacles keep disadvantaged children from participating. How important are these factors here? Is this true for here? Are there other factors that are more important?


How have these problems/obstacles affected the education that your child is receiving? Have you heard about any changes at school? Maybe from the teachers or other parents? Please give me examples. Did you like any of these changes?

63


Are there any improvements you would recommend?

10 Sustainability Do you give any form of support to the school? (Probe: time, volunteer, attend meetings, or in other ways?


Are there other important, important things that we haven’t talked about? If so, please describe these.

64

FGD GUIDE: Past Students & Current Students


Questions

1. Relevance: Getting started

[All questions are for past students unless noted otherwise] Please tell me about your school. How often do you usually attend? When did you start attending? What do you do when you are at school? Do you enjoy school? Remember when you came to school in the beginning from PPE, how was it going to primary school? Can you remember if school lessons were hard or easy to understand?

2. Relevance: Challenges

Please tell me about some of the things that bother you about school. Some children in other schools have told me that they are bothered by X & Y. Is this true for you? Are any of these things also bothersome for you? Which of these bothersome things are most important? How do they affect you? Do they affect boys too? Do you think that they affect boys differently? How do they affect the rest of your time in school? What do you do to deal with these things that may bother you? In your opinion, how well does it work?

3. Relevance: The initiative

Do you ever address one of these problems in your school? Please give me an example of a way that you address a problem in your school. Do you address this problem in other ways, outside of school? How is addressing your problem in school different?

4. Effectiveness: General

What do you like in school? [for current students too] Please tell me how going to school has changed the way you see yourself? Now? In the future? [Not sure we will get an answer to this question – but let’s try!]

5. Effectiveness: Challenges

Would you like to attend school more often? What keeps you from attending more often? How could things be changed to address this problem/obstacle? What do you think could be improved?

6. Effectiveness: Innovation

Since you started attending school, how has the school changed? How have these changes affected you?

7. Effectiveness: Innovation

What needs to change to make it easier for you to come to school?

8. Effectiveness: Impact

Since you started attending school, have you noticed any changes at home? Give me an example of what has changed at home? What about your parents changing since you started school? What about other family members changing since you started school?

9. Effectiveness: Sustainability

Tell me about your plans for the next year. Do you think you will go to school? What school do you think you will go to? What else might you do?

10. Effectiveness: Wrapping up

What are some other things about your school that are important, but that we haven’t talked about?

65

Annex 5 Classroom Observation Guide

Classroom Observation Form For the LUL Evaluation

Prior to observation, confirm from teacher(s) that this is the “regular routine” Date: ______Month, Year LUL School Code: [Use from Sampling Frame]_______

Name of school: Number of children observed: Girls_______ Boys________ Are all students present today? __Yes __No

Number of Teachers: Class Observation Time (check one): ___15-20 minutes or ___ 30 minutes

1. Is there text on walls? ____ Yes ____ No

2. Work designed and created by children is on display? ____ Yes ____ No

3. Classroom is warm and inviting? ____ Yes ____ No

4. How many walls? (Circle Only One) 0 1 2 3 4 WHAT IS THE DAILY ROUTINE? Just to get an idea about the usual routine, ask the teacher: “Does the daily LUL PPE routine include children……(ONLY check one)”:

5. Taking part in planning their day with adults?

____ Yes ____ No ___To Some Extent

6. Carrying out those plans through work time or free play?


7. Engaging in small group activities? ____ Yes ____ No ___To Some Extent

8. Engaging in large group activities? ____ Yes ____ No ___To Some Extent

9. Eating a snack/meal (if applicable)? ____ Yes ____ No ___To Some Extent

10. Participating in cleaning-up? ____ Yes ____ No ___To Some Extent

HOW DOES THE DAILY LUL PPE ROUTINE LOOK?

11. The children appear engaged and participating in the classroom?


12. Did the teacher include interactive activities (songs, games, role-play) today?


66

QUALITY OF INSTRUCTION: OPPORTUNITIES FOR SOCIAL SKILLS, SELF-EXPRESSION, PHYSICAL SKILLS, SENSORY SKILLS, WORKING WITH MATERIALS: 13. Children are involved in social

activities? ____ Yes ____ No ___To Some Extent

14. Children communicate attitudes and/or feelings?


15. Children listen to teacher(s)? ____ Yes ____ No ___To Some Extent 16. Children take part in tasks involving

attention to physical and bodily needs? ____ Yes ____ No ___To Some Extent

17. Children take part in caring for the class space?


18. Did you observe children using gross motor skills?


19. Did you observe children using fine motor skills?


20. Use of teaching aids during that day? ____ Yes ____ No ___To Some Extent 21. Children’s use of any reading skills (if

applicable)? ____ Yes ____ No ___If Applicable

22. Children’s use of any writing skills (if applicable)?

____ Yes ____ No ___If Applicable

23. Children’s use of any math skills (if applicable)?

____ Yes ____ No ___If Applicable

GENERAL NOTES

OBSERVED EXAMPLES OF HOW LEARNING IS TAKING PLACE IN THE LUL PRESCHOOLS: OBSERVED CHALLENGES (IF ANY)?:

67

Annex 6 Application of the Tanahashi Model

Shyamnagar

68

Budhata

69

Annex 7 Stakeholder List

UNICEF

Central: UNICEF Country Office personnel � Anjana Mangalagiri, Chief Education Programme � Isa Achoba, Chief, Social Policy and Monitoring & Evaluation � Mohammad Mohsin, Education Manager � Mohammad Golam Kibria, Education Section Officer � Shantanu Gupta, Monitoring and Evaluation Specialist, Social Policy Planning, Monitoring &

Evaluation Section Reference Group Members Dr. Manzoor Ahmed with the Institute of Educational Development, BRAC University; Field-office UNICEF personnel

1. Mohammad Badrul Hassan, Head of UNICEF Khulna Zone Office 2. Mohammad Tanvirul Islam, Education Officer 3. Mong Yai, Communicating for Development Officer in UNICEF Khulna Zone Office

Persia Nargis, Formative Evaluation Interpreter

Partners – Central Level

Directorate of Primary Education (DPE) with Ministry of Primary And Mass Education, (MOPME) � Md. Mafuzan Rahman (Jewel), Education Officer, Directorate of Primary Education � Dr. Gopika RanJan Chakrabortj, Deputy Director, Pre Primary Education, Directorate of Primary

Education � Md. Emran, Joint Secretary and Director, Monitoring and Evaluation, Directorate of Primary

Education (DPE) � Kawsar Sabina, Deputy Director, Monitoring and Evaluation, Directorate of Primary Education

(DPE) � Waliul Islam, Assistant Director, Monitoring and Evaluation, Directorate of Primary Education

Ministry of Women and Children Affairs, (MoWCA) � Bikash Kishore Das, Joint Secretary

Bangladesh Shishu Academy (BSA) � Md. Kafiluddin Kaiya, Deputy Secretary, BSA

BRAC Personnel � Dr. Safiqul Islam, Director, BRAC Education Programme � Dr. Ameena Ahmed, Programme Coordinator, BEP � Md. Faroque Hossain, Sr. Programme Manager, BEP � Md. Kaisar Ali Khan, Programme Manager, Monitoring, BEP � Perven Akhter Khanam Sr. Manager, SS(QA), BPPS � Afrina Hossain Management Professional Staff, BPPS

Partners – District and Sub-District (Upazila) Levels

Ministry of Primary And Mass Education, (MOPME) � Md. Mohiuddin, Assistant District Primary Education Officers, Satkhira � Sk. Ahidul Alam, Assistant District Primary Education Officers, Satkhira � Beonale Kumar Mallick, Upazilla Education Officer, Tala, Shatkhira

BSA Personnel � Asif Iqbal, District Officer, BSA Satkhira

BRAC BRAC Regional Manager (2 males)

� Meer Abdus Sattaer, BRAC Regional Manager, Khulna � Asaduzzaman, BRAC Area Manager, Satkhira

70

BRAC District Managers (5 males and one female) � Mafuza Sultana, Senior Branch Manager, Kalaroa � Biprokash Sarker, Senior Branch Manager, Tala � Manoj Biswas, Branch Manager, Kubria � Md. Hanif Mia, Branch Manager, Assasuni � Prokash Chandra Roy, Senior Branch Manager, Shyamnagar � Md Kamruzzaman, Senior Branch Manager, Satkhira Sader

Partners – Community Level

BRAC-FGD (4 males and 2 females) � Md. Habibur Rahaman, Quality Assurance � Md. Jamsed Ali, programme Organizer � Rashida Khatun, field Organizer � Azmira Khatun, programme Organizer � Most. Sahida Khatun, field Organizer

Government Primary School Head Teacher (1 male) � Sk. Farid Uddin Ahmed, Moddah Atarai Government Primary School, Tala

Teachers and Parents – Community Level

PPE Centre Teachers: � Six Teachers – Interviews � 10 Teachers – FGDs

Parents: � 16 parents (15 mothers and 1 father) - interviews � 11 Parents (all mothers) – FGDs

Current students receiving PPE services: � 7 students (all girls)

Past students or those who have already gone through the LUL-funded PPE: � 3 students (2 girls and one boy)