let us learn (lul) formative evaluation - unicef · let us learn (lul) formative evaluation ......
TRANSCRIPT
1
LET US LEARN (LUL)
FORMATIVE EVALUATION
UNICEF AFGHANISTAN COUNTRY
OFFICE
EVALUATION REPORT
2
LET US LEARN (LUL)
FORMATIVE EVALUATION
UNICEF AFGHANISTAN COUNTRY
OFFICE
EVALUATION REPORT
iii
Let Us Learn (LUL) Formative Evaluation
Final Report – UNICEF Afghanistan Country Office
© United Nations Children’s Fund, New York, 2012
United Nations Children’s Fund
Three United Nations Plaza
New York, New York 10017
January 2015
The purpose of publishing evaluation reports produced by the UNICEF Afghanistan Country Office is to fulfil a corporate commitment to transparency through the publication of all completed evaluations. The reports are designed to stimulate a free exchange of ideas among those interested in the topic and to assure those supporting the work of UNICEF that it rigorously examines its strategies, results, and overall effectiveness. This evaluation report for Afghanistan was commissioned by the UNICEF Evaluation Office and the Afghanistan Country Office (ACO) and constitutes a part of a global evaluation of Let Us Learn (LUL) that sought to examine lessons learned in UNICEF’s LUL programme in five countries. In the ACO, Siping Wang, Chief of Planning, Monitoring and Evaluation section, Etsuko Matsunaga, M&E Specialist, were the lead counterparts. In the ACO Education section, Cristina Brugiolo, Nazifa Aabedi, and Lailoma Hassani were the lead counterparts. The purpose of the report is a self-evaluation to examine lessons learned and to facilitate the exchange of knowledge among UNICEF personnel. The content of this report does not necessarily reflect UNICEF’s official position, policies or views. The designations in this publication do not imply an opinion on legal status of any country or territory, or of its authorities, or the delimitation of frontiers. The copyright for this report is held by the United Nations Children’s Fund. Permission is required to reprint/reproduce/photocopy or in any other way to cite or quote from this report in written form. UNICEF has a formal permission policy that requires a written request to be submitted. For non-commercial uses, the permission will normally be granted free of charge. Please write to the Afghanistan Country Office at the address below to initiate a permission request. The report was commissioned by UNICEF Afghanistan Country Office with the use of Let Us Learn Funding. Evaluation conducted and submitted by Kerrin Ann Barrett, Ph.D. For further information, please contact:
UNICEF Afghanistan Country Office
UNOCA, PO Box 54, Jalalabad Road
Paktia Kot
Kabul, Afghanistan
UNICEF Staff and Support Contact Information:
Evaluation Manager
Lisa Bender, Education in Emergencies ([email protected])
Elena Matsui ([email protected])
4
CONTENTS
TABLES AND FIGURES ................................................................................................................. 6 ACKNOWLEDGEMENTS ................................................................................................................ 8 ACRONYMS .................................................................................................................................. 10 Executive Summary ....................................................................................................................... 11 1 Project Description .................................................................................................................. 15
1.1 Introduction .................................................................................................................. 15
1.1.1. Project Context .......................................................................................................... 15 1.1.2 About Let Us Learn in Afghanistan ............................................................................ 16
1.2 ALC Programme Description: Focus on Equity ........................................................... 17 1.3 Key Achievements ....................................................................................................... 18
2 Methodology ............................................................................................................................ 19
2.1 Purpose of the Evaluation ........................................................................................... 19
2.1.1 Evaluation objectives ................................................................................................. 20
2.2 Scope ........................................................................................................................... 20 2.3 Approach ..................................................................................................................... 20
2.3.1 Mixed-methods approach .......................................................................................... 21 2.3.2 Sample size and sampling methods .......................................................................... 21 2.3.3 Evaluation instruments ............................................................................................... 22
2.4 Evaluation criteria ........................................................................................................ 22 2.5 Data Collection ............................................................................................................ 24 2.6 Ethical Safeguards....................................................................................................... 24 2.7 Enumerator Training .................................................................................................... 24 2.8 Field Work .................................................................................................................... 25 2.9 Mobile Phone Application Pilot .................................................................................... 25 2.10 Data Analysis ............................................................................................................... 26 2.11 Limitations of the methodology .................................................................................... 26
3 Findings ................................................................................................................................... 27
3.1 Relevance .................................................................................................................... 29
3.1.1 First cohort graduates primary school........................................................................ 30 3.1.2 Perspective transformation ........................................................................................ 30 3.1.3 Increase in community support .................................................................................. 30
3.2 Effectiveness ............................................................................................................... 30
5
3.2.1 Key objectives reached .............................................................................................. 30 3.2.2 Girls keen to continue education ................................................................................ 31 3.2.3 Multiplier effect significant in communities ................................................................. 31 3.2.4 Addressing needs of refugee children ....................................................................... 31 3.2.5 Hygiene improved ...................................................................................................... 31 3.2.6 Building of a civil society ............................................................................................ 32
3.3 Sustainability ................................................................................................................ 32
3.3.1 Importance of emphasizing community sponsorship ................................................. 32 3.3.2 Parents now seeing value in education ..................................................................... 32
3.4 Coherence ................................................................................................................... 32
3.4.1 Lack of stipends and attendance ............................................................................... 33
3.5 Equity ........................................................................................................................... 33
3.5.1 Increased learning outcomes ..................................................................................... 33 3.5.2 Increased awareness of human rights ....................................................................... 34 3.5.3 Impact of multiplier effect on communities ................................................................. 34 3.5.4 Community support for additional ALCs in neighboring villages and additional
grades .................................................................................................................... 34
3.6 Learning ....................................................................................................................... 35
3.6.1 Mature students learn better ...................................................................................... 35 3.6.2 Improved learning outcomes with smaller class sizes ............................................... 35 3.6.3 Policy ramifications of successful AL programming ................................................... 35
3.7 Innovation .................................................................................................................... 35
3.7.1 Completing 2 grades per year .................................................................................... 35 3.7.2 ALCs co-located in communities ................................................................................ 36 3.7.3 Proximity to villages and recognition by MoE ............................................................ 36
3.8 Challenges ................................................................................................................... 36
3.8.1 Effectiveness (Quality): Teaching and Learning Materials (TLM) .............................. 37 3.8.2 Effectiveness/Sustainability: Lack of MOE ownership ............................................... 38 3.8.3 Effectiveness: Transition and hub school support weak ........................................... 38 3.8.4 Effectiveness: WASH program .................................................................................. 39 3.8.5 Effectiveness/Sustainability/Equity: Lack of Teachers, especially female, to teach
girls and serve as role models ............................................................................... 39 3.8.6 Effectiveness/Sustainability: Coed classes are a problem for more conservative
areas of the country ............................................................................................... 39 3.8.7 Sustainability: Difficulties in monitoring in remote and insecure locations and in local
communities ........................................................................................................... 40
6
3.8.8 Scalability & Sustainability: 3.5M OOSC and Girls’ Education .................................. 40
4 Recommendations .................................................................................................................. 41
4.1 Effectiveness: Improve distribution of TLM via accountability, communication and
support for capacity building in MoE .......................................................................... 41 4.2 Effectiveness/Sustainability: Establish a tracking system for the graduates .............. 42 4.3 Sustainability: Strengthen MoE ownership of ALC programme .................................. 42 4.4 Effectiveness: Support student transition and hub school administration .................. 42 4.5 Effectiveness: Coordinate with UNICEF’s WASH team to improve sanitation at the
schools ....................................................................................................................... 43 4.6 Relevance: Continue supporting female teachers ....................................................... 43 4.7 Sustainability: Significantly strengthen monitoring in remote and insecure locations
and in local communities ............................................................................................ 44 4.8 Scalability & Sustainability: Addressing 3.5M OOSC and Girls’ Education ................. 44
5 Lessons Learned ..................................................................................................................... 46
5.1 Overview ...................................................................................................................... 46 5.2 Plan for and Monitor Teaching and Learning Material (TLM) Distribution ................... 46 5.3 Expand Monitoring and Pilot Mobile M&E ................................................................... 46 5.4 Consider Timeline of Innovation Integration ................................................................ 47
6 Conclusion ............................................................................................................................... 48
ANNEXES ...................................................................................................................................... 49
Annex 1 Methodology ................................................................................................................ 49 Annex 2 Soft Power Solutions Debrief Report ........................................................................... 61 Annex 3 Site Visits and Respondent List ................................................................................... 69 Annex 4 ALC Statistics .............................................................................................................. 74 Annex 5 List of Documents Consulted ....................................................................................... 75 Annex 6 Instruments (National Evaluator, Enumerator Team and Mobile) ............................... 76 Annex 7 Evaluation/Learning Framework ................................................................................ 135 Annex 8 Evaluation SOW ........................................................................................................ 143 Annex 9 Quantitative Data: Charts from Participant Groups ................................................... 150
TABLES AND FIGURES
Tables
Table 1 UNICEF LUL - ALC Data 2014 (Year 3) ........................................................................... 18
Table 2: Evaluation criteria ............................................................................................................ 23
Table 3: Cross-cutting themes ....................................................................................................... 23
7
Figures
Figure 1 Concept map of findings across 5 provinces .................................................................. 28
Figure 2 Girls continuation of education ........................................................................................ 30
Figure 3 Girls hoped-for future career paths ................................................................................. 33
Figure 4 Problems faced by teachers ............................................................................................ 36
Figure 5 Model of ALC intervention ................................................................................................ 49
8
ACKNOWLEDGEMENTS
This report was made possible thanks to the significant time, effort and contributions of many
people, both inside and outside UNICEF. The evaluator gratefully acknowledges the support of
the UNICEF Country Office (ACO) in Afghanistan, including sharing of documents and data in
support of the evaluation. She would also like to acknowledge the hard work and dedication of
the national enumerator team, Soft Power Solutions, in obtaining the quantitative and qualitative
data needed to tell the story of Let Us Learn’s Accelerated Learning Centers (ALCs).
The managerial and technical support provided by the ACO Planning, Monitoring and Evaluation
section and Education section and the UNICEF Bamyan field office were essential in ensuring the
coordination and safety of the evaluation team, including the national enumerator team. The
evaluator would like to acknowledge the assistance of Siping Wang, Chief of Planning, Monitoring
& Evaluation section and Etsuko Matsunaga, M&E Specialist, who provided technical support to
the evaluation, as well as Edmond Gaible of Natoma Group, who was Lead Evaluator on the
global effort. The staff of the ACO Education section: Cristina Brugiolo, Nazifa Aabedi, Ahmad
Fawad Jamshid, Lailoma Hassani, Panji Chamdimba, Khadija Bahram and Mirwais Fahez, were
also all very helpful to this evaluation effort, and without their contributions this evaluation would
not have been possible.
Special recognition goes to Nadima Sahar, Executive Director of Soft Power Solutions, for her
can-do attitude in the face of 300+ interviews and focus group discussions across five provinces.
Additional recognition is extended to Mike Dawson of Ustad Mobile, who developed a pilot mobile
application to gather data from the field.
Special recognition also goes to the national evaluation reference group for their substantive
advice and for their contributions to the data gathering instruments used by the national
enumerator team.
The evaluator wishes to note her appreciation of the many people who made time to meet with
her and the enumerator team during the course of the evaluation including central and local
Ministry of Education officials, and the many parents, children, ALC teachers, hub school
principals and community members who participated in the interviews, meetings and focus group
discussions conducted as part of the evaluation.
She would also thank staff from other international NGOs who contributed their time, information
and thoughts to this evaluation.
Additionally, the evaluator extends her sincere gratitude toward EO, New York: Kathleen
Letshabo, Elena Matsui and Lisa Bender, who provided the overall leadership and guidance in
managing the evaluation.
9
10
ACRONYMS
ACO Afghanistan Country Office
ALC Accelerated Learning Center
ALP Accelerated Learning Programme
BEE Basic Education and Equity
C4D Communications for Development
CBE Community Based Education
CFS Child Friendly Schools
CBS Community Based Schools
CO Country Office
DAC Development Assistance Council
ECD Early Childhood Development
EGRA Early Grade Reading Assessment
FGD Focus Group Discussion
GIRoA Government of the Islamic Republic of Afghanistan
GPE Global Partnership for Education
iNGOs International Non-governmental Organizations
ILE International Lead Evaluator
INSET In-service Teacher Training Programme
LUL Let Us Learn
M&E Monitoring and Evaluation
MDG Millennium Development Goal
MoE Ministry of Education
NE National Evaluator
NFE Non-formal Education
NGOs Non-governmental Organizations
OOSC Out of school children
PPE Pre-primary Education
SOW Statement of Work
UNDP United Nations Development Program
UNEG United Nations Evaluation Group
USAID United States Agency for International Development
11
EXECUTIVE SUMMARY
This report describes the findings and recommendations for the evaluation of the Let Us Learn
(LUL) programme in Afghanistan, which focuses on Accelerated Learning Centers (ALCs). LUL
is a unique private partnership that allows for flexible and innovative approaches to addressing
inequities in education access and outcomes. LUL funds programmes in five countries:
Afghanistan, Bangladesh, Liberia, Madagascar and Nepal. In so doing, LUL contributes to
UNICEF’s sharpened focus on educational equity, innovation and outcomes for learners through
support for pre-school education, learning about health and hygiene, and education of adolescent
girls and out-of-school children (OOSC). The foundation of the design of the LUL programme
rests on three “equity pillars”: reaching OOSC; expanding girls’ education; and improving the
quality of outcomes for learners. Activities that address each of these three pillars are in turn
implemented in ways that promote three “LUL themes”: equity, learning and innovation. These
themes are critical in terms of the approach and findings of the evaluation of LUL Afghanistan.
The aim of the LUL programme in Afghanistan is to provide a primary school education (Grades 1
to 6) for OOSC ages 9 to 15, primarily girls, who otherwise would not have a chance to attend
school and obtain an education. The programme is relatively new, beginning in 2011, and only
now the first cohorts of students are graduating Grade 6, more than 9,000 in all. LUL focuses on
the 12 provinces of the Central Region. The programme is innovative in that students attend two
grade levels per year, attending school year round, studying the same curriculum as that of
formal schools. In addition, the teachers, over 700, are drawn from the local villages where the
ALCs are located, thus community members are more trusting to let their girls, in particular,
attend classes.
Evaluation objectives
From a country perspective, the objectives of the evaluation of LUL in Afghanistan are threefold:
1) to examine the extent to which LUL is achieving intended outcomes in Afghanistan; 2) to
systematically document new learning, and to document new efforts in monitoring for equity, in
particular; and 3) to contribute to the global LUL evaluation and analysis of cumulative impact.
The evaluation is focused on learning to be shared within the organization.
Methodology
The methodological approach used was a self-evaluation exercise with a primary focus on
learning and a secondary focus on accountability, using a common evaluation/learning framework
developed for the global LUL evaluation. The methodology is primarily qualitative in nature, using
content analysis and constant comparison of narrative data to validate emerging themes. As this
was a pilot project, only the DAC criteria of relevance, effectiveness, sustainability and scalability
(as a sub-criterion of sustainability) were used to provide a general framework and were adapted
to the LUL programme in Afghanistan.
Five provinces out of the 12 provinces in the Central and Central Highland Regions served by
UNICEF were highlighted for this evaluation: Bamyan, Daikundi, Kabul, Panjshir, and Paktya. The
focus was on ALCs located specifically in 14 districts and 30 villages already selected and
identified by the ACO.
12
The NE and enumerator team conducted over 300 participant (teacher, community members, hub
school principals, students) interviews and focus group discussions. Supporting the innovations in
LUL’s approach, data gathering included the use of mobile phones. Concept mapping aided in
the data analysis to further refine relationships between findings.
Findings
This evaluation has determined based on the evidence that the LUL initiative in Afghanistan has
met its objectives, reaching intended participants and increased learning outcomes, and has even
shown significant effectiveness (impact) in many communities in terms of transformational
learning (defined by Mezirow (1991) as a change in perspective, or world view).
Results are very promising for this innovative program. Children and communities are being
given access to education, in many places for the first time in a generation due to conflict – or for
the first time at all - in many rural and geographically remote areas. Between December 2014
and January 2015, the first cohort graduated primary school. Opportunities now exist for female
teachers to teach in their local community, making it possible for girls to obtain an education.
While challenges naturally exist in such harsh circumstances, participants overwhelmingly report
positive changes in their homes and communities as the result of ALCs being opened in their
villages.
An increase in community support for ALCs, and in particular for girls’ education, was found
across all five provinces. In a number of instances there was a significant shift in perspective,
from being against girls’ education to encouraging girls to continue on to university.
The ALCs have significantly contributed to improving equity for girls in Afghanistan. Girls are
keen to continue their education, though face serious barriers, and the multiplier effect of
education is significant in communities. The oft-repeated adage of educating one girl means
educating an entire family was found to be true also in the communities surveyed in this
evaluation. Participants, especially the students, reported returning home from school and
sharing what they learned with siblings and parents. Several girls talked about teaching their
fathers to read at night.
In addition, an “auditing classes” phenomenon is very much a part of the ALC programme. Very
young children also attend regularly, as do much older girls, because they like to learn. In
addition, ALCs address the needs of refugee children, most of whom are older, OOSC returning
from Iran and Pakistan.
Increased learning outcomes were observed, primarily in the areas of literacy and numeracy, and
students seemed to be ahead of their formal school counterparts.
In terms of effectiveness/impact, the most important finding is that the building blocks of a civil
society begin in early grades. The Life Skills curriculum teaches respect for oneself, one’s family,
and in particular, for elders in the community. Most promising is the girls’ increased awareness of
their own rights, and human rights in general, in part as a result of the Life Skills curriculum.
Many conservative areas who were formally against girls’ education now actively encourage their
girls to attend the ALCs.
13
As part of the global LUL evaluation effort, the context within which this initiative is taking place
was considered, in particular whether or not WASH basic standards were being met. Particularly
in Bamyan province, overall better hygiene practices were reported as a result of WASH training
in the early grades.
Although overall the LUL initiative in Afghanistan is successful in terms of relevance and
effectiveness, there are still significant challenges that need attention and resolution. The primary
issue is lack of capacity in the MoE. On the central level at MoE, although there is a CBE policy
in place, CBE has been primarily under the purview of the NGO community in Afghanistan. MoE
does not fund the CBEs; they are funded solely with donor monies and implemented by a number
of NGOs, both national and international.
As a result of the lack of capacity in MoE, reliance on the ministry as an implementing partner for
LUL has meant that essential aspects of the programme are severely deficient1. Transition and
hub school support for graduating ALC students is weak, and no tashkeel exists for ALC teachers
to enter the formal school system. Additionally, the lack of teaching and learning materials (TLM)
was mentioned at nearly every school visited. The MoE does not have the capacity to manage
the distribution of TLM, and corruption is high. In addition, effecting both equity and sustainability,
in particular in the more rural and conservative areas of the Central Region, is the lack of an
adequate number of female teachers. In every province, monitoring was an issue, especially in
remote and insecure locations, which impacts both effectiveness and sustainability.
Lessons Learned and Recommendations
Lessons learned from the LUL implementation in Afghanistan highlight the importance of
community participation and good practices of processes to operationalize, monitor and engage
communities and other stakeholders to make LUL a success. Consideration should be given to a
lengthy timeline of innovation integration to give social mobilizers time to prepare communities for
CBE, especially educating girls in conservative areas.
Recommendations revolve around the lack of capacity in the MoE to fully implement the LUL
programme. With TLM essential for ALC effectiveness, key stakeholders were all in agreement
on this point: if the MoE is unable to deliver, UNICEF should organize distribution to ALCs until
such time as the MoE has built enough capacity in order to ensure timely distribution to all
Centers. Subcontracting to an INGO would be one way to resolve this ongoing problem.
In order to ensure proper monitoring of the ALCs, especially in remote and insecure locations, the
recommendation is to contract with a local (third party) NGO experienced in M&E, both qualitative
and quantitative techniques, and use mobiles for real-time/GPS tracking. A local 800 number and
call center should be made available so that any community member can call for free and report
on the ALCs (e.g., late teacher salaries, etc.).
1 The NE had a further conversation (after the LUL mission) with Arian, head of Planning in MoE, in January 2015, wherein he stated that he warned UNICEF that the MoE lacked capacity to implement the CBEs, including the proposed ALC/LUL initiative but that they decided to proceed regardless.
14
In order to improve sustainability of the ALCs, the recommendations are to: 1) strengthen MoE
ownership of the ALC programme through more frequent collaboration with government officials;
2) collaborate with MoE’s Planning Department to strengthen monitoring and pilot mobile M&E; 3)
continue supporting female teachers through encouraging MoE/TED to provide flexible teacher
training programming; and establish a tracking system for the graduates.
The ramifications of a programme that successfully covers two classes in one year in terms of
formal education bring up an important strategic issue for the MoE and donors. If funding were
made available, the ALCs could be expanded to fully cover the remaining 3.5M OOSC, as well as
many of the thousands of children projected to enter the school system in the next decade.
The ALCs in Afghanistan represent a promising way forward to improve equity for girls and
OOSC in remote areas of the country. Most encouraging are the findings relating to the creation
of civil society, wherein the finding showed an increase in respect for elders, a decrease in
violence and an increase in awareness of human rights, particularly amongst girls.
The challenge to the future of the ALCs lies in the hands of the new government which came into
power in 2014. Though the MoE is a strong partner in CBE implementation, capacity and budget
are nearly non-existent to support the current programme. Continuing support from
UNICEF/ACO will be needed to ensure the most marginalized girls and older children have an
opportunity to learn and contribute to their families, communities and the future growth and
stability of Afghanistan.
15
1 PROJECT DESCRIPTION
1.1 Introduction
This report describes the findings and recommendations for the evaluation of the Let Us Learn
(LUL) programme in Afghanistan, which focuses on Accelerated Learning Centers (ALCs). The
aim of the programme is to provide a primary school education (Grades 1 to 6) for out of school
children (OOSC) ages 9 to 15, primarily girls, who otherwise would not have a chance to attend
school and obtain an education. Nearly 10,000 children are in the programme, and LUL, formerly
known as the Basic Education & Equity (BEE) initiative, is a unique private partnership that allows
for flexible and innovative approaches to addressing inequities in education access and
outcomes. LUL funds programmes in five countries: Afghanistan, Bangladesh, Liberia,
Madagascar and Nepal. In so doing, LUL contributes to UNICEF’s sharpened focus on
educational equity, innovation and outcomes for learners through support for pre-school
education, learning about health and hygiene, and education of adolescent girls and out-of-school
children (OOSC). The foundation of the design of the LUL programme rests on three “equity
pillars”:
Reaching out-of-school children (OOSC)
Expanding girls’ education
Improving the quality of outcomes for learners
Activities that address each of these three pillars are in turn implemented in ways that promote
three “LUL themes”: equity, learning and innovation. These themes are critical in terms of the
approach and findings of the evaluation of LUL Afghanistan. The evaluation is framed as a self-
evaluation or evaluation focused on learning to be shared within the organization.
1.1.1. Project Context
Afghanistan intends to achieve MDG4 by 2020, and with massive donor aid and the strong
support of the Ministry of Education, education is now reaching record numbers of children,
especially girls, and there are non-formal and formal schools in many areas of the country where
previously there were none. Thousands of newly qualified teachers now teach throughout the
country, and in-service training is commonplace. Half of the teachers enrolled in the Teacher
Training Colleges (TTCs) are female. Yet the country still faces sizeable challenges in building its
education system, fundamental for future economic growth and stability. For more than two
decades the country has been in conflict, creating tremendous insecurity and disruption to
development and basic services, particularly girls’ education. Moreover, while the country’s
urban areas were well developed before the onset of the recent conflicts, the majority of the
remainder of the country had little, if any, access to formal schools, health and other livelihood
services and remain underserved.
16
It is estimated that 60% of boys and 42% of girls now enrol in primary school2, yet attendance
rates are much lower. Of those enrolled, the completion rate is only 30% for girls. In other
words, barely one in three girls will complete her primary education.3 At the secondary level, net
enrolment and attendance rates for girls are 15% and 6% respectively.4There are also serious
shortages of classrooms, which mean many children, particularly in urban areas, attend school in
shifts for as little as two hours per day.
1.1.2 About Let Us Learn in Afghanistan
In the past decade, the education system in Afghanistan has been recovering after years of
conflict and providing access to education in all provinces, including in many areas where
education was limited or lacking altogether, especially for girls. Nonetheless, a number of barriers
to education remain throughout most of the country and it is estimated 3.5M children are still not
in school. Insecurity, poverty, distance to school, lack of permanent classrooms and qualified
teachers (particularly females), limited teaching and learning material and restrictive cultural
practices are challenges that prevent most Afghan children from obtaining a quality education.
Thus, the country still has substantial progress to make in order achieve primary education for all.
In addition to this challenge, the backlog of children and adolescents out of school is an added
burden to an overstretched system that has difficulties coping with the already high demand for
education at all grade levels.
The LUL initiative in Afghanistan seeks to improve education in the 12 provinces in the Central
and Central Highland Regions. Target areas are those that have not been served by government
and donor programmes as extensively as have other provinces, by providing equitable access to
basic education for out of school children (OOSC) and adolescents, with particular focus on girls.
Other iNGOs (e.g., as BRAC, Save the Children and Aga Khan Foundation) have also been
implementing CBE, including accelerated learning classes, in Afghanistan for a number of years.
LUL programming in FY 2013-2014 included instruction in Grades 5 and 6, distribution of
teaching and learning materials and stationary, teacher training, and monitoring. Annex D shows
detailed data on the ALCs, while Table 1 UNICEF LUL - ALC Data 2014 (Year 3) shows data for
2014 on the first cohort of ALC students who will graduate primary school Grade 6. The number
of LUL ALCs has remained steady since the first year of implementation, with 384 in the first year,
and 383 remaining (one ALC was integrated into a formal school in Bamyan) in the 12 central
provinces.
The strong advocacy brought forward by iNGOs and the Ministry of Education (MoE) during the
past years to envisage alternative learning opportunities to out of school children resulted in the
development of CBE Policy and the inclusion of CBE as integral part of the Global Partnership for
Education (GPE) proposal, successfully awarded to the Government of the Islamic Republic of
2State of the World’s Children, UNICEF, 2012.
3 Millennium Development Goals in Afghanistan, UNDP, 2012
4State of the World’s Children, UNICEF, 2012.
17
Afghanistan (GIRoA) in 2012 for $55.7M. UNICEF was deemed the supervising body for this
grant money, which focused mostly on 13 provinces in the south, east, west and central regions.
1.2 ALC Programme Description: Focus on Equity
Since the inception of the LUL programme in Afghanistan, the UNICEF ALCs have been focusing
on equity and provision of equitable learning opportunities to all children, particularly girls and
children from marginalised communities. While there are much larger Community Based
Education (CBE) programs, including those with ALCs, this evaluation report focuses only on the
ALCs funded by UNICEF, all of which are located in the Central Region.
All CBE programs are funded outside the Ministry of Education (MoE)5 because it does not yet
have the funding nor the capacity to manage such a large scale programme. The UNICEF ALCs
are a relatively small programme implementation in comparison to, for example, BRAC’s Girls
Education Challenge CBE Project 5085 funded by DFID for three years, until 2017, which targets
150,000 girls across 10 provinces, 50,000 of whom are OOSC.
Similar to other CBE initiatives, ALCs offer fast-tracked curriculum lessons to OOSC and
adolescents, aged 9 to 15 years, using the same curriculum as in the formal school system. The
curriculum covered is two grades per year, in order to complete the full cycle of primary education
(6 grades) in three years. Children are encouraged to enroll in the nearest formal school after the
completion of the 6th grade to continue their education.
ALCs are created through community participation. Communities identify: a) educated members
who can be trained as ALC teachers, and b) learning spaces within the community. The MoE has
been involved from the onset and, with the support of UNICEF and other partners, developed the
CBE policy that provides the framework for the operation of Community Based Schools (CBS)
and ALCs. In the case of ALCs, MoE is responsible for providing textbooks while UNICEF trains
teachers and provides teachers’ salaries, teaching and learning material, as well as stationery,
carpets and firewood.
Similar CBE initiatives funded by other donors (e.g., Aga Khan Foundation, DFID) and
implemented by Save the Children and BRAC, among others, have also seen significant
outcomes and, like UNICEF’s implementation, have also been found to be sustainable over time
with donor assistance.
5 Known as “off-budget funding” because it is not included in the MoE’s own budget.
18
Table 1 UNICEF LUL - ALC Data 2014 (Year 3)
No. of
Districts
ALCs G5-6 (2014)
Number of Teachers Number of Students
Province
# of
ALCs Male Female Total Girls Boys Total Dari Pashtu
10 Parwan 34 29 39 68 783 0 783 611 172
6 Panjshir 23 18 28 46 502 18 520 520
7 Kapisa 28 42 14 56 579 140 719 538 181
7 Bamyan 17 19 15 34 240 93 333 333 0
8 Daikundi 40 27 53 80 851 241 1092 1092 0
12 Paktiya 20 28 12 40 500 0 500 25 475
15 Paktika 20 30 10 40 309 304 613 7 606
10 Khost 42 72 12 84 1057 100 1157 1157
14 Kabul 40 44 36 80 898 0 898 389 509
8 Ghazni 37 23 51 74 745 120 865 529 336
7 Logar 42 59 25 84 785 231 1016 240 776
9 Wardak 40 60 20 80 648 204 852 163 689
113 Total 383 451 315 766 7,897 1,451 9,348 4,447 4,901
1.3 Key Achievements
The most recent UNICEF figures on LUL (2014) showed that since the beginning of the project a
total of 384 community-based Accelerated Learning Centres (ALCs) have been established in
113 districts in the 12 target provinces, providing equitable access to basic education to a total of
9,348 children (7,897 girls) in rural deprived areas. In September 2014, 11 (5 girls) students of
one ALC integrated into formal school and the rest of the students integrated to a nearby ALC in
Bamyan, leaving a total of 383 ALCs.
19
In a recent 2013 report, the findings showed that the students in ALCs supported by UNICEF
successfully complete two grades in one year, and they do very well in language and life skills.
Additionally, the percentage of girls’ enrolment in ALCs is very high: 84 per cent. This is one of
the main achievements of the Let Us Learn project because girls in the program now have the
chance to improve their lives.
Previous evaluations of ALCs6, demonstrated that the number of drop-outs gradually increases
as the learners progress to higher grades, if effective measures to prevent school drop- out are
not taken. This kind of finding is not only experienced in the ALCs but also in the formal education
system. This is particularly the case for girls, especially after grades 3 and 4, where the drop-out
rate sometimes reaches 30 per cent in conservative and poverty stricken areas7. Yet, according
to UNICEF figures for the LUL ALCs, the drop- out rate for girls is less than 5%. The main
contribution factor for this lower dropout rate is due to the girls being able to attend school very
near their homes, and be taught by a local teacher, often a female.
During the past three years, the programme has delivered on its goals of providing support
mechanisms to the ALCs, though the delivery has been uneven in large part due to the lack of
capacity in the MoE. UNICEF provides funding under LUL for teacher training and provision of
teaching and learning materials. With this funding, MoE PEDs and DEDs have conducted
teacher capacity enhancement, with standard INSET teacher training given once per year to all
teachers in the ALC programme. During the 2014 reporting period a total of 768 community
teachers (317 female and 451 male) taught in the ALCs, with 384 teachers hired for Grades 1-3
and the remaining hired for Grades 4-6 (second year and a half). Nearly all were trained in Grade
3-6 textbook orientation.
Again through outreach by the PEDs and DEDs, parents and communities have been sensitized
to keep their children in school and to continue supporting the ALCs. These efforts have helped
to reduce the barriers girls face in achieving a primary education and concomitantly improved the
quality of education in the classroom. Capacity is also being built in the School Management
Shuras since the ALCs are an education model that requires a considerable degree of initiative
and contribution from the community.
2 METHODOLOGY
2.1 Purpose of the Evaluation
The purpose of this evaluation is to provide internal lessons learned for this pilot project in order
to further understanding of how this type of educational innovation is best managed and whether
or not it is relevant and effective for the target population. Additionally, lessons learned will aid in
6 Partnerships for Community Based Education in Afghanistan, PACE/A, and Primary Education 2003-2006, USAID.
7 State of the World’s Children, UNICEF, 2012.
20
determining whether or not the model can be scaled and applied in other geographical areas
suffering from similar barriers to equity for females and OOSC.
2.1.1 Evaluation objectives
From a country perspective, the objectives of the evaluation of LUL in Afghanistan are threefold:
1. To examine the extent to which LUL is achieving intended outcomes in Afghanistan;
2. To systematically document new learning, and to document new efforts in monitoring for
equity, in particular; and
3. To contribute to the global LUL evaluation and analysis of cumulative impact.
Results from the evaluation have shown significant and positive outcomes for all three equity
pillars across the five regions and 30 ALCs sampled. The three LUL themes are the warp and
weft of the findings, clearly emerging from the large data set to indicate the programme is
reaching those most in need of an education, while concurrently evolving to adapt and respond to
circumstances on ground.
2.2 Scope
Five provinces out of the 12 provinces in the Central and Central Highland Regions served by
UNICEF were highlighted for this evaluation: Bamyan, Daikundi, Kabul, Panjshir, and Paktya. The
focus was on ALCs located specifically in 14 districts and 30 villages already selected and
identified by the ACO.
The NE and enumerator team conducted over 300 participant (teacher, community members, hub
school principals, students) interviews and focus group discussions.
2.3 Approach
The approach used was a self-evaluation exercise with a primary focus on learning and a
secondary focus on accountability, using a common evaluation/learning framework developed for
the global LUL evaluation. The methodology is primarily qualitative in nature, using content
analysis and constant comparison of narrative data to validate emerging themes.
Data collection was conducted first by the NE in October/November, followed by the larger data
gathering effort by the enumerator team the latter part of November. This is the only evaluation
of the five LUL countries to require a national enumerator team. The NE conducted a training
workshop for the local enumerator team the first week in November that introduced the
methodology and questionnaires, along with the mobile application.
Methods and sources include a review of programme documents, interviews of key informants,
interviews and/or focus groups of important groups of beneficiaries, testimonials involving
beneficiaries (videography) and direct observations of activities with beneficiaries wherever
possible. A retrospective baseline has been established in consultation with programme staff and
21
relevant MoE staff to assess to extent possible contribution to quality of education (e.g., learning
outcomes, reduction in violence, as per the Child Friendly Schools framework).
2.3.1 Mixed-methods approach
The LUL Afghanistan evaluation combines qualitative and quantitative methods in alignment with
each of the other four national evaluations.
Qualitative methods comprised the chief means of collecting data regarding this LUL initiative.
Substantial focus on qualitative approaches has helped to ensure that the evaluation is
participatory, equitable, and open to unexpected information.
Focus Group Discussions (FGDs)
FGDs comprised the primary means of field-level data collection or collection of information about
the experiences, opinions and ideas of children, teachers, parents and community members at
large (e.g., School Management Shuras).
Semi-structured interviews
The NE and local enumerator firm (Soft Power Solutions) conducted semi-structured interviews
with key informants within the Ministry of Education (MoE), UNICEF ACO, and among other
stakeholders and partners in Kabul and the provinces. FGD responses have been coded to
enable these responses to be analyzed via content analysis (qualitative).
Quantitative methods included 1) an analysis of existing data relevant to the LUL program -
such data was used to help establish “retrospective baseline” information in relation to
programme impact, and to create a foundation for subsequent evaluative efforts that will benefit
from ongoing data-collection efforts on the part of the MOE and others; and 2) data gathered from
the field by the enumerator team that was coded according to a framework derived from the
global evaluation template and entered into an Excel spreadsheet, from which graphs were
created (See Annex I).
In-Depth interviews by the NE and narrative analysis were conducted in Kabul Province and
in two provinces outside Kabul, Panjshir and Bamyan, in order to answer the “how” question to
inform better understanding of outcomes from LUL on participants and the surrounding
community.
2.3.2 Sample size and sampling methods
Five provinces of Central and Central Highland Regions were highlighted for this evaluation:
Bamyan, Daikundi, Kabul, Panjshir, and Paktya. The focus was on ALCs located specifically in
14 districts and 30 villages already selected and identified by the ACO. These provinces were
selected by the ACO due to their accessibility and based on the ACO’s sampling framework,
Probability Proportional to Size (PPS).
In addition to key donor, NGO and MoE stakeholders, primary participants in the Afghanistan
study were teachers (male and female), School Management Shuras and community members,
and students, especially girls. Five provinces were selected, six ALCs in each, for a total of 30
22
field sites. UNICEF provided the sample, and in two cases where the ALCs were inaccessible
due to the onset of winter, provided alternative sites.
Reliability. To assess the reliability of collected information, and to increase the reliability of
findings, an effort was made to “triangulate” information—comparing accounts or statements on a
topic from multiple sources. The three primary methods of triangulation included:
Design of semi-structured interview and FGD scripts
These scripts included areas of overlap or redundancy to support triangulation of information in
anticipated categories.
Responsive re-structuring of interviews and FGDs
The NE and enumerator team noted key opinions and facts expressed by interview or FGD
respondents and subsequently augmented scripts and guidelines to support triangulation in
relation to information in new categories.
Developing inferences from review of existing datasets
To the extent that existing education datasets (e.g., enrolment, completion and other data) are
available and can be disaggregated, data of this sort has been used to “cross-check” results.
Validity. The LUL evaluation was designed to be maximally participatory, after the fact of the
evaluation design, and via this means will help ensure the validity of findings. Validity, in
qualitative research, refers to whether study findings are true and certain. “True” in the sense
that research findings accurately reflect the situation, and “certain” in that evidence supports the
research findings. Direct and indirect beneficiaries have not been involved in the design of the
evaluation, its approaches or the evaluation questions (which might be called participatory before
the fact); however, these beneficiaries have had, over the course of the evaluation process, the
opportunity to share opinions and experiences in an open-ended discussion format, ensuring that
their stories are captured.
2.3.3 Evaluation instruments
Instruments for this evaluation were qualitative. Three sets of instruments were created: 1) for
use by the NE; 2) for use by the enumerator team; and 3) for implementation on the mobile
phone.
Quantitative information, when possible to obtain, came from prior data-collection efforts (e.g.,
MOE school- and student-management information, partners’ records, ACO project-monitoring
reports, etc.), as well as coding the qualitative data set.
2.4 Evaluation criteria
Evaluation criteria were developed with reference to guidelines included in the initial Statement of
Work (SOW) and widely accepted frameworks and guidelines, chiefly that of the Development
Assistance Committee (DAC), and in consultation with UNICEF HQ personnel (Lisa Bender and
Kathleen Letshabo).
23
The DAC criteria provided a general framework and were adapted to the LUL programme in
Afghanistan. Overall, per Table 2: Evaluation criteria, the LUL evaluation focused on relevance,
effectiveness, and sustainability from among the DAC criteria and on the three LUL themes—
gender equity, learning and innovation. LUL pillars (OOSC, programmes for adolescent girls, and
outcomes for learners) were treated as crosscutting elements.
Table 2, below, summarizes the evaluation criteria that guided the LUL evaluation.
Table 2: Evaluation criteria
Criterion Source Working definition
Relevance DAC The extent to which the aid activity
addresses the priorities and policies of
recipients and donors
Effectiveness DAC + LUL SOW
and other documents
The extent to which the initiative achieved
stated objectives and the relationship of
those objectives to development goals
Sustainability DAC + LUL SOWs
and other documents
The potential for continuation of benefits
after the initiative has been completed,
with attention to start-up or developmental
costs (e.g., content design and
development) as well as partners’
priorities, policies and demand on the part
of local institutions
Scalability
(sub-criterion of sustainability)
USAID The extent to which a small-scale or pilot
activity is likely to be expanded to reach
larger populations
In Afghanistan, the following questions were explored:
• Relevance – To what extent do the ALCs address the priorities of the community?
• Effectiveness – To what extent did the ALCs attain their objective of reaching rural
populations, girls and out of school, older children and support female teachers?
• Sustainability – How can UNICEF ensure that the ALCs will continue to be funded by
MoE (integration) once LUL has ended?
The following cross-cutting themes augmented and modified these criteria, per Table 3:
Table 3: Cross-cutting themes
Theme Source Working definition
Gender equity LUL SOWs and other
evaluation-planning
documents
Support for girls’ participation
in school
24
Outcomes for learners LUL SOWs and other
evaluation-planning
documents
Changes in outcomes for
learners
Innovation LUL SOWs and other
evaluation-planning
documents
The use of new ideas,
practices or tools in response
to unanticipated challenges
The following questions were asked regarding cross-cutting issues:
• Gender equity – How do the ALCs contribute to gender equity in Afghanistan?
• Outcomes for learners – How do the ALCs improve learning and other outcomes for
learners?
• Innovation – How do the ALCs provide a response to the tremendous need for
improvement in access to primary education in Afghanistan?
2.5 Data Collection
Afghanistan was the only country of the five LUL participating COs to require a national team of
enumerators. The reason for this is primarily insecurity in the country, along with deeply held
cultural beliefs in the rural areas that prevent outsiders from going into those areas altogether, or
from effectively communicating with the local population. Additionally, the scale of the
programme is much larger than the other four countries.
Two data collection efforts were conducted for the LUL evaluation in Afghanistan, the first by the
NE and the second, more extensive, effort conducted by the national enumerator team, Soft
Power Solutions.
2.6 Ethical Safeguards
Ethical safeguards were taken in ACO and in all field visits by both the National Evaluator and
Soft Power Solutions. Respondents were advised that their responses would be presented as
anonymous and, if they wish, that their identities will not be recorded. As this evaluation explores
management aspects of a large, donor-funded program, assurances of confidentiality were critical
to obtaining the narrative data necessary to make appropriate recommendations in order that
improvement measures can be taken by UNICEF.
2.7 Enumerator Training
The NE trained the Soft Power Solutions enumerator team in qualitative data gathering and
introduced them to the instruments and mobile phone application during a three-day workshop in
Kabul. All travelled together to an ALC in Kabul in order to practice skills and test the
instruments. The NE provided feedback to the enumerators to help strengthen their approach
and skills before they went out in the field. (A separate report on the enumerator training was
filed in November 2014.)
25
2.8 Field Work
Due to the delay in awarding the enumerator contract and need to report on the LUL intervention
at the Nepal LUL donor conference in early November, the NE was on ground mid-October. She
conducted interviews with relevant stakeholders in Kabul (e.g., MoE officials, UNICEF ACO staff)
and during three site visits to Panjshir, Bamyan and Kabul province (rural area to the north of the
city).
The enumerator team commenced data gathering immediately the week following the NE mission
and training program, November 8, and continued through 29, in order to keep to a very
compressed timeline and travel to high mountain locations before deep snows closed access for
the winter months.
2.9 Mobile Phone Application Pilot
Supporting the innovations in LUL’s approach, data gathering included the use of mobile phones.
Given the remoteness of many of the areas and success of a similar UNICEF initiative in the
South that monitored teacher salaries via SMS, UNICEF/HQ and ACO supported a proof of
concept, which was developed and implemented by Ustad Mobile.
Separate mobile instruments were developed for the phones. Questions were yes/no and
multiple choice, with one or two text boxes for short answers. The total number of questions was
kept to between 10 and 12 to capture the essence of the interviews. All instruments were
translated into Dari, although interestingly, the data captured in the text boxes was entered in
English.
This innovation was well received by the enumerator team and holds promise for future
monitoring and evaluation of ALCs and other LUL programs. The enumerators reported that the
mobile application was much easier to use during the interview process than the paper-based
forms. In terms of monitoring for equity, mobile data gathering appears to be successful.
Ustad Mobile is originally an Afghanistan-based education technology company. They have a
number of successful mobile learning projects in Afghanistan, such as literacy for female police
and the Afghan military, and their platform is currently in use for many other course development
efforts worldwide. The development of mobile survey instrumentation was a new adaptation of
their core software platform.
All original, paper-based questionnaires were enhanced for the mobile application. The total
number of questions was limited to between 10 and 12, and mirrored the structured and open-
ended questions on the paper-based questionnaires. Likert scale questions (3 point scale),
Yes/No, and very limited text boxes were used to record answers. The mobile questionnaires
were translated into Dari by Soft Power Solutions.
Ustad Mobile created a wireframe mockup in English and uploaded several questionnaires (also
in English) to preview the application for the enumerators during the training workshop in
November, in anticipation of releasing the Dari beta version for testing during the site visit. Soft
26
Power Solutions purchased Android Samsung mobile phones for all enumerators, and the
software was downloaded onto each.
The enumerators were excited to use the new technology, and Soft Power Solutions was very
patient and supportive of the new technology. In one province, Daikundi, connectivity was a
problem because Etisalat does not have good coverage there. To solve the problem and be able
to upload data the team bought a different SIM card and then was able to complete the data
upload. All enumerators reported that data was much easier to gather using the mobile phone.
Data captured on the mobile phones is all in Dari, except for the few text box entries, which the
enumerators filled in using English.
With the data successfully being captured on the Ustad Mobile server, the developer then created
a backend application that disaggregates data by school code and publishes it onto an Excel
spreadsheet for review and download. The developer created a software script that
automatically fills in the spreadsheet with the most recent data capture. A link was provided to
Soft Power Solutions so that they could easily monitor the data and then clean/edit for analysis
and reporting.
Data gathered from the mobile phone application corroborates findings from the NE and
enumerator interviews with regard to relevance, effectiveness and sustainability, and highlights
the same ongoing issues such as lack of textbooks and unsuitable classroom space.
2.10 Data Analysis
The data analysis methodology used was primarily qualitative in nature, using content analysis
and constant comparison of narrative data to validate emerging themes. A codebook was
developed based on the lead evaluator’s suggested codes for the global evaluation and shared
with the enumerator team. The codes were then modified by both the NE and enumerator team
lead, with additional codes added as they emerged from the data analysis. AtlasTi, a qualitative
software analysis program, was used to support coding and analysis. Interview documents were
translated into English and uploaded into the program, where they were coded and queries run.
An ongoing conversation between the NE and enumerator team lead, who is a local national and
thus familiar with the culture and education system in Afghanistan, served to clarify and support
findings as they arose.
Concept mapping aided in the data analysis to further refine relationships between findings.
Concept mapping software developed by the IHMC in Florida was used to create and manipulate
the maps. This free software download enables graphical representation of concepts and
relationships, as well as collaboration on files through IHMC’s central servers. A pictorial
representation was also helpful in explaining and verifying findings to the ACO M&E officer.
2.11 Limitations of the methodology
Chief limitations of the methodology included:
Indeterminate quantitative data
The LUL evaluation has relied on prior data-collection and analysis efforts for its
27
quantitative component. Data on ALCs, including number of teachers and attendance
figures, was provided by UNICEF and has not been independently verified, (i.e., number
of girls in attendance remains nearly identical from Year 1 even though the NE visited
one school wherein four girls had dropped out, which was not indicated in the data
provided).
Lack of baseline data
A “retrospective baseline” using available data from before the start of initiatives and from
groups outside the beneficiary group has been attempted based on early reporting and
interview data.
Unrepresentative samples and resulting bias
The evaluation design relied on ACO to mitigate bias generated by unrepresentative
samples. Given the onset of Winter, geographical remoteness of many ALCs, and
delayed timeline, ALCs sampled are biased toward those that are fairly easily accessible.
Response bias
Response bias in the FGDs is indeterminate.
These limitations were mitigated to the extent possible by gathering a substantial amount of data,
more than 300 interviews and FGDs, from a number of sources (e.g., Shura members, teachers,
principals, students, community members) that were then triangulated.
In addition to limitations arising from design considerations, implementation of the LUL
Afghanistan evaluation has also incurred limitations. These limitations stem primarily from bias in
various forms and among various actors, and include bias in translation and unreliability of
responses, especially given the use of an enumerator team operating at a distance from the NE.
3 FINDINGS
Findings based on an analysis of both the NE and enumerator team data are congruent. Concept
mapping8 was used to determine relationships that emerged between findings to give a more
complete and nuanced picture of how and why this successful pilot programme is effective and
relevant to UNICEF’s objectives. More finely tuned linkages to sustainability also emerged from
the larger, enumerator data set.
Findings are presented according to the DAC criteria, wherein the three equity pillars (reaching
out-of-school children (OOSC), expanding girls’ education, and improving the quality of outcomes
8 Concept maps are graphical tools for organizing and representing knowledge. Novak and Cañas (2008), define concept as a perceived regularity in events or objects, or records of events or objects, designated by a label.
28
for learners) are interwoven, and three LUL themes (Equity, Learning, Innovation) are presented,
to aid in learning and to facilitate synthesis in the global LUL evaluation.
Figure 1 Concept Map of Findings across 5 provinces presents a synthesis of the findings,
together with relationships between emergent themes. The map is divided into two maps, with
the upper half showing positive outcomes and challenges, and the lower indicating community
and principal (formal school) results. The lower half of the map can be interpreted as the
foundation and trajectory for sustainability, wherein community (i.e., Shura) support for the school
and students is needed, along with a pathway into the formal school system (i.e., hub schools).
Through the mapping exercise regional differences emerged in viewpoints of the effectiveness
(impact) of schooling older children and girls, especially. Also emerging was the new awareness
of one’s human rights and need for contributing to the community, both building blocks of a civil
society.
Challenges can also be seen in the map, including from the perspectives of community and hub
school principals.
29
3.1 Relevance
Figure 4 Concept Map of Findings across 5 provinces
30
The early findings clearly indicate the ALC initiative in Afghanistan is relevant to the context of the
education system in the country, and the tremendous need for increasing both the number of
students in primary education and the quality of that education.
3.1.1 First cohort graduates primary school
The first cohorts of ALC students are graduating primary school (Grade 6) through March 2015.
ALCs in three provinces graduated January 10, with two more provinces slated to graduate in
February. The remaining 7 provinces will graduate March 20, 2015. Studying two classes per
year enabled older children to catch up to peers in formal school. Teachers reported that these
older children are more mature than children in formal schools, due to their age, and as a result
are more studious and focused on their schoolwork. The results observed in a number of
classrooms by the senior enumerator team members are Grade 6 students who perform and
behave as though they are in Grade 9.
3.1.2 Perspective transformation
Of critical import to the LUL programme and the future of Afghanistan, in particular, is that
perspectives of individual, community, and education authorities on the importance of education,
especially girls’ education, have changed as result of ALCs being established in communities.
Parents reported that the most important effect of the ALCs for boys and girls is that they are
being educated (60% of respondents). In the case of girls’ education, parents reported that the
most important effect of the ALCs was that both younger and older girls are obtaining an
education (78%). Many conservative areas who were formally against girls’ education per se, or
at least against girls leaving the village to attend school, are now supportive of girls’ education
since they have seen the many benefits to families and the community as a whole.
3.1.3 Increase in community support
The emphasis on community sponsorship of the educational initiative through provision of
classroom space and in agreement with the local PEDs has meant that the majority of community
members, even if initial support was limited, now support the ALCs. Many provide not only
classroom space, but also teaching and learning materials for students and teachers when
needed and where possible (e.g., whiteboards and markers).
3.2 Effectiveness
3.2.1 Key objectives reached
The key objectives for LUL’s first three years across the 12 provinces of Central Afghanistan:
providing access to primary education for OOSC, especially girls, and opportunities for female
teachers to teach, have been successfully met. The target numbers have been reached for
Centers, students (especially girls), and female teachers in ALCs established in the Central and
Central Highland Regions from 2012 till today - 384 ALCs with 9,348 learners, 7,897 girls and
1,451 boys in 12 provinces. Over 40% of teachers are female. Moreover, ALCs are providing
access to basic education in geographically remote areas, where none existed previously.
31
3.2.2 Girls keen to continue education
Girls, in particular, were keen to continue their education, and in many cases said their parents
were also supportive of them continuing on to Grade 7. Figure 2 Girls Continuation of Education
shows that the majority of girls prefer to continue at the ALCs, with only 21% hoping to join a
formal school. Unfortunately, the reality
for many girls is that the hub schools are
a great distance from the ALC, too far to
attend due to security issues and cultural
restrictions. Adding to this fact, the ALCs
do not have classes beyond Grade 6,
leaving this current cohort of girls without
a way to continue their education beyond
primary school.
3.2.3 Multiplier effect
significant in communities
The multiplier effect of ALCs in
communities is significant. Girls and boys
now have literacy and numeracy skills and help their families (e.g., read prescriptions, wedding
invitations, sign posts (directions), dial numbers on mobiles). In an unanticipated finding, family
quality of life is also improved.
An “auditing classes” phenomenon is very much a part of the ALC programme. Very young
children also attend, as do much older girls, because they like to learn. This translates into a far
wider reach than the LUL objectives originally intended (20 students per school). Rogers (2010),
who authored Diffusion of Innovation, observed this phenomena in developing nations where
early adopters of interventions in practice reach far more of the population than originally
planned.
3.2.4 Addressing needs of refugee children
Another population of beneficiaries of the ALCs are the many refugee children who have returned
to Afghanistan after years of war. They, too, attend ALCs, as they are too old to enter formal
school when they return home. Boys, especially, benefit from the accelerated learning, as they
are more likely to attend formal school in Grade 7 and beyond than the girls given the distances
involved between the villages and formal school hubs.
3.2.5 Hygiene improved
Overall, better hygiene practices were reported as a result of WASH training in the early grades.
Participants reported that they are now aware of the importance of hand washing, for example.
However, a gap exists between such awareness and the availability of facilities to practice their
knowledge with proper hand washing and toilet facilities absent in the majority of ALCs visited.
21%
79%
Would you continue at the ALC, if possible? If not, will you go to formal
school?
Will Join FormalSchool
Will Continue atALC
Figure 5 Girls Continuation of Education
32
3.2.6 Building of a civil society
Across all provinces, community members, parents and teachers report boys and girls are more
respectful toward elders, helping them when needed. Some areas have seen a reduction in
violence in the community (boys). In one community in Kabul province visited by the NE, village
elders reported that violence, drinking and drug use by young men had been eliminated through
the establishment of the ALC, because now the school serves as a focus for the boys and they
are no longer fighting and engaging in bad behavior. UNICEF ACO attributed this increase in
civility to the required Life Skills curriculum offered in the early grades.
3.3 Sustainability
With a relatively modest amount of money, UNICEF has been able to create educational
opportunity for children who otherwise would never have been able to attend school due to
factors of age, gender and distance.
3.3.1 Importance of emphasizing community sponsorship
The emphasis on community sponsorship of the educational initiative through provision of
classroom space and in agreement with the local Provincial Education Departments (PEDs) has
meant that the majority of community members, even if initial support was limited, now support
the ALCs. Many provide not only classroom space, but also teaching and learning materials for
students and teachers when needed and where possible (e.g., whiteboards and markers).
3.3.2 Parents now seeing value in education
Importantly, parents now understand the importance of education and see the value in letting
their children, in particular girls, attend school. Many parents are now encouraging their children
to continue to formal school and even university. Forty-seven percent of parents and community
members said that they support the ALCs through recommending to other parents that they send
their children to school or by encouraging children themselves to attend school.
In one ALC visited by the NE, the community members said even if UNICEF pulls funding, they
would continue to fund the school because the outcomes for the community at large have been
so great.
3.4 Coherence
Coherence is assessed in terms of whether or not this pathway makes sense, and how the ALCs
fit in with everything else that is being done in terms of girls’ education and OOSC’s access to
education.
There are many interventions in Afghanistan to improve the education situation for OOSC and
girls, ranging from small NGO-led projects to very large programmes, such as Save’s STAGES
and BRAC’s Girls Education Challenge. UNICEF’s LUL programme is congruent with these
interventions, and follows the same CBE policy from MoE.
33
In an important difference between implementations, the NE subsequent to the LUL mission had
an opportunity to visit BRAC’s CBEs, which are funded under DFID and target OOSC, all girls.
BRAC has adapted its successful model from Bangladesh to fit the Afghan context. Unlike
UNICEF LUL classrooms which are wanting for TLM, such as wall charts and stationery, BRAC
GEC classrooms are brightly decorated, with “wall magazines”, student artwork, many charts and
paper decorations lining the walls. Students have the full kit of pencils, pens, math sticks,
notebooks and textbooks. Girls all wear the same color headscarf, giving them a sense of pride
by wearing a “uniform”, which is a point many LUL students brought up – wanting to be “like the
formal school students” by wearing uniforms.
3.4.1 Lack of stipends and attendance
One difference between UNICEF and most of these educational interventions that target OOSC
and girls is the fact that UNICEF does not pay stipends or other forms of enticement to attend
classes. Despite the lack of such incentives, communities are very supportive of the ALCs and
there are no shortage of girls and OOSC who would like to attend. The (negative) effect of
paying stipends to attend classes on sustainability cannot be underestimated.
For one, the MoE does not have the budget to fund any of the CBE programming, let alone pay
stipends to girls. This is a strictly NGO phenomena, and controversial in that wanting to learn and
improve one’s life should come naturally, not as the by-product of what is in essence another
form of bribery in a country awash with bakhseesh.
3.5 Equity
The ALCs have significantly contributed to improving equity for girls in Afghanistan through
providing access to primary schooling where previously either none existed or was inaccessible
due to security and cultural restrictions on walking distances to school. Girls overwhelmingly
responded (91%) that without the ALCs they would be unable to attend school. Additionally,
older girls can now attend school, whereas they would have not had the opportunity previously
because even if the school was located in their village they would have been too old to participate
in the classes.
3.5.1 Increased learning outcomes
Learning outcomes also have significantly increased equity for girls, as now they are literate and
able to contribute to their families in a way they could not before. Early analysis of the data
gathered suggests that respect for girls increases in families and communities as a whole as a
result of their educational achievements.
34
3.5.2 Increased awareness of human rights
Participants are learning also about their rights as a girl, and as a human being. This awareness
has increased their confidence and belief that they have a positive future. The reasons for this
increased awareness are complex, though
foremost are 1) the Life Skills curriculum
taught in the early grades, and 2) literacy and
numeracy skills. Through the Life Skills
curriculum children learn to respect others,
especially elders, and most importantly,
themselves. Girls in the more northern
provinces, especially, expressed desire to
become teachers, doctors, lawyers, and
policewomen. (See Figure 3 Girls hoped-for
future career paths.) While some of this is
cultural, where doctors, teachers, lawyers and
police are viewed as highly respected in these
communities, the fact that girls now see
themselves having a future outside the home
is a critical step in increasing equity.
3.5.3 Impact of multiplier effect on
communities
The multiplier effect of ALCs in communities is
significant in terms of improving equity for
families and entire communities. With newfound literacy and numeracy skills, girls and boys are
helping their families with important life skills and facilitating social connections (e.g., read
prescriptions, wedding invitations, sign posts (directions), dial numbers on mobiles).
3.5.4 Community support for additional ALCs in neighboring villages and
additional grades
The LUL ALCs have educated approximately 9,000 students across 12 of the Central and Central
Highland provinces in Afghanistan. In terms of the relationship to population and demand, this is
a very small percentage of the need for primary education in the country. The United Nation
Statistics Division (UNSD)’s estimate of the Afghan population in 2013 is 34.4 millioni. The MoE
use UNPD’s estimates of population for planning purposes. Population growth according UNPD is
3%9, and if the population continues to grow at this rate, it will reach 42.6 million in 2020. Fifty
percent of the population is made up of children under 15 years old, and 36% of the population is
at school-age (about 12.4 million in 2013). With 3.5M OOSC children included in this 2013 figure,
the need for the LUL programme is great, and explains the results of the evaluation, where the
9 CSO estimates the population differently, pegging the growth rate 2.2%. With no nationwide census figures, there are conflicting demographic figures for the Afghan population.
Figure 6 Girls hoped-for future career paths
35
majority of respondents asked for an expansion of the programme into other villages and into
higher grades.
3.6 Learning
3.6.1 Mature students learn better
This pilot project has contributed to our understanding of how an educational innovation
contributes to student learning. Although assessment data has not been made available by the
PEDs to this point, classroom observations by the NE and seasoned enumerators from Soft
Power Solutions seem to indicate that students are learning, and many student may even be
ahead of their class peers in formal school. Several enumerators reported that the students they
observed seemed much more mature and knowledgeable than students they had observed in
formal schools on previous survey efforts.
3.6.2 Improved learning outcomes with smaller class sizes
An analysis of the data showed that respondents also felt that the smaller class sizes with the
additional attention from teachers aided in the positive learning outcomes for students. Many
classes have upwards of 30 to 40 or more students, making it impossible for teachers to provide
the individual attention necessary for optimal learning outcomes.
3.6.3 Policy ramifications of successful AL programming
The ramifications of a programme that successfully covers two classes in one year in terms of
formal education bring up an important strategic issue for the MoE and donors. If funding were
made available, the ALCs could be expanded to fully cover the remaining 3.5M OOSC, as well as
many of the thousands of children projected to enter the school system according to population
estimates – with the same learning outcomes, if not better, than their formal school peers.
3.7 Innovation
The ALC programme in Afghanistan has contributed two key innovations to LUL, and in the larger
sense supports innovation in the education sector in general, particularly in conflict-affected
nations.
Innovation #1: Out of school children (OOSC) ages 9 to 15 complete 2 grades per year = Gr 1-
2, 3-4 and 5-6.
3.7.1 Completing 2 grades per year
Students attend class the full year, even in winter, spending up to four hours in class per day.
Formal school class time is quite frequently only two hours per day, far below the recommended
number of hours. The primary reason for this shortened timeframe is due to overcrowding in
schools. The students are taught the same curriculum and use the same textbooks as in formal
school. Teachers are provided training in modern pedagogies through the standard
36
MoE/Teacher Education Directorate (TED) INSET training program given to teachers throughout
Afghanistan.
Modern pedagogies taught are learner-centered activities, such as collaborative project work and
questioning. Impact studies conducted in 2011-12 by Creative Associates under the BESST
programme (USAID) showed that the majority of teachers (over 75%) trained using the INSET
packages were successfully applying the techniques in their classrooms. Teachers observed in
this evaluation study have also been trained in INSET and the majority were practicing learner-
centered techniques in their classrooms.
The first cohort is now completing Grade 6, in February 2015, and transitioning to Grade 7
(Secondary School) in the formal school system. The accelerated learning schedule has retained
quality, while making it possible for older children who otherwise would have been unable to
attend school and to catch up to their younger peers, joining them in Grade 7.
Innovation #2: ALCs close to communities: children, especially girls, do not have to walk far to
attend classes.
3.7.2 ALCs co-located in communities
Children attend with local teachers in a building contained within or very close to their village. In
the first year of the LUL program, a survey was conducted to determine best locations for
schools. Whereas formal schools may be co-located between communities to maximize the
number of potential students attending, this creates an untenable situation for most conservative
and especially insecure areas of the country. Girls in those areas are at risk for insults and worse
yet, kidnapping and being trafficked, even over short distances. Thus, ALCs were located in
villages where there was no possibility for girls to attend school due to the distance involved in
walking to formal school.
3.7.3 Proximity to villages and recognition by MoE
In Afghanistan, CBE is considered an innovation because historically, formal schooling was the
only way a child could get a recognized primary school education. CBE has been available since
before the Taliban, and especially during their regime, when all girls school had to go
underground in people’s homes. However, teachers were not formally trained, standardized and
updated curricula was not available, and graduation could not be formally recognized by the MoE.
3.8 Challenges
Although overall the LUL initiative in Afghanistan is successful in terms of relevance and
effectiveness, and has every indication of being sustainable over time in many communities,
there are still significant challenges that need attention and resolution. Primarily, these
challenges revolve around education quality, and MoE ownership and programme administration.
Both quality and ownership are connected to a still weak and corrupt education system that
needs considerable capacity building. It is important to acknowledge that Afghanistan is not only
recovering from 30 years of conflict, as is oft mentioned in the literature, but it is building an
education system that never existed before, particularly in the geographically remote areas in
37
which LUL ALCs have been established. Many areas of the country now have primary and
secondary schools, and teacher training colleges, where none existed prior to 2001. Therefore,
institutional strengthening is essential at all levels in MoE, including building monitoring systems
that are appropriate for the environment in which LUL ALCs are located.
The overarching challenge for the ALCs lies in CBE in general: in effect, a parallel educational
system has been created that is funded by the donors to meet the critical need for primary
education in Afghanistan. Although the MoE is involved in all aspects of administering the ALCs,
there is no national budget for these schools. Therefore, in terms of sustainability, donor funding
will be required for the foreseeable future. One caveat is the finding in this evaluation that many
communities say even if UNICEF stops funding, they will support the school and find a way to
continue educating their children.
3.8.1 Effectiveness (Quality): Teaching and Learning Materials (TLM)
Of all the challenges mentioned by nearly every stakeholder and participant, the lack of teaching and learning materials (TLM) was number one. Lack of TLM was discovered in all provinces at nearly every ALC. TLM either do not reach ALCs on time or at all. TLM are fundamental for teaching effectiveness and improving learning outcomes. Seventy-three percent of teachers
reported that primary problem they face is lack of books and stationery. (See Figure 4 Problems faced by teachers
In an interview with the
ACO finance director, he
said that last year’s
(2013) delay in
distribution was caused in
large part by the order
itself being delayed due
to a lengthier than
anticipated approval time.
Learning from last year,
the finance director submitted all paperwork
well in advance and anticipates this year’s
TLM order will arrive on time for distribution in 2015.
UNICEF’s policy is to build country capacity wherever possible, thus, the TLM are given to the
MoE to distribute to the ALCs through the PEDs. However, with a combination of lack of
administrative capacity, the 2014 election standstill in the GIRoA, and ongoing corruption around
textbooks, even if the TLM arrive at the PED there is significant delay in distribution to the ALCs,
especially to the more remote schools.
The centralization and hierarchical nature of the MoE is an ongoing bottleneck to timely delivery
of TLM. During both the NE and enumerator team visits to the PEDs, nearly every PED Director
was in Kabul due to the election of the new President and Ministry appointments underway. This
12%
73%
3%12%
Please tell me about some of the problems that you face as a teacher
Lack of properlearning space
Lack of books andstationery
Poverty
Nothing
Figure 4 Problems faced by teachers
38
meant that any TLM that arrived could not be distributed because the Director’s signature is
needed.
TLM, in particular textbooks, are central to effective teaching and learning. With often only one
textbook in the class, students revert to rote learning, copying off the blackboards into their
books. This negatively affects the interactive teaching and learning taught in the INSET courses.
Without textbooks students cannot study in the home as effectively as they would if they had a
textbook, with its examples and pictures. This is a failure on the MoE’s part, as they are tasked
with textbook and TLM distribution, paid for by UNICEF funds.
There was also a regional difference found in awareness of TLM. In Paktya, participants did not
seem aware of the fact that they are missing equipment and TLM in the schools, whereas in the
northern provinces participants, especially teachers and students, are keenly aware that they do
not have enough TLM or other learning aids (eg., lab equipment) that the formal schools have.
3.8.2 Effectiveness/Sustainability: Lack of MOE ownership
Though supported by the MoE and with a policy in place, CBE has been primarily under the
purview of the NGO community in Afghanistan. This lack of support is in part due to the massive
need for primary education in the country (estimates are 3.5M children are still out of school)
combined with a lack of capacity in the GIRoA and MoE to administer another sizeable program.
The lack of capacity in MoE affects the effectiveness of the ALC programme over the longer term,
as well as its sustainability. MoE is a full partner with UNICEF in implementing the ALC
programme, from surveying school sites to monitoring Centers and training teachers. Yet no
budget exists for CBE and the relationship between the majority of hub schools and their
associated ALCs is weak. The staff in the M&E Unit in the MoE was largely unaware of the
UNICEF ALC programme, as an example of the need to improve awareness and ownership,
particularly in the area of monitoring and evaluation of the Centers.
3.8.3 Effectiveness: Transition and hub school support weak
The ALCs visited by the NE reported strong support by the hub school principal, and the Kabul,
Bamyan and Panjir PED focal points said that there is full coordination of attendance, exam
scores and other student documentation between the ALC and the hub school. However, the
focal points for those provinces also noted that ALCs in the remote areas were likely not to have
the same strong linkages with the hub schools due to remoteness and lack of monitoring and
support.
In terms of transition to the hub schools after graduation from Grade 6 and desire to continue
their education in Grade 7, in schools visited by the NE all children raised their hands. Girls, in
particular, were keen to continue their education, and in many cases said their parents were also
supportive of them continuing on to Grade 7. Unfortunately, the reality for many girls is that the
hub schools are a great distance from the ALC, too far to attend due to security issues and
cultural restrictions. This fact, in combination with cultural practices, i.e., early marriage, means
that many girls will most likely not continue their education beyond primary school. In interviews
with community and Shura members, a number remarked that when girls are married they often
relocate to another village to the husband’s home, which presents a new set of challenges to
39
continuation of her education. The new family may not be supportive of education, or the
secondary school may be too far from her new village.
3.8.4 Effectiveness: WASH program
As part of the global LUL evaluation effort, the context within which this initiative is taking place is
being considered. In the case of the ALCs, this means the villages and houses in which the
classroom is located. Interrelated with the ALCs is UNICEF’s WASH program. With the new
head of WASH on ground at the ACO, Rolf Luyendijk, an effort was made to survey the WASH
facilities at the ALCs using the three main criteria of UNICEF’s WASH programme:
W1. Is there a functional water point at or near the school?
W2. Does the school have toilet facilities?
W3. Does the school have handwashing facilities?
The majority of the ALCs were found lacking in all three criteria. Taking into account the
classroom environment surrounds, the effectiveness of the programme is negatively impacted.
Respondents said that lack of proper facilities was detrimental to learning in the school, especially
where there was no separate facility for girls in coed schools.
3.8.5 Effectiveness/Sustainability/Equity: Lack of Teachers, especially
female, to teach girls and serve as role models
In particular in the more rural and conservative areas of the Central Region, lack of an adequate
number of female teachers negatively affects access to education for girls. In conservative
Paktia, the enumerator team found that communities, while supportive of girls education, were
reticent to send their daughters to a class taught by a male. Lack of female teachers is a
nationwide challenge that the MoE’s TED has been actively trying to improve since 2005. The
ALCs, with their innovative programme that enables teachers to remain in their communities, and
even teach in their own homes, offers a pathway to young women to serve and earn money to
support their families.
Though lack of female teachers is a challenge in some areas of the country, in other areas, for
example, in Daikundi, community members said that they would not mind having even foreign
teachers, including Westerners, teach their children as long as the teacher is a professional.
3.8.6 Effectiveness/Sustainability: Coed classes are a problem for more
conservative areas of the country
Co-ed schools are viewed as a problem in the more conservative areas of the country, in some
cases as shocking. Nonetheless, the students like the opportunity to learn, and learn different
subjects as well. Parents recognize that without this ALC program their children will never have
the opportunity to have an education because they are OOSC – overage for formal education.
Therefore, despite being uncomfortable with gender mixing, the communities interviewed
40
understand that they must have this co-ed arrangement, or have no education. Being enrolled
and being able to continue to the next level is extremely important.
3.8.7 Sustainability: Difficulties in monitoring in remote and insecure
locations and in local communities
In every province, monitoring was an issue. Yet this difficulty is inherent in the LUL programme in
Afghanistan due to its providing access to education for OOSC and girls, especially in
geographically remote and rural areas. Thus, the design of the programme calls for creativity and
persistence in monitoring approaches.
As expected, those ALCs in and close to city centres, such as Bamyan, received regular visits
from the PED focal point for ALCs. Yet those ALCs more than an hour’s trip via bus or taxi were
visited at most twice per year. Since Winter was setting in and the timeline for the evaluation
effort was compressed, visits to the more remote sites could not be accomplished, though it can
be surmised from findings in less remote areas that monitoring is an even greater problem in the
remote ALCs.
UNICEF provides a transportation allowance for the focal points in the PEDs to travel to the ALCs
and monitor the classes, yet this allowance does not cover transportation costs to very remote
areas, which must also include an overnight stay. Moreover, the NE observed in both Bamyan
and Kabul province that focal points lacked appropriate transportation to reach the rural ALCs,
such as motorbikes and 4-wheel drive vehicles. More visits to ALCs would provide much needed
support to teachers, and facilitate distribution of TLM, while ensuring students are attending
classes regularly.
UNICEF implemented a small pilot SMS monitoring project in the southernmost provinces that
had teachers regularly reporting back to Kabul. One of the issues discovered was the significant
delay in salary distribution by the PED. UNICEF pays the teachers, and relies on the MoE to
distribute the salaries in cash to teachers. Unfortunately, the program was discontinued due to a
lack of programme funds for messaging, but ACO colleagues have asked for additional funds to
re-start the monitoring program due to its success in monitoring the ALC programme.
3.8.8 Scalability & Sustainability: 3.5M OOSC and Girls’ Education
This evaluation has conclusively determined that the LUL initiative in Afghanistan has reached
intended participants and increased learning outcomes, and has even shown significant
effectiveness (impact) in many communities in terms of transformational learning (defined by
Mezirow (1991) as a change in perspective, or world view). With a shift in perspective toward
realizing and valuing girls education, and education in general, the ALC programme has a strong
chance of being sustained on some level, even in the absence of full MoE and donor support.
Participants were also enthusiastic about continuing support of the ALCs, even in the absence of
donor support, offering in-kind contributions. The LUL ALC model of having the community
provide classroom space ensures the community is supportive of the intervention. In several
locations, the NE visited classes held in teacher’s homes.
41
In nearly every community visited, participants asked that the ALC not only be continued for a
second cohort (where there is established need; some areas do not have enough remaining
OOSC), but that it be expanded to Grades 7 to 9. While this request is outside the scope of
UNICEF’s LUL program, and its programme in general in Afghanistan due to the mission’s focus
on improving access to primary education for children, nonetheless, this finding points to a
definite need for non-formal secondary education based on the innovative model of the ALCs.
The cost-effective model of LUL’s ALCs can easily be expanded to reach deeper into rural and
geographically remote areas of the country. Other iNGOs, such as BRAC, Aga Khan and Save,
have also successfully implemented CBE and ALCs in remote areas, so coordination between
MoE and the iNGOs would be necessary in determine which areas resources could be shifted
from where there is not more need for an ALC (e.g., some areas in Kabul). With a sizeable
population of OOSC in Afghanistan remaining, the ALC model can be implemented and adapted
to meet that need and ensure at least a quality primary education for all.
4 RECOMMENDATIONS
Recommendations have been triangulated with those from UNICEF’s ACO, MoE, other relevant
key stakeholders in CBE, such as Save and BRAC, and suggestions from respondents in the five
provinces surveyed for this evaluation effort. In order of importance, recommendations 4.1 (TLM
distribution), 4.7 (strengthening M&E) and 4.3 (MoE ownership) rank highest in terms of ensuring
quality and future sustainability.
4.1 Effectiveness: Improve distribution of Teaching Learning
Material (TLM) via accountability, communication and support
for capacity building in MoE
With TLM essential for ALC effectiveness, key stakeholders were all in agreement on this point: if
the MoE is unable to deliver, UNICEF should organize distribution to ALCs until such time as the
MoE has built enough capacity in order to ensure timely distribute to all Centers. Concomitantly,
to the extent possible, UNICEF should collaborate with MoE colleagues to provide administrative
support to MoE/PEDs and DEDs, in particular to assist them in planning ahead for the school
year. Planning remains a weak point across all Ministries in Afghanistan; MoE is no exception.
The second piece to ensuring every child, every teacher, in the ALCs has TLM is to significantly
strengthen monitoring capacity to check on timely procurement and delivery of textbooks and
stationery to all ALCs, regardless of remoteness.
All three of these recommendations -- 1) organizing distribution; 2) providing technical support for
administration; and 3) improving monitoring -- require budget outlays. Yet the benefits of
improved teaching and learning effectiveness may well outweigh the monetary expenditure.
As an interim solution, the yearly INSET teacher training should cover how to teach without
students’ possessing textbooks, and without resorting to copying from the blackboard (e.g.,
collaborative activities, use of local materials for science, etc.) – defeating the purpose of the
learner-centered approach taken by MoE.
42
4.2 Effectiveness/Sustainability: Establish a tracking system
for the graduates
With the successful graduation of the first cohort from the ALCs, the goal for these girls remains
unclear. The majority will most likely not attend secondary school due to the distance from their
village to the hub school, or because they will marry young and go to live in another village. At
present there is no tracking proposed to find out what will happen to these girls, or how their lives
have improved compared to a control group who did not obtain a primary school education, or a
group who obtained a formal school primary education.
Establishing a longitudinal study for graduates of the ALCs in the MoE’s M&E Unit would serve to
provide valuable information on the benefits of a primary school education, in particular through
accelerated learning. In terms of educational policy and strategic thinking to solve critical
education issues in a rapidly growing population, a study of this type would also provide data to
establish whether or not an accelerated course of study is comparable to a formal school course
of study.
4.3 Sustainability: Strengthen MoE ownership of ALC
programme
Strengthening MoE ownership of the ALC programme is important for sustainability, as well as for
effectiveness and coherence. Through encouraging key stakeholders in MoE, primarily in the
CBE and M&E units, to take increasing ownership of the program by continued inclusion in
planning and monitoring activities, donors expect that eventually the ALCs will become integrated
within the MoE’s budget and fully supported by the GIRoA over time, with the NGOs eventually
taking a support role only. CBE should have a prominent role in the new education sector
strategic plan, with clear strategic objectives, targets and budget estimate.
The change in government to Ghani has brought about a virtual freeze in any activity since April
2014, and until the cabinet is finalized and appointments made, it will be difficult, if not impossible
to make any substantial progress in this area. Afghans interviewed by the NE are hopeful about
the change in government, and the improvements it will bring to the country.
Providing support for capacity building and strengthening linkages within and between key
stakeholders is another area that is somewhat outside the ACO’s mandate for LUL. Nonetheless,
if it is possible to provide additional support (i.e., funding) to MoE for training and capacity
building in the area of CBE, AL and M&E that would continue to raise awareness and build
competencies, that support would facilitate ownership of the ALC programme.
4.4 Effectiveness: Support student transition and hub school
administration
The ALC model is integrated into the MoE formal school system through the attendance records,
which are shared with the local “hub”, or formal secondary school. The hub school principal has
responsibility for ensuring a place in school for the ALC graduates in order to mainstream them
43
into the formal educational system. Students in the first cohort of graduates from the ALCs are
slated to transition in Grade 7 formal school, i.e., the local “hub” school, in 2015.
The initial review of hub school linkages showed that they were fairly strong, though these
schools were in locations that are relatively accessible. PEDs and DEDs should be supported and
encouraged to continue to support hub school linkages with ALCs where weak. Monitoring plays
a key role in this support, and must be significantly strengthened.
Several MoE officials, including focal points, said that school principals need further training in
how to effectively integrate with ALCs to ensure graduates of the ALC programme are fully
prepared and have a place in the secondary school. Preparatory activities could include an
orientation day at the secondary school for parents and children, as well as hosting a short
workshop for teachers on the ALCs and what to expect from those students.
The problem of transition for girls into formal school Grade 7 is serious and wide-ranging, but
outside the scope of the LUL intervention. Other NGO interventions, such as BRAC’s support of
CBE through secondary and even high school, enable girls to fully complete their education.
However, UNICEF’s policy and programmatic concern in Afghanistan is primary education.
Nonetheless, perhaps UNICEF/ACO could partner with BRAC or Save on their CBE initiatives for
secondary school education for girls. The ACO should also continue to provide encouragement
and support to MoE in the area of CBE, perhaps suggesting a transitional programme. In a
number of villages, if girls had access to safe transportation they could attend secondary school,
but there is no MoE budget for this type of support at present.
4.5 Effectiveness: Coordinate with UNICEF’s WASH team to
improve sanitation at the schools
The lack of proper toilet and handwashing facilities was found to negatively impact the learning
environment in the school. No toilet facilities mean children cannot always attend the full class
period, particularly girls in coed classroom situations. Because the ALCs are not part of the
MoE’s formal school system, they do not show up in WASH statistics and are not included in
WASH programming. At the request of the new Chief of the WASH programme in Kabul, Rolf
Luyendijk, the ALCs were surveyed using the WASH in Schools Monitoring Package basic three
critieria. Mr. Luyendijk stated that he is eager to coordinate with the Education Department with
regard to the ALCs in order to improve sanitation conditions, and he has included CBEs in his
planned programming. The recommendation is for the Education Department and WASH team to
coordinate and organize an action plan to ensure the basic WASH criteria are met at every ALC.
4.6 Relevance: Continue supporting female teachers
The lack of female teachers continues to be challenging for ALCs, negatively impacting
effectiveness because parents are uncomfortable sending their daughters to school in mixed
gender classrooms and in those where the teacher is male. This situation is particularly evident
in the more conservative areas of the Central Region, such as Paktia. While outside the policy
scope of UNICEF’s mission in Afghanistan, continued support and championing of teacher
training for females in the MoE and in collaboration with the donor and NGO community will serve
44
to foster and sustain progress in this critical education area. Other intervention support around
increasing the number of female teachers would be: 1) encouraging flexible timings in TTCs; 2)
extending ALCs to higher grades (7 through 12); 3) providing monetary support through
scholarships and transportation costs as incentives to go to TTC; and 4) provide daycare for
young mothers without family support would improve sustainability over the longer term.
Extending ALCs to Grade 12 would one avenue toward making it possible for females to attend
TTCs and become qualified teachers. Perhaps partnering with another NGO to use some of the
current ALCs for higher grades would be one possible step toward solving the problem of lack of
female teachers.
4.7 Sustainability: Significantly strengthen monitoring in
remote and insecure locations and in local communities
Improving programme monitoring is one of the key recommendations thus far in this evaluation
study. The recommendation is to increase the monitoring budget to enable monitors to reach
remote ALCs, and to reach more ALCs more frequently (e.g., include motorbikes in budget).
Secondly, focal points in the MoE/PED recommended hiring a third party monitoring firm to cover
MOE areas since they are unable to monitor due to restrictions in budget and time.
Another idea that holds promise is the use of mobile technology (e.g., extend successful SMS
program, implement mobile monitoring). This evaluation proved the concept of gathering data via
mobile, and the successful SMS program implemented by the ACO that tracked, among other
data, teacher salaries, demonstrates the effectiveness of this form of data gathering and
reporting. With the majority of the ALCs well away from urban areas, yet within cell tower range,
using mobile technology to monitor the ALCs, with data points such as attendance, assessments,
textbooks, TLM and UNICEF funded items (e.g., teacher salaries, wood) routinely gathered and
uploaded to a central server, would ensure the programme is being implemented as envisioned.
Problems would be immediately reported so that action could be taken by UNICEF and the MoE
to resolve them soonest, textbook delivery being at the top of the list.
4.8 Scalability & Sustainability: Addressing 3.5M OOSC and
Girls’ Education
The need for primary education in Afghanistan is still massive, with many girls and older children
such as those served by the ALCs unable to attend school. In the majority of the schools
surveyed, the PEDs and community requested that UNICEF continue with a second cohort of
ALC students where there is established need. In all four provinces except Paktya, respondents
asked that ALCs be opened in neighboring villages. Taking an equity approach, UNICEF should
work more on these conservative and poverty stricken areas to effect change in the number of
girls in school.
With the beginning of a promising new government, the ALCs themselves need to be looked at
strategically. Questions as to the advisability of what in effect is a parallel system of schooling
and the future of the graduates need to be looked at closely in terms of scalability and
sustainability. In many cases, especially in the south of the country, mainstreaming of girls into
45
the formal education system is impossible due to the lack of girls schools and female teachers.
CBE offers one solution, though again the lack of trained female secondary school teachers is a
limiting factor.
One recommendation arises from the history of the education system in Afghanistan, where in the
1960s and 70s, the MoE gave certifications to teachers based on their educational level, e.g.,
teachers with only a Grade 9 could teach up to say, Grade 6, and teachers with only a Grade 6
could teach Grades 1 to 3. This flexible qualification system enabled the qualification of many
more teachers. All teachers were given in-service training and observed, ensuring the quality of
the education being provided. Older respondents in this evaluation study were of the opinion that
the educational quality was much higher a generation ago than it is today, even with many more
graduates of TTCs.
A number of respondents also asked for vocational training, in order to provide skills upon
graduation that they could then use to provide an income for their families. Although outside the
LUL initiative in Afghanistan, the notion of skills training, perhaps partnering with other NGOs,
should be considered, especially for girls.
Locations were sometimes problematic, since the ALC model requires the community to provide
classroom space. Mosques, especially, were considered by the teachers to be an unsuitable
location for classes due to the number of special services, funerals, and other religious events
occurring fairly regularly. Respondents suggested finding other options besides mosques, even
building a classroom facility, though building schools is outside the scope of LUL.
At nearly every site visited, it is clear that the MoE and participants are supportive of extending
the reach and number of ALCs into geographically remote areas. While scalability is not per se
part of this evaluation effort, the ALC model is scalable and, if funding were to become available,
hundreds more ALCs could be implemented in communities across Afghanistan. ACO
colleagues and other stakeholders in the CBE community suggested bringing in NGOs to
implement if MOE unable to do so.
The early findings suggest that it takes approximately two to three years for a community initially
reticent about sending its girls to school to accept an education intervention. Once the parents
see the benefits of sending their girls to school, (e.g., the benefits of becoming literate) they
become supportive of the ALC, which significantly increases sustainability in the community for
future classes and generations to come.
“By 2020 all children in Afghanistan, boys and girls alike, will be
able to complete a full course of primary education.”
Afghanistan’s MDG for Education
46
5 LESSONS LEARNED
5.1 Overview
This section presents lessons learned in the course of the evaluation the ALCs. In all likelihood,
however, these lessons represent knowledge commonly held within the UNICEF ACO Education
Office; adhering to these lessons in the course of development programming is not in all
instances possible. For this reason, and because the extent of knowledge within ACO is
unknown, this section should be properly considered as lessons learned, lessons to be learned
and lessons reinforced.
5.2 Plan for and Monitor Teaching and Learning Material (TLM)
Distribution
The problem of textbooks has plagued the education system in Afghanistan and has taken many
forms over the past decades, from the Soviet-era war-based curricula up to present day
corruption that finds textbooks locked in warehouses and cabinets – if they have even been
printed, and illegally sold in marketplaces. By many accounts, the donors have paid for the
textbooks now three times over, with the Danes providing most of the funding. (One NGO
interviewed for this evaluation had just signed off on $1M USD of textbooks for another CBE
programme.)
As part of UNICEF’s ongoing capacity building strategy for MoE, responsibility for the textbook
distribution rested on the central Ministry providing the textbooks to the PEDs, who then are to
distribute the textbooks to the ALCs through their offices and the DEDs. Stationery distribution is
also the responsibility of the Ministry, as is delivering wood for ALCs in the cold climates (all paid
for by UNICEF). Nearly every site visited reported problems with TLM distribution, ranging from
having no textbooks and stationery at all, to only obtaining copies of textbooks weeks into the
school year. Some school children were observed to have brand new backpacks, when these
were to have been distributed at the start of the programme, which means many children
(probably in the most remote areas) more than likely did not receive a backpack.
The lesson learned is to take a closer look into materials distribution historically in the
country/region and the level of corruption before deciding on a strategy for distribution, then
taking careful action to lower the risk of loss. Actions can include instituting simple tracking
systems for goods, and multiple checkpoints. Implementing a strong monitoring plan and system
will also increase the likelihood that TLM reaches its intended recipients.
5.3 Expand Monitoring and Pilot Mobile M&E
Monitoring of the ALCs is difficult by design. The objective of reaching OOSC in remote areas
calls for increased focus on monitoring. Without regular monitoring there is no evidence that
children are attending school, that they have adequate TLM, or that teachers are using their
INSET training.
47
The mobile data gathering component used in the data gathering showed that it is possible for
local Afghan staff and contractors to use mobile technology to gather data from the field;
however, this needs to be planned and tested well before data gathering starts. This is one
solution to address the lack of monitoring.
Afghan staff working for SoftPower solutions (the enumerator contractor) were able to enter data
from the field which was then downloaded via the web on low cost smartphones that were
purchased. Therefore, if a sufficiently simple survey system is developed it would be desirable to
trial a system with a select group of teachers to allow them to provide information directly to
UNICEF to provide continuous data during project implementation.
This technology holds promise to simultaneously: 1) help solve the problem of a lack of
programme monitoring due to accessibility issues; and 2) build capacity in M&E through providing
a simple, widely accessible instructional platform for basic M&E concepts and skills.
5.4 Consider Timeline of Innovation Integration
From analysis of the data emerged a timeline for acceptance of the ALC innovation, in particular
toward girls education. This timeline is modeled in Figure 5 Model of ALC Intervention. In the
first approximately two years of the program, respondents indicated that the more conservative
members of the community were unsure of the benefit to having their girls attend the ALC. Once
the girls began to apply their newfound knowledge in the home, particularly in conservative
Paktya, parents began to see the value in sending their girls to school. This cycle took about two
years (Grades 1 to 4). By the beginning of the third year (Grade 5) communities began to accept
and appreciate that the intervention was a benefit to them, although Paktya remained focused on
the benefit to family, rather than the community at large as observed in the other four provinces.
At the end of the third year (completion of Grade 6 and primary school), there was a noticeable
change in values in many communities who previously were unsupportive of education, especially
for their girls.
In terms of innovation, this timeframe follows Roger’s (1983) diffusion of innovation model, where
early adopters, change agents and opinion leaders all play a vital role in the acceptance of a new
idea into the group. Once acceptance reaches critical mass, the idea takes off, and most of the
community will then follow.
48
For the ALCs, this means the second cohort will have much more parental and community
support than the first cohort. If ALCs are scaled up, they will most likely face the same skepticism
that the initial cohort experienced. MoE and ACO support will be needed to socialize the idea
and then support the ALCs in the early years, until the value of the education to family first, then
communities, can be experienced and appreciated.
6 CONCLUSION
The ALCs in Afghanistan represent a promising way forward to improve equity for girls and
OOSC in remote areas of the country. Through providing a primary school education to these
children who otherwise would not have had the opportunity for an education, families and entire
communities benefit from increased literacy and numeracy. Most encouraging are the findings
relating to the creation of civil society, wherein the finding showed an increase in respect for
elders, a decrease in violence and an increase in awareness of human rights, particularly
amongst girls. In the majority of communities visited, respondents asked that ALC grades and
the schools themselves be extended and expanded, respectively.
The challenge to the future of the ALCs lies in the hands of the new government which came into
power in 2014. Though the MoE is a strong partner in CBE implementation, capacity and budget
are nearly non-existent to support the current programme, let alone scale it to meet the demand
of a growing population, half of which are school age. Continuing support from UNICEF/ACO will
be needed to ensure the most marginalized girls and older children have an opportunity to learn
and contribute to their families, communities and the future growth and stability of Afghanistan.
Figure 5 Model of ALC Intervention
49
ANNEXES
Annex 1 Methodology
The annex presents for the full methodology for the evaluation, which has been shortened in the
body of the final report due to restrictions on overall length.
The approach used was a self-evaluation exercise with a primary focus on learning and a
secondary focus on accountability, using a common evaluation/learning framework developed for
the global LUL evaluation. The methodology is primarily qualitative in nature, using content
analysis and constant comparison of narrative data to validate emerging themes. The NE and
enumerator team covered five provinces and more than 30 ALCs, for a total of over 300
participants (teacher, community, hub school principal, students) surveyed, including those in
focus groups.
Data collection was conducted first by the NE in October/November, followed by the larger data
gathering effort by the enumerator team the latter part of November. The NE conducted a
training workshop for the local enumerator team the first week in November that introduced the
methodology and questionnaires, along with the mobile application.
Methods and sources include a review of programme documents, interviews of key informants,
interviews and/or focus groups of important groups of beneficiaries, testimonials involving
beneficiaries (videography) and direct observations of activities with beneficiaries wherever
possible. A retrospective baseline has been established in consultation with programme staff and
relevant MoE staff to assess to extent possible contribution to quality of education (e.g., learning
outcomes, reduction in violence, as per the Child Friendly Schools framework).
Mixed-methods approach
The LUL Afghanistan evaluation combines qualitative and quantitative methods in alignment with
each of the other four national evaluations.
Qualitative methods comprised the chief means of collecting data regarding this LUL initiative.
Substantial focus on qualitative approaches has helped to ensure that the evaluation is
participatory, equitable, and open to unexpected information.
Focus Group Discussions (FGDs)
FGDs comprised the primary means of field-level data collection or collection of information about
the experiences, opinions and ideas of children, teachers, parents and community members at
large (e.g., School Shuras).
Semi-structured interviews
The NE and local enumerator firm conducted semi-structured interviews with key informants
within the Ministry of Education (MOE), UNICEF ACO, and among other stakeholders and
partners in Kabul and the provinces.
Videos
A video camera crew recorded videos documenting the influence of LUL on participants in Herat.
50
The videos have the potential to demonstrate the ways in which narrative reflection contributes to
the participatory construction of knowledge. In other words, these methods have a degree of
indirect value in relation to participatory evaluation, but the also have direct value for participants
in terms of building both competency and knowledge.
FGD responses were initially coded by the enumerator firm, followed by the national evaluator,
to enable these responses to be aggregated and analyzed via descriptive statistics (i.e.,
means/medians, percentages, cross-tabulations) and content analysis (qualitative). Designing,
testing and implementing a coding method will also support comparisons of this 2014 formative
evaluation with formative and summative evaluations that are conducted subsequently.
In addition, results of FGDs in relation to specific evaluation questions will be summarized by the
national evaluator, and supporting and/or insightful comments from respondents are being
transcribed for inclusion in reports and other documents.
Quantitative methods focused on analysis of existing data relevant to the LUL program; such
data will be used to help establish “retrospective baseline” information in relation to programme
impact, and to create a foundation for subsequent evaluative efforts that will benefit from ongoing
data-collection efforts on the part of the MOE and others. EMIS (Education Management
Information System) data collected by the MOE, for example, will be reviewed and assessed so
as to support future determination of the impact of LUL on broad indicators related to education
access and quality, such as school leaving, school completion, Net Enrolment Ratios,
advancement of boys and girls, among other categories.
In-Depth interviews by the NE and narrative analysis were conducted in Kabul Province and
in two provinces outside Kabul, Panjir and Bamyan, in order to answer the “how” question to
inform better understanding of outcomes from LUL on participants and the surrounding
community.
Sample size and sampling methods
Five provinces were highlighted for this evaluation: Bamyan, Daikundi, Kabul, Panjshir, and
Paktya. The focus was on ALCs located specifically in 14 districts and 30 villages already
selected and identified by the ACO. These provinces have been selected by the ACO due to
their accessibility and based on the ACO’s sampling framework, Probability Proportional to Size.
In addition to key donor, NGO and MoE stakeholders, primary participants in the Afghanistan
study are teachers (male and female), School Management Shuras and community members,
and students, especially girls. Five provinces were selected, and six ALCs in each, for a total of
30 field sites.
PPS is used with surveys in which the probability of selecting a sampling unit (e.g, school, village,
zone, district) is proportional to the size of its population. It gives a probability (i.e., random,
representative) sample. The basic steps to develop the sample are as follows:
1. List all schools in the project area.
2. Calculate the running cumulative population and find the total.
3. Determine the number of sites that will be visited and the total sample size desired.
4. Divide the total population of the project area by the number of sites to visit. The result is
called the Sampling Interval (SI).
51
6. Choose a Random Start (RS) number and calculate the series, each number of which
corresponds to a site on the list. Continuing in this manner, the desired number of sites will
be selected.
Reliability. To assess the reliability of collected information, and to increase the reliability of
findings, evaluators will make efforts to “triangulate” information—comparing accounts or
statements on a topic from multiple sources. The three primary methods of triangulation will
include:
Design of semi-structured interview and FGD scripts
These scripts will, in advance of data collection, include areas of overlap or redundancy to
support triangulation of information in anticipated categories.
Responsive re-structuring of interviews and FGDs
The NE and enumerator team will note key opinions and facts expressed by interview or FGD
respondents and will augment scripts and guidelines to support triangulation in relation to
information in new categories.
Developing inferences from review of existing datasets
To the extent that existing education datasets (e.g., enrolment, completion and other data) are
available and can be disaggregated, data of this sort will be used to “cross-check” results.
Reliability of the evaluation findings will be problematized by many factors that are, at this point,
unknowable. To the extent that these influences can be mitigated by a combination of structured
evaluation processes and triangulation, evaluation findings will be reliable.
Validity. The LUL evaluation is designed to be maximally participatory, after the fact of the
evaluation design, and via this means will help ensure the validity of findings. Direct and indirect
beneficiaries have not been involved in the design of the evaluation, its approaches or the
evaluation questions (which might be called participatory before the fact); however, these
beneficiaries have had, over the course of the evaluation process, the opportunity to share
opinions and experiences in an open-ended discussion format, ensuring that their stories are
captured.
Evaluation instruments
Instruments for this evaluation are qualitative. Three sets of instruments were created: 1) for use
by the NE; 2) for use by the enumerator team; and 3) for implementation on the mobile phone.
Quantitative information will be derived from prior data-collection efforts where available (e.g.,
MOE school- and student-management information, partners’ records, ACO project-monitoring
reports, etc.).
Data Collection
Afghanistan is the only country of the five LUL participating COs to require a national team of
enumerators. The reason for this is primarily insecurity in the country, along with deeply held
cultural beliefs in the rural areas that prevent outsiders from going into those areas altogether, or
from effectively communicating with the local population. Additionally, the scale of the
programme is much larger than the other four countries, and LUL funding is only supporting part
of a larger implementation of ALCs throughout most of the provinces in Afghanistan.
52
Enumerator Training
The NE and enumerator team travelled to an ALC in Kabul in order to practice skills and test the
instruments. (A separate report was filed in November 2014.)
Mobile Phone Application
Supporting the innovations in LUL’s approach, data gathering included the use of mobile phones.
This innovation has been well received by the enumerator team and holds promise for future
monitoring and evaluation of ALCs and other LUL programs. In terms of monitoring for equity,
mobile data gathering appears to be successful.
Data Analysis
The data analysis methodology used was primarily qualitative in nature, using content analysis
and constant comparison of narrative data to validate emerging themes. A codebook was
developed based on the lead evaluator’s suggested codes for the global evaluation and shared
with the enumerator team. The codes were then modified by both the NE and enumerator team
lead, with additional codes added as they emerged from the data analysis. AtlasTi, a qualitative
software analysis program, was used to support coding and analysis. Interview documents were
translated into English and uploaded into the program, where they were coded and queries run.
An ongoing conversation between the NE and enumerator team lead, who is a local national and
thus familiar with the culture and education system in Afghanistan, served to clarify and support
findings as they arose.
Concept mapping aided in the data analysis to further refine relationships between findings.
Concept mapping software developed by the IHMC in Florida was used to create and manipulate
the maps. This free software download enables graphical representation of concepts and
relationships, as well as collaboration on files through IHMC’s central servers. A pictorial
representation was also helpful in explaining and verifying findings to the ACO M&E officer.
Limitations of the methodology
Chief limitations of the methodology included:
Indeterminate quantitative data
The LUL evaluation has relied on prior data-collection and analysis efforts for its
quantitative component. Data on ALCs, including number of teachers and attendance
figures, was provided by UNICEF and has not been independently verified, (i.e., number
of girls in attendance remains nearly identical from Year 1 even though the NE visited
one school wherein four girls had dropped out).
Lack of baseline data
A “retrospective baseline” using available data from before the start of initiatives and from groups
outside the beneficiary group has been attempted based on early reporting and interview data.
Unrepresentative samples and resulting bias
53
The evaluation design relied on ACO to mitigate bias generated by unrepresentative samples.
Given the onset of winter, geographical remoteness of many ALCs, and delayed timeline, ALCs
sampled are biased toward those that are fairly easily accessible.
Response bias
Response bias in the FGDs is indeterminate.
These limitations, were mitigated to the extent possible by gathering a substantial amount of data,
more than 300 interviews and FGDs, from a number of sources (e.g., Shura members, teachers,
principals, students, community members) that were then triangulated.
Evaluation criteria and questions
Evaluation criteria
Evaluation criteria were developed with reference to guidelines included in the initial Statement of
Work (SOW) and widely accepted frameworks and guidelines, chiefly that of the Development
Assistance Committee (DAC), and in consultation with UNICEF HQ personnel (Lisa Bender and
Kathleen Letshabo).
The DAC criteria provided a general framework and were adapted to the LUL programme in
Afghanistan. Overall, per Table 2: Evaluation criteria, the LUL evaluation focused on relevance,
effectiveness, and sustainability from among the DAC criteria and on the three LUL themes—
gender equity, learning and innovation. LUL pillars (OOSC, programmes for adolescent girls, and
outcomes for learners) were treated as crosscutting elements.
DAC evaluation criteria
The five criteria developed by the DAC, a committee of the Organisation for Economic
Cooperation and Development (OECD) were presented originally in “Principles for evaluation of
development assistance (OECD, 1991). These five criteria are:
Relevance
The extent to which the aid activity addresses the priorities and policies of recipients
and donors.
Effectiveness
A measure of the extent to which an aid activity attains its objectives
Efficiency
Outputs—qualitative and quantitative—measured in relation to inputs; activity costs
should be in line with national norms and provide intended goods and services
Impact
The positive and negative changes produced by a development intervention, directly
or indirectly, intended or unintended
Sustainability
The extent to which the benefits of an activity are likely to continue after donor funding
has been withdrawn; includes both financial and environmental sustainability
The DAC criteria, however, represent a framework for the evaluation of scaled, fully implemented
development-assistance programmes, not for pilot-scale programmes and programmes in the
early stages of implementation, such as those being assessed in the LUL evaluation. For this
54
evaluation, emphasis among the DAC criteria will fall on relevance, effectiveness and
sustainability; for reasons discussed in the Inception Report, efficiency and impact will be de-
emphasized within the LUL evaluation. Per discussion and agreement at the UNICEF HQ
conference in early October, 2014, the notion of coherence is also included as part of this
evaluation.
Relevance is assessed in terms of initiatives’ objectives in relation to beneficiaries’
views of education- and equity-related challenges, and in relation to beneficiaries’
perceptions of deficits in overall social and economic well-being.
Effectiveness is assessed in relation to stated objectives and relevant challenges (as
identified by stakeholders), and in relation to development goals.
Sustainability is assessed in terms of the relationship of initiatives to partners’
priorities, policies and demand on the part of local institutions, with attention to start-
up or developmental costs (e.g., content design and development) whether costs are
reportedly in line with those of similar programmes and/or services.
Coherence is assessed in terms of whether or not this pathway makes sense, and
how the ALCs fit in with everything else that is being done in terms of girls education
and OOSC’s access to education.
Effectiveness is of critical importance in the Afghanistan evaluation and the multi-country
evaluation. Per the DAC Glossary of Evaluation Terms, effectiveness will be considered as:
The extent to which the development intervention’s objectives were achieved, or are expected to
be achieved, taking into account their relative importance.
Effectiveness and objectives. Per the DAC recommendations, effectiveness is assessed in
relation to objectives as framed in initiative proposals and designs. As appropriate, objectives will
be augmented by information pertaining to scope and scale (e.g., number of schools, etc.).
Effectiveness and development goals. In addition, however, effectiveness will be gauged from
the perspective of development goals as expressed in programme documents (e.g., proposals),
policies and priorities of partners and government stakeholders, and via statements of direct and
indirect beneficiaries.10 Underlying this perspective, results more broadly, arraying objectives
“achieved” and “likely to be achieved” in relation to overall educational, social and economic
improvement.
De-emphasis of efficiency and impact. Two of the DAC criteria—efficiency and impact—are
de-emphasized in the LUL evaluation. These criteria are most pertinent to larger-scale or mature
development-assistance programmes, rather than to pilot projects. Impact as defined by the DAC,
similarly, presents an inappropriate standard for the short timeframes, adaptive approaches and
limited scales typical of pilot projects. Current and potential results will be derived from the
comparison of objectives (as part of assessing effectiveness), relevance and scalability.
10 Direct beneficiaries will be considered as students and other youth participating in LUL initiatives, accessing LUL resources or receiving attention (instruction) from teachers or others who participate in LUL training or professional development; indirect beneficiaries will include students’ family members and members of the communities in which students live.
55
Scalability
As part of the effort to appropriately assess the pilot-scale initiatives launched under LUL,
evaluation criteria will include “scalability” as a sub-criterion of sustainability. The working
definition of scalability will be:
The extent to which a small-scale or pilot activity is likely to be expanded to reach
larger populations.
Framed in this way, scalability complements and shares characteristics with DAC criteria,
including sustainability (increasing scale implies continuing benefits), efficiency (costs are a key
factor in scalability), effectiveness (ineffective initiatives will not be scaled) and impact (low-impact
initiatives should not be scaled). Partners’ interest in and willingness to scale are key factors and,
again, are related to sustainability as well.
In keeping with the views of many sources with regard to the DAC, this degree of
complementarity is to be expected: “Criteria often overlap, and the same data can be employed
for different criteria” (Better Evaluation, 2014).
Gender equity, learning and innovation
ALNAP (the Active Learning Network for Accountability and Performance in Humanitarian Action)
states that appropriate crosscutting themes that can sharpen the focus of the five DAC criteria
(ALNAP, 2006).
Gender equity will be considered in relation to the three DAC-derived criteria, as follows:
Relevance: the extent to which the intervention reflects perceptions of need by girls
and women, the extent to which the intervention is informed by in-depth consultation
with girls and women
Effectiveness: the extent to which activities and achieved objectives mitigate
challenges to girls’ education
Sustainability: the extent to which programme activities address priorities and policies
in ways that benefit girls in particular
Education quality. For the purposes of this evaluation, the quality of education is framed as
“outcomes for learners,” and is considered more broadly than the commonly used term, “learning
outcomes.”
Innovation and broadening programme emphasis. Innovation as a cross-cutting theme is
primarily considered, within the LUL evaluation, as adaptive response to challenges. However, an
additional aspect of the LUL evaluation involves assessing the innovative aspect of LUL, itself.
Summary of evaluation criteria
Table 2, below, summarizes the evaluation criteria that guided the LUL evaluation.
Table 2: Evaluation criteria
Criterion Source Working definition
Relevance DAC The extent to which the aid activity
addresses the priorities and policies of
recipients and donors
56
Effectiveness DAC + LUL SOW and
other documents
The extent to which the initiative
achieved stated objectives and the
relationship of those objectives to
development goals
Sustainability DAC + LUL SOWs and
other documents
The potential for continuation of benefits
after the initiative has been completed,
with attention to start-up or
developmental costs (e.g., content
design and development) as well as
partners’ priorities, policies and demand
on the part of local institutions
Scalability
(sub-criterion of sustainability)
USAID The extent to which a small-scale or
pilot activity is likely to be expanded to
reach larger populations
In Afghanistan, the following questions were explored:
• Relevance – To what extent do the ALCs address the priorities of the community?
• Effectiveness – To what extent did the ALCs attain their objective of reaching rural
populations, girls and out of school, older children and support female teachers?
• Sustainability – How can UNICEF ensure that the ALCs will continue to be funded by
MoE (integration) once LUL has ended?
The following cross-cutting themes will augment and modify these criteria, per Table 3:
Table 4: Cross-cutting themes
Theme Source Working definition
Gender equity LUL SOWs and other
evaluation-planning
documents
Support for girls’ participation
in school
Outcomes for learners LUL SOWs and other
evaluation-planning
documents
Changes in outcomes for
learners
Innovation LUL SOWs and other
evaluation-planning
documents
The use of new ideas,
practices or tools in response
to unanticipated challenges
The following questions were asked regarding cross-cutting issues:
• Gender equity – How do the ALCs contribute to gender equity in Afghanistan?
• Outcomes for learners – How do the ALCs improve learning and other outcomes for
learners?
• Innovation – How do the ALCs provide a response to the tremendous need for
improvement in access to primary education in Afghanistan?
Evaluation questions
The following specific questions guided development of approaches to gathering information from
proposed direct and indirect beneficiaries, such as students, teachers, families and community
57
members, and from partners and other stakeholders. (Questions derived from the three cross-
cutting themes list those themes at the end of the question.)
Innovation. Within the LUL evaluation, innovation is seen as contributing to effectiveness, and
sustainability (and potentially to efficiency and impact). Innovation, framed as the creative
response to unanticipated challenges, also reflects the resilience of programme design and
implementation.
The following working definition of innovation is proposed:
Innovation: The use of a new idea, practice or tool in response to an unanticipated need,
challenge, obstacle or desire.11
(For additional perspective on this approach, refer to UNEG, 2011).
For purposes of this evaluation, innovation includes the introduction of an approach or activity
that is new to an LUL country, or to schools or development agencies in that country. For
example, the ALC initiative is considered an innovation, because the introduction of accelerated
learning is new to Afghanistan schools.
“Newness,” then, is a relative concept in any case, and especially in relation to national and
international development activities. Emphasis should fall on adaptive responses of any sort—
dynamic changes in programme implementation, for example, that address unanticipated
problems. Innovation conceived in relation to adaptive responses maintains the conceptual
connection to resilience that is of primary importance for pilot activities.
Relevance
What are key challenges to children’s well-being and learning?
How are these or other challenges specific to girls’ learning and well-being?
Equity
How does the LUL initiative (as designed) address these challenges?
Equity
In what ways does the design and/or implementation of the LUL initiative address
challenges to student learning, their well-being, and their success in education and in
life?
Effectiveness
What is (or are) the objective(s) of the LUL initiative?
To what extent has the LUL initiative accomplished the stated objectives?
How has the LUL initiative, to the degree it has accomplished its objectives, affected
the learning and well-being of students, their families, and their communities?
11 This working definition draws on the definition of innovation by the pioneer innovation researcher, Everett M. Rogers, “an idea, practice, or object that is perceived as new by an individual or other unit of adoption” (Rogers, 1983).
58
What changes in terms of the education environment, educational opportunities or
out-of-school opportunities for girls have resulted from the LUL initiative?
Equity
When challenges arose, what adaptive responses emerged?
Innovation
In what ways did those adaptive and/or innovative responses address the needs of
girls and women?
Innovation and equity
What challenges to achieving change have not been addressed?
Innovation
How could the effectiveness of the LUL initiative be improved?
Innovation (and Equity)
Sustainability and scalability
How does the LUL initiative relate to MOE or partners’ priorities?
What current or potential partners are likely to have interest in scaling (or replicating12)
the initiative?
How enduring are the changes that have resulted from the LUL initiative?
What are their potential longer-term effects in relation to girls and women?
Equity
In what ways have activities resulted in outputs (e.g., learning resources,
administrative processes, etc.) that will reduce costs of scaling (or replicating)?
How could the sustainability of the LUL programme be improved?
Innovation
Coding framework for assessment of qualitative results
This evaluation effort followed multi-country guidelines for coding FGD and interview results that
were developed by the Lead Evaluator, and further refined during the October UNICEF workshop
in New York City. An initial coding framework based on the DAC criteria was developed and
shared with the enumerator team. Twelve codes were developed based on the global evaluation
framework. (See Annex A.) AtlasTi, a qualitative software program, is being used to code and
support the data analysis given the large volume of data gathered.
Data Collection
Two data collection efforts were conducted for the LUL evaluation in Afghanistan, the first by the
NE and the second, more extensive, effort conducted by the national enumerator team, Soft
Power Solutions. Due to the delay in awarding the enumerator contract and need to report on the
LUL intervention at the Nepal LUL donor conference in early November, the NE was on ground
12 Replication is considered to be a re-staging of a pilot-scale project in a different area with a similar number of treatment sites. Replication might be motivated by the intention to deliver proven benefits to a different population, to field-trial, review and refine implementation processes in different conditions, to confirm the “transferability” of prior pilot-phase results, or for other reasons.
59
mid-October. She conducted interviews with relevant stakeholders in Kabul and during three site
visits to Panjshir and Bamyan provinces, as well as to a rural area North of Kabul in Kabul
province.
The results of those interviews were analyzed and preliminary findings and recommendations
submitted to the ACO for review and comment before final submission to the Nepal conference
team.
Coding Framework
Codebook Let Us Learn: Accelerated Learning Centers in Afghanistan
Main RQ
Are LUL/ALCs achieving their intended outcome in Afghanistan?
Sub-Research
Questions
What is LUL/ALCs' progress and learning in relation to the programme's three
equity pillars?
Reaching out-of-school children (OOSC)
How have the program
innovations affected the ALCs'
outcomes?
Expanding girls' education
How have the ALCs affected
equity for girls education?
Improving the quality of outcomes for
learners
How has the quality of outcomes
for learners improved as a result
of the innovations in the ALC
approach?
DAC Criteria Description
Code
Abbreviation
Relevance Girls Education is taking place RelGE
OOSC are in school RelOOSC
Learning is taking place RelLRN
Effectiveness Challenges present in ALC CHAL
Challenges regarding Teaching &
Learning Materials CHAL-TLM
Girls Education is having a
positive effect on the girls Ged-Effects
60
ALC has changed attitudes and
behavior in community ALC-Impacts
Sustainability Community supporting ALC CommALC
Community supporting Teachers CommT
Community supporting Students CommS
Hub school linked to ALC SUS-Hub
Students are planning on going to
Grade 7 Formal School SUS-Gr7
61
Annex 2 Soft Power Solutions Debrief Report
UNICEF Debriefing Report
December 11, 2014
Submitted to:
UNICEF Education Department – Dr. Kerrin Ann Barrett (Technical Advisor and Education and
Evaluation Specialist at Nadya Mundo)
Submitted by:
Soft Power Solutions Consulting Services (SPS)
Authorized Contact
Nadima Sahar Executive Director Soft Power Solutions Tel: 0787055815
62
Introduction:
The UNICEF Let Us Learn (LUL) Program is widely perceived to be predominantly relevant to and
effective in addressing the education needs of out of school girls and as having a real and tangible
impact on the lives of people participating in the program in Kabul, Panjshir, Bamyan, Daikundi and
Paktya. Based on the debriefing session, the program has effectively helped the participants by
enhancing their education level while developing long-term benefits through increasing their self
confidence and making their communities more open minded towards girls’ education. Of those
participating in the ALCs, the program is viewed very positively not only for how the program has
helped its participants, but also for improving the perceptions people have toward girls’ education.
The strength of the program, which is largely driven by the ALCs and ownership and active
involvement of the community and the shura; also leads to the strongest critique of the program – that
the program needs to be upgraded to at least 9th
grade. Participants appreciate all they have gained
from the program, but express concern that due to the distance between the HUB school and their
villages, some of them may not be able to continue their education.
Although there were specific concerns about the program – such as the lack of a proper classroom
and adequate stationary and books – the overwhelming positive impact the program has directly had
on the lives of the participants is broadly viewed as something that needs to continue throughout the
country.
The sections below highlight the positives, challenges and recommendations for each of the five
provinces:
KABUL
Positives: A highly positive and successful aspect of the program, which is monitored on a
monthly basis by a PED representative, is that it locally owned and encourages the
involvement of stakeholders such as families and shura in the decision making process.
While the families provide the space, the shura members ensure the safety of the
participants and address any challenges that they face in coordination with the PED
representative.
Based on the enumerator team’s assessment, the participants’ level of education
/understanding is higher than those enrolled in formal schools. For example, the level of
understanding and comprehension of those enrolled in 6th
grade at ALCs is similar to those enrolled
the 9th
grade at formal schools.
“I want to complete my education and become an engineer in the future so that I could rebuild
my village.” Male Participant, Kabul
“My father took me out of school when I was very young so that I could assist him with animal
husbandry. Now, I am like a blind person because neither I can write nor I can read. However, I am
fortunate that both my daughter and son are literate. I want them to go out and learn as much as
there is to learn so that they do not face what I am facing now.” Shura Member, Kabul
63
“If I get married and if my husband does not allow me to continue my education, I would speak
with him at length, encourage him and educate him about the benefits of education until he
changes his mind and allows me to complete my education.” Female Participant,
Kabul
“…Earlier when we used to go to my father’s grave, we would always ask someone to help us
identify it since none of us could read the tombstone. Now, my daughter can read it and identify it
easily.” Parent, Kabul
“My aunt wanted to take my cousin to the doctor. When we arrived at the market, there was a
barbers shop next to the doctor’s office. Instead of going to the doctor’s office, my aunt had gone to
the barber’s shop and had asked the barber to give my cousin an injection.
Thankfully, since I could read, I told her that the doctor’s office is beside the barber’s shop and
that she had came to the wrong address…” Female Participant, Kabul
“Initially, I was not in favor of my daughter’s enrollment in the program. My daughter was too
hesitant to share her passion / desire of getting educated with me. Therefore, she had shared it
with her mom, who in turn shared it with me. The first time her mom asked, I flatly rejected the idea
of enrolling my daughter in the program. Then again, her mother came to me and said that she is
very keen in getting educated. Again, I refused her and told that such programs would turn her into
an infidel. Then my wife came to me again and said that my daughter had told her that so and so
from our village are also enrolled in the program, but they have not become infidels and also that
the program has religious subjects and teaches them how to pray, fast etc. Then I changed my
mind and enrolled her. Now I am happy. She assists her mom with the house chores, does her
studies, and has become well behaved. I want to enroll my other daughter too because it is a very
good program.” Parent, Kabul
“…Earlier, the male members of my family were respected more than the female members not
only because they were the bread winners but also because they were more educated. They were
given a priority in everything. Now that we are educated and can read and write, we feel we are
respected more within our families…we feel as equals and are considered as equals by our family
members as well.” Female Participant, Kabul
“I think this program (LUL) provides girls with a great opportunity to become educated and
productive members of the society.” PED Representative, Kabul
“This is a male dominated society and most of the doctors in our village are male. I want to
become a doctor so that a woman does not have to go to a male doctor to get treated”
Female Participant, Kabul
“…we cannot even read the Holy Book, but we are fortunate that our kids can and now they are
helping us learn more about our religion by reading us the Holy Book and teaching us other religious
issues.” Shura Members, Kabul
64
“My husband was living away from us and sending us letter and money from time to time. Since no
one could read in our house, therefore, we always had to take his letter to the neighbor to read. We
were a bit uncomfortable with this since our financial matters were out in the open and I always
feared that now the neighbor knows how much my husband has sent me, one day they might as
well rob us. Now, my younger daughter has the ability to read it for us.” Parent, Kabul
“Prior to their enrollment in this program, the students were wandering around in the streets
aimlessly and wasting their time, but now they have become well behaved. They pay proper
respects to the elders, say hello and know how to speak property.” Shura Member, Kabul.
“The community is very supportive of the program so much so that in our village, one of the local
businessmen donated 6 boxes of stationary along with a carpet for the classroom and wood for
winter. He also gave us and the students biscuits.” Teacher, Kabul
Challenges: Three of the main challenges facing the ALCs in Kabul were lack of a proper
classroom, stationary and books. The classroom space; provided by the community
members; though well maintained and with access to water, is under-equipped and lacks
basic necessities such as proper sanitation and heating / cooling facilities. This was more
evident in one of the districts where due to lack of a classroom, they had a coed class for
the girls and boys. This has led to a clash between the community residents so much so that
the ALC was closed for 15 days. Even though this issue was resolved later, however, they
still maintained that having separate classrooms for boys and girls is ideal considering the
cultural and religious barriers to coed education. In terms of stationary and books, a majority
of the ALCs surveyed, did not have sufficient access to them. In one of the ALCs, the team
had observed that they had only one book which was being shared by all students.
Recommendations: Almost all were in favor of increasing the number of ALCs in their
villages due to the community residents’ heavy demand for education. They were also
unanimously in favor of upgrading the program from 6 to 9 grade. They maintained that
since the distance to formal HUB school is a bit longer, therefore, a number of the
participants would not be able to continue their education after graduating from the ALC.
However, if the program is upgraded to at least 9th
grade, this would address this
challenge. Another recommendation that most people had was increasing the number of
teachers who are more technical and experienced. Lastly, they suggested having
equipments similar to those of formal schools such as a lab, flip charts and more English
and computer courses.
BAMYAN
Positives: Overall people are very supportive of the ALCS because it has enhanced their
capacity and level of education. The Centers get monitored on a monthly basis by their
respective HUB schools because the PED cannot come to the ALCs in remote areas due
to the distance.
65
“They used to give their daughters hand to marriage at 12 or so but now they are educated and the
girls know more about their rights…they have encouraged their parents and community members to
gradually get rid of this tradition.” PED, Bamyan
“…girls have become more active in our community as a result of their participation in the ALCs –
now whenever there is a community activity; they outnumber and outrun the boys in terms of their
participation and performance.” Shura, Bamyan
“In the past, whenever we received a letter, no one knew how to read it and we would go to the
adjacent village to find someone who could read it for us. Now fortunately, our kids have acquired
the ability to solve such problems...” Shura, Bamyan
“… I am ignorant and therefore blind. Whenever we go outside or attend a social gathering, my
daughter teaches me how to speak and behave properly among people.” Parent, Bamyan
“…I want them to complete their education, stand on their feet, become financially self sufficient and
assist the community.” Parent, Bamyan
“…Parent’s rights, praying, and reading the Holy Book are some of the things that I have learned
from my kids...” Parent, Bamyan
“…Earlier, whenever we would ask them something, they would run away, but now they say hello and
have become well mannered. I think not only them, but the community, as a whole has also become
more open minded.” Shura, Bamyan
“In the past, we were day dreaming about becoming a teacher, a doctor or an engineer. We always
thought that we would never be able to fulfill these dreams. Since our enrollment in the ALC, we
know that have become a step closer to our dreams and believe that if we try harder, one day we
would be able to achieve them. Now we no longer think of ourselves as weak individuals. We have
become stronger and more confident.” Participant, Bamyan
There was a time when families would get their daughters married at 12, but now is a time when a
12 year old girl has been selected from this village to participate in the 1 minute film representing
Afghanistan. That speaks for the positive change and the impact that this program has had on the
lives of the participants, their families and this community at large.” Teacher, Bamyan
Challenges: Some of the main challenges facing the ALCs in Bamyan is lack of a proper
classroom / space for learning, books, stationary, heating facilities and the distance between
the HUB school and the ALCs. Similar to Kabul, lack of a proper space for learning is a
major challenge in Bamyan. For example, one of the ALCs is being held at a mosque.
Whenever, there is an activity (e.g. a funeral etc), they have to cancel their classes, which
leads to disconnect in their studies. The space also has heating problems. Even though they
receive wood from the PED, but they receive it in the middle of the winter. Lastly, the HUB
schools are 7-8 hours away from the ALCs, which makes it difficult for those who graduating
to enroll.
66
Recommendations: Since there is a good demand for ALCs and the current ALCs do not
have the capacity to enroll more students; therefore, they have recommended for additional
ALCs to be opened in their villages. They have also requested for the ALCs to be upgraded
to 9th
or more grade because the HUB schools are very far. The community residents have
volunteered to provide in kind contribution in the form of a classroom, heating facilities and
others should that happen. Lastly, they have requested for sufficient stationary, especially
flip charts that they could use for group work.
DAIKUNDI
Positives: Overall people are very happy with the program and two of the positive aspects
highlighted by them are that the ALCs are located within their village and that it is
accelerated. Also a number of the respondents maintained that the teachers, who receive
a training once a year, are technical and teach the students effectively. As for monitoring, it
was less especially in more remote and rural areas because of the distance between the
PED and the ALCs (an average of 6-7 hours). Therefore, they were not monitored as
frequently as they should be.
When asked about whether having a male teacher was an impediment to their participation almost
all answered “it does not matter who the teacher is as long as we get educated. We could care less
if he is a male or a foreigner or someone else, as long they educate us.” Participant, Daikundi
“This program has enabled us to read and write. Now I can read signs and board as I pass them and
feel proud and confident.” Participant, Daikundi
“Earlier we could not do something as basic as making a call or finding someone from the contacts
list or even adding someone. Now my kids can help me do it.”Parent,
Daikundi
“Every day, my daughter teaches me whatever she learns at her class; so indirectly I am also
benefitting and getting educated through this program.” Shura Member, Daikundi
“…Most families are in favor of their girls getting educated because it has a replicating effect. By
educating one girl, you educate one community.” PED, Daikundi
Challenges: Similar to Kabul and Bamyan, the three main challenges facing the ALCs in
Daikundi is lack of books, a proper space and sufficient wood for the winter. Based on the
team’s observation and assessment, in a number of the ALCs, they were using outdated
books. In a few of the ALCs, they teachers and the parents had bought books for the
students. In one specific ALC, they had only two books that were used by all students. In
terms of the classrooms, most of them were being held at the teachers’ houses. In one
specific classroom, there were only 6 students when the attendance sheet had a list of 25
participants. When the team inquired, they learned that although there were 25 students in
that class, but the class was so small that it could only accommodate 6 students at a time.
Another classroom was being held at a teacher’s car garage. Lastly the team added that the
67
demand for the ALCs is high. There were older married women in addition to the girls who
had voluntarily enrolled themselves in the ALCs so that they could get educated.
Recommendations: Similarly, they recommended the ALCs to be expanded to other
areas in Daikundi and to be upgraded to 9 grade. They also positively reflected on their
experience using mobile phones. Their recommendation was to use Areeba instead of
Etisalat in the future since Etisalat does not have a good coverage in Daikundi.
PAKTYA
Positives: People have a very positive attitude towards the program as they believe that it
is reaching to out of school girls and empowering them by making education accessible
within their own village, closer to their houses. The program gets monitored by the PED on
a monthly basis.
“Girls’ access to education was very less in comparison to boys since a majority of them had to stay
at home either due to security or the tradition. This program is affecting this tradition by providing
girls with a better access to education. This has also affected the parent’s mentality as they have
become more open minded and thus supportive of their girls enrollment in ALCs.” Shura, Paktya
“Their attitude, in general, and more specifically towards the elders have improved. They respect the
elders and their parents more now…” Teacher, Paktya
“They have acquired the ability to understand and solve a problem in much better way.
Their overall hygiene has also improved.” Parent, Paktya
“In my opinion, one of the good things about this program is that it also teaches them religious
subjects and things like praying, reading the Holy Book, the five pillars of Islam and other
fundamentals of Islam.” Shura, Paktya
“Aside from the girls, we also have women participants who come with their kids and attend the
sessions. They say that even if they learn a single word, they would be the happiest and the luckiest
women.” Teacher, Paktya
Challenges: Some of the main challenges include lack of up to date books, stationary and a
proper classroom. Most of the classrooms are held at either at the teacher’s or shura
members house, which are neither warm or not have a proper carpet or a bathroom. As for
the books, most are out of date and have been purchased by the family members. A good
example of this was Chamkani, which had only one book and a poor attendance because
most girls were hesitant to attend due to security constraints.
Recommendations: A majority of the teachers in Gardez were male teachers and the
community members were more in favor of having female teachers so that the female
participants can feel comfortable. Although the teachers received professional development
trainings once a year, but they were not sufficient and additional trainings were
recommended so that the teachers become more technical. Lastly, they want the program
68
to continue and have suggested upgrading it to 12th
grade since the formal HUB schools
are at least 1.5 to 2 hours away from the ALCs.
PANJSHIR
Positives: All ALCs are monitored by HUB schools and the CBE focal point on a monthly
basis in Panjshir. One of the major affects of the program is that it has not only increased the
participants’ capacity, but has also made the family members and the community members
more open minded and supportive of girls’ education. The family members are so
encouraged by the positive effects of this program that now they want their children to finish
high school and even continue on to university.
“If we do not allow our daughters to study and if one of them becomes sick tomorrow, who will treat
them then? We cannot allow a male doctor to treat neither can we let them die, but if we educate our
girls, then we can solve this and many other such problems.”Parent, Panjshir
“They did not know how to behave earlier, but now they behave like a lot better. They have become
well mannered and have started paying more attention to others and respecting others.” Shura,
Panjshir
“They are able to read and write. They can read invitations whether it is a wedding invitation or a
funeral. They also know how to read the invitations be it a wedding or a funeral and inform us of the
venue, time etc. Earlier we used to take it to the village leader or a teacher to read it for us.” Parent,
Panjshir
“I will tell you very honestly, during war I was a fighter and had burned a few schools myself – but
now I regret having done that and encourage everyone both inside and outside my family to seek
education because I know the value of more today than ever. I know that if there is anything that can
heal this society and nation, it is education.”Parent, Panjshir
“I did not know to insert credit into my phone. I always had to ask someone to do it for me. Now my
kids now it and have taught me. This program has truly been a blessing for all of us.” Parent,
Panjshir
Challenges: Although the HUB school (UNICEF provides TLM through PEDs and DEDs)
provides them with stationery, it is not sufficient to address their needs. Also, they had one
book per class, which is a challenge for the students as most families cannot afford to buy
them books.
Recommendations: Similar to the ALCs in other provinces, the main recommendations
were upgrading the ALCs to 12th grade due to the distance between the villages and the
formal HUB Schools (3 hours of walking), which makes it difficult especially for the younger
participants to commute. They also recommended receiving wood earlier because the
weather gets cold earlier in there than other areas. Lastly, they requested for additional
ALCs both within their villages and throughout Panjshir.
69
Annex 3 Site Visits and Respondent List
List of Respondents interviewed and ALCs visited by Province (Soft Power Solutions
Enumerator Team)
Province District Sites Respondents Interviewed
Bamyan
Shiber
Bareki Dahan Sadbarg
1. Shura Members
2. Students
3. Teacher
4. Parents
Sadbarg e Kaloo
1. Students
2. HUB School Principal
3. Teacher
4. Parents
5. Shura Members
Yakawlang Qala e Bala Sar e Kanak
1. Shura Members
2. Teacher
3. HUB School Principal
4. Teacher
5. Parents
6. Students
Waras
Dahan e Nala Ghejin
1. Shura Members
2. HUB School Principal
3. Teacher
4. Students
5. Parents
Kamar Petap
1. HUB School Principal
2. Teacher
3. Parents
4. Students
Center Seya Khar Toghi
1. HUB School Principal
2. Students
3. Family Members
4. Teacher
5. Shura Members
6. PED Director / Manager
7. ALC Focal Point at PED
Daikundi Shahristan Katut 1. Parents
2. Shura
70
3. Students
4. Teacher
Shewana Qala
1. Parents
2. Teacher
3. Shura
4. Students
Ghuchgerd
1. Teacher
2. Shura Members
3. Students
4. Teacher
Miramor
Safid Sang
1. Teacher
2. Shura Members
3. Students
4. Parents
Mir Ghulam
1. Teacher
2. Shura Members
3. Students
4. Parents
Katta Seb
1. Teacher
2. Shura Members
3. Students
4. Parents
Kabul
Shakar Dara
Kareez e Mir
1. Teacher
2. Students
3. Shura Members
4. Family Members
5. Parents
Surkh Blandi
1. Students
2. Shura Member
3. HUB School principal
4. Teacher
5. Families
Mirbacha Kot Sara I Khwaja
1. Students
2. Teacher
3. Shura Members
4. Parents
5. Hub School Principal
Farza Farza - Mesli 1. Students
71
2. Teacher
3. Families
4. Hub School Principal
Qarabagh
Bagh e Alam
1. Students
2. Teacher
3. Shura Members
4. Family Members
5. HUB School Principal
Qaria Bazar
1. Student
2. Teacher
3. Family Member
4. Shura Member
5. HUB School Principal
Panjshir
Unaba
Unaba
1. Shura Member
2. Family Member
3. HUB School Principal
4. Students
5. Teacher
Zaman Koor
1. Students
2. Shura Members
3. Teacher
4. Family Members
Shutol
Koraba
1. HUB School Principal
2. Family Member
3. Shura Member
4. Students
5. Teacher
Saighwar
1. Teacher
2. Family Members
3. Shura Member
4. Students
Rukha
Khuja
1. Family Members
2. Shura Members
3. Students
4. HUB School Principal
Piawasht
1. Student
2. Family Members
3. Shura Members
72
4. Students
Paktya
Gardez
Choonai
1. Students
2. HUB School Principal
3. PED Manager
4. PED Focal Point
5. Shura Member
6. Teacher
7. Family Member
Rabat
1. Family Members
2. Students
3. Head of Shura
4. HUB School Principal
5. Teacher
Mootai
1. HUB School Principal
2. Shura Member
3. Students
4. Teacher
5. Family Member
Sar e Ghondai
1. Family Member
2. Shura Member
3. Teacher
4. Students
Chamkani
Peerkalai
1. Shura Member
2. Family Member
3. Students
4. Teacher
5. HUB School Principal
Kuja Merab Kalai
1. Students
2. Teacher
3. Families
4. Shura Members
List of Respondents interviewed and ALCs visited by Province (National Evaluator)
Province/
District
Site/Organization Respondents
Kabul MoE Arian, Director of Planning
Muhammad Hosein Vahidi, M&E Officer, M&E
Unit Jalaludin Atayee, Director of M&E Unit
73
Province/
District
Site/Organization Respondents
Kabul UNICEF ACO Cristina Brugiolo, Education Specialist
Siping Wang, M&E Specialist
Etsuko Matsunaga, M&E Specialist
Ahmad Fawad Jamshid, Education Program
Assistant
Mirwais, Finance Program Assistant
Lailoma Hassani, Education Officer
Nazifa Aabedi, Education Officer
Panji Chamdimba, Education Specialist
Khadija Bahram, Education Officer
Khatera Afghan, Education Specialist
Richard Stanley, Technology Specialist
Zabihullah Kamran, Information Officer
Rajat Madhok, Communication Specialist
Kabul PED Mohammad Naeem Bahee, General Education
Director
Kabul
Province
PED/DED – District Kalakan Faisal Rhaman, District Director
Rasul Nawabi, Director
Abdul Hamid, MoE Focal Point for ALCs
Kabul Zemah Now ALC
Konoka ALC
Qala e Khoja
Teacher
Students, Community Members, Shura Members
Bamiyan UNICEF Field Office Hussain Mohseni, Education Officer, UNICEF
ALC Focal Point
Bamiyan PED Dehati, Deputy Director
Ali Madit Khan, ALC/CBE Focal Point
Habibullah Haidri, ALC/CBE Focal Point
Bamiyan Seya khar Toghy ALC, Village of
Ahangaran, Bamyan City
Sultan, Teacher
Students, Community Members
Bamiyan Shinwari ALC, Parwan District Teachers, Students, Community Members
Bamiyan Kohna Qala Sad barg ALC,
Kowakul Village, Shiber District
Parotali and Chamali, Teachers
Students, Community Members
Bamiyan Bareki Dahan Sadbarg ALC,
Shiber District
Mohammed Salfi, Teacher
Students, Community Members
Panjir PED Deputy Director, UNICEF ALC Focal Point
Panjir Qormbo ALC, Unaba District Ahmad Shah and sister, Marja, Teachers
Students, Community Members
Panjir Unaba ALC, Unaba District Teachers, Students, Community Members
Kabul Save the Children Marianne O’Grady, Programme Director,
STAGES
Kabul Sahak ALC, Bagrami District *Observation Visit with Soft Power Solutions
74
Annex 4 ALC Statistics
The table below was provided by the UNICEF Education Office and reflects their most up to date information as of October, 2014, for the ALCs in the 12 Central
Regions of Afghanistan.
Male Female Total Girls Boys Total Dari
10Parwan
34 15 19 34 783 0 783 611
Number of Teachers
Statistics of the Accelerated Learning Classes ( ALCs)in the Central Regions - 2014
# o
f
Dis
tric
ts
Province
ALC G1-2 ( 2012)
# of
ALCs
Number of Sudent
Pashto Male Female Total Girls Boys Total Dari Pashtu Male Female Total Girls Boys Total Dari Pashtu Male Female Total Girls Boys Total Dari Pashtu
172 34 29 39 68 783 0 783 611 172 34 29 39 68 783 0 783 611 172 34 29 39 68 783 0 783 611 172
Number of Teachers # of
ALCs
Number of Teachers
Statistics of the Accelerated Learning Classes ( ALCs)in the Central Regions - 2014
ALC G3-4 ( 2013)
Number of Teachers CommnetsNumber of SudentNumber of Sudent
Total in 2014 ALC G5-6 ( 2014)
Number of Sudent# of ALCs
# of
ALCs
6 Panjshir 23 8 15 23 502 18 520 520 0 23 23 23 46 502 18 520 520 0 23 18 28 46 502 18 520 520 23 18 28 46 502 18 520 520
7 Kapisa 28 25 3 28 579 140 719 538 181 28 42 14 56 579 140 719 538 181 28 42 14 56 579 140 719 538 181 28 42 14 56 579 140 719 538 181
7 Bamyan 18 7 11 18 245 99 344 344 0 18 21 15 36 245 99 344 344 17 19 15 34 240 93 333 333 0 17 19 15 34 240 93 333 333 0
8 Daikundi 40 10 30 40 851 241 1092 1092 0 40 27 53 80 851 241 1092 1092 0 40 27 53 80 851 241 1092 1092 0 40 27 53 80 851 241 1092 1092 0
12 Paktiya 20 18 2 20 500 0 500 25 475 20 28 12 40 500 0 500 25 475 20 28 12 40 500 0 500 25 475 20 28 12 40 500 0 500 25 475
15 Paktika 20 20 0 20 309 304 613 7 606 20 30 10 40 309 304 613 7 606 20 30 10 40 309 304 613 7 606 20 30 10 40 309 304 613 7 606
10 Khost 42 39 3 42 1057 100 1157 1157 42 72 12 84 1057 100 1157 1157 42 72 12 84 1057 100 1157 1157 42 72 12 84 1057 100 1157 1157
14 Kabul 40 28 12 40 898 0 898 389 509 40 44 36 80 898 0 898 389 509 40 44 36 80 898 0 898 389 509 40 44 36 80 898 0 898 389 509
8 Ghazni 37 11 26 37 745 120 865 529 336 37 23 51 74 745 120 865 529 336 37 23 51 74 745 120 865 529 336 37 23 51 74 745 120 865 529 336
7 Logar 42 39 3 42 785 231 1016 240 776 42 59 25 84 785 231 1016 240 776 42 59 25 84 785 231 1016 240 776 42 59 25 84 785 231 1016 240 776
9 Wardak 40 39 1 40 648 204 852 163 689 40 60 20 80 648 204 852 163 689 40 60 20 80 648 204 852 163 689 40 60 20 80 648 204 852 163 689
113Total
384 259 125 384 7,902 1,457 9,359 4,458
Dari Pashto
1 Parwan 27 7
2 Panjshir 23
3 Kapisa 20 8
4 Bamyan 18
5 Daikundi 40
6 Paktiya 20
7 Paktika 20
8 Khost 42
9 Kabul 17 23
10 Ghazni 23 14
11 Logar 9 33
12 Wardak 8 32
Total 185 199
No. Classes Province
4,901 384 458 310 768 7,902 1,457 9,359 4,458 4,901 383 451 315 766 7,897 1,451 9,348 4,447 4,901 383 451 315 766 7,897 1,451 9,348 4,447 4,901
75
Annex 5 List of Documents Consulted
Summary Results Matrix: Government of Afghanistan – UNICEF Country Programme, 2010 –
2013
Afghanistan Country Programme Document 2010-2013
Education Interim Plan 2011-13 version 5 (GIRoA/MoE)
Combined BEE/Let Us Learn Human Interest Stories (2013)
Table 1. One-year and six-month extensions of ongoing country/area programmes approved by
the Executive Director (2013)
Mid Term Review of UNICEF Afghanistan Country Programme 2010-2013, Executive Summary
UNICEF Afghanistan: Country Programme Expenditure And Costs (December 2011)
Afghanistan Aid Effectiveness in the Education Sector in 2010
Bringing Education to Afghan Girls: A Randomized Controlled Trial of Village-Based Schools
(Burde, D., 2013)
76
Annex 6 Instruments (National Evaluator, Enumerator Team and
Mobile)
Interview instruments
Overview: Interview Instruments (National Evaluator)
The following interview instruments are tied to the evaluation matrix that appears in multi-country LUL
inception report so as to best support the multi-country evaluation of LUL, and are focused on the
evaluation of the LUL ALCs in Afghanistan. They are intended for use by the National Evaluator; a more
simplified version has been created for the local enumerator team, including mobile instruments.
Instruments included here address:
Central stakeholder: UNICEF CO personnel
(and field-office personnel)
Central stakeholder: MOE personnel
Other stakeholder: NGO personnel
Field visits: District Education Officer (DEO) [same as MOE personnel]
Field visits: School principals (hub schools)
77
Afghanistan INTERVIEW guide:
ALCs Programme
Guide for interview with UNICEF CO personnel
Criterion: Topic or theme
Relevance
Relevance: Getting started Please tell me about your CO’s activities in relation to the
ALCs. Please describe your activities in relation to:
Planning
Design
Implementation
Monitoring
Relevance: Getting started 2 Please tell me about your own involvement with the ALCs. Are
you familiar with it? If so, how?
Relevance: Challenges In your opinion, what are the biggest challenges that students
face in relation to education?
How do these challenges affect student learning?
Relevance: Challenges & equity Are these challenges different for girl students?
If so, please describe these differences.
(e.g., differences could stem from impact, intensity, frequency,
etc.)
Relevance: Challenges & Equity How do the ALCs address challenges to girls’ education?
Which of these challenges do the ALCs address most
successfully?
(For each challenge named)
In your opinion, is this challenge important? Please tell me
why or why not.
78
Effectiveness: General What are the most important changes brought about by the
ALCs?
Why are these changes important?
Effectiveness: Equity In what ways do the ALCs specifically change conditions for
girls in school?
Why are these changes important?
Effectiveness: Equity What important challenges to girls’ education do the ALCs not
address?
How could the programme be changed to address these
challenges?
Effectiveness: Learning &
Innovation
The ALCs may have been implemented differently in different
schools. If relevant, please describe the various effects and
the importance of these variations.
In your opinion, which of these are most important and most
effective?
Effectiveness: Learning &
Innovation 2
What obstacles has the ALCs encountered in relation to its
launch and implementation?
How have these obstacles affected the programme?
Has the programme been changed to respond to these
obstacles? If it has, please tell me how.
How did the UNICEF CO view or contribute to these
responses?
In what ways did other organizations contribute?
How effective were these responses? And why?
Effectiveness: Learning Are there potential changes to the programme that might
make it more effective, but that we haven’t discussed? If so,
please describe these potential changes.
79
Sustainability What is the relationship of the ALCs to MOE policies and
priorities?
Please describe UNICEF’s relationship with the MOE.
Please describe the role of the MOE in supporting the ALCs.
How can the MOE best support the ALCs in the next 3 years?
Sustainability Please describe UNICEF’s relationships with partners
participating in ALC .
In what other programmes and activities are you partners?
Are there past partnerships that are important? If so, please
describe these.
Please describe the role of these partners in the ALCs.
In what ways has the partnership been effective?
In what ways has the partnership been ineffective?
Do you have any specific recommendations regarding
partnerships in relation to the ALCs?
Sustainability: Scalability In your opinion, what are next steps for the ALCs?
Do you believe that the ALCs should be scaled up in
Afghanistan? If so, please describe the measures that you
think are necessary.
Please describe UNICEF’s most-effective potential role, in
your opinion, in scaling up the ALCs.
What is the role of the LUL programme?
80
Effectiveness: Wrapping up Do you have specific recommendations with regard to the
ALCs?
Do you have recommendations with regard to the LUL
programme overall?
Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
Afghanistan INTERVIEW guide:
ALCs
Guide for interview with MOE personnel
Criterion: Topic or theme Question
Relevance: Getting started Please tell me about the MOE’s relationship to the ALCs. How
were you or other personnel involved?
How do the ALCs fit into the government of Afghanistan’s
overall education strategy?
Do the ALCs connect to other policies or strategies or
priorities? If so, please tell me about these.
(Are these priorities formalized in any documents or otherwise
adopted?)
Relevance: Challenges In your opinion, what are the biggest challenges that students
face in relation to education?
How do these challenges affect student learning?
Relevance: Challenges & equity Are these challenges different for girl students?
If so, please describe these differences.
(Differences could stem from different impact, intensity,
frequency, etc.)
81
Relevance: Challenges & Equity How do the ALCs address challenges to girls’ education?
Which of these challenges do the ALCs address most
successfully?
(For each challenge named)
In your opinion, is this challenge important? Please tell me
why or why not.
Effectiveness: General What are the most important changes brought about by the
ALCs?
Why are these changes important?
Effectiveness: Equity In what ways do the ALCs specifically change conditions for
girls in school?
Why are these changes important?
Effectiveness: Equity What important challenges to girls’ education do the ALCs not
address?
How could the programme be changed to address these
challenges?
Effectiveness: Learning &
Innovation
What obstacles has the ALCs encountered in relation to its
launch and implementation?
How have these obstacles affected the programme?
Has the programme been changed to respond to these
obstacles? If it has, please tell me how.
How did your organisation view or contribute to these
responses?
In what ways did other organizations contribute?
How effective were these responses? And why?
Effectiveness: Learning Are there potential changes to the programme that might
make it more effective, but that we haven’t discussed?
If so, please describe these potential changes.
82
Sustainability What is the relationship of the ALCs to Ministry of Education
policies and priorities?
Please describe the role of the MOE in supporting the ALC
initiative.
Sustainability Please describe MOE’s relationship with UNICEF.
In what ways is this partnership effective?
In what ways could it be improved?
How can UNICEF best support the ALCs in the next 3 years?
Sustainability Please describe the role of UNICEF with regard to the ALCs.
In what ways has UNICEF been effective?
In what ways has UNICEF been ineffective?
Do you have any specific recommendations for UNICEF in
relation to the LUL programme?
Sustainability: Scalability In your opinion, what are next steps for the ALCs?
Do you believe that the ALCs should be scaled up in
Afghanistan?
If so, please describe the measures that you think are
necessary.
Effectiveness: Wrapping up Do you have other specific recommendations with regard to
the ALC initiative?
Do you have recommendations with regard to the LUL
programme overall?
Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
83
Afghanistan INTERVIEW guide: ALCs
Guide for interview with Other Stakeholders
(NGOs, etc.)
Criterion: Topic or theme Question
Relevance: Getting started Please describe the relationship of your organisation to
UNICEF.
Please tell me about your organisation’s relationship to the
ALCs. How were you or other personnel involved?
How do the ALCs fit into your organisation’s overall strategy?
Is this strategy reflected in policy or other documents?
Relevance: Challenges In your opinion, what are the biggest challenges that students
face in relation to education?
How do these challenges affect student learning?
Relevance: Challenges & Equity Are these challenges different for girl students?
If so, please describe these differences.
Relevance: Challenges & Equity How do the ALCs address challenges to girls’ education?
Which of these challenges do the ALCs address most
successfully?
Effectiveness: General What are the most important changes brought about by the
ALCs?
Why are these changes important?
84
Effectiveness: Equity In what ways do the ALCs specifically change conditions for
girls in school?
Why are these changes important?
Effectiveness: Equity What important challenges to girls’ education do the ALCs not
address?
How could the programme be changed to address these
challenges?
Effectiveness: Learning &
Innovation
If there are challenges, has the programme been changed to
respond to these obstacles? If it has, please tell me how.
How did your organisation view or contribute to these
responses?
Effectiveness: Learning Are there potential changes to the programme that might
make it more effective, but that we haven’t discussed? If so,
please describe these potential changes.
Sustainability What is the relationship of the ALCs to Ministry of Education
policies and priorities?
Please describe your organisation’s relationship with the
MOE.
Please describe the role of the MOE in supporting the ALC
initiative.
How can the MOE best support the ALCs in the next 3 years?
Sustainability Do you have any specific recommendations for UNICEF in
relation to LUL?
85
Sustainability: Scalability In your opinion, what are next steps for the ALCs?
Do you believe that the ALCs should be scaled up in
Afghanistan? If so, please describe the measures that you
think are necessary.
Without committing to anything in particular, do you believe
that your organisation would be able to support scaling up of
the ALCs? If so, please elaborate.
Effectiveness: Wrapping up Do you have specific recommendations with regard to the ALC
initiative?
Do you have recommendations with regard to the LUL
programme overall?
Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
86
Afghanistan interview guide: ALCs
Guide for interviews with school principals (hub schools)
Criterion: Topic or theme Question
Relevance
Relevance: Getting started
How long have you been principal of this school?
What is your school’s involvement with the ALC?
Relevance: Challenges Are there challenges to the ALC being a part of your school
hub?
If so, what are these?
Are children being mainstreamed from the ALC into formal
school?
If not, why not?
Relevance: Equity Do any of these challenges affect girls more than boys?
Relevance: Equity Please tell me, based on your own observation, how girls are
doing in the ALC compared to boys in the ALC.
Relevance: Challenges & Equity Please tell me about the ALCs in your school.
In your opinion, which of the innovations in ALCs are most
important? (small class sizes, access for girls, learning
materials, etc.)
87
Relevance: Relevance How does the ALC address the challenges to girls’ education?
Effectiveness: General Has the programme changed girls’ education in your school
hub? If so, please describe these changes.
Effectiveness: Sustainability &
Impact
How are the changes brought about by the ALCs received by
family members?
How are these changes received by other members of the
community?
Effectiveness: Challenges 2 What keeps more girls from participating in the ALCs?
Effectiveness: Learning &
Innovation
Since the ALCs started in your school hub, what problems has
it encountered?
(Problems might include low attendance, parental disapproval,
lack of a room, etc.)
How have these problems affected the programme or girls
who participate in it?
Has the programme been changed to respond to these
problems? If it has, please tell me how.
(Evaluator will note the degree to which these changes have
been initiated by the school or programme-wide.)
Has the change been successful?
Effectiveness: Learning &
Innovation
Are there other changes that might improve the ALC
programme?
Effectiveness: Impact Have the ALCs changed attitudes or behaviors among the
girls’ families? In the community?
If so, please tell me how.
88
Sustainability What can the school hub/formal school system do to further
encourage the success of the ALCs and reach more OOSC?
Effectiveness: Wrapping up Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
Overview: Focus Group Guides
The following FGD guides are tied to the evaluation matrix from the multi-country evaluation inception
report, so as to best support the multi-country evaluation of LUL. These instruments will be used by the
National Evaluator for her field visits. Where focus groups are not available, the instruments will serve as
individual interview guides.
A more simplified version for the local enumerator team, along with draft mobile instruments, follows in
the next section.
Due to the remote locations of many ALCs, focus groups for participating teachers may not be able to be
held; rather, individual interviews will be conducted. Moreover, these interviews will need to be
conducted by the enumerator team.
Instruments included here address:
Field visits: FGD guide for communities and families, including Shura members
Field visits: FGD guide for participating ALC students, girls and boys
Field visits: FGD guide for participating ALC teachers (male/female)
89
Afghanistan FGD guide: ALCs
Guide for FGDs with communities and families, including Shura members
Criterion: Topie or theme Question
Relevance
Relevance: Getting started Please tell me about your main reason for attending this
meeting.
Relevance: Challenges Please tell me about the importance of education. Is it
important that your children complete school? If so, please tell
me why.
What might keep one of your children from completing the
ALC? From attending formal school?
Do you have a plan to help your child address these
problems? If so, please describe it.
Relevance: Challenges 2 Does your child’s ALC (and, if relevant, school hub) also help
address these problems? If so, please tell me how.
Relevance: Equity What are differences between girls’ education and boys’
education?
Is it more important for boys to be educated than for girls? If
so, please tell me why.
Please tell me about your hopes for your children, in relation
to education.
Do you believe that your child will finish primary school?
Secondary school?
Relevance: Awareness Please tell me about the ALC programme in your community.
90
Effectiveness: General What are the most important effects of the ALC for girls? For
boys?
How has the programme changed learning or going to school
for your child?
Are these changes important? Why (or why not)?
Effectiveness: Challenges What factors do you think keep more girls from participating in
the ALC?
Effectiveness: Learning &
Innovation
How have these problems affected the ALCs or girls who
participate in it?
Do you know if the ALC programme been changed to respond
to these problems? If it has, please tell me how.
Has the change been successful?
Effectiveness: Learning &
Innovation
Are there other changes that might improve the ALC
programme?
Effectiveness: Impact Have the availability of the ALCs changed your attitude about
your child’s schooling?
If so, please tell me how.
Sustainability Do you support the ALCs in any way—with your time, or in
other ways?
If so, please tell me how.
Effectiveness: Wrapping up Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
91
Afghanistan FGD guide: ALCs
Guide for FGD with participating girls
Criterion: Topic or theme Question
Relevance: Getting started Please tell me about your Accelerated Learning programme:
How often do you usually attend? When did you start
attending?
What grade are you in?
How old are you?
How many grades have you completed so far in the ALC?
Do you enjoy learning?
Do you enjoy going to school in the ALC?
Relevance: Challenges Please tell me about some of the things that are difficult about
attending AL courses.
Do they affect boys too? Do you think that they affect boys
differently?
How do they affect the rest of your activities and time in
school?
92
Relevance: The initiative Would you have had a chance to attend school if you were not
in the ALC?
What prevented you from attending formal school?
Effectiveness: General What are you learning in school now?
Are you doing well in school, making good marks?
How has being able to attend school changed your life?
Effectiveness: Challenges Are you attending the ALC regularly?
If you are not attending often, what keeps you from attending
the ALC?
How does the distance to school affect your attendance?
Are you planning on attending school past Grade 6/ALC
graduation?
What keeps you from being able to continue to formal school
in Grade 7?
How do family obligations affect your attendance at the ALC?
How could things be changed to address this problem?
93
Effectiveness: Innovation Since you started attending the ALC, how has the programme
changed, if at all?
How have these changes affected you?
Have you had access to learning materials?
What are they?
Have these been helpful?
Do you get enough attention from the teacher to help you
learn?
What would you improve in your classroom?
Effectiveness: Innovation Are there other changes that might make the AL programme
better?
Effectiveness: Impact Since you started attending the AL courses, have things
changed at home in any way?
If so, please tell me how.
Effectiveness: Sustainability Tell me about your plans for the next year .
Will you continue at the ALC?
If not, will you go to formal school?
Do you think you will continue your education after you marry?
If not, why not?
Would you like to continue your education after you marry?
What would make it possible for you to continue your
education after you are married?
94
Effectiveness: Wrapping up What are some other things about the ALC that are important,
but that we haven’t talked about?
95
Afghanistan FGD guide: ALCs
Guide for FGD with participating boys
Criterion: Topic or theme Question
Relevance: Getting started Please tell me about your Accelerated Learning programme:
How often do you usually attend? When did you start
attending?
What grade are you in?
How old are you?
How many grades have you completed so far in the ALC?
Do you enjoy learning?
Do you enjoy going to school in the ALC?
Relevance: Challenges Please tell me about some of the things that are difficult about
attending AL courses.
Do they affect girls too? Do you think that they affect girls
differently?
How do they affect the rest of your activities and time in
school?
96
Relevance: The initiative Would you have had a chance to attend school if you were not
in the ALC?
What prevented you from attending formal school?
Effectiveness: General What are you learning in school now?
Are you doing well in school, making good marks?
How has being able to attend school changed your life?
Effectiveness: Challenges Are you attending the ALC regularly?
If you are not attending often, what keeps you from attending
the ALC?
How does the distance to school affect your attendance?
Are you planning on attending school past Grade 6/ALC
graduation?
What keeps you from being able to continue to formal school
in Grade 7?
How do family obligations affect your attendance at the ALC?
How could things be changed to address this problem?
97
Effectiveness: Innovation Since you started attending the ALC, how has the programme
changed, if at all?
How have these changes affected you?
Have you had access to learning materials?
What are they?
Have these been helpful?
Do you get enough attention from the teacher to help you
learn?
What would you improve in your classroom?
Effectiveness: Innovation Are there other changes that might make the AL programme
better?
Effectiveness: Impact Since you started attending the AL courses, have things
changed at home in any way?
If so, please tell me how.
Effectiveness: Sustainability Tell me about your plans for the next year.
Will you continue at the ALC?
If not, will you go to formal school?
Do you think you will continue your education after you marry?
If not, why not?
Would you like to continue your education after you marry?
What would make it possible for you to continue your
education after you are married?
98
Effectiveness: Wrapping up What are some other things about the ALC that are important,
but that we haven’t talked about?
Afghanistan FGD guide: ALCs
Guide for FGD with participating teachers
Criterion: Topic or theme Question
Relevance
Relevance: Getting started Please tell me about your experience and education as a
teacher.
How long have you been a teacher? Have you taught at other
schools?
Have you received education or training in relation to
teaching? If so, please tell me about it.
Did you participate in training in relation to the ALCs?
If so, please describe that training.
How do you use what you learned in your teacher training at
school?
Relevance: Challenges Please tell me about some of the problems that you face as a
teacher.
Which of these problems are most important? How do they
affect you directly?
Do these problems affect student learning? If so, please tell
me how.
99
Relevance: Challenges 2 What challenges do your students face in relation to their
education?
Which of these challenges are most important?
How do students and their families address these challenges?
Relevance: Equity Do any of these challenges affect girls more than boys? In
what ways?
Relevance: Equity 2 Please tell me, based on your own observation, how girls are
doing in your school compared to boys.
If you observe differences, what do you think causes these
differences?
Relevance: Learning Which aspects of the ALCs are most important for improving
access and quality of education?
Relevance: Challenges & Equity How does the ALC address the challenges to girls’ education?
Effectiveness: General What are the most important effects of the ALCs for girls?
How has the ALC changed girls’ education in your
community?
Why have the ALCs been effective in this way?
Effectiveness: Sustainability &
Impact
How are these changes received by family members?
How are these changes received by other members of the
community?
100
Effectiveness: Learning &
innovation
Do you have learning materials for yourself? For your
students?
How do you assess learning in your classroom?
What are the criteria for students to move to the next grade?
What assessments are given after Grade 3? After Grade 6?
How do they measure learning?
What keeps more girls from participating in the after-school
programme?
Effectiveness: Learning &
Innovation 2
Since the ALCs started in your school, what problems has it
encountered?
How have these problems affected the programme or girls
who participate in it?
Has the programme been changed to respond to these
problems? If it has, please tell me how.
Has the change been successful?
Effectiveness: Learning &
Innovation 3
Are there changes that might improve the programme in other
ways?
101
Effectiveness: Impact Have the ALCs changed attitudes or behaviors among the
girls’ families?
If so, please tell me how.
Sustainability Do you support the ALC programme in any other way—with
your time, by encouraging girls outside of school or in other
ways?
If so, please tell me how.
Effectiveness: Wrapping up Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
Overview: Focus Group Guides for National Enumerator Team
The following FGD guides are tied to the evaluation matrix from the multi-country evaluation inception
report, so as to best support the multi-country evaluation of LUL. These instruments will be used by the
National Enumerator Team, Soft Power Solutions, for their field visits, along with the mobile instrument.
Where focus groups are not available, the instruments will serve as individual interview guides.
These instruments are a simplified version of the instruments used by the National Evaluator. They have
been developed for the local enumerator team, and mirror the mobile instruments. In areas where mobile
cannot be used due to security, paper-based instruments will be used. In addition, the enumerator team
will be encouraged to gather, where possible, additional narrative data on the LUL ALC program. These
guides will aid the enumerators in conducting interviews in the field.
Due to the remote locations of many ALCs, focus groups for participating teachers may not be able to be
held; rather, individual interviews will be conducted.
Instruments included here address:
Field visits: FGD guide for communities and families, including Shura members
Field visits: FGD guide for participating ALC students, girls and boys
Field visits: FGD guide for participating ALC teachers (male/female)
Field visits: Interview guide for Hub School Principals
Classroom Observation Guide
102
National Enumerator Team
Afghanistan FGD guide: ALCs
Guide for FGDs with communities and families, including Shura members
Criterion: Topie or theme Question
Relevance
Relevance: Awareness
Please tell me about the ALC programme in your community.
Please tell me about the importance of education. Is it
important that your children complete school? If so, please tell
me why.
Relevance: Challenges
What might keep one of your children from completing the
ALC? From attending formal school?
Relevance: Challenges 2 Does your child’s ALC (and, if relevant, school hub) also help
address these problems? If so, please tell me how.
Relevance: Equity Please tell me about your hopes for your children, in relation
to education.
Do you believe that your child will finish primary school?
Secondary school?
Effectiveness: General What are the most important effects of the ALC for girls? For
boys?
How has the programme changed learning or going to school
for your child?
103
Effectiveness: Challenges What factors do you think keep more girls from participating in
the ALC?
Effectiveness: Learning &
Innovation
How have these problems affected the ALCs or girls who
participate in it?
Do you know if the ALC programme been changed to respond
to these problems? If it has, please tell me how.
Effectiveness: Learning &
Innovation
Are there other changes that might improve the ALC
programme?
Effectiveness: Impact Have the availability of the ALCs changed your attitude about
your child’s schooling?
If so, please tell me how.
Sustainability Do you support the ALCs in any way—with your time, or in
other ways?
If so, please tell me how.
Effectiveness: Wrapping up Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
National Enumerator Team
Afghanistan FGD guide: ALCs
Guide for FGD with participating girls
Criterion: Topic or theme Question
104
Relevance: Getting started Please tell me about your Accelerated Learning programme:
How often do you usually attend?
How many grades have you completed so far in the ALC?
Do you enjoy going to school in the ALC?
Relevance: The initiative Would you have had a chance to attend school if you were not
in the ALC?
What prevented you from attending formal school?
Relevance: Challenges Please tell me about some of the things that are difficult about
attending AL courses.
Do you think that they affect boys differently?
Effectiveness: General What are you learning in school now?
Effectiveness: Challenges
How does the distance to school affect your attendance?
How do family obligations affect your attendance at the ALC?
How could things be changed to address this problem(s)?
105
Effectiveness: Innovation Since you started attending the ALC, how has the programme
changed, if at all?
Have you had access to learning materials? What are they?
Have these been helpful?
Do you get enough attention from the teacher to help you
learn?
What would you improve in your classroom to make the AL
programme better?
Effectiveness: Impact
How has being able to attend school changed your life?
Effectiveness: Sustainability Tell me about your plans for the next year.
Will you continue at the ALC? If not, will you go to formal
school?
Would you like to continue your education after you marry?
What would make it possible for you to continue your
education after you are married?
Effectiveness: Wrapping up What are some other things about the ALC that are important,
but that we haven’t talked about?
106
National Enumerator Team
Afghanistan FGD guide: ALCs
Guide for FGD with participating boys
Criterion: Topic or theme Question
Relevance: Getting started Please tell me about your Accelerated Learning programme:
How often do you usually attend?
How many grades have you completed so far in the ALC?
Do you enjoy going to school in the ALC?
Relevance: The initiative Would you have had a chance to attend school if you were not
in the ALC?
What prevented you from attending formal school?
Relevance: Challenges Please tell me about some of the things that are difficult about
attending AL courses.
Do you think that they affect girls differently?
Effectiveness: General What are you learning in school now?
107
Effectiveness: Challenges
How does the distance to school affect your attendance?
How do family obligations affect your attendance at the ALC?
How could things be changed to address this problem(s)?
Effectiveness: Innovation Since you started attending the ALC, how has the programme
changed, if at all?
Have you had access to learning materials? What are they?
Have these been helpful?
Do you get enough attention from the teacher to help you
learn?
What would you improve in your classroom to make the AL
programme better?
Effectiveness: Impact
How has being able to attend school changed your life?
Effectiveness: Sustainability Tell me about your plans for the next year.
Will you continue at the ALC? If not, will you go to formal
school?
Would you like to continue your education after you marry?
What would make it possible for you to continue your
education after you are married?
Effectiveness: Wrapping up What are some other things about the ALC that are important,
but that we haven’t talked about?
108
National Enumerator Team
Afghanistan FGD guide: ALCs
Guide for FGD with participating teachers
Criterion: Topic or theme Question
Relevance
Relevance: Getting started Please tell me about your experience and education as a
teacher.
Did you participate in training in relation to the ALCs? If so,
please describe that training.
Relevance: Learning Which aspects of the ALCs are most important for improving
access and quality of education?
Relevance: Challenges Please tell me about some of the problems that you face as a
teacher.
Do these problems affect student learning? If so, please tell
me how.
Relevance: Challenges 2 What challenges do your students face in relation to their
education?
How do students and their families address these challenges?
Relevance: Equity Do any of these challenges affect girls more than boys? In
what ways?
Relevance: Equity 2 Please tell me, based on your own observation, how girls are
doing in your school compared to boys.
If you observe differences, what do you think causes these
differences?
109
Relevance: Challenges & Equity How does the ALC address the challenges to girls’ education?
Effectiveness: General What are the most important effects of the ALCs for girls?
How has the ALC changed girls’ education in your
community?
Effectiveness: Sustainability &
Impact
How are these changes received by family members?
How are these changes received by other members of the
community?
Effectiveness: Learning &
innovation
Do you have learning materials for yourself? For your
students?
How do you assess learning in your classroom?
What are the criteria for students to move to the next grade?
What assessments are given after Grade 3? After Grade 6?
Effectiveness: Learning &
Innovation 2
Since the ALCs started in your school, what problems has it
encountered?
Has the programme been changed to respond to these
problems? If it has, please tell me how.
Effectiveness: Impact Have the ALCs changed attitudes or behaviors among the
girls’ families?
If so, please tell me how.
110
Sustainability Do you support the ALC programme in any other way—with
your time, by encouraging girls outside of school or in other
ways?
If so, please tell me how.
Effectiveness: Wrapping up Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
National Enumerator Team
Afghanistan interview guide: ALCs
Guide for interviews with school principals (hub schools)
Criterion: Topic or theme Question
Relevance
Relevance: Getting started
How long have you been principal of this school?
What is your school’s involvement with the ALC?
Relevance: Relevance How does the ALC address the challenges to girls’ education?
111
Relevance: Challenges Are there challenges to the ALC being a part of your school
hub?
If so, what are these?
Are children being mainstreamed from the ALC into formal
school?
If not, why not?
Relevance: Equity Do any of these challenges affect girls more than boys?
Please tell me, based on your own observation, how girls are
doing in the ALC compared to boys in the ALC.
Effectiveness: General Has the programme changed girls’ education in your school
hub? If so, please describe these changes.
Effectiveness: Sustainability &
Impact
How are the changes brought about by the ALCs received by
family members?
How are these changes received by other members of the
community?
Effectiveness: Challenges What keeps more girls from participating in the ALCs?
Effectiveness: Learning &
Innovation
In your opinion, which of the innovations in ALCs are most
important? (small class sizes, access for girls, learning
materials, etc.)
Has the programme been changed to respond to problems
early on in implementation? If it has, please tell me how.
112
Effectiveness: Impact Have the ALCs changed attitudes or behaviors among the
girls’ families? In the community?
If so, please tell me how.
Sustainability What can the school hub/formal school system do to further
encourage the success of the ALCs and reach more OOSC?
Effectiveness: Wrapping up Are there other aspects of the ALCs that are important, but
that we haven’t talked about?
If so, please describe these.
113
Classroom Observation Guide
Classroom Observation Form For the LUL/ALCs Evaluation
Prior to observation, confirm from teacher(s) that this is the “regular routine”
Date: ______ November, 2014 LUL/ALC School Code: [Use from
Sampling Frame]_______
Name of school:
Number of children observed:
Girls_______ Boys________
Are all students present today? __Yes __No
Were non-students (very young/older) present today?
__Yes __No
Number of Teachers:
Names of Teachers:
Class Observation Time (check one):
___15-20 minutes or ___ 30 minutes
Is there text on walls: ____ Yes ____ No
Work designed and created by children is on display? ____ Yes ____ No
Classroom is warm and inviting?____ Yes ____ No
HOW DOES THE DAILY ALC ROUTINE LOOK? (check one)
The children appear engaged and participating in the classroom?
____ Yes ____ No ____ To Some Degree
Did the teacher include interactive activities (songs, games, role-play) today?
____ Yes ____ No ____ To Some Degree
114
Quality of Instruction: Opportunities for Social Skills, Self-Expression, Physical Skills, Sensory
Skills, Working with Materials (check one):
Children are involved in social activities? ____ Yes ____ No ___A bit
Children listen to teacher(s)? ____ Yes ____ No ___A bit
Use of teaching aids during that day? ____ Yes ____ No ___A bit
Children’s use of any reading skills (if applicable)? ____ Yes ____ No NA
Children’s use of any writing skills (if applicable)? ____ Yes ____ No NA
Children’s use of any math skills (if applicable)? ____ Yes ____ No NA
Children’s use of any creative (art) skills (if applicable)? ____ Yes ____ No NA
WASH
W1. Is there a functional water point at or near the school? ____ Yes ____ No
W2. Does the school have toilet facilities? ____ Yes ____ No
W3. Does the school have handwashing facilities? ____ Yes ____ No
GENERAL NOTES
OBSERVED EXAMPLES OF HOW LEARNING IS TAKING PLACE IN THE ALC CLASSROOM:
OBSERVED CHALLENGES (IF ANY)?:
115
Overview: Mobile Instruments for Enumerators
The following four (4) instruments were created specifically for use on mobile phones in order to support
the data gathering process, and to serve as a proof of concept for future M&E efforts. The questions
inputted onto the mobile phones capture the key elements of the instruments devised for the Let Us Learn
global evaluation effort. The tables below indicate the DAC criteria in the left column, general question in
the middle column, and the actual questions and possible answers inputted into the mobile phone
application in the right column.
Instruments included here address:
Field visits: mobile instrument for hub school principals
Field visits: mobile instrument for communities and families, including Shura members
Field visits: mobile instruments for participating ALC students, girls and boys
Field visits: mobile instrument for participating ALC teachers (male/female)
116
Instruments for Enumerator Team: Optimized for Mobile
Target Group: Guide for Hub School Principals – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Relevance: Getting
started
How long have you been principal of
this school?
What is your school’s involvement
with the ALC?
1. How long have you been principal
of this school?
1. Less than 1 year
2. 1-2 years
3. 2-5 years
4. 5 or more years
2. What is your school’s involvement
with the ALC?
1 – I do not know we have an ALC
as part of our hub
2 – My school supports the ALC
Relevance: Challenges
Are there challenges to the ALC
being a part of your school hub?
If so, what are these?
Are children being mainstreamed
from the ALC into formal school?
If not, why not?
3. Are there challenges to the ALC
being a part of your school hub?
1 – Yes
2 – No
4. Are children being mainstreamed
from the ALC into formal school?
1 – Yes
2 – No
117
Target Group: Guide for Hub School Principals – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
5. What are the challenges to the
ALC being a part of your school
hub?
TEXT BOX #1
Relevance: Equity
Do any of these challenges affect
girls more than boys?
6. The challenges affect girls more
than boys.
1 – Yes
2 – No
3 – No boys are in the school
Relevance: Equity
Please tell me, based on your own
observation, how girls are doing in
the ALC compared to boys in the
ALC.
7. Girls are doing better than the
boys in the ALC
1 – Yes
2 – No
3 – No boys are in the school
Relevance: Challenges
& Equity
Please tell me about the ALCs in
your school.
In your opinion, which of the
innovations in ALCs are most
important? (small class sizes, access
for girls, learning materials, etc.)
8. Which of the innovations in ALCs
are most important?
1 – Small class sizes
2 – Access for girls
3 – Access for older out of school
children
4 – Learning materials
118
Target Group: Guide for Hub School Principals – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
5 – All of these
Effectiveness:
Sustainability & Impact How are the changes brought about
by the ALCs received by family
members?
How are these changes received by
other members of the community?
9. Parents and the community
support the ALCs.
1 – Yes
2 – No
Effectiveness: Impact Have the ALCs changed attitudes or
behaviors among the girls’ families?
In the community?
If so, please tell me how.
10. The attitudes and behaviors
among the girls families have
positively changed:
1 – Yes
2 - No
Sustainability
What can the school hub/formal
school system do to further
encourage the success of the ALCs
and reach more OOSC?
11. The hub/formal school could do
more to encourage the success of
the ALCs and reach more OOSC.
1 – Yes
2 - No
Effectiveness:
Wrapping up
Are there other aspects of the ALCs
that are important, but that we
haven’t talked about?
If so, please describe these.
12. TEXT BOX #2
13. Enumerator – I have more
narrative data that will not fit in the
text box.
1 – Yes
2 - No
WASH in Schools 3 basic indicators:
W1. Is there a functional water point
at or near the school?
1 – Yes
119
Target Group: Guide for Hub School Principals – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Is there a functional water point at or
near the school?
Does the school have toilet facilities?
Does the school have handwashing
facilities?
2 – No
W2. Does the school have toilet
facilities?
1 – Yes
2 - No
W3. Does the school have
handwashing facilities?
1 – Yes
2 - No
120
Target Group: Teachers in ALCs – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Relevance: Getting
started
Please tell me about how you
became a teacher and how much
education you have had:
1. How you became a teacher:
I went to teacher training college
I was the most knowledgeable person
to teach in my community.
UNICEF trained me
2. Education level:
Grade 1-3
Grade 4-6
Grade 7-9
High School (Gr 10-12)
Teacher training college (Gr 13-14)
University degree
Relevance: Getting
started
Please tell me about your teacher
training:
3. How many teacher trainings from
UNICEF have you participated in?
[Enter number (1 to 9)]
4. The teacher training I received from
UNICEF improved my effectiveness as
a teacher:
1 - not at all
2 – a little
3 – a lot
121
Target Group: Teachers in ALCs – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Relevance: The
initiative
How does the ALC address the
challenges to girls’ education?
To out of school girls and boys?
5. The ALCs address the challenges to
girls’ education.
1 – Yes
2 – No
6. The ALCs address the challenges to
OOSC and older children, including
boys.
1 – Yes
2 - No
Relevance:
Challenges
Please tell me about some of the
problems that you face as a
teacher.
What challenges do your students
face in relation to their education?
7. Please describe, in brief, what
challenges you face as a teacher.
1 – Lack of materials
2 – No child care for my children
3 – Lack of parent support
4 – Not enough training
5 – Other: TEXT BOX #1
Effectiveness:
Innovation
Are there other changes that might
make the programme better?
8. The program could be improved.
1 – Not at all
2 – a little
3 – a lot
122
Target Group: Teachers in ALCs – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Effectiveness: Impact How are these changes received
by family members?
How are these changes received
by other members of the
community?
9. My life has changed since I began
teaching in the ALC.
1 - not at all
2 - for the better
3 - for the worse
Effectiveness:
Sustainability
How supportive of your role as a
teacher in the ALC are parents and
the community?
10. The parents and my community
school management Shura are helpful
and supportive to me as a teacher.
1 – not supportive
2 – a little supportive
3 – very supportive
Effectiveness:
Sustainability
Did you register with the hub
school?
What kind of support are you
receiving from the hub school?
11. Did you register with the hub
school?
1 – Yes
2 - No
12. Are you receiving support from the
hub school?
1 – not supportive
2 – a little supportive
3 – very supportive
123
Target Group: Teachers in ALCs – 12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Effectiveness:
Wrapping up
What are some other things about
teaching in an ALC that are
important, but that we haven’t
talked about?
13. What are some other things about
teaching in an ALC that are important,
but that we haven’t talked about?
TEXT BOX #2
14. Enumerator – I have more
narrative data that will not fit in the text
box.
1 – Yes
2 - No
WASH in Schools 3 basic indicators:
Is there a functional water point at
or near the school?
Does the school have toilet
facilities?
Does the school have handwashing
facilities?
W1. Is there a functional water point at
or near the school?
1 – Yes
2 – No
W2. Does the school have toilet
facilities?
1 – Yes
2 - No
W3. Does the school have
handwashing facilities?
1 – Yes
2 - No
124
Target Group: Guide for FGDs with communities and families, including Shura members –
12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Relevance: Getting
started
Please tell me about how you
became a teacher and how much
education you have had:
1. How you became a teacher:
I went to teacher training college
I was the most knowledgeable person
to teach in my community.
UNICEF trained me
2. Education level:
Grade 1-3
Grade 4-6
Grade 7-9
High School (Gr 10-12)
Teacher training college (Gr 13-14)
University degree
Relevance: Getting
started
Please tell me about your teacher
training:
3. How many teacher trainings from
UNICEF have you participated in?
[Enter number (1 to 9)]
4. The teacher training I received from
UNICEF improved my effectiveness as
a teacher:
1 - not at all
125
Target Group: Guide for FGDs with communities and families, including Shura members –
12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
2 – a little
3 – a lot
Relevance: The
initiative
How does the ALC address the
challenges to girls’ education?
To out of school girls and boys?
5. The ALCs address the challenges to
girls’ education.
1 – Yes
2 – No
6. The ALCs address the challenges to
OOSC and older children, including
boys.
1 – Yes
2 - No
Relevance:
Challenges
Please tell me about some of the
problems that you face as a
teacher.
What challenges do your students
face in relation to their education?
7. Please describe, in brief, what
challenges you face as a teacher.
1 – Lack of materials
2 – No child care for my children
3 – Lack of parent support
4 – Not enough training
5 – Other: TEXT BOX #1
Effectiveness:
Innovation
Are there other changes that might
make the programme better?
8. The program could be improved.
1 – Not at all
126
Target Group: Guide for FGDs with communities and families, including Shura members –
12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
2 – a little
3 – a lot
Effectiveness: Impact How are these changes received
by family members?
How are these changes received
by other members of the
community?
9. My life has changed since I began
teaching in the ALC.
1 - not at all
2 - for the better
3 - for the worse
Effectiveness:
Sustainability
How supportive of your role as a
teacher in the ALC are parents and
the community?
10. The parents and my community
school management Shura are helpful
and supportive to me as a teacher.
1 – not supportive
2 – a little supportive
3 – very supportive
Effectiveness:
Sustainability
Did you register with the hub
school?
What kind of support are you
receiving from the hub school?
11. Did you register with the hub
school?
1 – Yes
2 - No
127
Target Group: Guide for FGDs with communities and families, including Shura members –
12 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
12. Are you receiving support from the
hub school?
1 – not supportive
2 – a little supportive
3 – very supportive
Effectiveness:
Wrapping up
What are some other things about
teaching in an ALC that are
important, but that we haven’t
talked about?
13. What are some other things about
teaching in an ALC that are important,
but that we haven’t talked about?
TEXT BOX #2
14. Enumerator – I have more
narrative data that will not fit in the text
box.
1 – Yes
2 - No
Target Group: Boy Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Relevance: The initiative Would you have had a chance to
attend school if you were not in the
ALC?
1. Would you have had a chance to
attend school if you were not in the
ALC?
128
Target Group: Boy Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
What prevented you from attending
formal school?
1 – No
2 – Yes
Effectiveness: General What are you learning in school
now?
Are you doing well in school,
making good marks?
How has being able to attend
school changed your life?
2. I am learning the following
subjects in school:
1 – reading and writing
2 – math
3 - science
4 – all of the above
3. I am making good marks in
school.
1 – No
2 – Yes
Relevance: Challenges Please tell me about some of the
things that are difficult about
attending AL courses.
Do they affect girls too? Do you
think that they affect girls
differently?
How do they affect the rest of your
activities and time in school?
4. There are some things that are
difficult about attending the ALC
because I am a boy.
1 – Yes
2 – No
5. The difficulties are: TEXT BOX #1
Effectiveness:
Challenges Are you attending the ALC
regularly?
6. The short distance to the ALC
from my home makes it possible for
me to go to school.
129
Target Group: Boy Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
If you are not attending often, what
keeps you from attending the ALC?
How does the distance to school
affect your attendance?
Are you planning on attending
school past Grade 6/ALC
graduation?
What keeps you from being able to
continue to formal school in Grade
7?
How do family obligations affect
your attendance at the ALC?
How could things be changed to
address this problem?
1 – Yes
2 – No
Effectiveness:
Innovation
Since you started attending the
ALC, how has the programme
changed, if at all?
How have these changes affected
you?
Have you had access to learning
materials?
What are they?
Have these been helpful?
7. I have my own learning materials.
1 – Yes
2 – No
130
Target Group: Boy Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Do you get enough attention from
the teacher to help you learn?
What would you improve in your
classroom?
Effectiveness: Impact Since you started attending the AL
courses, have things changed at
home in any way?
If so, please tell me how.
8. My life has changed for the better
since I began going to school in the
ALC.
1 – Yes
2 - No
Effectiveness:
Sustainability
Tell me about your plans for the
next year .
Will you continue at the ALC?
If not, will you go to formal school?
Do you think you will continue your
education after you marry? If not,
why not?
Would you like to continue your
education after you marry?
What would make it possible for
you to continue your education after
you are married?
9. I plan to go to formal school past
Grade 6.
1 – Yes
2 – No
10. If you are not continuing with
school, why not?
1 – my parents will not let me
2 – I am getting married
3 – the formal school is too far away
from my home
4 – family responsibilities (work,
etc.)
Effectiveness: Wrapping
up
What are some other things about
the ALC that are important, but that
we haven’t talked about?
11. TEXT BOX #2
131
Target Group: Girl Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Relevance: The
initiative
Would you have had a chance to
attend school if you were not in the
ALC?
What prevented you from attending
formal school?
1. Would you have had a chance to
attend school if you were not in the
ALC?
1 – No
2 – Yes
Effectiveness: General What are you learning in school
now?
Are you doing well in school, making
good marks?
How has being able to attend school
changed your life?
2. I am learning the following
subjects in school:
1 – reading and writing
2 – math
3 - science
4 – all of the above
3. I am making good marks in school.
1 – No
2 – Yes
Relevance: Challenges Please tell me about some of the
things that are difficult about
attending AL courses.
Do they affect boys too? Do you
think that they affect boys
differently?
4. There are some things that are
difficult about attending the ALC
because I am a girl.
1 – Yes
2 – No
5. The difficulties are: TEXT BOX #1
132
Target Group: Girl Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
How do they affect the rest of your
activities and time in school?
Effectiveness:
Challenges
Are you attending the ALC
regularly?
If you are not attending often, what
keeps you from attending the ALC?
How does the distance to school
affect your attendance?
Are you planning on attending
school past Grade 6/ALC
graduation?
What keeps you from being able to
continue to formal school in Grade
7?
How do family obligations affect
your attendance at the ALC?
How could things be changed to
address this problem?
6. The short distance to the ALC
from my home makes it possible for
me to go to school.
1 – Yes
2 – No
Effectiveness:
Innovation Since you started attending the
ALC, how has the programme
changed, if at all?
How have these changes affected
you?
7. I have my own learning materials.
1 – Yes
2 – No
133
Target Group: Girl Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
Have you had access to learning
materials?
What are they?
Have these been helpful?
Do you get enough attention from
the teacher to help you learn?
What would you improve in your
classroom?
Effectiveness: Impact Since you started attending the AL
courses, have things changed at
home in any way?
If so, please tell me how.
8. My life has changed for the better
since I began going to school in the
ALC.
1 – Yes
2 - No
Effectiveness:
Sustainability Tell me about your plans for the next
year .
Will you continue at the ALC?
If not, will you go to formal school?
Do you think you will continue your
education after you marry? If not,
why not?
Would you like to continue your
education after you marry?
9. I plan to go to formal school past
Grade 6.
1 – Yes
2 – No
10. If you are not continuing with
school, why not?
1 – my parents will not let me
2 – I am getting married
3 – the formal school is too far away
from my home
4 – family responsibilities (work, etc.)
134
Target Group: Girl Students in ALCs – 10 Questions, 2 short answer
Criterion: Theme Question Answers/Mobile Input
What would make it possible for you
to continue your education after you
are married?
Effectiveness:
Wrapping up
What are some other things about
the ALC that are important, but that
we haven’t talked about?
11. TEXT BOX #2
135
Annex 7 Evaluation/Learning Framework
Overview
This section links evaluation questions to specific methods of data collection and analysis and is based
on the global evaluation matrix and analysis plan.
Question Method Indicators
Relevance
What are key challenges to
children’s well-being and
learning?
Document review; Field-site
FGDs and interviews; Central
stakeholder interviews
Review: Statements or analyses
of challenges
Students: Descriptions of factors
that interfere with school
attendance, completion and
success that link to LUL
objectives
Community: Attitudes about and
support for girls’ and boys’
education that link to LUL
objectives
Teachers: Descriptions of
environmental challenges to
student attendance, completion
and success; descriptions of
academic challenges that link to
LUL objectives
Central: UNICEF CO personnel
describe challenges that link to
LUL objectives
136
Question Method Indicators
How are these or other
challenges specific to girls’
learning and well-being?
Document review; Available
quantitative data; Field-site
FGDs and interviews; Central
stakeholder interviews
Review: Statements or analyses
of challenges
Girl students: Descriptions of
challenges with specific
relevance to girls
Community: Attitudes about and
support for girls’ and boys’
education
Teachers: Attitudes about girls’
and boys’ education
Central: UNICEF ACO personnel
describe challenges specific to
girls
Quantitative: Girls’ (performance
/ completion / advancement in
comparison to boys
How does the LUL initiative (as
designed) address these
challenges?
Document review; Field-site
FGDs and interviews
Review: Descriptions of
initiatives that identify
relationship to challenges that
confront girls
Girl students: Descriptions of
activities that identify the
relationship to challenges that
confront girls
Teachers: Descriptions of
activities that identify the
relationship to challenges that
confront girls
137
Question Method Indicators
In what ways does the design
and/or implementation of the
LUL initiative address
challenges to student learning,
their well-being, and their
success in education and in life?
Field-site FGDs and
interviews; Central-stakeholder
interviews
Girl students: Descriptions of
improved environment, increased
motivation, other improvements
in school
Central: Descriptions of specific
components or activities in the
context of change or results that
have direct bearing on
challenges
Effectiveness
What is (or are) the objective(s)
of the LUL initiative?
Document review; Central-
stakeholder interviews
Review: Descriptions of
objective(s)
Central: Descriptions of
objectives
To what extent has the LUL
initiative accomplished the
stated objectives?
Document review; Field-site
FGDs and interviews; Central-
stakeholder interviews
Review: Performance monitoring
information
Girl students: Descriptions of
activities or changes that link to
objectives
Teachers: Descriptions of
activities or changes that link to
objectives
Families: Descriptions of
activities or changes that link to
objectives
Central: UNICEF CO personnel
or others describe activities and
results that link to objectives
138
Question Method Indicators
How has the LUL initiative, to
the degree it has accomplished
its objectives, affected the
learning and well-being of
students, their families, and their
communities?
And
What changes in terms of the
education environment,
educational opportunities or out-
of-school opportunities for girls
have resulted from the LUL
initiative?
Document review; Field-site
FGDs and interviews; Central-
stakeholder interviews
Review: Performance-monitoring
information; case-studies,
narratives or analyses that
describe results
Girl students: Descriptions of
initiative activities in relation to
improvements in school
environment, motivation, self-
conception, etc.
Teachers: Descriptions of
activities in relation to
improvements in girls’ motivation,
participation, etc.
Families: Descriptions of
changed attitudes, in particular
increased understanding of or
support for girls’ schooling
Central: Success stories, etc.
139
Question Method Indicators
When challenges arose, what
adaptive responses emerged?
Field-site FGDs and
interviews; Central-stakeholder
interviews
Girl students: Descriptions of
problems and obstacles to
initiative participation or
effectiveness (broadly framed),
and of changes in
implementation in response
Community: Descriptions of
problems and challenges
encountered by initiative
(especially that they themselves
communicated), and of changes
in implementation in response
Teachers: Descriptions of
problems and challenges in
classroom implementation or
girls’ participation (especially that
they themselves communicated),
and of changes in
implementation in response
Central: Descriptions of problems
in planning, design,
implementation, follow-up, etc., ,
and of changes in
implementation in response
In what ways did those adaptive
and/or innovative responses
address the needs of girls and
women?
Field-site FGDs and
interviews; Central-stakeholder
interviews
Teachers: Descriptions of
adaptive responses or innovative
approaches to address
challenges
Central: Descriptions of adaptive
responses or innovative
approaches to address
challenges
140
Question Method Indicators
What challenges to achieving
change have not been
addressed?
Field-site FGDs and
interviews; Central-stakeholder
interviews
Girl students: Descriptions of
unmet challenges or ongoing
problems
Teachers: Descriptions of unmet
challenges or ongoing problems
Community: Descriptions of
unmet challenges or ongoing
problems
Central: UNICEF ACO personnel
and others describe unmet
challenges or ongoing problems
How could the effectiveness of
the initiative be improved?
Field-site FGDs and
interviews; Central-stakeholder
interviews
Girl students: Descriptions of
potential solutions to unmet
challenges
Teachers: Descriptions of
potential solutions to unmet
challenges
Community: Descriptions of
potential solutions to unmet
challenges
Central: Descriptions of potential
solutions to unmet challenges
Sustainability and scalability
How does the initiative relate to
MOE or partners’ priorities?
Document review; Central-
stakeholder interviews
Review: Statements of policy and
planning goals, objectives,
priorities, etc.
Central: Linkage of initiative to
policy and planning priorities
141
Question Method Indicators
What are unit costs for the
initiative and what are their
ramifications for scaling?
Document review; Field-site
FGDs and interviews; Central-
stakeholder interviews
Review: Budget information;
other relevant information
(number of schools, regions,
etc.)
Local implementers: Descriptions
of variant local costs (e.g.,
generator/petrol, transportation,
etc.)
Central: Assessment (formal or
informal) of relative costs
What current or potential
partners are likely to have
interest in scaling (or replicating)
the initiative?
Central-stakeholder interviews Central: Descriptions of plans or
other evidence of intention to
scale (or replicate)
In what ways have activities
resulted in outputs (e.g., learning
resources, administrative
processes, etc.) that will reduce
costs of scaling or replicating
Review of initiative design;
Review of programme
documents; Central-
stakeholder interviews
Review: Identification of content
resources, expertise or other
items that can be re-used
Central: reports of support and/or
interest on the part of ministry or
other partner-organization
personnel
Document review / central:
connection to policies or plans
142
Question Method Indicators
How enduring are the changes
that have resulted from the
initiative?
and
What are their potential longer-
term effects?
Field-site FGDs and
interviews; Central-stakeholder
interviews
Teachers: Descriptions of
attitudinal or practical change
associated with success
Head teachers: Descriptions of
attitudinal change and of
practical measures for ongoing
teacher development related to
the initiative
Central: Explanations of
connection between initiative and
policy or planning goals and
values; mainstreaming of
initiative in MOE budget
How could the sustainability of
the initiative be improved?
Programme review; Field-site
FGDs and interviews; Central-
stakeholder interviews
Review: Identification of potential
overlap and recombination,
standardization, and other areas
Teachers/Head-teachers:
Identification of potential
pathways for integration or
mainstreaming (e.g., in school
requirements, budgets, etc.)
Central: Descriptions of possible
improvements to address
effectiveness or sustainability
143
Annex 8 Evaluation SOW
TERMS OF REFERENCE (TORS) FOR THE EVALUATION OF
LET US LEARN - AFGHANISTAN
Title Terms of reference for the evaluation of Let Us Learn Initiatives
(LUL) in Afghanistan, and scope of work (SOW) for (National
Evaluator)
Purpose To provide technical support for the evaluation of Let Us Learn
Initiatives in Afghanistan and contribute to the global evaluation
Contract
Evaluation Timeline July 2014 through 26 December 2014
A. Background
The Let Us Learn initiative (formerly known as the Basic Education & Equity initiative) is a unique private
partnership that allows for flexible and innovative approaches to addressing inequities in education access
and outcomes. Challenges and barriers to education, particularly amongst excluded and marginalized
children and youth have been identified in five diverse country contexts, Afghanistan, Bangladesh, Nepal,
Madagascar and Afghanistan. Each country is targeting the hardest to reach children by sharpening the
equity focus in both programming and monitoring of results. 20 million USD have been designated for the
period 2011-2014.
The programme is focused on three equity pillars: reaching out-of-school children, expanding girls’
education and improving quality outcomes for learners. Each of the participating Country Offices is working
to address inequity through targeted approaches, adapted to address their specific contexts. Each
programme is conducting a country-level evaluation that will also contribute and inform a global-level
evaluation.
Afghanistan:
In Afghanistan, the evaluation will focus on its innovative approach towards establishing Accelerated
Learning Programmes (ALPs) in community-based schools for children, especially girls, to receive and
finish their primary education. This initiative provided educational opportunities for young girls as well as
providing an innovative opportunity to recruit young women into the teaching profession. In particular, the
initiative has opened and made operational a total of 384 ALCs in 12 provinces in the central region,
benefitting 9,359 and out of these, 7,902 are girls (84%).
Some elements for consideration will include:
Tracking enrolment, retention and transition;
Collecting evidence on academic performance and/or learning outcomes;
Analysis of changes in attitudes/behaviours amongst key stakeholders;
Review of practices related to supply provision, monitoring practices, community teacher training,
etc.
Analysis of equity focus and innovative approaches;
144
B. Rationale for the evaluation
The partners involved in launching Let Us Learn agreed that an evaluation would be carried out at the end
of the programme. The evaluation effort is intended to ensure accountability, strengthen any future
programming and contribute to the global equity dialogue. In order to capture information across a broad
range of goals, a series of evaluation activities are being proposed. Evaluative activity is broadly defined to
include evidence gathering, performance monitoring and measuring results. Outside of the technical
assessment, consideration will also be given to improving organisational effectiveness by examining
development and implementation processes, as well as programme supervision.
One primary focus of evaluation will be to engage all COs and HQ in a learning process around the Let Us
Learn (LUL) initiative, documenting new efforts in monitoring for equity, innovations, partnership and good
practice. A series of 5 discrete evaluations and a synthesis/consolidation are planned, along with
participatory products such as testimonials and ‘most significant change’ stories involving beneficiaries.
We are aware that some of these analyses may be be challenged by a lack of baseline data and varied
programme approaches across all of the participating COs. Activities should build evidence to fill
information gaps and inform any next phase of the LUL, thus retroactive baselines may need to be
established to measure change. Mixed methodologies will be utilized given the diverse range of activities
taking place across COs but should fall within a global methodological framework.
The initial audience for the country-level evaluation products will be internal with discussions held on what
elements best contribute to the global evaluation. Each country office will have a dedicated evaluator to
review their programmes. These evaluators will work closely with the lead evaluator at the global level to
integrate efforts into the global evaluation. The country level evaluations should be complete in October,
the global synthesis in November and the final evaluation package needs to be completed by December
2014.
C. Evaluation approach and scope
Activities included and excluded from evaluation. The self-evaluation study will examine only
activities/results that are agreed upon with the respective country offices. Processes related to planning,
coordination and monitoring will be included. Selected programme activities along each of the three pillars
of the programme will be included as will thematic analysis related to equity and innovation.
Attribution and/or contribution. While evaluating outcomes also often determines if a programme/project
has added value to country-level results as stated in the Country Programme, it is often difficult to
attribute results to only one source of inputs, actions, or actors in a field where there are multiple actors
and inputs. Therefore the evaluation will seek to outline, to the extent possible, LUL’s contribution to
overall results through a contribution analysis. From a country perspective, the objectives of the
evaluation are threefold;
4. To examine the extent to which LUL projects are achieving intended outcomes at the country
level; and,
145
5. To systematically document new learning, and to document new efforts in monitoring for equity, in
particular.
6. To contribute to the global evaluation and analysis of cumulative impact.
The proposed approach is to conduct a self-evaluation1 exercise in Afghanistan with a primary focus on
learning and a secondary focus on accountability, utilizing a common evaluation/learning framework.
Methodologies and scope will be determined in consultation with the Country Office and with technical
inputs from the lead evaluator.
Evaluation Criteria and questions: Since this is an evaluation of innovation (equity innovations), the
OECD/DAC criteria should be used only to the extent that they are relevant. However, we expect the
evaluator to propose criteria that is more appropriately suited to evaluation of innovations. Similarly, the
evaluation criteria will be the basis for developing evaluation questions.
Attribution and/or contribution. While evaluating outcomes also often determines if a programme/project
has added value to country-level results as stated in the Country Programme, it is often difficult to attribute
results to only one source of inputs, actions, or actors in a field where there are multiple actors and inputs.
Therefore the evaluation will seek to outline, to the extent possible, LUL’s contribution to overall results
through a contribution analysis.
D. Methodology
Sample: A draft stakeholder analysis has been conducted as part of preliminary planning for the evaluation,
hence a sampling framework that reflects a range of activities, stakeholders and will be developed for each
country. Detailed information on sampling will be included in the inception report.
Design and data collection methods: Data will be collected from multiple sources, using both qualitative
and quantitative methods. A menu of methods and sources include a review of programme documents,
interviews of key/elite informants, interviews and/or focus groups of important groups of beneficiaries,
testimonials involving beneficiaries (One Minute Jrs.), and direct observations of activities with beneficiaries
wherever possible. A retroactive baseline will be established in consultation with programme staff and
relevant education authorities to assess to extent possible contribution to quality of education (leaning
outcomes, reduction in violence, etc. as per the Child Friendly Schools framework). The evaluator will
provide technical oversight and support to the national enumerators (including orientation, training, piloting
of tools, quality assurance, etc.).
Review of documents and procedure: A desk-based review will be used as an important tool to
examine the theory of change, restate and/or amend it to reflect what happened in practice
(empirical ToC); assemble evidence of activities or outputs. LUL focal points in each country will
avail key documents for a systematic desk review. These will include LUL programme documents
(proposals, monitoring frameworks, reports, human interest stories, etc.), sector plans, other
education evaluations and assessments as relevant.
146
Data Collection: Primary data – to be collected through national enumerators through a
competitive institutional contract. Their work will include interviews and focus groups, self-
administered surveys, and possibly small scale assessments – will add to the existing knowledge
about the activities, and will be used to gauge perceptions of stakeholders on the utility and impact
of interventions.
E. Evaluation responsibilities and management
The LUL Afghanistan evaluation will be supported by:
An international evaluator
An in-country manager and reference group, responsible for technical supervision of the evaluator,
and for clearance of all evaluation products, respectively;
National enumerators contracted through a competitive process;
An international Lead evaluator (ILE), who will provide technical guidance on the overall approach
to enable learning across countries; and,
The LUL global coordinator and Evaluation Office in HQ, on administrative matters, supervision of
the international lead Evaluator, and overall quality assurance.
The international evaluator (IE) in Afghanistan will be responsible for the professional conduct of the
evaluation in accordance with the terms of reference, the United Nations Evaluation Group (UNEG) Norms
and Standards for Evaluation1 and the UNEG Code of Conduct for Evaluation1. He/she will be expected to
perform the following tasks:
review and finalize the evaluation approach and work plan;
customize, review and validate data collection tools, (the self-evaluation module, which may include
questionnaires, interview questions and protocols);
supervise data collection and data processing;
design and facilitate consultative workshops and meetings;
prepare all country reports – inception, draft and final reports, and develop PowerPoint
presentations;
convene a forum to validate the findings of the evaluation;
attend and make a presentation at the global reporting workshop; and,
ensure that the International Lead Evaluator and reference group is regularly informed of the
progress of the evaluation, any possible causes of delays and issues to resolve.
The Evaluation Manager (ideally the Monitoring and Evaluation Officer) will provide overall guidance and
contribute directly to its quality assurance activities. The Afghanistan LUL country team will support the self-
evaluation by providing support to arrange for meetings, write letters of introduction and/or accompany the
evaluator to meetings where necessary.
Reference Group: Our recommendation is to establish a reference group in each participating country.
Membership will be determined within country for each of the five country offices.
147
The Reference Group Chair is responsible for oversight of evaluation activities, while members will advise
on the following:
Consultant’s terms of reference and work plan, including any adjustments required at inception or
other phases
Approval of all evaluation products, including the final report
The international lead evaluator (ILE), contracted by the UNICEF Evaluation Office, will be responsible
for leading the evaluation efforts from a global perspective. Specifically, he/she will be responsible for the
following:
develop the evaluation approach and methodology, including the evaluation/learning framework;
develop a self-evaluation module/guide and other materials as he/she sees fit;
facilitate a virtual planning workshop for national consultants and focal points in each participating
country;
pilot the evaluation methodology in one country, and effect the necessary revisions;
review data collection tools and protocols used at the country level;
undertaking quality assurance missions during data collection in up to two participating countries;
review country reports - draft and final reports;
ensure that the evaluation managers (Evaluation Specialist and Education Specialist in New York)
are regularly informed of the progress of the evaluation, possible causes of delay and issues to
resolve.
The Evaluation Specialist and Education Specialist (in New York) will manage the work of the
international lead evaluator, provide overall guidance to the evaluation, and contribute directly to its quality
assurance activities.
F. Evaluation processes and products
Evaluation Planning Workshop (via Webinar): The International Lead Evaluator will develop initial
draft of the evaluation approach, and convene an orientation and planning webinar for the national
evaluators, to finalize the approach to the evaluation. The planning workshop is expected to agree on the
methodology, including the sampling frame for each country, a set of evaluation tools with necessary
customization to each country, and a tentative work plan that reflects all evaluation activities for country
teams and for the international lead evaluator.
Inception Reports: An inception report will be produced at the global level by the International Lead
Evaluator, and adapted/customized for each country by the National Evaluator. In addition to presenting
a short summary of the programme context, these reports will be used to confirm a common understanding
of the description of what is being evaluated, and the logic or theory of change for each programme. The
report will include, inter alia,
148
Evaluation purpose and scope – a clear statement of the objectives of the evaluation and the
main aspects and limitations of the evaluation
Evaluation criteria and questions – question to be answered by the evaluation, and the criteria
which will be used to assess performance, including questions that address critical human rights
and gender equality issues
Evaluation methodology – expanding on the methodology section in the TOR, a sampling
strategy, a description of data collection methods and data sources (including a rationale for their
selection), draft data collection instruments, a discussion on reliability and validity of the evaluation,
and a discussion on the limitations of the methodology. This section should include instruments to
assess relevant human rights and gender equality aspects.
Evaluation matrix and analysis plan – a mapping that identifies evaluation questions, how they
will be answered through the selected methods, and a data analysis plan;
Evaluation work plan and timeline – a revised work and travel plan
Structure for the final report will be proposed in the inception report, and may be revised later in
the evaluation
Resources requirements – detailed budget for the evaluation, tied to evaluation activities, work
plan, deliverables.
The inception report will be 10-15 pages, including the revised work plan, and will be presented at a formal
meeting of the in-country reference group, and LUL extended management team (HQ Manager plus Chiefs
of Education/LUL focal points in participating countries).
Evaluation Report: The final evaluation report should include, but not limited to the following:
An analysis of concepts, trends and critical themes in innovating solutions for equitable provision
of education in respective contexts;
An analysis of overarching programme goals and themes (OOSCI, equity, innovation, etc.), nothing
those with most demonstrable impact;
An overview of each of the country evaluations, analyzing trends, areas of divergence and any
lessons learned;
An assessment of UNICEF’s mandate, strengths and weaknesses relating to UNICEF’s strategic
and programmatic choices for equity, against a set of agreed evaluation criteria;
An analytical framework which UNICEF can apply to improve the effectiveness of its support for
equity interventions, at the country level, and an assessment of the role of the regional and global
offices in support of equity in education;
Derived from the findings of the evaluation, recommendations for improving coordination, and
successful scale up of equity focused programming.
The evaluation report should not exceed 30 pages (excluding the executive summary and annexes).
Annexes will include the TOR, description of methodology (including evaluation/learning framework), list of
background materials used, list of people interviewed, PowerPoint presentations, and workshop materials.
G. Desired Competencies for Evaluator
149
Evaluator must offer the following demonstrated experience, knowledge and competencies:
1. Significant knowledge and experience of evaluation concepts and approaches;
2. Good knowledge in gender in the Education Sector, and experience in evaluating equity and
education innovations;
3. Facilitation skills, particularly design of stakeholder consultation exercises as well participatory
methods;
4. Strong quantitative and qualitative data collection skills;
5. Strong quantitative and qualitative data analysis skills;
6. Excellent language and communication skills in English;
7. Demonstrated report writing skills, in English; and
8. Computer literacy in Word, Excel and PowerPoint;
Evaluator must remain in strict adherence with UNEG ethical guidelines and code of conduct.
H. Proposed resource commitments
It is anticipated that the resource commitments would as follows:
50-60 person days for National Evaluators for each country; DSA for international and/or in-country
travel (# of days in-country TBD)
Travel costs for one international mission and 4-6 days DSA for national consultant (reporting
workshop)
Travel costs for one international mission and 4-6 days DSA for UNICEF/LUL Afghanistan focal
point or UNICEF M&E Special (reporting workshop)
I. Tasks for the National Evaluator (Afghanistan)
The National Evaluator will be responsible for leading the evaluation for the LUL Afghanistan Phase
1. Specifically, she will be responsible for the following:
Adapt and customize the evaluation approach and methodology, and learning framework to the
Afghanistan context, and develop and work plan;
Develop draft evaluation tools aligning them with the overall evaluation methodology;
Participate in the piloting of the evaluation methodology in Afghanistan, and update the tools
accordingly;
Execute the evaluation in Afghanistan; support national enumerators to collect and analyse data
as per the agreed methodology;
150
Participate in reference group meetings, including a dissemination meeting where s/he will present
the evaluation findings.
Draft country reports and/or updates - draft and final reports;
Ensure that the evaluation manager and lead evaluator is regularly informed of the progress of the
evaluation, possible causes of delay and issues to resolve.
J. Timeline, time allocation and deliverables
Task Output/
deliverables
Person
days
Deadline
Participate in conceptualization and development
of evaluation design and approach (learning
framework, methodology, work planning, country-
evaluation module, etc.)
Afghanistan inception
report (first draft)
2 days July,
2014
Mission to Afghanistan: Inception meeting for
consultations with the LUL team, KII with team
Afghanistan inception
report, including trip report
as appendix (second draft)
2 days August,
2014
Training of additional data collectors; pilot testing
of the evaluation instruments in Afghanistan, and
data collection, KII with partners and
stakeholders as possible
Afghanistan evaluation
instruments
6 days August,
2014
Technical backstopping with country office on
outstanding data collection activities; data
analysis and report writing
Afghanistan evaluation
report (zero draft)
8 days Sept,
2014
Mission to Nepal: Quality review and reporting
meeting:
Afghanistan evaluation
report – quality reviewed
(1st draft)
6 days Sept 2014
Follow-up mission to Afghanistan: more data
collection, cleaning, analysis and sharing 2nd
draft of report
*as needed and requested by NCO
Afghanistan evaluation
report (2nd draft); PPT
presentation (first draft)
8 days Sept/Oct
2014
Revision of reports and final reporting Afghanistan evaluation
report (final draft); Final
PPT presentation
8 days Oct 30,
2014
TOTAL 40 days
In summary, the national evaluator for Afghanistan is responsible for the following deliverables:
151
Inception report: This report will include, among others, the revised evaluation approach and
methodology, revised evaluation work plan, country-evaluation module and other materials for the
orientation and planning webinar and facilitating the workshop;
Afghanistan evaluation reports: These will be developed by the national evaluator - the ILE
will provide quality control/review;
Evaluation synthesis report: First, second and final drafts, according to the UNICEF House
Style and UNICEF standards for evaluation reports. These will be developed by the national
evaluator - the ILE will provide quality control/review; and,
PowerPoint presentation for evaluation synthesis report: The national evaluator will be
expected to present at 1-2 reporting/dissemination events.
152
ANNEX 9 Quantitative Data: Charts from Participant Groups
Soft Power Solutions created the charts using AtlasTi, a qualitative software analysis program. They are derived from instances in the narrative data, which are given a numerical value in AtlasTi. Thus, for the first question on Relevance: Awareness, respondents could have answered “yes” directly regarding the importance of education, or an affirmative answer was implied from their response when it was analyzed.
Family Members
1. Relevance: Awareness
Figure 1: Importance of education (per family member)
2. Relevance: Challenges
Figure 2: What prevents children from completing ALC (per family member)
100%
Please tell me about the
importance of education. Is it
important that your children
complete school?
Yes
5%
5%
19%
8%
3%
60%
What might keep one of your children from
completing the ALC?
Family problems
Early Marriages
Insecurity
Lack of learning
materialsPoverty
153
3. Relevance: Equity
Figure 3: Hopes for their children’s future (per family member)
Figure 4: Will your child finish primary school? (per family member)
84%
16%
Please tell me about your hopes for your
children, in relation to education
Complete their
education
Serve their
community
95%
5%
Do you believe that your child will finish
primary school?
Yes
Maybe
154
Figure 5: Will your child finish secondary school? (per family member)
4. Effectiveness: General
Figure 6: Important affects of ALC on girls (per family member)
Figure 7: Important effects of ALC on boys (per family member)
92%
8%
Do you believe that your child will finish
secondary school?
Yes
Maybe
3%
59%11%
8%
19%
What are the most important affects of the ALC for
girls?
Older girls are educated
Girls are educated
Has solved long distance
problem to schools
Their behavior has
improved
It is accelerated
155
5. Effectiveness: Challenges Figure 8: Factors preventing girls from joining ALC? (per family member)
6. Effectiveness: Learning & Innovation Figure 9: Problems affect on girls (per family/community member)
62%
14%
5%
19%
What are the most important effects of the ALC for
boys?
Boys are educated
Solved long distance problem to
schools
Their behavior has improved
It is accelerated
3%3%3%
13%
24%
8%5%
41%
What factors do you think keep more girls from participating
in the ALC?
Lack of parents awareness about the
importance of educationLack of heating materials
Early marriages
Long distance
Poverty
House responsibilities
Cultura traditions / customs
Nothing
156
7. Effectiveness: Learning & Innovation Figure 10: Improving ALC (per family member)
8. Effectiveness: Impact
27%
54%
19%
How much have these problems affected the ALCs
or girls who participate in it?
Some affect
A lot of affect
No affect
3%
47%
12%
14%
6%
6%
6%6%
Are there other changes that might improve the ALC
programme?
Increasing teacher's salary
Sufficient learning materials
Additional ALCs in remote areas
Continuation of ALCs
Encouraging families to enroll their
children
Proper management of ALCs by
UNICEF
Separate classes for girls and boys
No
157
Figure 11: Change in Attitude (per family member)
9. Sustainability Figure 12: Support for ALC (per family member)
10. Effectiveness: Wrapping up Figure 13: Other important aspects (per family member)
100%
Has the availability of the ALCs changed your attitude
about your child’s schooling?
Y…
11%
84%
5%
Do you support the ALCs in any way—with your time, or in other
ways?
Yes; by encouraging other families to
send their children
Yes; by sending our childre
Yes; through other ways
158
Teachers Final Report
1. Relevance: Getting started Figure 1: Education level (per teacher)
Figure 2: Training participation (per teacher)
24%
33%
24%
19%
Are there other aspects of the ALCs that are
important, but that we haven’t talked about?
Want additional ALCs
Classes to be upgraded to
9th gradeWant ALCs continuation
No
6%
73%
9%
12%
Please tell me about your education as a teacher?
10th grade graduate
12th grade graduate
14th grade graduate
Bachelors Degree
159
2. Relevance: Learning Figure 3: Most important aspects of improving quality of education (per teacher)
3. Relevance: Challenges
97%
3%
Did you participate in training in relation to the
ALCs?
Yes
No
38%
9%6%
6%
12%
12%
17%
Which aspects of the ALCs are most important for improving
access and quality of education?
Provision of books and stationary
Seminars for teachers
Proper student & teacher
attendanceProper learning materials
Additional classes
Better teaching environment
Better teaching methods
160
Figure 4: Teachers’ problems (per teacher)
Figure 5: Problems effect on student learning (per teacher)
4. Relevance: Challenges 2 Figure 6: Students education challenges (per teacher)
12%
73%
3%
12%
Please tell me about some of the problems that you
face as a teacher
Lack of proper learning
space
Lack of books and
stationery
Poverty
Nothing
84%
16%
Do these problems affect student learning?
Yes No
161
1. Relevance: Equity Figure 7: Challenges effects on girls v. boys (per teacher)
1. Effectiveness: General Figure 8: Most important ALC effects on girls (per teacher)
17%
50%
6%
6%
9%
3%9%
What challenges do your students face in relation to
their education?
Lack of proper learning
space
Lack of books and
stationery
ALC timing
Distance
Insecurity
Poverty
Nothing
35%
65%
Do any of these challenges affect girls more than boys?
Yes
No
162
2. Effectiveness: Sustainability & Impact Figure 9: Families response to changes (per teacher)
Figure 10: Community members’ responses to changes (per teacher) \
91%
9%
What are the most important effects of the ALCs for
girls?
They are literate
They have become well
mannered
94%
6%
How are these changes received by family
members?
They are happy
They support the
program
163
3. Effectiveness: Learning & innovation Figure 11: Teaching and learning materials (per teacher)
Figure 12: Class assessment (per teacher)
50%50%
How are these changes received by other
members of the community?
They are happy
They support the
program
38%
35%
27%
Do you have teaching and learning materials for
yourself? For your students?
Yes
No
Not Sufficient
164
4. Effectiveness: Learning & Innovation 2 Figure 13: ALC problems (per teacher)
Figure 14: Program changes to respond problems (per teacher)
29%
59%
12%
How do you assess learning in your classroom?
Group & individual work
Daily evaluations
Homework
15%
26%59%
Since the ALCs started in your school, what problems has
it encountered?
Lack of learning space
Lack of learning materials
No problem
165
5. Sustainability Figure 15: Support for ALC (per teacher)
6. Effectiveness: Wrapping up Figure 16: Other aspects / recommendations (per teacher)
24%
47%
29%
Has the programme been changed to respond to
these problems?
Partly
Completely
No
29%
18%
53%
Do you support the ALC programme in any other way—
with your time, by encouraging girls outside of school or
in other ways?
Yes; by encouraging people
to send their children to
ALCs
Yes; by encouraging
children to join ALCs
Yes; through other ways
166
Principles
Figure 1: ALC challenges (per principle)
Figure 2: Barriers to Girls Participation in ALCs (per principle)
6%6%
12%
3%
6%
67%
Are there other aspects of the ALCs that are important,
but that we haven’t talked about?
Additional ALCs
Continue ALCs
Upgrade ALCs to 9th grade
Payment of salary on time
Increase in salary
No
17%
10%
7%
13%
53%
Are there challenges to the ALC being a
part of your HUB school?
Lack of books and
stationery
Lack of a proper
classroom
Lack of heating
material
Distance to HUB
School
No Problem /
Challenge
167
Figure 3: Important Innovations (per principle)
Figure 4: HUB School support to ALC (per principle)
27%
33%
37%
3%
What are the barriers to girls' participation in the
ALCs?
No Barrier
Family Restrictions
Insecurity
Long Distance
27%
33%
30%
10%
In your opinion, which of the innovations in ALCs
are most important?
Accessibility for older aged
girls
Proximity to villages
Completing 2 grades per
year
All
168
Male Students
1. Relevance: The initiative Figure 1: Chance of attending formal school (per male student)
Figure 2: Challenges to attending formal school (per male student)
7%
33%
60%
What can the school hub/formal school system do to
further encourage the success of the ALCs and reach
more OOSC?
Provide a classroom /
building
Provide learning materials
Encourage families to send
their children to ALC
87%
13%
Would you have had a chance to attend school if you
were not in the ALC?
No
Yes
169
2. Relevance: Challenges Figure 3: Challenges to Attending ALC (per male student)
3. Effectiveness: General Figure 4: Favorite subjects (per male students)
6%
88%
6%
What prevented you from attending formal school?
Being Overaged
Distance to School
Lack of Time
19%
81%
Challenges to attending ALC Courses (Per Male
Students)
Lack of a proper
classroom
Lack of stationery /
learning materials
170
4. Effectiveness: Sustainability Figure 5: Continue at ALC or join formal school (per male student)
Figure 6: Future plans / aspirations (per male student)
33%
13%
6%7%
7%
27%
7%
What is your favorite subject?
Dari
Holy Quran
Painting
Pashto
Geography
Math
Science
100%
Will you continue at the ALCor go to formal school?
Will Continue at ALC
171
5. Effectiveness: Wrapping up Figure 7: Other important aspects of ALC (per male student)
Female Students
25%
44%
6%
6%
6%
13%
What do you want to be when you grow up?
Doctor
Teacher
Translator
Pilot
Astronaut
Don't Know
19%
44%
37%
What are some other things about the ALC that are
important, but that we haven’t talked about?
The Need for a Proper
Classroom
ALC to be Expanded to 9th
Grade
Nothing
172
1. Relevance: The initiative Figure 1: Chance of attending formal school (per female student)
Figure 2: Challenges to attending formal school (per female student)
2. Relevance: Challenges Figure 3: Challenges to attending ALC (per female student)
91%
2% 7%
Would you have had a chance to attend school if
you were not in the ALC?
No
Yes
Maybe
6%
74%
6%
1%
12%1%
What prevented you from attending formal school?
Being Overaged
Distance to School
Family Restrictions
Lack of Time
Insecurity
Other
173
3. Effectiveness: General Figure 4: Favorite subjects (per female student)
Figure 5: Family obligations affect on attendance (per female student)
1%
47%
1%13%
17%
21%
Distance to ALC
Lack of Learning
Materials
Family Restrictions
Lack of a Proper
Classroom
Insecurity
No Challenges
27%
6%
12%23%
6%
5%
10%
5%
6%
What is your favorite subject?
Dari
Pashto
Painting / Drawing
Holy Quran
Math
Science
Calligraphy
Geometry
Biology
174
Figure 6: Addressing ALC challenges (per female student)
1. Effectiveness: Sustainability Figure 7: Continue at ALC or join formal school (per female student)
43%
57%
How do family obligations affect your
attendance at the ALC?
Affects
Does not Affect
64%13%
6%
17%
How could things be changed to address the ALC
problem(s)?
Additional Learning
Materials
Change in Class
Timing
Do Not Know
No Problem at ALC
175
Figure 8: Education after marriage (per female student)
4. Effectiveness: Wrapping up
Figure 9: Aspirations / Future Plans (per female student)
21%
79%
Will you continue at the ALC? If not, will you go
to formal school?
Will Join Formal
School
Will Continue at ALC
68%
5%
27%
Would you like to continue your education after
you marry?
Yes
No
Maybe
176
23%
0%
46%
0%
6%
3%
7%
8%
6% 1%
What do you want to be when you grow up?
Doctor
Parliament Member
Teacher
Judge
Lawyer
Journalist
Engineer
Midwife
Nurse
Do Not Know