Completion Report

Project Number: 37475 Loan Number: 2294-INO(SF) March 2014

Indonesia: Madrasah Education Development Project This document is being disclosed to the public in accordance with ADB’s Public Communications Policy 2011.

CURRENCY EQUIVALENTS

Currency Unit – rupiah (Rp)

At Appraisal At Project Completion (31 October 2006) (31 December 2012)

Rp1.00 = $0.00011 $0.00010 $1.00 = Rp9,180 Rp9,780

ABBREVIATIONS

ADB – Asian Development Bank BAPPENAS – Badan Perencanaan dan Pembangunan Nasional

(National Development Planning Agency) CDD – community-driven development CPMU – central project management unit DCU – district coordinating unit GAP – gender action plan MDC – madrasah development center MDP – madrasah development plan MOEC – Ministry of Education and Culture MORA – Ministry of Religious Affairs PCU – provincial coordinating unit PIC – project implementation consultant PSC – project steering committee TA – technical assistance

GLOSSARY

madrasah aliyah – senior secondary madrasah madrasah ibtidaiyah – primary madrasah madrasah tsanawiyah – junior secondary madrasah

NOTES

(i) The fiscal year (FY) of the government and its agencies ends on 31 December. FY before a calendar year denotes the year in which the fiscal year ends, e.g., FY2009 ends on 31 December 2009.

(ii) In this report, "$" refers to US dollars.

Vice-President S. Groff, Operations 2 Director General J. Nugent, Southeast Asia Department (SERD) Country Director A. Ruthenberg, Indonesia Resident Mission (IRM), SERD Team leader S. Hasanah, Senior Project Officer, IRM, SERD Team members W. Kubitzki, Principal Portfolio Management Specialist, IRM, SERD Suzana, Senior Operations Assistant, IRM, SERD

In preparing any country program or strategy, financing any project, or by making any designation of or reference to a particular territory or geographic area in this document, the Asian Development Bank does not intend to make any judgments as to the legal or other status of any territory or area.

CONTENTS

Page BASIC DATA ii MAP vii

I.  PROJECT DESCRIPTION 1 

II.  EVALUATION OF DESIGN AND IMPLEMENTATION 2 A.  Relevance of Design and Formulation 2 B.  Project Outputs 2 C.  Project Costs 7 D.  Disbursements 8 E.  Project Schedule 8 F.  Implementation Arrangements 8 G.  Conditions and Covenants 8 H.  Related Technical Assistance 8 I.  Consultant Recruitment and Procurement 9 J.  Performance of Consultants, Contractors, and Suppliers 9 K.  Performance of the Borrower and the Executing Agency 10 L.  Performance of the Asian Development Bank 10 

III.  EVALUATION OF PERFORMANCE 10 A.  Relevance 10 B.  Effectiveness in Achieving Outcome 11 C.  Efficiency in Achieving Outcome and Outputs 11 D.  Preliminary Assessment of Sustainability 12 E.  Impact 13 

IV.  OVERALL ASSESSMENT AND RECOMMENDATIONS 14 A.  Overall Assessment 14 B.  Lessons 14 C.  Recommendations 15 

APPENDIXES 1. Design and Monitoring Framework 16 2. Participating Districts 21 3. Output 2: Teaching and Learning Resources and Facilities Upgraded to National Standards 23 4. Results of Accreditations 25 5 Academic Performance of Project Madrasah Students in the National Exams 26 6. Student Enrollments at Project Madrasahs 28 7. Gender Action Plan Targets and Achievements 31 8. Project Cost and Financing Plan 38 9. Fund-Channeling Mechanism 39 10. Project Implementation Schedule 40 11. Implementation Arrangements 43 12. Status of Compliance with Loan Covenants 45 13. Technical Assistance Completion Report 50 14. Economic and Financial Reevaluation 53

BASIC DATA A. Loan Identification 1. Country 2. Loan Number 3. Project Title 4. Borrower 5. Executing Agency 6. Amount of Loan 7. Project Completion Report

Number

Indonesia 2294-INO(SF) Madrasah Education Development Project Republic of Indonesia Directorate General of Islamic Education, Ministry of Religious Affairs Original loan amount: $50.0 million (SDR33,638,000) Actual loan amount: $50.2 million (SDR32,326,766) PCR: INO 1436

B. Loan Data 1. Appraisal – Date Started – Date Completed 2. Loan Negotiations – Date Started – Date Completed 3. Date of Board Approval 4. Date of Loan Agreement 5. Date of Loan Effectiveness – In Loan Agreement – Actual – Number of Extensions 6. Closing Date – In Loan Agreement – Actual – Number of Extensions 7. Terms of Loan – Interest Rate – Maturity (number of years) – Grace Period (number of years)

18 September 2006 2 October 2006 16 September 2006 16 September 2006 15 December 2006 15 March 2007 13 June 2007 13 June 2007 0 30 September 2012 03 February 2014 1 1% during the grace period; 1.5% thereafter 32 years 8 years

9. Disbursements a. Dates Initial Disbursement

7 September 2007

Final Disbursement

16 July 2013

Time Interval

69 months

Effective Date

15 June 2007

Original Closing Date

30 September 2012

Time Interval

63.5 months

iii

b. Loan 2294(SF)-INO: Amount ($’000)

Category or Subloan

Original

Allocation

Last Revised

Allocation

Amount

Canceled

Net Amount

Available

Amount

Disbursed

Undisbursed

Balance 1 Equipment 882.93 690.32 0.00 690.32 676.67 13.652 Human Resources

Development 5,480.41 2,799.90 0.00 2,799.90 2,564.98 234.92

A Overseas Training

900.77 1,004.11 0.00 1,004.11 997.76 6.36

B In-Country Training

4,579.64 1,795.79 0.00 1,795.79 1,567.22 228.57

3 Madrasah Development Plan

35,256.27 41,259.84 0.00 41,259.84 41,064.60 195.24

4 Surveys and Testing

337.42 299.27 0.00 299.27 266.12 33.15

5 Consulting Services 2,057.19 2,172.68 0.00 2,172.68 2,083.83 88.85 A International 960,22 587.00 0.00 587.00 584.15 2.85 B National 1,096.97 1,585.68 0.00 1,585.68 1,499.68 86.006 Project

Management Support

2,189.49 2,703.04 0.00 2,703.04 2,424.51 278.54

7 Interest Charges 1,256.02 1,154.96 0.00 1,154.96 1,146.75 8.208 Unallocated 2,540.28 1,146.22 0.00 1,146.22 0 1,146.22 Total 50,000.00 52,226.23 0.00 52,226.23 50,227.46 1,998.77Notes: 1. The increase in the total revised allocation was due to fluctuations of exchange rates.

2. Numbers may not sum precisely because of rounding.

c. Loan 2294-INO(SF): Amount (SDR’000) Category or Subloan

Original

Allocation

Last Revised

Allocation

Amount

Canceled

Net Amount

Available

Amount

Disbursed

Undisbursed

Balancea 1 Equipment 594.00 469.00 0.00 469.00 460.05 8.952 Human Resources

Development 3,687.00 1,815.00 0.00 1,815.00 1,660.88 154.12

A Overseas Training

606.00 645.00 0.00 645.00 640.83 4.17

B In-country Training

3,081.00 1,170.00 0.00 1,170.00 1,020.05 149.95

3 Madrasah Development Plan

23,719.00 26,500.00 0.00 26,500.00 26,371.91 128.09

4 Surveys and Testing

227.00 195.00 0.00 195.00 173.25 21.75

5 Consulting Services

1,384.00 1,412.00 0.00 1,412.00 1,353.71 58.29

A International 646.00 377.00 0.00 377.00 375.13 1.87 B National 738.00 1,035.00 0.00 1,035.00 978.58 56.426 Project

Management Support

1,473.00 1,750.00 0.00 1,750.00 1,567.35 182.65

7 Interest Charges 845.00 745.00 0.00 745.00 739.62 5.388 Unallocated 1,709.00 752.00 0.00 752.00 0.00 752.0

Total 33,638.00 33,638.00 0.00 33,638.00 32,326.77 1,311.23a  The undisbursed balance of SDR1.31 million (equivalent to $1.99 million) was cancelled when the loan account

was closed on 3 February 2014. Note: SDR1 = $1.52. 

iv

10. Local Costs (Financed) - Amount ($ million) 0.00 - Percent of Local Costs 0.00 - Percent of Total Cost 0.00 C. Project Data

1. Project Cost ($ million)

Cost Appraisal Estimate Actual Foreign Exchange Cost 0.00 0.00 Local Currency Cost 0.00 0.00 Total 0.00 0.00

2. Financing Plan ($ million)

Cost Appraisal Estimate Actual Implementation Costs Borrower Financed 21.43 17.87 ADB Financed 48.74 49.08 Other External Financing (madrasahs and community contributions)

0.00 3.10

Total 70.17 70.05

IDC Costs Borrower Financed 0.00 0.00 ADB Financed 1.26 1.15 Other External Financing 0.00 0.00 Total 1.26 1.15

Grand Total 71.43 71.20

ADB = Asian Development Bank, IDC = interest during construction. 3. Cost Breakdown by Project Component ($ million)

Component Appraisal Estimate Actual A. Furniture, equipment, and vehicles 1.26 1.16 B. Human resources development 22.11 18.54 C. Consulting services 2.06 2.08 D. Madrasah block grants 36.98 44.16 E. Survey and testing 0.34 0.31 F. Project management supports 2.19 2.42 G. Incremental recurrent costs 1.54 1.21 H. Taxes and duties 0.10 0.17 G. Contingencies 3.59 0.00 H. Financing charges during implementation 1.26 1.15

Total 71.43 71.20

v

4. Project Schedule

Item Appraisal Estimate

Actual

Date of Contract with Consultants Project implementation consultants Q2 2007 December 2008 Individual consultants to support management Q1 2007 March 2008 Upgrading teacher to degree level program Q4 2007 March 2009 Professional certification program Q4 2007 March 2009 Short-term training on subject content, methods, and new

curriculum Q4 2007 September 2008

Improvement of governance and management Q3 2007 September 2009 Development and implementation of teaching and learning

assessment and quality assurance system Q4 2007 September 2009

Introduction of performance-based planning and budgeting Q2 2008 September 2009 Establishment of madrasah advocacy program Q1 2008 March 2010 Preparation of madrasah development plans Q2 2007 June 2007 Transfer of block grants of project madrasahs Q4 2007 October 2009 Implementation of madrasah development plans Q1 2008 June 2010 Establishment of Central Project Management Unit Q2 2007 June 2007 Establishment of provincial and district coordinating units Q2 2007 June 2007 Development of project management information system Q3 2007 September 2009 Development of project website Q4 2007 November 2010 Midterm review Q4 2009 November 2010 Final reviews Q1 2012 June 2012 External monitoring and evaluation Q4 2007 November 2009 5. Project Performance Report Ratings

Ratings Implementation Period

Development Objectives

Implementation Progress

June 2007–January 2008 Satisfactory Satisfactory February 2008–December 2009 Satisfactory Potential Problem January 2010–December 2012 Satisfactory Satisfactory D. Data on Asian Development Bank Missions

Name of Mission

Date

No. of Persons

No. of Person-

Days Specialization of Membersa

Fact-finding 17 Jul–12 Aug 2006 4 40 a, b, c, d, e, f, g Appraisal 18 Sep–2 Oct 2006 4 40 a Inception 25– 28 Sep 2007 4 16 e, h, i, j Review 1 25 Apr–15 May 2008 3 12 d, k, l Review 2 2–18 Dec 2008 3 12 d, k, l Review 3 2–26 Jun 2009 2 14 d, k Review 4 1–16 Dec 2009 2 12 d, k Review 5 8 Apr–3 May 2010 2 14 d, k Review 6 6 Dec 2010–17 Jan 2011 2 10 d, m Review 7 23 May–14 Jul 2011 2 10 d, m Review 8 16 Dec 2011–10 Jan 2012 2 10 d, m Review 9 4 May–19 Jun 2012 2 18 m, o, p Special project administration 25–29 Jun 2012 3 2 p Project completion review 7 Mar–9 Sep 2013 2 20 m, p

vi

a a = senior project implementation specialist, b = project specialist, c = project implementation specialist, d = project officer, e = implementation specialist, f = education consultant, g = financial analyst, h = senior project management specialist, i = principal human resources development specialist, j = assistant project analyst, k = senior portfolio management specialist, l = gender and development advisor, m = principal portfolio management specialist, n = senior project officer, o = senior social development specialist, p = senior project officer.

I. PROJECT DESCRIPTION

1. The education system in Indonesia has two main subsystems: general school education, which is under the administration of the Ministry of Education and Culture (MOEC); and madrasah education, which is under the responsibility of the Ministry of Religious Affairs (MORA).1 While the education system under the MOEC is decentralized, MORA has maintained oversight of public and private madrasahs. Under the Law on National Education (No. 20, 2003), madrasahs were formally integrated into the national education system, and follow the national curriculum and education standards set by the government. Of approximately 235,000 primary and junior secondary education institutions, about 82% are schools and the remaining approximately 18% are madrasahs. About 90% of madrasahs are private and 10% are run by the state. In 2013, 87% of about 50 million students in the primary and junior secondary education system were enrolled in MOEC schools, while the remaining 13% were enrolled in madrasahs. Madrasahs mostly serve poor rural students and attract more female students. Thus, madrasah education plays a crucial role in the achievement of the government’s key education goal of universal 9-year basic education, as well as the Millennium Development Goals for education and gender equity. 2. The law requires madrasahs to deliver high-quality education services in line with the minimum service standards for basic education. However, the majority of madrasahs are less well-endowed than MOEC schools. This affects the quality of madrasah education, which is considered low. Many madrasahs, particularly the private ones, face challenges in meeting the minimum service standards for learning outcomes and the teaching and learning environment without additional investment and support. The Madrasah Education Development Project supported the government’s efforts to improve madrasah education to meet the national education standards. Over the long term, the project was expected to help raise levels of educational attainment among new entrants to the labor force, and increase formal employment rates and household incomes. The medium-term outcome of the project was improved quality, efficiency, and effectiveness of primary, junior secondary, and senior secondary madrasah education. The project had four outputs: (i) improved teacher professionalism according to national standards; (ii) upgraded teaching and learning resources and facilities; (iii) increased internal efficiency; and (iv) strengthened governance, management, and sustainability that meet national standards. The project design and monitoring framework is in Appendix 1. 3. The Asian Development Bank (ADB) financed the project through a loan with an original allocation of about $50.0 million.2 The project covered 500 primary, junior, and senior secondary private and public madrasahs in 27 districts in the provinces of Central Java, East Java, and South Sulawesi (Map). The list of participating districts and the selection criteria of madrasahs is in Appendix 2. The project aimed to raise education standards in the 500 project madrasahs to benefit about 120,000 poor students, including about 60,000 girls, who were enrolled. MORA was the executing agency for the project.

1 Three levels of madrasah education are formalized: madrasah ibtidaiyah, which is equivalent to elementary

school of the MOEC education system; madrasah tsanawiyah, which is equivalent to junior secondary school; and madrasah aliyah, which is equivalent to senior secondary school.

2 ADB. 2006. Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical Assistance Grant to the Republic of Indonesia for the Madrasah Education Development Project. Manila. The approved loan amount was SDR33.64 million. At project completion, the total loan disbursed was SDR32.33 million (equivalent to about $50.23 million).

2

II. EVALUATION OF DESIGN AND IMPLEMENTATION

A. Relevance of Design and Formulation

4. The project was formulated in the context of the National Medium-Term Development Plan, 2005–2009 (Rencana Pembangunan Jangka Menengah Nasional), which gave strategic directions to the government’s efforts to increase access to quality education. Improving the quality of and access to education was a key government priority, as outlined in medium-term development plans (2005–2009 and 2010–2014). 3 The project design was consistent with government strategies that recognize the importance of investments in education for sustaining inclusive growth and providing access to economic opportunities. The project included targeted support to 500 madrasahs in poor and remote areas to help address the increasing and urgent need to provide good education opportunities in poor communities. This approach is considered essential for tackling the widening regional inequalities in the provision of quality education. The project design was aligned with ADB’s country strategy and program, 2006–2009, which identified the following education outcomes: (i) improved expansion and equity; (ii) better education quality, relevance, and competitiveness; and (iii) enhanced governance, accountability, and public image. 5. Project preparatory technical assistance (TA) was provided to assist the government in formulating the project.4 A team of consultants conducted a comprehensive sector assessment. Surveys conducted at the start of the project preparation showed that only 45% of private madrasah buildings were in an adequate condition and only 10% of children had the required textbooks. As a result of the poor teaching and learning environments, only 70% of primary madrasah students progressed to junior secondary level; many students switched to general schools or dropped out. A critical issue for the madrasah education system is to improve student performance and, simultaneously, reduce the difference in performance between madrasahs and general schools. Generally, research has shown that madrasah graduates have difficulty competing with general school graduates of the same age cohort and gender at both the junior and senior secondary levels. The gap is widest among senior secondary graduates. The project scope addressed these issues. The project design was prepared in close consultation with the major project stakeholders including senior officials from the MOEC and MORA, madrasah principals and teachers, parents, and community members involved in private madrasahs. B. Project Outputs

1. Teacher Professionalism Improved According to National Standards 6. Under this output, by 2012, at least 60% of teachers in project madrasahs were to attain bachelor’s degree level (S1) qualifications, and at least 25% of teachers in project madrasahs were to achieve professional certifications. 5 To meet these targets, the project provided resources for (i) about 800 teachers and 200 principals to upgrade their qualifications to bachelor’s degree level, and (ii) about 1,200 teachers and principals to undertake professional 3 Government of Indonesia. 2009. Overview of the Indonesia’s Medium-Term Development Plan, 2004–2009.

Jakarta. The strategy outlines key policy priorities and direction of the government. Indonesia’s Constitution requires that at least 20% of the public expenditure per year be directed to education. Accordingly, education's share of public expenditure increased from about 12% (approximately Rp42 trillion) in 2001 to about 19% (approximately Rp209 trillion) in 2009. Since 2009, the government has met and surpassed the 20% target.

4 ADB. 2004. Technical Assistance to the Republic of Indonesia for the Madrasah Education Development Project. Manila. (TA 4547-INO, $1.2 million, with $600,000 cofinancing provided by the Government of Australia, approved on 23 December 2004).

5 The 2006 baseline data indicated that the proportion of teachers with S1 qualifications was about 50% and no project madrasah teacher had achieved professional certification.

3

certification programs. Additional short-term training focused primarily on subject content in the sciences, mathematics, and English; and on teaching methodologies. The project also supported the development of teacher learning groups for primary and junior secondary schools, which act as forums for teachers to help each other solve day-to-day teaching and learning problems. 7. The targets under the S1 and certification programs were not met. Of the 453 teachers that enrolled in the project-financed S1 upgrading program, 403 teachers, including 163 female teachers, had attained S1 qualifications by project completion. Feedback received from madrasah teachers suggested that family-related matters were the main reasons for the underachievement. The bachelor’s degree programs offered under the project were conducted at universities located far from the home villages of the teachers.6 Teachers were reluctant to stay away from home for long periods of time to complete their bachelor’s degrees.7 However, at project completion, more than 8,500 (80%) of about 10,430 teachers in the project madrasahs had attained the S1 qualification.8 In addition to the project-financed S1 program, teachers attended alternative S1 programs, financed from their own resources or other government scholarship programs. 9 The teacher certification program could not be fully completed due to a change in the government’s teacher certification regulations.10 Thus, at project completion 149 teachers, including 73 female teachers, had become qualified for certification under the project. At project completion, about 35% of project madrasah teachers had obtained professional certification as a result of the project and other government-financed programs. 8. Some of the unutilized funds originally earmarked for the S1 and certification programs were reallocated to provide more short-term training courses for teacher professional development and to provide additional block grants to madrasahs.11 At project completion, 5,279 teachers had participated in training on subject content, including sciences, mathematics, and English; and teaching methodologies. In addition, more than 9,500 teachers, of which 4,950 were women, received nonsubject-content training, which had not been included in the original project design.12 A survey carried out in 2012 confirmed that teachers greatly appreciated the training opportunities provided by the project. Interviewed teachers suggested that the nonsubject-content training contributed to improving their overall professionalism and teaching capability in the classroom.13 9. In addition to training organized by MORA, the project provided a series of self-help programs for teachers that were financed by project block grants and organized directly by the

6 To ensure quality, only reputable universities were selected to implement the project S1 program. Most of these

universities are located far from the project madrasahs and teachers’ residences. 7 Many teachers have other additional sources of income. Leaving their hometown meant that they would lose this

income. Other teachers, particularly women, cited that leaving their home villages meant that they could not attend to domestic matters, such as taking care of their young children and husbands.

8 When the S1 program started in late 2008, the proportion of project madrasah teachers with S1 qualifications was 76%. The project contributed to an increase of 4%.

9 By financing their S1 program themselves, teachers could choose universities close to their homes. 10 Under the previous certification program, teachers with S1 qualification would qualify for the certification after

completing 9 months of training, mainly in pedagogy.  However, under the revised regulations, the number of teachers to be granted teacher certification was restricted by a quota system and should be integrated with the government-financed certification program. The teachers should also complete certain criteria, e.g., teaching hours and period of service, in addition to having the requisite educational qualifications.

11 A change in project scope was not required during project implementation, as short-term training is also part of the human resources development category. These training were organized by MORA.

12 This included training on library management, science laboratory management, lesson study, and making simple practical equipment for science lessons. Some teachers participated in more than one training program.

13 MORA. 2012. Evaluation of Impact of MEDP Inputs toward Achieving the Project Goals. Jakarta.

4

project madrasahs through teacher working groups and subject teacher groups. Through the teacher groups, teachers from neighboring schools and madrasahs met regularly to share knowledge, and exchange information and experiences about subject contents, lesson planning, and teaching techniques. By project completion, more than 6,320 teacher group meetings had been conducted involving more than 65,200 teachers.14

2. Teaching and Learning Resources and Facilities Upgraded to National Standards

10. Under this output, at least (i) 70% of the project madrasahs were to be accredited by the end of 2008, (ii) 80% of project madrasahs were to have facilities that meet the national standards, and (iii) 80% of the project madrasahs were to have teaching and learning materials that meet the national standards. To meet the targets, the project provided an original allocation of about $38.5 million for block grants to be transferred directly to the 500 project madrasahs. Funds were to be used to rehabilitate and construct classrooms, libraries, laboratories, and sanitation facilities; procure equipment, teaching aids, textbooks, and furniture; and finance training for madrasah teachers, student remedial programs, and small scholarships for poor students. The project adopted a community-driven development approach. To determine investment needs to be financed by block grants, the management of project madrasahs, teachers, and community members jointly prepared individual madrasah development plans (MDPs). The MDPs were based on comprehensive self-assessments and included detailed investment plans with prioritized upgrading and training needs. Facilitators were hired to help madrasah teams prepare the MDPs and facilitate procurement, training, and civil works. 11. However, the guidelines for preparing the MDP and setting the ceilings for block grants available for each project madrasah were imprecise. This led to confusion, and resulted in the proposed MDP amounts exceeding the $38.5 million block grant allocation. Thus, the MDPs had to be revised to be within the ceiling. The revision was performed by the executing agency without consulting the participating madrasahs. This created frustration among the project madrasahs, because the proposed investment plans were reduced and/or altered without proper explanation. To address the complaints from the project madrasahs, the executing agency had to review the MDPs jointly with the concerned madrasahs, which delayed the finalizing of the plans. Starting from the third quarter of 2009, the first cycle of block grants was transferred to the 500 project madrasahs. In 2010, to accommodate the large demand from the project madrasahs to improve teaching and learning environments, the overall allocation for block grants was increased from the original $38.5 million to about $41.3 million.15 The second and third tranches of block grants were transferred in 2010 and 2011 after the madrasahs had submitted financial and technical reports on the utilization of the previous funds. In 2012, after all three tranches had been transferred, MORA provided additional block grants to 60 madrasahs that had performed well to utilize the unused loan proceeds. 12. At project completion, the 500 project madrasahs had used block grants to (i) construct 1,043 new classrooms and renovate 802 classrooms; (ii) construct 210 new school clinics and upgrade 23 school clinics; (iii) construct 272 new science laboratories and rehabilitate 28 science laboratories; (iv) establish 289 new computer laboratories and upgrade 33 laboratories; (v) construct 17 new language laboratories and renovate 1 language laboratory; (vi) construct 348 new libraries and renovate 72 libraries; (vii) construct 843 new toilets and rehabilitate 110 toilets; (viii) procure thousands of textbooks and teaching aids; and (ix) finance teacher training

14 The figures represent the number of teachers attending the meetings. Most teachers participated in more than one

meeting. 15 The increase was financed through reallocation of loan proceeds from the unallocated category.

5

programs. Furthermore, the project also provided 48,300 new items of furniture for clinics, science and computer laboratories, libraries, and classrooms. More than 150,000 students and more than 10,000 teachers benefited from new facilities and equipment and teacher training program. Details of the facilities provided under the project are in Appendix 3. 13. In 2012, the executing agency carried out a survey of the project madrasahs to evaluate the quality of the new and upgraded infrastructure and equipment. The results indicated that about 85% of the new or rehabilitated classrooms, libraries, clinics, and laboratories were well constructed and met national standards. 16 However, some sanitation and hand-washing facilities were substandards and thus needed further attention. The provision of teaching and learning resources and the upgrading of facilities through the MDPs had increased the number of madrasahs that were accredited. At the start of the project, in 2008, 374 project madrasahs (about 75% of the total) were accredited. 17 At project completion, in 2012, 446 project madrasahs (89%) were accredited. Detailed accreditation results are in Appendix 4. 14. The improved teaching and learning environments also contributed to an improvement in the academic performance of the project madrasah students in the national exams at all levels (grade 6 for madrasah ibtidaiyahs, 9 for madrasah tsanawiyahs, and 12 for madrasah aliyahs). At the madrasah ibtidaiyah level, the national exams average score increased by 0.69 point; at the madrasah tsanawiyah level by 0.55 point; and at the madrasah aliyah level by 0.37 points for students specializing in natural sciences, by 0.42 points for students specializing in social sciences, and by 0.61 points for students specializing in religion.18 Appendix 5 summarizes the achievement rates.19 Overall, the project achieved the key performance indicators under this output. About 89% of the project madrasahs are accredited, and 85% of the upgraded and new facilities as well as teaching and learning materials meet the national standards. These measures improved teaching and learning in classrooms and resulted in better academic performance of students as described above.

3. Internal Efficiency Increased to National Standards

15. The performance indicators under this output stipulated that by 2012: (i) madrasah tsanawiyah enrollment is increased by at least 5%; (ii) transition rates from madrasah ibtidaiyah to junior secondary education (madrasah tsanawiyah or junior high schools) is increased to 95% for both female and male students; (iii) repetition rates in project madrasah ibtidaiyahs and madrasah tsanawiyahs are reduced to 1%; (iv) dropout rates in madrasah ibtidaiyahs and madrasah tsanawiyahs are reduced to zero; and (v) 85% of project madrasahs achieve a ratio of year 1 to final year students of at least 85% for madrasah ibtidaiyahs, and 80% of madrasah tsanawiyahs and madrasah aliyahs. 16. At project completion, about $1.75 million in block grants were used to support remedial programs and scholarships. Students benefited from remedial coaching programs to improve their learning outcomes, and 10,265 poor students including 4,397 (43%) female students, received small scholarships to help them continue to the next level of education. The coaching programs, which were mainly conducted by the madrasah teachers before or after the regular

16 Standards for size, lighting, and provision of furniture are based on Ministry of National Education Regulation No.

24, 2007. 17 Level A accreditation is the highest. 18 Comparison between SY 2008/09 and SY 2011/12. The national exam scores are between 0.00 to 10. 19 For comparison, the academic performance of the students of madrasahs that were located near the project

madrasahs but did not receive project support (nonproject madrasahs) are also presented in Appendix 5. The data suggest the academic performance of students of nonproject madrasahs is lower than that of students of the project madrasahs.

6

school sessions, contributed to improving student achievement in the national exams. Although the overall performance of students in the national exams improved, it was noted that mathematics appeared to be a weak subject in all project madrasahs.20 Targeted teacher training programs are required to further improve teaching methods and subsequent learning and student achievements, particularly in mathematics. The project completion review mission also noted that the majority of teachers at project madrasahs lack the skills to foster critical thinking. Based on lessons learned under the project, the executing agency has developed training programs for teachers to improve their proficiency to promote the development of students’ analytical skills. The programs will be implemented as part of the new subject content and lesson-study training that is planned to commence in SY2013/14. 17. In SY2008/09, a total of 118,570 students (51% female and 49% male) were enrolled in the 500 project madrasahs. At project completion in 2012, the number of students had increased to 123,665 (the share of female to male students remained unchanged). Enrollment in madrasah tsanawiyahs increased by 6.1%, exceeding the project target of at least 5%. At project completion, the transition rates of students from madrasah ibtidaiyah to madrasah tsanawiyah increased to 99.7% for both female and male students, exceeding the project target of 95%. Although the overall repetition rate for madrasah ibtidaiyah students decreased from 1.78% at the beginning of the project to 1.46% at project completion, the target of reducing the repetition rate to 1% was missed. However, the repetition rate for female students decreased from 1.33% to 0.98%. Although dropout rates at project madrasahs were very low, the zero dropout target could not be met. At project completion, the dropout rate for madrasah ibtidaiyah students increased by 0.02% to 0.14%. As most project madrasahs are located in poor communities, the economic circumstances of low-income families were considered the main reasons for dropping out. This factor was beyond the project’s control. In general, dropout rates in nonproject madrasahs in neighboring communities were slightly higher. In 2012, the ratio of year 1 to final year students was 96.1% for madrasah ibtidaiyahs, 88.6% for madrasah tsanawiyahs, and 83.8% for madrasah aliyahs. The figures indicate that the target ratios of 85% for madrasah ibtidaiyahs, and 80% for madrasah tsanawiyahs and madrasah aliyahs were achieved. Details on student enrollment, transition, and repetition rates for project and nonproject madrasahs are in Appendix 6.

4. Governance, Management, and Sustainability of Project Madrasahs Strengthened to Meet National Standards

18. Under this output, the project supported a range of activities to (i) strengthen madrasah committees, (ii) improve madrasah-based management and accountability, (iii) design and conduct preventive maintenance programs, (iv) improve education sector management, (v) establish provincial madrasah quality assurance working groups to promote teaching and learning assessment and quality assurance system development, and (vi) promote advocacy and public information programs. The performance indicators were: (i) 80% of project madrasahs have result-based MDPs; (ii) at least 80% of the madrasah principals attain S1 qualifications by 2012; (iii) 80% of project madrasah principals achieve professional certification; (iv) 70% of project madrasah display annual income and expenditures on school notice boards; (v) 20% of madrasahs’ budgets are allocated for quality improvement; (vi) 95% of project madrasahs have functioning madrasah committees with at least 30% female members; (vii) there are at least three quality assurance specialists in each MDC; (viii) 100% of project districts increase funding for madrasahs. 20 Overall, in Indonesia the student performance in mathematics is weak (World Bank. (2010). Policy Brief, Inside

Indonesia’s Mathematics Classrooms: A Trend in International Mathematic and Science Study Video Study of Teaching Practices and Student Achievement. Ragatz, A. et al. Jakarta). Specific teaching techniques targeting at the weakness are needed to improve the learning process.

7

19. At project completion, all 500 project madrasahs had developed results-based MDPs. These were prepared based on comprehensive self-assessments, and included detailed investment plans with prioritized upgrading and training needs. The MDPs were the basis for the block grants transferred to each madrasah. Observations during review missions indicated that most madrasahs regularly update their MDPs, which they first prepared in SY2007/08. The introduction of results-based planning and adoption of a community-driven approach was considered a useful and efficient planning and management tool. A total of 441 madrasah principals (88%) have attained S1 qualifications and about 70% of the principals have obtained professional certifications. While the S1 programs were mostly financed by the principals themselves, the certification programs were financed under other government programs. 20. Observations indicated that project funds were administered transparently and efficiently. The vast majority of the project madrasahs displayed their annual income and expenditure reports publicly on school notice boards. All madrasahs had established functioning committees, and 1,134  (23%) of the 4,950 committee members were women. Most of the MDP budget (85%) was allocated for the upgrading of facilities and procurement of equipment, and the rest was spent on training and other quality-improvement measures. About 30% the district administrations were providing additional funding for madrasahs, mostly to upgrade and expand facilities. Although the target of 100% increased financing from districts was not met, the project’s advocacy programs had encouraged local governments to increase their financing for madrasahs.21 21. The project also included some actions to maximize the benefits to women and girls (teachers and students). The project gender action plan (GAP) included targets for female teacher participation in degree upgrading (bachelor’s degree), professional certification, short-term training activities, and improvement in teachers’ capacity to identify gender biases in textbooks and to eliminate gender biases and stereotyping in the teaching and learning process. At project completion, project had achieved or partially achieved 9 of the 11 GAP targets. About 40% (163 of 403) of degree upgrading (S1) participants were female teachers, exceeding the 30% target. Women made up 53% (4,955 of 9,433) of participants in training related to subject content, curriculum development, and teaching methodology. A total of 73 women teachers (49% of 149 teachers) participated in professional certification. Both achievements exceeded the 40% target. A total of 4,397 female students (43% of 10,265 students) received transition scholarships and about 183,420 (42% of 436,081) benefited from remedial coaching programs. All 500 madrasahs have well-functioning madrasah committees, but only 23% (1,134 of 4,950) of the committee members are women. In 473 madrasahs, separate toilets were constructed for female and male students. The detailed gender-related targets and achievements at project completion are in Appendix 7. C. Project Costs

22. At appraisal, the project cost was estimated at $71.4 million. ADB was to contribute 70% of the project costs through the provision of a SDR33,638,000 ($50.0 million) loan from the Asian Development Fund. At project completion, the total project cost was estimated at $71.2 million, which consisted of $50.2 million from the loan proceeds (71%), $17.9 million (25%) from government counterpart funds, and $3.1 million (4%) from the madrasahs. The project cost details and financing plans at appraisal and completion are in Appendix 8. 21 Because madrasahs are under the management of MORA, many local administrators believe that madrasah

financing is the central government’s responsibility. Thus, they prioritize their budgets to increase the financing of the general schools, which are under their responsibility. Other local governments indicated that limited budgets had prevented them from increasing their support to madrasahs.

8

D. Disbursements

23. The loan proceeds were disbursed according to ADB’s Loan Disbursement Handbook (2007, as amended from time to time). The project’s fund-channeling mechanism was straightforward with direct transfer from the project imprest account to the madrasah account. About $41.1 million (79% of the total loan proceeds allocation) were transferred to madrasah accounts for the financing of the madrasah block grants. In total, $50.2 million of the loan proceeds (96% of the total loan amount of $52.2 million) was disbursed to finance project investments. To facilitate disbursement of the loan proceeds, an imprest account was open at Bank Indonesia. The imprest account was established, managed, replenished, and liquidated in accordance with the provisions of ADB’s Loan Disbursement Handbook. Overall, MORA managed the imprest account satisfactorily, with no significant issues during project implementation, and the counterpart funds were sufficiently and provided in a timely manner. Appendix 9 illustrates the fund-channeling mechanism. E. Project Schedule

24. The loan was declared effective in June 2007. Due to the delays in reviewing the MDPs and procuring consultants, implementation at the project madrasahs started in late 2009, after the release of the first tranche of block grants. However, the executing agency made up for the delays and the three scheduled tranches of block grants were disbursed within the original implementation period. The government extended the loan period by 3 months so that the unutilized loan proceeds could be allocated as additional block grants to 60 madrasahs that had performed well. Thus, the closing date of the loan was revised from 30 September 2012 to 31 December 2012. Appendix 10 provides the original and actual implementation schedules. F. Implementation Arrangements

25. The implementation arrangements outlined in the report and recommendation of the President were generally followed, and were considered appropriate for efficient project implementation. No change in the implementation arrangements was required. The project implementation arrangements are in Appendix 11. G. Conditions and Covenants

26. Most of the loan covenants were fully complied with. Five covenants were partly complied with. Due to limited internet access, particularly in remote project madrasahs, comprehensive online project information and monitoring systems could not be established in all madrasahs. A project website was established but was not regularly updated. Thus, information such as the list of participating bidders, name of the winning bidder, basic details on bidding procedures adopted, amount of contract awarded, and the list of goods and services procured was mostly not provided. One loan covenant on the establishment of a liaison office for the National Board of Education Standards was cancelled due to a change in government regulations related to this. The partial compliances did not affect the overall project implementation. The loan covenants and their compliance status are in Appendix 12. H. Related Technical Assistance

27. The project included a technical assistance (TA) grant for $995,000 to support the executing agency in establishing an independent monitoring and evaluation system to observe the project’s progress and assess its benefits and impacts. In addition, the TA provided support to the executing agency to raise public confidence in the quality, efficiency, and cost-

9

effectiveness of madrasah education. Overall, the TA provided effective support to help the executing agency to monitor the project implementation and evaluate the initial impact of project interventions. The TA completion report is in Appendix 13. I. Consultant Recruitment and Procurement

28. Consultants were recruited according to ADB’s Guidelines on the Use of Consultants (2007, as amended from time to time). Originally, the project provided funding to engage a firm to provide a team of project implementation consultants (PICs) consisting of 60 person-months of international consultant inputs and 183 person-months of national consultant inputs to assist MORA in the overall project implementation. In addition, the project provided funding to support project management through individual consultants to support the central project management unit (CPMU), and the provincial coordinating units (PCUs) and district coordinating units (DCUs). To assist the 500 madrasahs, the project financed more than 100 facilitators, who worked in teams in the project districts. 29. The recruitment of the project implementation consulting firm was delayed for about 1 year. Organizational matters related to the delegation of project administration from the ADB headquarters to the Indonesia Resident Mission and insufficient understanding of ADB’s procedure for the recruitment of consultants on the part of executing agency staff contributed to the delays. About 55 person-months of international consultants and 120 person-months of national consultants were recruited under the consulting firm’s contract. Due to the poor performance of the consulting firm and of several experts contracted, MORA shortened the PIC contract period. After the PIC contract had expired, MORA recruited individual consultants to continue the support during the remaining project duration. MORA also recruited a consultant team in 2012 to evaluate the project’s impacts.22 30. In addition to the PICs, about 470 person-months of national consulting services were recruited to assist the CPMU in project management and carrying out the project impact evaluations and assessments. About 560 person-months of national consulting services were recruited to help the provincial and district coordinating units. In addition, about 1,960 person-months of consulting services for facilitators were recruited to assist the 500 project madrasahs. Facilitators were selected by the CPMU in close cooperation with PCUs. At the beginning, facilitators were recruited as individual consultants by the CPMU. However, due to difficulties in contract management, the CPMU changed the approach and engaged a firm to provide and manage the facilitator teams. The change proved to be very effective. J. Performance of Consultants, Contractors, and Suppliers

31. The performance of the consultants was rated partly satisfactory. Weak performance and coordination between the consultants and MORA, and poor advice from the consultants contributed to the delays in project start-up, particularly during the review and evaluation of the MDPs. The PIC team prepared project manuals and guidelines. However, feedback from madrasahs pointed out that the guidelines were imprecise and not useful, thus the guidelines had to be revised. Due to weak performance of the PIC firm and team, MORA shortened the contract period by about 1 year. In their place, the executing agency recruited individual

22 As specified in the terms of reference, the TA-financed independent monitoring and evaluation consultant focused

mostly on the establishment of appropriate monitoring and evaluation systems at the madrasah, local, and central levels. Surveys were carried out using randomly selected madrasahs to monitor the progress of the project and evaluate the initial project impact. A year before the project closure, it was noted that additional impact evaluations involving all project madrasahs were required to complement work of the TA-financed consultant. A consultant team financed by the loan proceeds was recruited to carry out the work.

10

consultants to support the project management. The performance of the individual consultants in supporting the CPMU was considered satisfactory. Overall, the support of the facilitators to assist madrasahs in the formulation of the MDPs, procurement of equipment, construction, organizing training measures, and recording of work progress and expenditures was satisfactory. Poorly performing facilitators were replaced during implementation. The performance of consultants to support PCUs and DCUs varied widely from highly satisfactory to unsatisfactory. No contractors were recruited under the project. K. Performance of the Borrower and the Executing Agency

32. The performance of the government (borrower) and the executing agency was rated satisfactory. Both parties demonstrated strong commitment to the success of the project. The executing agency and borrower made efforts to ensure that the counterpart funds provided were sufficiently and timely. To support project implementation, MORA appointed by decree national, provincial, and district project coordinating units. At project start-up, the project management was considered weak, which caused delays in the finalization of the MDPs. After several changes in CPMU personnel and by appointing qualified personnel as project managers, the executing agency’s project management improved and it was able to complete all project activities within the original project schedule. Counterpart funds were provided on time. L. Performance of the Asian Development Bank

33. The performance of ADB was rated satisfactory. ADB provided regular guidance to the project during implementation. During the 6-year implementation period, ADB fielded 11 project administration missions for a total of about 210 staff person-days, or about 35 staff person-days per year. The mission teams mainly comprised project implementation specialists and project officers. The frequency and number of staff-person days per mission appeared sufficient for effective supervision. After 1 year of implementation, project administration was delegated to the Indonesia Resident Mission. This move was highly appreciated by the government and improved the overall administration of the project. Resident mission staff helped the executing agency expedite the transfer of the first block grant to the madrasahs by providing guidance and advice to speed up the review and approval of MDPs by the CPMU. ADB also provided support on all aspects of project management, including management of consultants and facilitators. ADB encouraged the change in contract method of the facilitators from individual to firm contracts. The change proved to be very effective in addressing issues previously faced.

III. EVALUATION OF PERFORMANCE A. Relevance 34. The project was rated relevant at appraisal, and remained so throughout the implementation period. The project design was fully aligned with the national development priorities and directly responsive to the need to improve the provision of basic education services for remote and poor communities. The project design complemented the government’s medium-term development plans (2005–2009 and 2010–2014), which focused on addressing disparities in the education system by strengthening the madrasah education system, regarded as weaker than the public schools administered by the Ministry of Education and Culture (MOEC). Project design was based on the results of a comprehensive subsector assessment. The project’s main components, including upgrading facilities, training teachers, and adopting a community-driven development approach, are in line with government strategies to strengthen the quality of education services. Supported by other development partners (in particular the Australian Agency for International Development), the government will continue to use its own

11

resources to further strengthen madrasah education, especially by improving continuous professional development schemes for teachers and upgrading madrasah facilities to improve the overall teaching and learning environments. Project design was consistent with ADB’s country strategy, which highlighted quality basic education as one of its key priorities in promoting pro-poor growth. B. Effectiveness in Achieving Outcome

35. The project is rated effective in achieving its expected outcome. The intended outcome of the project was improved quality, efficiency, and effectiveness of primary, junior secondary, and senior secondary madrasah education. The outcome indicators refer to (i) improvements in student performance at all levels, including reducing the gap in quality between the madrasah and general education systems; (ii) reduction of district disparities in madrasah student performance, with significant improvements in the project madrasahs; (iii) improvements in progression and retention rates within the madrasah system; and (iv) enhancement of community perceptions on madrasah education quality. Performance targets set in the DMF are slightly different from the indicators specified in the report and recommendation of the President and are more focused on the achievements at the project madrasahs. The targets include (i) improvement in the academic performance of grades 6 and 9 in standardized tests conducted in SY2007/08 and SY2011/12 for core subject areas; (ii) improvement in accreditation of 70% of project madrasahs by at least one level by 2012; and (iii) more positive community perceptions. 36. The project has substantially improved teaching and learning environments at the project madrasahs and has subsequently contributed to improvements in student performance at all madrasahs supported by the project. The performance targets stipulated in the DMF were met. Academic achievement of students from project madrasahs during the national exams and the accreditation level of project madrasahs have been improved (paras. 12–13). Communities’ perception of project madrasahs has improved as well, as demonstrated by the increased enrollment rates and positive feedback from parents who appreciated the better teaching and learning environments at the project madrasahs. 37. However, the achievement of the broader development objective of improving the quality, efficiency, and effectiveness of the madrasah education system and reducing the gap in quality between the madrasah and the general education systems was not fully realized. Project accomplishments are principally visible at the 500 project madrasahs. Fresh strategies are required to improve the coordination between district MORA offices and district MOEC administrations in order to harmonize education planning at the district level and create synergies among schools and madrasahs. MORA has understood the limited impact of the project on the wider system, and is making efforts to apply the lessons learned from the project to the entire madrasah system, particularly (i) the adoption of a results-based planning approach at the madrasahs; (ii) the involvement of community members in planning and decision making; and (iii) improvement in access to and quality of teacher training opportunities, particularly by fostering teacher working groups and subject teacher groups, and strengthening teaching methods. C. Efficiency in Achieving Outcome and Outputs

38. The project is rated efficient. The project was completed on time with a minor extension of 3 months to distribute additional block grants to madrasahs that had performed well. No additional resources were required to complete the work. Overall, MORA managed the project well. Key performance indicators stipulated in the project concept were met. The project resources were used efficiently. In total, about $50.2 million of the loan proceeds (96% of the

12

total loan amount of $52.2 million) were disbursed to finance project investments. Most of the investment (about $41.1 million) was used to upgrade project madrasahs. More than 150,000 students, mainly from poor families, and more than 10,000 teachers benefited from the project. Most of the project madrasahs were located in poor and remote areas, and were managed by private foundations. Most madrasahs had not received any major support previously and depended on contributions from the communities, which were inadequate to build up environments conducive to teaching and learning. Thus, the project made a major impact on improving the teaching and learning environments of the project madrasahs, which would otherwise not have had the resources to upgrade the quality of their education services. 39. Disadvantaged students from poor families who had no access to quality education before the project particularly benefited from project interventions, especially those related to (i) improving teacher professionalism; (ii) upgrading teaching and learning resources and facilities, including establishing new science and language laboratories and computer rooms; (iii) improving accreditation levels of madrasahs; and (iv) strengthening governance and management. These interventions contributed to better academic achievements for about 150,000 students, providing the foundation for better income-earning and employment opportunities and opening prospects for further advanced education. Observations during review missions and feedback from monitoring reports confirmed that the madrasah students and parents had a positive attitude toward the project and its impact on learning results.

40. Students and parents took pride in the improved condition of their madrasahs, which are generally perceived to be better and more complete than surrounding schools or other madrasahs. Students were more confident of studying in madrasahs and no longer have a sense of inferiority. With better facilities and infrastructure, students were able to carry out more activities, found it easier to complete their academic tasks, and had become more motivated to study. Students and parents also perceived that their teachers performed better after receiving the additional training facilitated by the project. Most of the teachers now use learning media, including multimedia (laptops with liquid crystal displays), which students considered were helpful for their learning and encouraged them to improve their performance. Teachers expressed their appreciation for the training opportunities, which improved their teaching capability in the classroom and content knowledge in the key subjects of the national exams. The economic and financial reevaluations suggested that the project is considered to be financially sustainable, as the bulk of expenditures do not continue beyond the end of the project. The economic and financial reevaluations are in Appendix 14. D. Preliminary Assessment of Sustainability

41. The project is rated likely sustainable. The government is strongly committed to further strengthening the basic education quality, including madrasah education, and considers investments in education essential for national development and inclusive economic growth. The project’s main components of upgrading of physical facilities, teacher development, and community participation are mainstream activities of the government. Education, including madrasah education, will continue to receive 20% of total public expenditure, as required by Indonesia’s Constitution. It is anticipated that the newly upgraded facilities under the project will be sustained because ownership by the communities surrounding madrasahs is generally high. 42. Although most project madrasahs’ achievements are encouraging, field observations provided limited evidence of uniform changes in teacher competencies and understanding of improved pedagogy for effective teaching throughout the madrasah system. Continued support for teachers will be required. Some teachers, particularly those in remote and isolated madrasahs, need more support to use the newly acquired inputs and/or training effectively.

13

There were also few visible signs of improved teaching and learning methods at some madrasahs. Furthermore, field observations suggested that some facilities, such as libraries, laboratories, and clinics, were not used and developed to the greatest benefit. They were equipped with ad hoc additions of learning materials, books, and laboratory equipment. Some books purchased for libraries were rarely used by students. More support is required to develop plans and strategies on how the new equipment and teaching aids can be integrated into the teaching process (particularly teaching aids for biology, chemistry, and physics). Teachers will need additional support to improve their understanding and knowledge of the use of sophisticated teaching and learning aids, as well as books. E. Impact

43. At this stage, it is too early to assess the long-term impact of the project. Most of the project madrasahs are located in poor communities. Most students enrolled in these madrasahs come from low-income families, who had limited or no access to quality education before the project intervention. Improving teaching and learning environments for disadvantaged communities resulted in improved academic achievements of students enrolled in project madrasahs, which will contribute to improved opportunities for madrasah graduates to progress to higher levels of education and/or find better employment and income-earning opportunities. Parents and teachers felt that with the new facilities, and science and information technology laboratories, children would be more prepared for admission to higher education, thereby contributing to a higher transition rate. Another early indication of impact is the increased participation and cohesiveness of community members around the madrasahs. The new facilities, laboratory equipment, and training aids, which are a novelty in remote areas, have generated a lot of enthusiasm, particularly among parents and members of madrasah committees. They seem to be more actively involved in the affairs of their children’s madrasahs. The positive community perception is also demonstrated by higher enrollment rates in madrasahs. 44. Community participation in the development of MDPs and the subsequent implementation of upgrading activities, and control over the resources by the madrasah committees contributed to systematically mobilizing the communities and committee members to prioritize their development needs and improve overall planning and management. Transparent planning, procurement of goods and services, management and disbursement of funds, and implementation of activities based on jointly agreed procedures contributed to the establishment of good governance practices. While the project has contributed to a substantial improvement in the quality of education at the 500 project madrasahs, the expectation that the project will improve the overall madrasah education system and reduce the gap in quality between the madrasah and the general education systems appears to be overambitious in retrospect. The capacity of the executing agency was fully absorbed in implementing the activities at the 500 project madrasahs in the 27 districts; therefore the resources and time frame were insufficient to address the quality gap between the MOEC and MORA education systems and overall district disparities in student performance. 45. The project increased women’s membership in the madrasah committees and thus improved their capacity to influence decision making, including on how project resources were spent. The gender action plan (GAP) was integral to the project, given that women and girls were intended to be significant beneficiaries, and contributed to the achievement of the overall project outcome. Through the implementation of the GAP, the project made it more likely that

14

madrasahs would provide a suitable learning environment for both boys and girls. Based on the achievements, the project was rated successful in terms of gender equality results.23

IV. OVERALL ASSESSMENT AND RECOMMENDATIONS A. Overall Assessment 46. The project is rated successful on the basis of a review of its relevance, effectiveness, efficiency, sustainability, and poverty and gender impact. The project was implemented as originally planned. Generally, the design was considered straightforward and relevant throughout implementation. Key performance indicators under the four outputs were largely achieved. Adopting a community-driven development approach for channeling project funds directly to madrasahs based on jointly developed MDPs was considered a successful and efficient implementation strategy. B. Lessons

47. Ensure government ownership and good management. Government ownership and good management are essential. MORA demonstrated strong ownership of the project by making sufficient financial, human, and technical resources available in a timely manner. Good coordination and communication among all project stakeholders, i.e., ADB, government agencies, madrasahs, consultants, and communities, are key factors in the success of the project. Creating positive synergies and a harmonious working environment among key stakeholders throughout the life of the project is essential to successful project implementation. 48. Promote a community-driven development approach. The success of the MDP-based improvements in each participating madrasah suggests that a community-driven development approach could provide a model for the development of all madrasahs. The requirement to jointly formulate an MDP in each project madrasah created an environment for participatory planning at the level of the individual madrasah for the formulation of both long- and short-term plans, and became the starting point of a development process that had the full support of the wider community. Even though project madrasahs were funded entirely through block grants provided from project funds, the MDP-based approach to madrasah strengthening ensured community ownership through the process of MDP formulation. As a result, community participation remained high. This was demonstrated by community cash and in-kind contributions totaling about $3 million to support the provision of teaching and learning facilities. 49. Systematically involve district administrations. A systematic approach toward involving the district administration and MOEC education offices is required to realize wider development impact. It would be beneficial to harmonize madrasah development with MOEC district education plans and integrate MDPs into the annual district development plans. Greater capacity-building of district-level MORA staff will be essential for sustaining quality improvements in the madrasah education system. Greater collaboration with the MOEC is also essential to ensure more efficient use of government resources. 50. Support institutional development beyond training. The project was also intended to build the institutional and professional capacity of the madrasah education management system. To that end, the project provided funds for MORA officials to take doctoral and master’s degrees overseas. However, institutional capacity building requires more that training. It also requires

23 The assessment is based on criteria specified in the updated corporate results framework and new Gender

Equality and Women's Empowerment Operational Plan (2013).

15

streamlining of the organizational structure, job functions, professional skills, resources allocation, and accountability system. Further efforts are needed to create and implement a capacity-development plan that will support the institutional capacities of MORA at all levels, starting from the central level. C. Recommendations

1. Project Related 51. Improve education management information system. More work is needed to improve the quality of district education statistics so that they include information on MOEC schools and madrasahs. There is also a need to foster a learning culture that uses the data in planning and decision making. Local officials must be committed and well trained. It is essential that madrasah teachers continue to share the knowledge and experience they gained from training programs with other teachers who may have not had the training opportunity. This will require support from the respective madrasah and MORA district offices in knowledge management and professional teacher development. 52. Ensure continuous monitoring and evaluation. Continuous monitoring and evaluation that feeds back into project implementation is critical to ensure the achievement of the expected results. Projects should develop a comprehensive monitoring and evaluation framework at the start, and conduct baseline studies in the first year. 53. Timing of the project performance evaluation report. It is recommended that project performance evaluation is carried out 2 years after the closure of the project.

2. General

54. Private madrasahs cannot rely on community funding alone if they are to achieve minimum service standards and eventually national education standards. Private madrasahs make a large contribution to student enrollments and to the overall performance of the education sector; therefore additional government funds are required to enable private madrasahs to improve the quality of the education they offer. Modifying the current legal framework and funding mechanisms under the decentralized system will allow better coordination and management of basic education at the district level. 55. With decentralization, local governments need to pay significantly more attention to the quality of madrasahs, including teacher placement, training, and professional development, particularly for madrasahs located in remote areas. In addition, while district offices manage and oversee madrasah education, the quality of their oversight and management continues to be variable and lacks a comprehensive approach to education planning in the district as whole. More capacity building of district-level MORA staff will be essential for sustained quality improvements in the madrasah education system. Greater collaboration with the MOEC is also necessary to ensure more efficient use of government resources.

16 Appendix 1

DESIGN AND MONITORING FRAMEWORK

Design Summary

Performance Indicators/

Targets

Data Sources and Monitoring

Mechanisms Assumptions

and Risks

Accomplishment Impact: In partnership with government and other donors, contribute to: • Increased level

of educational attainment and productivity among new entrants in the labor force

• Increased rates of formal employment and household incomes

Proportion of workforce with basic education qualifications is increased to 60% by 2020. Numbers of junior secondary graduates progressing to senior secondary education are increased to 80% by 2020. There is growth in real wages among new entrants in the labor force.

Population census, socioeconomic surveys (susenas) and tracer studies Labor force surveys

Assumptions Overall labor market profile and workforce skill demand profiles are sustained. Investment in the economy increases. Government commitment to harmonizing policies and financing of education within Ministry of National Education and Ministry of Religious Affairs (MORA) is sustained. Risks Attitudes toward madrasah graduates are slow to change. Fiscal outlook and projected education resource increases are not sustainable.

Data from the susenas indicated that the proportion of workforce with basic education (up to junior high school) is 69.4%. In 2007, when the project started, the proportion was 74.9%. Assuming that the current government policies on education are not changed, it is expected that the target to increase the proportion of workforce with basic education qualifications to 60% by 2020, can be achieved. In 2012, the number of project madrasah tsanawiyah graduates who continued to senior secondary education was 87.4%, an increase from 86.6% in 2008. In 2008, the average minimum wage for new entrants were Rp745,709. In 2012, the average minimum wage for new entrants were Rp1,008,093.

Outcome: Improved quality of madrasah education at primary (madrasah ibtidaiyah), junior secondary (madrasah tsanawiyah), and senior secondary (madrasah aliyah) levels

Academic performance improved on standardized tests for grade 6, and for grade 9 in core subject areas conducted in SY2007/08 and SY2011/12. At least 70% of project madrasahs have improved their accreditation by one level by 2012.

Annual National Examination scores and standardized test scores (e.g., PISA, TIMMS) NBSMA results Baseline and final evaluation studies

Assumption Madrasahs have equitable access to accreditation services. Risks Higher academic standards lead to lower student performance in examinations over the short term.

Academic performance on the national exams of the project madrasah students improved at all levels (madrasah ibtidaiyah, madrasah tsanawiyah and madrasah aliyahs). About 69% of project madrasahs are expected to improve their accreditation level by one level (at the time

Appendix 1 17

Design Summary

Performance Indicators/

Targets

Data Sources and Monitoring

Mechanisms Assumptions

and Risks

Accomplishment Community perceptions of madrasah education quality are more positive.

Teachers lack incentives to improve teaching quality and learning outcomes.

this project completion report is completed, the assessments by the national board of school-madrasah accreditation were not completed). Surveys carried out by the executing agency suggested that community perceptions of madrasah education are more positive.

Outputs 1. Teacher

professionalism improved in project madrasahs in accordance with national standards

2. Increased

availability of teaching and learning resources and facilities in project madrasahs to meet or exceed national standards

3. Internal

efficiency of project madrasahs increased to meet national

At least 60% of teachers in project madrasahs attain S1 qualifications by the end of 2012 (baseline 50%).

At least 25% of teachers in project madrasahs achieve professional certification (baseline 0%).

At least 70% of project madrasahs are accredited by the end of 2008. At least 80% of project madrasahs have facilities (classrooms, libraries, laboratories) that meet national standards. At least 80% of project madrasahs have teaching and learning materials in general subjects that meet national standards. Madrasah tsanawiyah enrollments increase by 5%. Transition rates from primary madrasahs to

Annual MORA EMIS reports NBSMA results Records on teacher certification Progress reports on MDP implementation MORA EMIS Baseline and final evaluation studies Implementation progress reports on MDPs MDP implementation progress reports Minutes of madrasah

Assumption Sufficient numbers of teachers will be eligible and willing to attend the degree upgrading program. Assumption Central and local governments provide adequate resources to enable madrasahs to meet minimum accreditation standards. Risk Government and communities are unable to fully sustain recurrent cost of project schools. Assumption Financial incentives are effective in attracting and retaining poor

More than 8,500 teachers (80%) of 10,430 teachers in the project madrasahs, (80%) had attained the S1 qualifications. About 35% of project madrasah teachers have obtained professional certification. 54 of 500 project madrasahs (11%) have not been accredited. Thus 89% of project madrasahs have been accredited. About 85% of facilities built under the project meet the national standards. About 85% of project madrasahs have teaching and learning materials in general subjects that meet national standards. Madrasah tsanawiyah enrollments increased by 6.14%. The transition rates are 99.6% for male students

18 Appendix 1

Design Summary

Performance Indicators/

Targets

Data Sources and Monitoring

Mechanisms Assumptions

and Risks

Accomplishment standards

4. Governance,

management, and sustainability of project madrasahs strengthened to meet national standards

junior secondary education (madrasah tsanawiyah or general junior secondary school) increase to 95% for both males and females. Repetition rates in project madrasah ibtidaiyah and MT are reduced to 1% by 2012. Dropout rates in project madrasah ibtidaiyah and madrasah tsanawiyahs are reduced to zero by 2012. 85% of project schools achieve ratio of Year 1 to final year students of at least 85% for madrasah ibtidaiyah, and 80% for madrasah tsanawiyahs and madrasah aliyah, by 2012. 80% of project madrasahs have results-based MDPs. At least 80% principals attain S1 qualifications by the end of 2012 (baseline 51%). At least 80% of principals in project madrasahs achieve professional certification (baseline 0%). 70% of project madrasahs display annual income and expenditure on school notice boards. 20% of school budget is allocated for quality

committee meetings Monitoring and evaluation reports of consultants and central project management unit (CPMU) Madrasah, district, provincial, and national budget and expenditure reports

students in madrasahs. Risk Financial investments for increasing student retention and transition, and reducing dropouts lead to greater effectiveness but lower efficiency. Assumptions MDCs will become quality assurance accredited. Local communities value transparency in financing and management of madrasahs . Local governments have sufficient resources to allocate to madrasahs. Risk Local governments do not allocate funds to madrasah in the absence of a regulation to do so.

and 99.8% for female students. In 2012, the repetition rates in project madrasah ibtidaiyahs was 1.46% and madrasah tsanawiyahs was 0.46%/ In 2012, the dropout rates in project madrasah ibtidaiyahs was 0.14%, and in Madrasah tsanawiyahs was 0.71%. In all project madrasahs, in 2012 the students ratio from year 1 to year 6 (final year) for madrasah ibtidaiyahs was 96.1%, and 88.6% for madrasah tsanawiyahs; and 83.8% for madrasah aliyahs. All project madrasahs (100%) have result-based MDPs. 88% of madrasah principals have attained S1 qualifications, About 70% of the principals have obtained professional certifications. The vast majority of the project madrasahs displayed their annual income and expenditure publicly at school notice boards. The majority of the MDP budgets (85%) was

Appendix 1 19

Design Summary

Performance Indicators/

Targets

Data Sources and Monitoring

Mechanisms Assumptions

and Risks

Accomplishment improvement. 95% of madrasahs have functioning madrasah committees with at least 30% female members. Committees meet once every month. At least three quality assurance specialists are certified in each MDC. 100% of project districts increase funding for madrasahs.

allocated for the upgrading facilities and procurement of equipment; about and 15% was allocated for training and other quality improvement measures. All madrasahs had established functioning madrasah committees. (1,134 of 4.950 committee members (23%) were women). This target was not met. The MDC in the project provinces were inactive due to insufficient government’s support. About 30% of the district administration provided additional or increase funding for madrasahs.

Activities with Milestones 1. Teacher professionalism improved.

1.1. MDCs and assessment teams undertake accreditation of all project madrasahs (baseline and end of project).

Baseline: By end 2008. Complete: 2012. 1.2 About 1,000 teachers and principals trained for bachelors (S1)

degree, 30% female. Commence: 2007. Complete: 2012.

1.3 1,200 teachers and principals achieve professional certification, 40% female. Commence: 2007. Complete: 2012.

1.4 5,300 teachers and principals, especially women, undergo content upgrading and methodology training, including preparation for implementing the new

competency-based curriculum. Commence: 2007. Complete: 2012.

2. Teaching−learning resources and facilities upgraded.

2.1 Provide block grants to about 500 madrasahs to build/renovate and equip classrooms, laboratories, and libraries. Commence: 2007. Complete: 2012.

2.2 Provide block grants to about 500 madrasahs to purchase teaching and learning materials for students and teachers.

Commence: 2007. Complete: 2012. 2.3 Provide block grants to teacher learning groups for training for

new curriculum and new education standards

Input (at appraisal) Furniture, equipment, and vehicles: $1.26 million Human Resources Development: $22.11 million Consulting services: $2.06 million • Foreign: 60

person-months • Local: 183 person-

months Madrasah block grants: $36.98 million Surveys and testing: $0.34 million Project management

Accomplishment (at completion)

Furniture, equipment, and vehicles: $0.86 million Human Resources Development: $18.54 million Consulting services: $2.08 million • Foreign: 55

person-months • Local: 120 person-

months Madrasah block grants: $44.2 million, (including community contributions) Surveys and testing: $0.31 million Project management

20 Appendix 1

Commence: 2007 Complete: 2012

3. Internal efficiency increased.

3.1 Provide scholarships to 4,000 madrasah ibtidaiyah and MT graduates to improve transition rates from madrasah ibtidaiyah to madrasah tsanawiyahs, and madrasah tsanawiyahs to madrasah aliyah. Commence: Repeat: annually until 2012.

3.2 Provide block grants to about 350 madrasahs to establish remedial programs to reduce repetition. Commence: 2007. Repeat: annually until 2012.

4. Governance, management, and sustainability strengthened.

4.1 200 principals, especially women, trained for S1 degree. Commence: 2007. Complete: 2012.

4.2 220 principals, especially women, achieve professional certification.

4.3 Establish madrasah committees, advocate membership of women in madrasah committee s, and conduct regular meetings to discuss MDP and use of block grants. Commence: 2007. Complete: 2012.

4.4 Train key madrasah personnel, madrasah committee s, and madrasah owner-operators (yayasan) in madrasah-based management. Commence: 2007. Complete: 2010

4.5 Develop performance- and results-based MDPs in consultation with yayasan, madrasah committee s, and other key stakeholders.

Commence: 2007. Complete: 2012. 4.6 Train principals on new national standards, requirements for

accreditation, and financial management 4.7 Conduct on-the-job training of provincial and district staff, and

supervisors in madrasah-based management, performance–based planning, budgeting, financial management, district/supervisor support to madrasah for quality improvement, and monitoring and evaluation. Commence: 2008. Complete: 2012.

4.8 Upgrade MDCs in quality assurance, and train at least three people per MDC as certified madrasah accreditation assessors, along with supervisors and selected principals. Commence: 2007. Complete: 2009.

4.9 Advocate with district mayors for nomination of proponents of madrasah education to district education councils. Commence: 2008. Complete: 2012.

4.10 Publicize positive developments in the madrasah system Commence: 2008. Complete: 2012.

4.11 Establish joint education planning between MORA district office and the district education office (dinas) in all districts.

Commence: 2008. Complete: 2012.

support: $2.19 million Incremental recurrent cost: $1.54 million

support: $2.42 million Incremental recurrent cost: $1.21 million

CPMU = central project management unit, EMIS = education management information system, MDC = madrasah development center, MDP = madrasah development plan, MORA = Ministry of Religious Affairs, NBSMA = National Board of School–Madrasah Accreditation, PISA = Program for International Student Assessment, S1 = strata 1 (undergraduate) degree, TIMMS = Trends in International Mathematics and Science Study.

Appendix 2 21

PARTICIPATING DISTRICTS

Table A2.1: Participating Districts in East Java Province

Number of Madrasahs No District MI MT MA Total

1. Bangkalan 9 7 2 18 2. Bojonegoro 10 13 2 25 3. Jember 7 8 2 17 4. Jombang 11 13 2 26 5. Kediri 7 8 2 17 6. Lamongan 16 14 4 34 7. Malang 14 15 4 33 8. Nganjuk 12 14 3 29 9. Ngawi 7 8 2 17 10. Pacitan 6 9 2 17 11. Ponorogo 7 8 2 17 12. Trenggalek 7 8 2 17

Total 113 125 29 267 MA = madrasah aliyah, MI = madrasah ibtidaiyah, MT = madrasah tsanawiyah. Source: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta.  

Table A2.1: Participating Districts in Central Java Province Number of Madrasahs

No District MI MT MA Total 1. Banjarnegara 7 8 2 17 2. Batang 7 8 2 17 3. Blora 7 8 2 17 4. Cilacap 7 8 2 17 5. Demak 7 8 2 17 6. Grobogan 7 8 2 17 7. Pemalang 7 8 2 17 8. Rembang 7 8 2 17 9. Tegal 7 8 2 17 10. Wonosobo 9 7 1 17

Total 72 79 19 170 MA = madrasah aliyah, MI = madrasah ibtidaiyah, MT = madrasah tsanawiyah. Source: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta.

Table A2.3: Participating Districts in South Sulawesi Province

Number of Madrasahs No District MI MT MA Total

1. Banjarnegara 3 6 2 11 2. Batang 7 8 2 17 3. Blora 3 6 2 11 4. Cilacap 4 6 3 13 5. Demak 4 6 1 11 Total 21 32 10 63

MA = madrasah aliyah, MI = madrasah ibtidaiyah, MT = madrasah tsanawiyah. Source: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta.

22 Appendix 2

Criteria for Selection of Project Madrasahs A project madrasahs were selected based on the flowing criteria:

The madrasah must have

1. stable student enrollment or improved enrollment during the last several years; 2. a commitment to excellent performance on the basis of the quality of graduates who

take national examinations and an evaluation of the managerial capacity of the madrasah and the learning process;

3. a satisfactory level of good health, as seen by its climate and environment that are conducive to learning and useful to the students;

4. ownership of the land it occupies (government-owned land in case of national or state madrasahs, and ownership by a foundation in the case of private madrasahs) as proven by a certificate of ownership or other letters of ownership duly authenticated by authorized officials;

5. a commitment and high level of attention and priority to the learning process as a way to improve the quality of education;

6. a commitment to develop a management system that is transparent and accountable and shows support and is able to generate various support from the community and local government in terms of creative and innovative ideas through measures aimed at improving the quality of student learning;

7. a plan to submit for improvement of the madrasah and investment is rational, satisfactory, effective, and efficient and contains innovative programs to improve the quality of educational services as an integral part of the holistic madrasah development plan, as covered in the proposal submitted.

Appendix 3 23

OUTPUT 2: TEACHING AND LEARNING RESOURCES AND FACILITIES

UPGRADED TO NATIONAL STANDARDS

Table A3.1: Textbooks and Teaching Aids Provided

No. Type of Facilities Units

1. Textbooks 568,308

2. Aids for madrasah medical clinics (health kits) 2,910

3. Teaching aids for science laboratories 25,127

4. Teaching aids for computer laboratories 8,742

5. Teaching aids for language laboratories 1,415

6. Equipment for library 76,532

7. Teaching aids for learning media 10,158

Source: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta.

Table A3.2: Built and Rehabilitated Facilities

No. Type New

Construction Renovation/

Rehabilitation Total

1. Classrooms 1,043 802 1,845

2. Madrasah medical clinics 210 23 233

3. Science laboratories 272 28 300

4. Language laboratories 17 1 18

5. Computer laboratories 289 33 322

6. Libraries 348 72 420

7. Toilets 843 110 953

Source: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta.

24 Appendix 3

Table A3.3: Purchased Furniture

No. Type of Furniture Total 1 For classroom 26,434

2 For library 8,258

3 For science laboratory 5,257

4 For computer laboratory 5,873

5. For language laboratory 813

6 For madrasah medical clinics 1,669

  Total 48,304 Source: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta.

1. In 2012, the executing agency carried out a survey involving the 500 project madrasahs to evaluate the infrastructure and equipment provided in relation to national standards. Results of the surveys indicated that most of (about 85%) classrooms built or rehabilitated met the national standards in terms of size, lighting, and provision of furniture (e.g., student chairs and desks). However, most of these classrooms did not meet the standards for the provision of other facilities such as hand-washing facilities and display tables: 75% of the classrooms did not have hand-washing facilities and 45% were not equipped with facilities to display students’ work. Most of the libraries (about 90%) met the standards for size, lighting, and location; but only about 65% meet the standards for provision of books (textbooks and other enrichment books) and furniture.

2. All the science laboratories provided under the project met the national standards for size, lighting, furniture, and science teaching aids; but only 70% met the standards for the provision of sinks and collection of garbage. All the computer and language laboratories met the national standards for size, lighting, furniture, and provision of related teaching aids. All the school clinics met the standards for size, lighting, and furniture. All toilets met the standards for size and availability of water. However, more than 60% of the toilets were not equipped with garbage bins. About 85% of toilets in primary madrasahs (madrasah ibtidaiyahs) were separate (for female and male students); and in junior secondary madrasah (madrasah tsanawiyahs) and senior secondary madrasahs (madrasah aliyahs) all toilets were separate. However, most of the teachers’ toilets were not separate (70% at madrasah ibtidaiyahs, 55% at madrasah tsanawiyahs, and 35% at madrasah aliyahs).

Appendix 4 25

RESULTS OF ACCREDITATIONS

Table A4: Detailed Results of Accreditation

Number of Madrasahs

with Accreditation Level Not Yet School Year A B C Total Accredited Total

2008/09 65 262 47 374 126 500 2011/12 129 278 39 446 54 500

Note: A is the highest level of accreditation. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

1. The number of madrasahs that have no accreditation decreased from 126 (25% of total project madrasahs) to 54 (11%) (Table A4). The accreditation levels of 76 of the 446 accredited madrasahs are being reviewed by the National Board of School–Madrasah Accreditation (NBSMA), as the validity of their accreditation has expired. About half of these madrasahs have been reassessed by the NBSMA to determine their latest accreditation levels; but at the time the project completion review was finalized, results of the assessments had not been released by the NBSMA. The other madrasahs (including the 54 madrasahs with no accreditation) are still waiting to be assessed. Due to limited resources, the NBSMA applied a quota system during the assessment of schools, including madrasahs. Thus, many project madrasahs that have applied for the assessments, have to wait for several years before their madrasahs are included in the quota list. 2. One of the key indicators under the project outcome is that at least 70% of the project madrasahs have improved their accreditation by one level (in 2012). Table A4 suggested that 64 madrasahs have increased their accreditation level from B to A; 80 madrasahs from C to B and 72 madrasahs from not accredited to accredited (C). Thus, the total madrasahs that have increased their accreditation level is 216 (43% of the total 500 project madrasahs). This suggested that the target of 70% has not been met. However, considering the results achieved by the project madrasahs and the accreditation standards, it is very likely that the accreditation level of the 76 madrasahs that were being reassessed will increase. Similarly for the 54 unaccredited madrasahs, it is very likely that their accreditation will increase at least to C level. Thus it may be concluded that 346 project madrasahs (69%) have increased their accreditation level. Thus the target of 70% of the project madrasahs have improved their accreditation by one level can be construed as likely to be achieved (once the assessment by NBSMA is completed).

26 Appendix 5

ACADEMIC PERFORMANCE OF PROJECT MADRASAH STUDENTS IN THE NATIONAL EXAMS

Table A5.1: Academic Performance of Project Madrasah Ibtidaiyah

Students in the National Exams Education Exam Number of Students Year Participants Passing the Exams Pass Rate (%) Average Score 2008/09 6,880 6,822 99.16 6.48 2011/12 7,428 7,420 99.89 7.17

Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A5.2: Academic Performance of Project Madrasah Tsanawiyah Students in the National Exams

Education Exam Number of Students Year Participants Passing the Exams Pass Rate (%) Average Score 2008/09 18,331 17,890 97.59 6.72 2011/12 18,343 18,083 98.58 7.27

Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A5.3: Academic Performance of Project Madrasah Aliyah Students in the National Exams

Education Exam Number of Students Average Score Year Participants Passing the Exams Pass Rate (%) NS SS RL 2008/09 5,322 5,294 99.47 7.28 7.11 8.53 2011/12 4,624 4,528 97.92 7.65 7.53 9.14

NS = students specializing in natural science, RL = students specializing in religion, SS = students specializing in social science. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A5.4: Academic Performance of Nonproject Madrasah Ibtidaiyah Students in the National Exams

Education Exam Number of Students Year Participants Passing the Exams Pass Rate (%) Average Score 2008/09 129 127 98.45 6.43 2011/12 124 121 97.58 6.56

Note: Nonproject madrasahs are madrasahs located near the project madrasahs that did not receive support from the project or other external sources. They were randomly selected for purposes of comparison to evaluate the project impacts. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Appendix 5 27

Table A5.5: Academic Performance of Nonproject Madrasah Tsanawiyah Students in the National Exams

Education Exam Number of Students Year Participants Passing the Exams Pass Rate (%) Average Score 2008/09 299 292 97.66 6.87 2011/12 289 283 97.92 6.57

Note: Nonproject madrasahs are madrasahs located near the project madrasahs that did not receive support from the project or other external sources. They were randomly selected for purposes of comparison to evaluate the project impacts. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A5.6: Academic Performance of Nonproject Madrasah Aliyah Students in the National Exams

Education Exam Number of Students Average Score Year Participants Passing the Exams Pass Rate (%) NS SS RL 2008/09 228 226 99.12 6.85 6.61 … 2011/12 278 275 98.92 6.55 6.56 …

… = not available, NS = students specializing in natural science, RL = students specializing in religion, SS = students specializing in social science. Note: Nonproject madrasahs are madrasahs located near the project madrasahs that did not receive support from the project or other external sources. They were randomly selected for purposes of comparison to evaluate the project impacts. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

28 Appendix 6

STUDENT ENROLLMENTS AT PROJECT MADRASAHS

1. This output aimed to increase the internal efficiency of madrasahs through (i) facilitation of after-school students remedial programs to improve students’ grades and learning outcomes, and to reduce the repetition and dropout rates; and (ii) provision of scholarships for transition programs to help students to continue to the next level of education. Financing of the programs was included under the block grants. Tables A6.1–A6.12 summarize the achievements of the project madrasahs.

Table A6.1: Number of Students at Project Madrasahs, SY2008/09 and SY2011/12 Number of Students School Madrasah Ibtidaiyah Madrasah Tsanawiyah Madrasah Aliyah Year Male Female Total Male Female Total Male Female Total 2008/09 22,448 21,868 44,316 28,714 29,348 58,062 7,383 8,809 16,192 2011/12 23,385 22,310 45,695 29,754 30,375 60,129 7,914 9,927 17,841

Change 4.2% 2.0% 3.1% 3.6% 3.5% 3.6% 7.2% 12.7% 10.2% SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.2: Student Enrollments at Project Madrasahs, SY2008/09 and SY2011/12

Student Enrollments School Madrasah Ibtidaiyah Madrasah Tsanawiyah Madrasah Aliyah Year Male Female Total Male Female Total Male Female Total 2008/09 4,195 3,990 8,185 10,788 11,205 21,993 2,789 3,104 5,893 2011/12 4,415 4,061 8,476 11,723 11,620 23,343 3,244 4,020 7,264

Change 5.24% 1.78% 3.56% 8.67% 3.70% 6.14% 16.31% 29.51% 23.26% SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.3: Transition Rates from Project Madrasah Ibtidaiyahs

to Junior Secondary Education, SY2008/09 and SY2011/12 Graduates Continuing to Higher Level Did Not Continue School Year Male Female Total Male Female Total

% of Total Male Female Total

% of Total

2008/09 3,353 3,469 6,822 3,309 3,447 6,756 99.03 44 22 66 0.972011/12 3,694 3,726 7,420 3,681 3,718 7,399 99.72 13 8 21 0.28

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.4: Transition Rates from Project Madrasah Tsanawiyah to Senior Secondary Education, SY2008/09 and SY2011/12

Graduates Continuing to Higher Level Did Not Continue School Year Male Female Total Male Female Total

% of Total Male Female Total

% of Total

2008/09 8,795 9,185 17,980 7,602 7,966 15,568 86.6 1,193 1,219 2,412 13.4 2011/12 8,772 9,331 18,083 7,656 8,153 15,809 87.4 1,116 1,158 2,274 12.6

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Appendix 6 29

Table A6.5: Repetition Rates for Project Madrasah Ibtidaiyah Students, SY2008/09 and SY2011/12

School Number of Students Repeating Students Repetition Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 22,448 21,868 44,316 500 290 790 2.23 1.33 1.78 2011/12 23,385 22,310 45,695 449 219 668 1.92 0.98 1.46

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.6: Repetition Rates for Project Madrasah Tsanawiyah Students, 2008 and 2012

School Number of Students Repeating Students Repetition Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 28,714 29,348 58,062 141 79 220 0.49 0.27 0.38 2011/12 29,754 30,375 60,129 167 105 272 0.56 0.35 0.45

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.7: Dropout Rates for Project Madrasah Ibtidaiyah Students,

SY2008/09 and SY2011/12

School Number of Students Dropped Out

Students Dropout Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/2009 22,448 21,868 44,316 39 12 51 0.17 0.05 0.12 2011/2012 23,385 22,310 45,695 39 27 66 0.17 0.12 0.14

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.8: Dropout Rates for Project Madrasah Tsanawiyah Students,

SY2008/09 and SY2011/12

School Number of Students Dropped Out

Students Repetition Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 28,714 29,348 58,062 338 219 557 1.18 0.75 0.96 2011/12 29,754 30,375 60,129 243 182 425 0.82 0.60 0.71

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.9: Repetition Rates for Nonproject Madrasah Ibtidaiyah Students, 2008 and 2012

School Number of Students Repeating Students Repetition Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 301 553 854 6 11 17 2.0 2.0 2.0 2011/12 343 520 863 11 10 21 3.2 1.9 2.4

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

30 Appendix 6

Table A6.10: Repetition Rates for Nonproject Madrasah Tsanawiyah Students, SY2008/09 and SY2011/12

Academic Number of Students Repeating Students Repetition Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 461 536 997 1 0 1 0.2 0 0.1 2011/12 565 546 1,111 0 0 0 0.0 0 0.0

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.11: Dropout Rates for Nonproject Madrasah Ibtidaiyah Students,

SY2008/09 and SY2011/12

School Number of Students Dropped Out

Students Dropout Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 301 553 854 0 0 0 0 0 0 2011/12 343 520 863 0 0 0 0 0 0

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Table A6.12: Dropout Rates for Nonproject Madrasah Tsanawiyah Students,

SY2008/09 and SY2011/12

School Number of Students Dropped Out

Students Dropout Rate (%) Year Male Female Total Male Female Total Male Female Total 2008/09 461 536 997 2 1 3 0.4 0.2 0.3 2011/12 565 546 1,111 1 1 2 0.2 0.2 0.2

SY = school year. Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

Appendix 7 31

GENDER ACTION PLAN TARGETS AND ACHIEVEMENTS

A. Project Description 1. The project was expected to help raise levels of educational attainment among new entrants to the labor force, and increase formal employment rates and household incomes. The project’s outcome was improved quality, efficiency, and effectiveness of primary, junior secondary, and senior secondary madrasah education. The project had four outputs: (i) improved teacher professionalism according to national standards; (ii) upgraded teaching and learning resources and facilities; (iii) increased internal efficiency; and (iv) strengthened governance, management, and sustainability that meet national standards. 2. The project aims to raise education standards in around 500 madrasah schools, thereby giving 120,000 students, including around 60,000 girls, the immediate benefit of better education. If the improved standards are sustained over 10–15 years, roughly 300,000 students will benefit directly from better education, and about 140,000 students will benefit directly from improved academic credentials, leading to improved access to post-secondary education and formal paid employment. The project was categorized as Effective Gender Mainstreaming (EGM). B. Gender Issues and Gender Action Plan Features 3. The gender issues in madrasah education related to (i) the quality of madrasah education to prepare female and male students to be more competitive in higher-level education and in the labor market on a sustainable basis, (ii) the disparity of participation between female and male staff in management and leadership, and (iii) the lower proportion of female staff in higher school levels and management. 4. The project gender action plan (GAP) features included the following:

(i) Establishing targets for women beneficiaries by ensuring that women make up at least 30% of teacher participants in degree upgrading (enrollment in bachelor’s degree), 40% of participants in professional certification, and 40% of participants in short-term training activities (e.g., training on subject content, curriculum development, classroom teaching methodology).

(ii) Establishing targets to improve capacity of teachers to identify gender biases and stereotyping in textbooks (i.e., in the content and illustrations), and to eliminate gender bias and stereotyping in the teaching and learning process.

(iii) Ensuring that at least 50% of the participating madrasahs have separate well-functioning toilets for boys and girls and that the facilities constructed and upgraded (e.g., school toilets) are gender sensitive.

(iv) At least 50% of transition scholarships are allocated for female students to encourage girls to stay at school.

(v) Advance the quality of education by ensuring that at least two master’s degrees and one doctoral fellowship are provided for women; at least 95% of madrasahs have a functioning madrasah committee, of which at least 30% of members are women; and gender training is included in madrasah-based management and madrasah leadership.

32 Appendix 7

C. Overall Assessment of Gender-Related Results and Achievements 5. By project completion, the project had achieved or partially achieved 9 of the 11 gender action plan targets. Table A.7 details the project’s gender-related achievements. Female teachers comprised 40.5% (163 of 403) of degree-upgrading (bachelor’s) participants, exceeding the 30% target. Women accounted for 52.5% (4,955 of 9,433) of the teachers who participated in training on subject content, curriculum development, and classroom teaching methodology. Women teachers made up 49% (73 of 149) of the teachers who participated in professional certification. Both training achievements exceeded the 40% target. In 473 madrasahs, separate toilets were provided for female and male students. Female students were the beneficiaries of about 42% (183,420 of 436,081) of remedial programs and received 43% (4,397 of 10,265) of the transition scholarships awarded. All 500 madrasahs have well-functioning madrasah committees. However, only 23% (1,134 of 4,950) of the committee members were women.

6. The project has also made strong efforts to fulfill the qualitative target for incorporating gender training in madrasah-based management and madrasah leadership. Gender equality issues have been accommodated in the modules of madrasah leadership development training, (e.g. Modules on “Gender Responsive Leadership in Madrasah” and “Gender-Sensitive Madrasah Culture). These modules encourage the training participants to discuss and identify barriers to women taking up and functioning in leadership positions; integrate gender perspective in madrasah management, in what manner women can participate in the madrasah management and how to improve women’s participation in madrasah management. D. Gender Equality Results

1. Benefits, Participation and Access to Project Resources

7. Improved girls’ access to education through scholarships. The project provided transition scholarships to 4,397 female students - about 43% of total 10,265 students who received scholarships - to help them continue their education and transit to the next higher level of schooling. Parents interviewed during a project review missions expressed appreciation of the scholarships provided to their daughters, remarking that this will enable their daughters to stay in school, and that even if the scholarships are withdrawn, they will find ways to keep their daughters in school. This demonstrates that parents are beginning to appreciate the importance of education to their daughters. 8. Better academic achievement for boys and girls. From 2009 up to 2012, remedial scholarships have been provided to 183,420 girls1 –about 42% of total 436,081 students who participated in remedial program. The remedial coaching programs were introduced to improve students’ learning outcomes and reduce repetition and drop-out rates. A survey2 reported that girls are doing extremely well academically and are considered by both female and male teachers to be more “active” as a group in classroom and to spend more time in the libraries, compared to male students, who were more reticent in the classroom, and spend more time playing sports. The provision of the remedial program was considered to contribute to the

1 The number represented the number of the remedial program participants. A student might participate in more than

one program. 2 Ministry of Religious Affairs. Survey report on Mapping of Gender Equality on Management and Teaching Learning

process in Madrasah of MEDP Target, February 2012. Jakarta.

Appendix 7 33

improvement of academic performance of students in project madrasahs improved as detailed in Appendix 5 of this project completion report. 9. Improved teaching-learning facilities. To improve teaching quality, teaching-learning resources were upgraded. Madrasahs were equipped with 539,524 books, 22,698 aids for science laboratory, aids for clinics, computer laboratory, language laboratory, library equipment, and teaching aids for learning media. The project constructed new buildings and renovated old buildings for clinics, science laboratories, computer laboratories, language laboratories, libraries, classrooms and toilets. , 10. Overall, 473 out of the 500 (94.6%) participating madrasahs have separate well- functioning toilets for female and male students, with 1 toilet serving an average of 109 female students (1:109 ratio) as compared to 1:160 for boys. About 11% of madrasahs provide sanitary items for girls who have their menstruation. Of the 500 madrasahs, 227 also constructed separate toilets for male and female teachers. However none of madrasahs provided separate changing rooms for female and male students. As reasoned out by teachers, this is not necessary since madrasahs are close to students’ home. 11. New classrooms and laboratories and upgraded facilities have created a pleasant learning environment and encourage students’ attendance in school. Female students remarked that the improved facilities made them happy to go to school, and the provision of separate toilets was crucial in persuading their mothers to allow them to attend classes even during menstruation. 12. Improved qualification of teachers. The bachelor’s degree program (S1 program) was offered for teachers to upgrade their qualification to degree level. About 163 out of 403 teachers (40.4%) who participated in degree (S1) upgrading were women, exceeding the 30% target. Those who were not able to continue were constrained by the distance of the universities from their homes, especially for women who have children or other family responsibilities. For male teachers, staying far from their homes means losing extra income from tutorials, small businesses, etc. Nevertheless, the project has created positive catalyst by enhancing awareness for women teachers in 500 madrasahs to upgrade their qualification by obtaining degree. 13. Aside from degree upgrading, the project has supported 73 women teachers (49% of the total 149 teachers) to obtain professional teachers certification, exceeding the target of 40%. The project also provided short trainings to 4,955 (52.5% of 9,533) women teachers. These trainings focused primarily on subject content in the sciences, mathematics and English, and on classroom teaching methodologies and curriculum development. These training helped teachers improve their teaching skills and enabled them to make learning more interactive and “fun” for students. Teachers have expressed that they are more creative in encouraging students to actively participate in class, through discussion and group activities.

2. Strategic Changes in Gender Relations

14. There was evidence indicating that the project has led to some strategic benefits at the local institutional level. Membership in madrasah committees by mothers made them feel empowered, as result of which they gained the confidence to speak out at madrasah committee meetings, often against predominantly male views. Several women madrasah committee members indicated that they were able to fully participate in madrasah committee meetings, that their involvement was appreciated, and that they were able to exert influence over the decision

34 Appendix 7

making in a range of activities, especially related to how resources from the block grant were spent. 15. Female teachers’ improved teaching skills and methodologies also contributed to the increased respect for female teachers, from both male and female students. Parents and MC members have expressed that the Project has enhanced the perception of female and male teachers in the community.

3. Contribution of Gender Equality Results to Overall Loan Outcomes and

Effectiveness

16. The GAP is integral to the project and has made significant contribution to the overall project and outcome, especially since women and girls are intended to be significant beneficiaries of the project. By considering gender equality through the implementation of the GAP, the project has ensured that all students, boys and girls alike, will improve their learning outcomes, thus, improving the madrasahs’ academic performance and enhancing the perception of madrasah education quality by parents and the community. 17. The training and upgrading of qualification for both male and female teachers significantly contributed to the achievement of the project goal of improving the quality of madrasah education. The gender design provided opportunity for women teachers to improve their teaching capacity through training and degree upgrading. It was evidenced during PCR mission that women teachers’ capacities and confidence are significantly improved. Training had enabled women teachers to change the way they taught students, by innovation, through more interactive learning and by making learning more fun for students. Their improved skills increased respect from students in the class, and improved positive perception of both female and male teachers by the community. This will eventually increase parents’ trust to enroll their children in madrasahs.

E. Lessons and Recommendation

18. The series of trainings conducted by madrasahs for teachers have contributed in improving the quality of teaching and the qualification of teachers. However, a survey3 of the same revealed that the majority of madrasahs are not gender responsive and that the design of learning process and materials does not yet accommodate and consider the differences, needs, aspirations and experiences between males and females. There was also potential for neutral learning processes and materials to become further gender biased since the learning potentially gives advantages to boys in term access. There has been limited success in training teachers to identify gender biases and stereotypes, and to eliminate the same. Many teachers were unable to identify gender bias in textbooks, and even if they recognize such, they seem unable to address this in the classroom. Systematic gender training should be provided to teachers with support of local government. In addition, the regular meetings for teacher learning/working groups are very suitable medium for discussions and lateral learning for this matter. 19. Project has appointed gender focal point in central project management unit (CPMU) whose task was to monitor the implementation of the GAP. In a positive note, the CPMU and its consultant team at national level are aware of the gender targets set for the project. However, observations during review missions including the project completion mission revealed that this is not the case for main stakeholders at the provincial and district level, including the madrasah

3 Ibid 2

Appendix 7 35

facilitators. For future projects, CPMU needs to ensure that these gender targets are known and understood by provincial and district implementation units and madrasah through systematic awareness-raising activities and trainings. 20. A key challenge is the low representation of women in madrasah committees. The GAP required an average target of 30% female madrasah committee members for all participating madrasahs. Although all participating madrasahs have functioning madrasah committees and overall, the membership for women is 23%, many of the madrasah committees have no female members at all. Considering that traditionally, women rarely participate in school management and in other community activities, gender training for parents, teachers, and all stakeholders is crucial to enable them to understand the changing roles of women in the community and at home. Other reasons for women’s limited participation are reportedly women’s lack of time and capacity. For future projects, capacity-building activities and setting meetings at a time convenient for mothers will encourage more women to participate. In addition, a target of female membership for each madrasah committee, rather than an aggregate target for all madrasah committee, would ensure more female membership in each madrasah. The quantitative target should also be accompanied with qualitative target to ensure that madrasahs developed a strategy to improve the target.

Table A.7: Project Achievements on Gender-Related Targets

Targets Achievements Issues and Challenges Recommendation Output 1: Teacher professionalism is improved according to national standards.

At least 30% of the participants in the degree upgrading (enrollment in bachelor’s degree [S1]) are women.

The target was achieved.

163 female teachers (40% of a total of 403) participated in degree upgrading (bachelor’s degree).

To ensure quality, the Executing Agency selected only reputable universities to implement the project S1 program. Most of these universities are located far from the project madrasahs and teachers’ residences. Many teachers were reluctant to participate in the degree-upgrading program offered by the project. Family-related issues and distance from universities to teachers’ home towns were the main reasons. Many teachers have other additional sources of income. Leaving their hometown meant that they would lose this income, which in many cases were higher than the income from teaching. Other teachers, particularly women, cited that leaving their hometown meant that they could not attend to domestic matters, such

More flexible program, e.g. by allowing more universities to be selected by teachers. To ensure quality, the reputable universities can be included as supervisors. Improvement in teachers’ salary, so that teachers will not need to have additional incomes.

36 Appendix 7

as taking care of their young children and husbands.

At least 40% of participants in professional certification are women.

The target was achieved.

73 female teachers (49% of a total of 149) participated in professional certification.

The project certification program could be not fully implemented due to a change in the government regulation on the certification system.

At least 40% of participation in short-term training activities are women (e.g., training on subject content, curriculum development, and classroom teaching methodology).

The target was achieved.

4,955 female teachers (52% of a total of 9,533) participated in subject content and classroom teaching methodology training, and curriculum development.

Capacity of teachers to identify gender biases and stereotyping in textbooks (i.e., in the content and illustrations) is improved.

MORA carried out studies to evaluate the textbooks used in the project madrasahs. The results indicated that about 4% of the textbooks used in project madrasah ibtidaiyahs are considered gender-biased in terms of content and illustrations. At the higher level (madrasah aliyah) the figure is 7%.

Discussions with project madrasah teachers indicated that many teachers have not realized that the books they are using are gender-biased. This suggested that the capacity of teachers to identify gender biases and stereotyping needs further improvement.

The Government needs to provide more more capacity building activities, gender trainings to improve teachers’ understanding and knowledge on gender biases and stereotyping.

Output 2: Teaching and learning resources and facilities are upgraded to national standards.

At least 50% of the participating madrasahs have separate well-functioning toilets for boys and girls.

Facilities MI MT MA Separate toilets for students 87% 100

% 100%

Separate toilets for teachers 28% 56

% 64%

473 (95% of 500) project madrasahs provided separate toilets for female and male students. One toilet is used by an average of 109 girls (1:109) as compared to 1:160 for boys. 227 project madrasahs (45%) also provided separate toilets for female and male teachers.

In general, the toilets meet the standards as specified under the Minister of National Education Regulation No. 24, 2007.

Surveys carried out in 2012 suggested that most students were happy with the condition of the toilets.

Ensure that the facilities

Observations during review and completion review missions suggested

Many project madrasah principals (managers)

Government is to carry out gender

Appendix 7 37

constructed and upgraded (e.g., school toilets) are gender sensitive (consider needs of female and male students).

that only a few project madrasahs (mostly those with women principals) provided or constructed toilets or other facilities that are gender sensitive, e.g. the provision of sanitary items for female students, who have their menstruation.

were not aware of the different needs of female and male students.

awareness campaigns that include the materials/ modules on the provision of gender sensitive facilities to accommodate different needs of female and male students. Guidelines for construction and provision of facilities need to be improved to include (i) gender-sensitive aspects e.g. to accommodate female students’ needs during menstruation; and (ii) security and safety measures of toilets.

Output 3: Internal efficiency is increased to meet national standards, especially student progression and transition rates.

At least 50% of transition scholarships are provided for female students.

183,420 female students (42% of a total of 436,081) benefited from the remedial coaching program.

Note: 436,081 is the total number of all the remedial program participants. Some students participated in more than one remedial program.

4,397 female students (43% of a total of 10,265) received small scholarships to help them continue their education and transition to a higher level of education.

The targets were missed by 8% and 7% for the transition and scholarship targets. More male students participated in the remedial coaching program as the academic performance of female students is better compared to that of male students, i.e. more male students needed remedial coaching.

Supervision by the district coordinating units and madrasah facilitators needed to be strengthned to ensure that the selection of scholarship recipients was truly based on the agreed selection criteria.

Output 4: Governance, management and sustainability strengthened to meet national standards.

At least two master’s degrees and one doctoral fellowship are provided for women.

Nine staff of the Ministry of Religious Affairs (MORA) were provided fellowships to obtain master’s and doctoral degrees.

Only 1 of 7 master’s degree students was a woman (12.5%). Neither of the two doctoral students was a woman.

MORA’s female staff were reluctant to study abroad due to family reasons.

The executing agency should be more flexible to allow staff to obtain their master’s or doctoral degrees from local universities.

At least 95% of madrasahs have a functioning madrasah committee.

All of the project madrasahs (100%) have functioning madrasah committees and most of them meet regularly.

At least 30% of madrasah committee members are women.

Overall, women’s representation on madrasah committees was only 23% (1,134 of 4,950 madrasah committee members).

About 90% of the female committee members were considered active members. About 70% of the male committee members were considered active.

The female committee members are usually madrasah teachers. The community leaders and parents are mostly represented by men. “Occupied with domestic affairs” and “feeling of not having the capacity to be

Capacity building activities, gender trainings and setting meetings at a time convenient for mothers will ensure that more women will participate in MCs and other school

38 Appendix 7

Source: Ministry of Religious Affairs. 2012. Evaluation of Impact of MEDP Inputs Toward Achieving the Project Goals. Jakarta.

a madrasah committee member” were often-cited reasons why women did not join or were not active in the committees.

activities.

Gender training is included in Madrasah-Based Management and Madrasah Leadership training modules.

Gender equality issues have been included in most of the madrasah-based management and leadership training modules.

Leadership development training included gender training (in module V: Gender-Responsive Leadership in Madrasah, and Women Leadership in Madrasah; and module VI: Gender-Sensitive Madrasah Culture).

Madrasah-based management modules covered gender perspectives well, e.g., how to integrate a gender perspective in madrasah management, in what manner women can participate in madrasah management, and how to improve women’s participation in madrasah management.

Appendix 8 39

PROJECT COST AND FINANCING PLAN ($’000)

No. Project Component and Activities ADB GOI Others Total ADB GOI Others Total(madrasahs) (madrasahs)

1 Furniture Equipment and Vehicles 883 376 0 1,259 677 480 0 1,1572 Human Resources Development 5,480 16,634 0 22,114 2,570 15,972 0 18,5423 Consulting Services 2,058 0 0 2,058 2,083 0 0 2,0834 Madrasah Block Grants 35,257 1,726 0 36,983 41,065 0 3,100 44,1655 Surveys and Testing 337 0 0 337 266 43 3096 Project Management Supports 2,189 0 0 2,189 2,424 0 2,4247 Incremental Recurrent Costs 0 1,541 0 1,541 0 1,211 1,2118 Taxes and Duites 0 100 0 100 0 165 1659 Contingencies 2,540 1,052 0 3,592 0 0 0

10 Interest Charges 1,256 0 0 1,256 1,147 0 1,14750,000 21,429 0 71,429 50,232 17,871 3,100 71,203

ActualAppraisal Estimates

Total ADB = Asian Development Bank, GOI = Government of Indonesia. Sources: Ministry of Religious Affairs. 2012. Completion Report Madrasah Education Development Project. Jakarta and Asian Development Bank estimates. 1. The actual project cost estimate is about $0.2 million lower than the original cost estimate. This was because of the partial use of the budgets allocated for the human resources development program (the bachelor’s degree and professional certification programs). At project completion, about $2 million in unused loan proceeds was cancelled. However, ADB’s share in US dollars increased from $50.0 million to $50.2 million because the US dollar depreciated against the SDR. At appraisal, ADB was to contribute to 70% of the total project costs and the remaining 30% was to be provided by the government. 2. At project completion, the total project cost was estimated at $71.2 million, consisting of $50.2 million from the loan proceeds (71%), $17.9 million (25%) from government counterpart funds, and $3.1 million (4%) in contributions from the madrasahs. Thus, ADB’s contribution to the project costs about the same as the appraisal estimate. The government’s contribution at project completion, however, had decreased from 30% to 25%. This is because of the change in the financing percentage for the madrasah development plan (MDP) component. Originally, the government was to contribute about 5% to the overall MDP financing. However, to avoid further delays due to frequent holdups in the issuance of the government’s annual budgets, and to simplify the administration of the block grant transfers, the government and ADB agreed to increase ADB’s MDP financing to 100%. To balance the financing plan, the government’s contribution to the human resources development program, i.e., in-country training, was increased to 22% from 13%. The budget allocated for the MDP is about 80% of the total loan proceeds, while the in-country training constituted about 2% of the total loan proceeds. 3. Even though about $2 million unused loan proceeds were cancelled at project completion, the actual project costs were almost the same as the appraisal project costs. This is because the actual costs included the madrasah contributions, which were not included at appraisal.

40 Appendix 9

FUND-CHANNELING MECHANISM

CPMU DGIE, MORA

I N D E P E N D E N T

M O N I T O RI N G

ADB

Imprest Account,

Bank Indonesia

MOF

DG Treasury

SPM

Provincial MORA Office (PCU)

3

2

4

5

District MORA Office (DCU)

Project Madrasah

Facilitators 1

KPPN Operational

Bank

SPPD

REPLENISHMENT

Project Madrasah Bank Account

6 7

Contractor/ Supplier

Madrasah Report on Funds Utilization and Commitment

submitted to PCU through DCU and consolidated by CPMU as

basis for second and third tranche release

MDP not approved

1a

Fund Flow

ADB = Asian Development Bank; CPMU = central project management unit; DCU = district coordinating unit; DG = Director General; DGIE = Director General, Islamic Education; KPPN = Ministry of Treasury, Finance; MDP = Madrasah Development Plan; MOF = Ministry of Finance; MORA = Ministry of Religious Affairs; PCU = project coordinating unit; SPM = payment order; SPPD = disbursement order. Source: Asian Development Bank

Appendix 10 41

PROJECT IMPLEMENTATION SCHEDULE

Year 2007 2008 2009 2010 2011 2012 Quarter 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 A. Activities Outside Madrasah Block Grants

Output 1 Improve Teacher Professionalism

1.1 Develop and implement program to upgrade teachers to degree level

1.2

Develop and implement program for professional certification

1.3

Develop and implement short-term training in subject content, methods and implementation of new curriculum

Output

4

Strengthen Governance, Management, and Sustainability

4.1 Improve Governance and Management a. Establish and strengthen madrasah committees

b. Train Ministry of Religious Affairs ( MORA) central, provincial,

and district staff, supervisors, principals and madrasah

owner-operators (yayasan) in leadership and madrasah-based management

c. Train provincial, district, madrasah staff, madrasah committees

and yayasan in madrasah development planning and financial

management

d. Train principals and supervisors in the operation and management of block grants

e. Develop and implement financial management information system in madrasah

f. Design and implement preventive maintenance program

4.2

Develop and Implement Teaching/Learning Assessment and Quality Assurance System

a. Establish provincial quality assurance working groups

b. Design and implement training in quality assurance and accreditation for central, provincial, and district MORA staff, and madrasah principals and owners

c. Review madrasah proposals for accreditation and make

recommendations

d. Train and certify madrasah development center staff, supervisors

and principals as assessors e. Madrasah development center assessors, supervisors, and

42 Appendix 10

Year 2007 2008 2009 2010 2011 2012 Quarter 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

principals participate in assessment and accreditation teams 4.3 Introduce Performance-Based Planning and Budgeting

a. Develop and implement performance-based planning and budgeting training for provincial and district MORA offices

b. Identify candidates and implement overseas training program

c. Train provincial and district staff in leadership development and madrasah development planning

4.4 Establish Advocacy Programs a. Develop and implement advocacy programs b. Establish madrasah linkages website c. Establish and facilitate Madrasah Development Forum d. Disseminate information for accessing funding sources B. Madrasah Block Grants

Output 1

Improve Teacher Professionalism and Student Performance

a. Identify, develop, and provide support to teacher learning groups b. Support teachers’ formal training costs (transport, per diem)

Output 2

Upgrade Essential Teaching/Learning Resources and Facilities

a. Provide funds for textbooks and learning support materials b. Provide funds for equipment and software c. Rehabilitate and construct required learning facilities

Output 3 Improve Internal Efficiency

3.1 Offer Remedial and Transition Programs a. Design and implement student remedial program b. Design and implement transition scholarships program

3.2 Expand Participation Capacity in Selected Schools

Appendix 10 43

Year 2007 2008 2009 2010 2011 2012 Quarter 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 (madrasah ibtidaiyah and madrasah tsanawiyah)

Project Implementation

a. Establish central project management unit b. Establish provincial and district coordinating units c. Select and engage project implementation consultants d. Develop Project Management Information System e. Develop project website f. Midterm Project Review g. Final Project Review

Technical Assistance for External Monitoring and Evaluation

Original schedule Actual schedule Source: Asian Development Bank

44 Appendix 11

IMPLEMENTATION ARRANGEMENTS 1. The Ministry of Religious Affairs (MORA) was the executing agency for the project, while the Directorate General of Islamic Education of MORA was responsible for the overall management of the project. A central project management unit (CPMU) was established. A project steering committee (PSC) was established to guide the executing agency and CPMU on general policy directions, intersector coordination, and strategic directions. Members of the PSC included representatives of MORA, the Ministry of Finance, the National Development Planning Agency (BAPPENAS), the Ministry of Home Affairs, and the Ministry of Education and Culture. The Director General for Islamic Education was the chair of the PSC and concurrent project director. The Director of the Directorate for Madrasah Education was the secretary to the PSC and concurrent deputy project director. A project technical committee (PTC) was established to guide the CPMU in the overall project implementation. Members of the PTC included representatives of MORA, the Ministry of Finance, the Ministry of Home Affairs, and the Ministry of Education and Culture (echelon 2). The Director of Education and Religious Affairs, BAPPENAS, was the chair of the PTC.

2. The executing agency established the CPMU under the guidance of the project director and headed by a project manager. The CPMU was responsible for day-to-day project implementation; planning and budgeting; procurement; disbursement; monitoring; supervision; oversight of implementation at province, district, and madrasah levels; and submission of the required reports to the government and ADB. The CPMU had about 15 staff, including MORA staff and consultants working together on each project component and supported by finance and administrative staff. Contractual staff were hired as specialists in training, quality assurance, monitoring and evaluation, civil works, and information and communication technology. 3. In each of the project provinces, a project coordinating unit (PCU) was established within MORA’s provincial office to manage, coordinate, monitor, evaluate, and consolidate the provincial and district activities. The PCU was headed by the provincial MORA education division (Kepala Bidang Mapenda). The PCU, including project advisors, and finance and administrative staff, coordinated the district and madrasah activities. At the district level, a district coordinating unit (DCU) was established in MORA’s district office. The DCUs oversaw implementation at the madrasah level, particularly the madrasah development plan block grants. Each of the DCUs was headed by the district MORA education section (Kepala Seksi Mapenda), assisted by supervisors, facilitators, and finance and administrative staff.

Appendix 11 45

ADB = Asian Development Bank, BAPPEDA = Regional Development Planning Agency , BAPPENAS = National Development Planning Agency, CPMU = central project management unit, DCU = District Coordinating Unit, DME = Directorate for Madrasah Education, MOF = Ministry of Finance, MONE = Ministry of National Education, MORA = Ministry of Religious Affairs, MPPPA = Madrasah Development and Empowerment Council, PCU = project coordinating unit, PMIS = project monitoring information system. Source: Asian Development Bank

PROJECT DIRECTORDEPUTY

PROJECT DIRECTORADB

STEERING COMMITTEE Chairperson: Secretary General MORA Secretary: Director DME,MORA Member: MORA, BAPPENAS, MOF, MONE (Director level)

TECHNICAL COMMITTEE Chairperson: Dir. Education and Religious Affairs, Bappenas Member: MORA, Bappenas , MOF, MONE (Sub -director level)

PROJECT MANAGER

SECRETARY

FINANCE -ADMINISTRATION•Finance/Accounting•Procurement and Civil Works

Administration•Monitoring and Evaluation

TECHNICAL TEAM (CONSULTANTS)•Teacher Quality •Teaching -Learning Resources•Internal Efficiency •Madrasah Based Management

TREASURER

CPMU

PROVINCIAL COORDINATING COMMITTEE Chairperson: BAPPEDADeputy Chairperson: Head of Provincial Office, MORAMember: Provincial. Office of Education, MDC, MPPPA

PROVINCIALCOORDINATING UNIT (PROVINCIAL MORA

OFFICE)

MADRASAH DEVELOPMENTCENTER (MDC)

PROJECT ADVISERS•Quality Assurance, and

Teacher Quality •Management and Planning•PMIS

FINANCE -ADMINISTRATION•Finance/A and Accounting•Procurement and Civil Work•Monitoring and Evaluation

DISTRICT EDUCATION COUNCIL

MADRASAH DEVELOPMENT FORUM

DISTRICTCOORDINATING UNIT

(DISTRICT MORA OFFICE)

MADRASAH FACILITATORSFINANCE - ADMINISTRATION•Finance/Accountant•Supervisor

MADRASAH MADRASAHCOMMITTEEYAYASAN

PCU

DCU

MADRASAH UNIT

PROJECT DIRECTORDEPUTY

PROJECT DIRECTORADB

STEERING COMMITTEE Chairperson: Secretary General MORA

Member: MORA, Bappenas , MOF, MONE (Sub director level)

PROJECT MANAGER

SECRETARY

FINANCE -ADMINISTRATION•Finance/Accounting•Procurement and Civil Works

Administration•Monitoring and Evaluation

TECHNICAL TEAM (CONSULTANTS)•Teacher Quality •Teaching -Learning Resources•Internal Efficiency •Madrasah-Based Management

TREASURER

CPMU

PROVINCIAL COORDINATING COMMITTEE Chairperson: BAPPEDADeputy Chairperson: Head of Provincial Office, MORAMember: Provincial. Office of

PROVINCIALCOORDINATING UNIT (PROVINCIAL MORA

OFFICE)

MADRASAH DEVELOPMENTCENTER (MDC)

PROJECT •Quality Assurance, and

Teacher Quality •Management and Planning•PMIS

FINANCE -ADMINISTRATION•Finance/A and Accounting•Procurement and Civil Work•Monitoring and Evaluation

DISTRICT EDUCATION COUNCIL

MADRASAH DEVELOPMENT FORUM

DISTRICTCOORDINATING UNIT

(DISTRICT MORA OFFICE)

MADRASAH FACILITATORSFINANCE - ADMINISTRATION•Finance/Accountant•Supervisor

MADRASAH MADRASAHCOMMITTEEYAYASAN

PCU

DCU

MADRASAH UNIT

46 Appendix 12

STATUS OF COMPLIANCE WITH LOAN COVENANTS

Covenant

Reference in Loan

Agreement

Status of Compliance The Borrower shall (i) maintain, or cause to be maintained, separate accounts for the Project; (ii) have such accounts and related financial statements audited annually, in accordance with appropriate auditing standards consistently applied by independent auditors whose qualifications, experience, and terms of reference are acceptable to ADB; (iii) furnish to ADB, as soon as available but in any event not later than 9 months after the end of each related fiscal year, certified copies of such audited accounts and financial statements and the report of the auditors relating thereto (including the auditors' opinion on the use of the Loan proceeds and compliance with the financial covenants of the Loan Agreement as well as on the use of the procedures for imprest account/statement of expenditures), all in the English language; and (iv) furnish to ADB such other information concerning such accounts and financial statements and the audit thereof as ADB shall from time to time reasonably request.

Article IV Section 4.02(a)

Complied with. The accounts were audited annually by the (BPKP), the agency assigned to carry out the audits. The audit reports were submitted on time. ADB found the audit reports acceptable. The auditors provided unqualified opinions for all the financial statements and project accounts (FY2007–FY2012). Findings mostly related to overpricing, overpayments, procurement-related matters, purchase of goods that were not in accordance with the specifications, and improper bookkeeping at the madrasahs. At project completion, the auditors’ recommendations had been followed up.

The Borrower shall enable ADB, upon ADB’s request, to discuss the Borrower’s financial statements for the Project and its financial affairs related to the Project from time to time with the auditors appointed by the Borrower pursuant to Section 4.02(a) here above, and shall authorize and require any representative of such auditors to participate in any such discussions requested by ADB, provided that such discussions shall be conducted only in the presence of an authorized officer of the Borrower unless the Borrower shall otherwise agree.

Article IV Section 4.02(b)

Complied with.

The Borrower shall enable ADB’s representatives to inspect the Project, the Goods and Works financed out of the proceeds of the Loan, and any relevant records and documents.

Article IV Section 4.03

Complied with.

The Ministry of Religious Affairs (MORA) shall be the executing agency and, with the support of the Directorate General of Islamic Education, shall be responsible for the overall management of the Project.

Schedule 5, para. 1

Complied with.

A project steering committee (PSC) shall be established prior to the Effective Date. The PSC shall provide guidance to the central project management unit (CPMU) on general policy directions, intersector coordination, and strategic directions. The PSC shall be chaired by the Secretary General of MORA and shall consist of representatives from MORA, the

Schedule 5, para. 2

Complied with. .

Appendix 12 47

Covenant

Reference in Loan

Agreement

Status of Compliance Ministry of Finance, the National Development Planning Agency (BAPPENAS), and the Ministry of National Education (MONE). The Director of the Directorate for Madrasah Education (DME) shall be the PSC Secretary. A project technical committee (PTC) shall be established and shall be chaired by the Director of Religious Affairs and Education of BAPPENAS and shall provide guidance to the CPMU on the overall project implementation. Members of the PTC shall include representatives from MORA, the Ministry of Finance, and MONE.

Schedule 5, para. 3

Complied with.

A CPMU shall be established within the Directorate General of Islamic Education prior to the Effective Date. The Director General of the GDIE shall be the Project Director and shall be responsible for providing guidance to the CPMU. The CPMU shall be responsible for (i) day-to-day project implementation; (ii) planning and budgeting; (iii) procurement; (iv) disbursement; (v) monitoring and supervision; (vi) overseeing project implementation at province, district, and madrasah levels; and (vii) submitting reports to the Borrower and ADB. The CPMU shall have about 15 staff, including MORA staff and consultants working together on each project component, supported by financed and administrative staff. CPMU shall receive support from contractual staff hired as specialists in training, quality assurance, monitoring and evaluation, civil works, and information and communication technology.

Schedule 5, para. 4

Complied with. The CPMU was supported by a project implementation consultant team and a number of individual consultants providing supports for the overall project management and implementation.

A provincial coordinating (PCC) shall be established in each of the project provinces to oversee project implementation at the province level. The PCC shall be chaired by the head of the provincial office of education (Dinas Pendidikan), the MORA provincial office (Kanwil Depag), the Madrasah Development Center (MDC), and the Madrasah Development and Empowerment Council.

Schedule 5, para. 5

Complied with.

A project coordinating unit (PCU) shall be established within each MORA provincial office to manage, coordinate, monitor, evaluate, and consolidate activities at the provincial and district levels. The PCU shall be headed by a staff member of the provincial MORA education division (Kepala Bidang Mapenda), assisted by finance and accounting, procurement and works, and monitoring and evaluation staff, and supported by consultants.

Schedule 5, para. 6

Complied with.

DCU shall be established within each MORA district office in each of the project districts. The DCU shall oversee implementation at madrasah level, particularly the madrasah development plan (MDP) block grants. Each of the DCU shall be headed by a staff member of the district MORA education section

Schedule 5, para. 7

Complied with.

48 Appendix 12

Covenant

Reference in Loan

Agreement

Status of Compliance (Kepala Seksi) and assisted by supervisors, facilitators, and finance and administrative staff. The Borrower shall provide counterpart funds for project implementation in a timely manner. The Borrower shall make timely submission of annual budgetary appropriation requests and ensure prompt disbursement of appropriated funds during each year of project implementation.

Schedule 5, para. 8

Complied with. Overall, counterpart funds provided were sufficient and released on time.

The Borrower shall ensure the smooth transfer of loan proceeds and counterpart funds from central level to the project provinces, project districts, and the Madrasah, in accordance with the flow of funds mechanism agreed between the Borrower and ADB.

Schedule 5, para. 9

Complied with.

Madrasah within the project districts shall be selected for eligibility for funding under the block grant based on (a) school size, (b) student poverty level, (c) teacher qualifications, and (d) geographical proximity.

Schedule 5, para. 10

Complied with. Madrasahs were selected during project preparation with close cooperation between ADB, with the assistance from the project preparatory technical assistance consultants, and MORA.

To be eligible for a block grant, a madrasah shall be required to develop a 4-year MDP and submit the MDP to the DCU for review, evaluation, and further processing. Approval of the submitted MDP shall be given by CPMU.

Schedule 5, para. 11

Complied with. All madrasahs prepared their individual MDPs, which were reviewed and approved by the CPMU.

Upon approval of the MDP, a Madrasah shall receive one block grant. The block grant shall be used to implement and support the activities proposed under the MDP. The MDP may cover various activities within the scope of project components.

Schedule 5, para. 12

Complied with. The MDPs were made as the basis for the transfer of the block grants. All 500 participating madrasahs received the block grants.

To ensure the smooth transfer of the block grants to each of the selected madrasah, immediately upon approval of a block grant, each madrasah will establish a bank account at a local branch of a commercial bank acceptable to ADB.

Schedule 5, para. 13

Complied with. All madrasahs opened individual bank accounts.

Not later than 9 months after the effective date, MORA shall establish a liaison office for the National Board of Education Standard and the National Board of School Madrasah Accreditation within the Secretary General’s or Director General’s office of MORA and the liaison office shall prepare a position paper on the strategic implications of implementing national school and madrasah accreditation regulations for madrasah within 3 months of its establishment.

Schedule 5, para. 14

This covenant was dropped because of a change in government regulations. This has no impact on the overall project implementation.

Not later than the end of the year 2007, the Borrower and MORA shall complete a draft 5-year strategic plan for implementing government education policies in Madrasah, which shall form the basis for dialogue on a harmonized MONE and MORA strategic plan as directed by Parliament.

Schedule 5, para. 15

Partly complied with. The strategic plan was completed in 2010. The delay did not affect the overall project implementation.

Appendix 12 49

Covenant

Reference in Loan

Agreement

Status of Compliance Not later than 12 months after the effective date, MORA shall design and standardize a comprehensive project monitoring information system (PMIS) for the different levels of management including the central Directorate of Madrasah Education, MDC, districts, and Madrasah and develop a users' manual.

Schedule 5, para. 16

Partly complied with. The PMIS was developed, but a comprehensive PMIS for all levels was not fully established.

A joint review shall be conducted by ADB and the Borrower at least twice a year. A midterm review shall be carried out during the third year of the Project. The reviews shall include (i) review of the project scope, design, implementation arrangements, institutional development and capacity building; (ii) assessment of project implementation against projections and performance indicators; (iii) review of compliance with loan covenants; (iv) identification on critical issues, problems, and constraints; and (v) recommendations for changes in project design or implementation as needed. One month after the review takes place, the CPMU shall submit to ADB a comprehensive report on each issue.

Schedule 5, para. 17

Complied with. Review missions were fielded twice a year.

To ensure that women benefit equally from the Project, the Borrower and MORA shall ensure that the Project is carried out in accordance with ADBs Policy on Gender and Development (1998) and the gender strategy contained in the Gender Action Plan that has been prepared and agreed between the Borrower and ADB. Specifically, the Borrower and MORA shall ensure that: (a) under component 1(a), at the national level, at least 30% of the participants in degree upgrading, and 40% for professional certification, are women; (b) under component 1(b), at the district level, female participation in the short-term training will be in accordance with the proportion of eligible teachers for each subject, at each level; (c) under Component 3(b), at the district level, at least 50% of the students receiving the scholarship shall be girls; and (d) under Component 4(c), at least three candidates for the master’s degree and the doctoral programs shall be women.

Schedule 5, para. 18

Partially complied with. Most targets under the gender action plans (GAP) were achieved. . Detailed discussions of GAP implementation and ahievements are provided in Appendix 7.

MORA shall create a project website to disclose information about various matters on the Project, including procurement related to the Project. With regard to procurement, the website shall include information on, among others, the list of participating bidders, name of the winner bidder, basic details on bidding procedures adopted, amount of contract awarded, and the list of goods and services procured.

Schedule 5, para. 19

Partly complied with. The website was established in 2010. However, it was not regularly updated. Information such as the list of participating bidders, name of the winning bidder, basic details on bidding procedures adopted, the amount of the contract awarded, and the list of goods and services procured was mostly not provided.

50 Appendix 12

Covenant

Reference in Loan

Agreement

Status of Compliance Within 9 months after the Effective Date, MORA shall establish within the CPMU a Complaints and Action Task Force to receive and resolve complaints/ grievances or act upon reports from stakeholders on misuse of funds or other irregularities. The Task Force shall (a) review and address grievances of stakeholders of the Project, in relation to either the Project, any of the service providers, or any person responsible for carrying out any aspect of the Project; and (b) set the threshold criteria and procedures for handling such grievances, for proactively and constructively responding to them, and for providing the stakeholders with notice of such mechanism.

Schedule 5, para. 20

Partly complied with. The Complaints and Action Task Force was established in 2010 (3 years after the effectiveness). The task force was mostly inactive.

Although no significant environmental impacts have been identified, the sitting, design, and construction and operation of school facility rehabilitation work undertaken under the Project will be implemented in line with the Governments environmental laws and regulations and ADB’s Environment Policy (2002).

Schedule 5, para. 21

Complied with. No significant environmental impacts were identified.

The Borrower and MORA and shall ensure that no block grant is approved if the rehabilitation or construction of new school facilities will involve involuntary resettlements according to ADB’s Policy on Involuntary Resettlement (1995). To be eligible for block grants, the Madrasahs are required to confirm that no land acquisition or resettlement is required under the Project. Construction of new classrooms shall be added to existing Madrasah on unoccupied land already owned by the Madrasah.

Schedule 5, para. 22

Complied with. No land acquisition or resettlement was required.

MORA shall ensure that the construction of two-story buildings complies with construction safety standards and is confirmed by a qualified engineer. Madrasahs shall not be allowed to construct a new classroom above an existing classroom without first obtaining confirmation from a qualified engineer that the existing classroom and its foundation is strong and solid enough for the additional upper-level classroom.

Schedule 5, para. 23

Complied with. A team of engineers was assigned to check and verify the buildings to ensure the compliance with the construction safety standards. The team concluded that the buildings constructed and financed under the project meet the standards.

Appendix 13 51

TECHNICAL ASSISTANCE COMPLETION REPORT

TA 4891-INO: Independent Monitoring and Evaluation of the Madrasah Education Development Project

Amount Approved: $995,000

Executing Agency Directorate General of Islamic Education in the Ministry of Religious Affairs (MORA)

Source of Funding ADB: $995,000 Government: $305,000

Amount Undisbursed: $19,254

Amount Utilized: $975,746

TA Approval Date:

TA Signing Date:

Fielding of First Consultant:

TA Completion Date Original: 31 March 2012

Actual: 30 April 2013

15 December 2006

12 June 2007 16 January 2008 Account Closing Date Original: 31 July 2012

Actual: 14 October 2013

Description. The Madrasah Education Development Project (MEDP) included a technical assistance (TA) grant. The TA commenced in 2008 with the aim to assist the government to establish a monitoring and evaluation (M&E) system to observe project activities in the 500 project madrasahs located in 27 districts, and to help raising public confidence in the quality, efficiency, and cost-effectiveness of madrasah education. The original TA completion date was 31 March 2012. However, due to delays in the project start-up, and the subsequent extension of the closing date of the MEDP, ADB and the executing agency agreed to extend the TA to ensure that the intended outputs, outcome, and impact could be achieved. Thus, the TA completion date was extended to 31 December 2012. In late 2012, to utilize the undisbursed TA proceeds, the executing agency requested ADB to finance a study on MEDP best practices as part of the overall project evaluation. To accommodate this request, the TA closing date was further extended to 30 April 2013. Expected impact, outcome, and outputs. The expected impact of the TA was increased results orientation and external focus for the project’s M&E systems. The expected outcome was improved results-based sector performance monitoring at the central level, and strengthened institutional capacity at the district and madrasah levels. The expected output was strengthened internal and external accountability and associated governance systems, with the following performance targets: (i) program, financial, quality compliance, progress monitoring, and benefits and impact evaluation conducted in 100% of randomly selected madrasahs by 2012; (ii) 50% of project madrasahs trained on M&E, financial management, and sector performance assessment by 2012; and (iii) 100% of randomly selected madrasahs audited annually. In general, the TA contributed as expected to the impact and outcome, and achieved the output targets as specified in the design and monitoring framework. Delivery of inputs and conduct of activities. In 2008, ADB contracted a firm to provide one international expert for 12 person-months intermittently; and four national experts, including a monitoring and evaluation specialist, a quality assessment specialist, a financial and compliance monitoring specialist, and a database design specialist. Due to poor performance, the international expert was replaced after 2 months. However, the performance of the replacement expert was also considered unsatisfactory. Thus, it was agreed not to engage a third international expert, but instead to utilize the remaining resources for international expertise to extend the national expert team. In 2011, the firm requested to resign from the assignment. ADB agreed and engaged the team of national experts directly through individual contracts. The performance of the national experts was appreciated by the executing agency and considered satisfactory throughout the TA. The executing agency provided office space for the consultant team and supported their activities appropriately. Of the $995,000 TA proceeds allocated, $975,746 were disbursed (98%) and $19,254 were cancelled. The expert team was able to establish an effective information system to monitor the utilization of block grants at the 500 project madrasahs. Data on the utilization of block grants related to (i) rehabilitation and

52 Appendix 13

construction of classrooms, libraries, laboratories, and sanitation facilities; (ii) procurement of equipment, teaching aids, textbooks, and furniture; (iii) training of madrasah teachers; (iv) student remedial programs; and (v) scholarships for poor students was collected timely and in a consistent manner. To verify information received from project madrasahs, the team conducted regular field visits to discuss the project progress with principals, teachers, students, parents, and community members. The findings of the field visits and information collected through the regular M&E mechanism were summarized in periodic monitoring and evaluation reports, which were considered useful by the executing agency for overseeing the activities at the project madrasahs. The team also conducted training of madrasah staff on data collection, accounting, and preparation of financial reports; and provided advice on establishing simple and efficient madrasah-based M&E systems to track the overall progress of project-financed activities as well as student achievements. Furthermore, the TA team provided valuable advice to the CPMU on revising the project implementation guidelines to manage block grants. In addition, the team conducted (i) two detailed student achievement evaluations, (ii) a parent and community attitude surveys, and (iii) an appraisal of initial impacts at the project madrasahs. Findings of the student achievement evaluations confirmed that students from project madrasahs achieved higher scores in key subjects during the project implementation period. The parent and community attitude surveys demonstrated that parents and the community consider that the project has improved teaching and learning environments and teacher performance, and that project madrasahs have better and more complete facilities than nearby schools. Many parents and community members commented on their overall satisfaction with the improvements of the upgraded madrasah infrastructure and facilities, and the improved teaching, which encouraged other families from the community to send their children to madrasahs. Using unutilized funds, the ADB contracted two individual experts to conduct final project review and summarize the achievements and lessons from the project. The findings and recommendations of the report are considered useful by MORA for developing strategies to replicate the project activities using government resources. Evaluation of outputs and achievement of outcome. The TA was successful in providing effective support to help the executing agency monitor project implementation and evaluate the initial impact of project interventions. The TA also succeeded in (i) strengthening institutional capacity of madrasah management and committees to facilitate and oversee the upgrading activities financed by the block grants, and (ii) establishing straightforward and practical M&E mechanisms at the project madrasahs. In retrospect, the outcome related to improving results-based sector performance monitoring at the central level was too ambitious. The M&E activities were focused on the implementation of activities at the project madrasahs. The audits by Indonesia’s Agency for Control and Finance Development on the madrasahs’ project accounts suggested that, overall, the project madrasahs complied with the project institutional integrity policy. This is evidenced by the unqualified opinion for the all project accounts of the MEDP (during 2008–2013). The project’s main components—upgrading of physical facilities, teacher development, and community participation—are mainstream activities of the government. The education sector (including madrasah education) will continue to receive 20% of public expenditure, as required by Indonesia’s Constitution. Since 2009, the government has met and surpassed the 20% target. It is anticipated that the newly upgraded facilities under the project will be sustained because the ownership by the community surrounding madrasahs is generally high and sufficient resources required for the operation and maintenance will be provided.

Appendix 13 53

Overall Assessment and Rating. Overall, the TA was rated successful in achieving its envisaged outputs and outcome. The TA established an effective M&E system to monitor the utilization of the block grants at the 500 project madrasahs, which was considered useful by the executing agency. The student achievement evaluations, parent and community attitude surveys, and appraisal of initial impacts at the project madrasahs provided additional valuable information for monitoring project progress, addressing challenges, and determining the benefits of the project interventions. The TA was efficient in utilizing its resources. The savings from the TA from reduced inputs for international consultancies were used to extend the contracts of national consults conduct a final review of project achievements.

Major Lessons. MORA demonstrated strong ownership of the TA by making sufficient financial, human, and technical resources available. The key contributor to the success of the TA was the good interaction with the project madrasahs. The TA team established clear and simple guidelines for data collection at the beginning of the project, and provided good advice to project madrasahs to ensure that key data were collected consistently and in a timely manner. While the regular M&E activities collected precise information on the utilization of the block grants and summarized project inputs (e.g., construction and training), the student achievement evaluations and parents and community attitude surveys were considered useful for monitor the effects of the project interventions on the quality of education at the project madrasahs. Recommendations and follow-up actions. Generally, the establishment of clear M&E indicators and straightforward data collection and reporting mechanisms are essential to determine the challenges of project implementation and address upcoming issues early on. To get information from the community level in the decentralized context of Indonesia, efficient reporting arrangements are essential and should be set up at the beginning of a project. Involvement of district administrations in M&E activities is useful to support data collection and verify information. Student achievement evaluations and parent and community attitude surveys were useful tools for obtaining additional feedback on the impact of the project interventions and public perception of the project. No immediate follow-up actions are foreseen. Based on the success of the overall project and the good M&E support provided under this TA, the government is considering replicating the project with financing from its own resources. At present, the government is not considering requesting further loans from development partners for basic education.

Prepared by: Wolfgang Kubitzki Designation: Principal Portfolio Management Specialist, IRM Siti Hasanah Senior Project Officer, IRM In preparing any country program or strategy, financing any project, or by making any designation of or reference to a particular territory or geographic area in this document, the Asian Development Bank does not intend to make any judgments as to the legal or other status of any territory or area.

54 Appendix 14

ECONOMIC AND FINANCIAL REEVALUATION

1. The project supported 500 madrasahs in 27 districts in the provinces of Central Java, East Java, and South Sulawesi. It included four outputs: (i) improved teacher professionalism; (ii) upgraded teaching and learning resources and facilities; (iii) increased internal efficiency; and (iv) strengthened governance, management, and sustainability to meet national standards. The major beneficiaries of the project were about 150,000 students, who were enrolled at the madrasahs during the project implementation period, and more than 10,000 teachers, who participated in various training programs. Most of the investment was used to upgrade the teaching and learning environments of project madrasahs, and strengthen teaching methods and content knowledge of teachers employed at the project madrasahs. Of the $50.2 million disbursed loan proceeds, $41.1 million was used for block grants that were transferred directly to project madrasahs to finance (i) rehabilitation and construction of classrooms, libraries, laboratories, and sanitation facilities; (ii) procurement of equipment, teaching aids, textbooks, and furniture; (iii) training of madrasah teachers; (iv) student remedial programs; and (v) small scholarships for poor students. Generally, the project resources were used efficiently. More than 95% of the total loan amount allocated for the project was disbursed to finance investments under the four outputs. 2. The economic rationale for public investment in basic education and the link between education and employment is well established. In general, studies confirmed that there are strong and consistent private returns to investment in education. However, quantifying benefits in economic terms is difficult as education by itself is not sufficient to raise incomes. Job opportunities and salary structures depend on various other economic factors. Considering that approximately 18% of Indonesia’s primary and junior secondary education institutions are madrasahs, which to a large extent serve poor communities, there is a general consensus that there is an unambigious role of the government to support madrasah education because it contributes to poverty alleviaton and the reduction of income inequality. Targeting disadvantaged communities through education programs is considered essential to promote inclusive growth and sustainable ecomonic development, and generates spillover benefits in improving citizenship and civic knowledge. For discussion purposes the following scenarios are considered: 3. The private rate of return to an additional year of schooling is typically about 10%. Estimated social rates of return are generally lower than private ones, higher for women, and higher in remote and underdeveloped areas. Generally, it can be assumed that the poor display lower education attainment than the nonpoor, and in particular lack managerial and basic technological skills required by the labor market. The project included pro-poor measures particularly aimed at improving enrollment and completion rates of students from poor families. The majority of the project madrasahs were located in remote and/or poor communities. Most of the madrasahs had not received any major support previously and depended on limited contributions from the communities, which were inadequate to develop environments conducive to teaching and learning. 4. The project had a major impact on improving teaching and learning environments at disadvantaged madrasahs, which would otherwise not have had the resources to upgrade the quality of their education services. Observations during review missions and feedback from monitoring reports confirmed that madrasah students and parents had a positive attitude toward the project and its impact on learning results. The positive attitude resulted in increased enrollment rates and improved academic performance of students in the national exams at all levels. This, in turn, provides the foundation for better income-earning and employment opportunities and opens prospects for further advanced education. Disadvantaged students from poor families, who had no access to quality education before the project, benefited from project interventions particularly

Appendix 14 55

through (i) after-school remedial programs to improve learning outcomes and to reduce the repetition and dropout rates, and (ii) scholarships for transition programs to help students to continue to the next level of education. 5. Assuming that (i) about 15,000 graduates from project madrasahs will enter the labor market annually and engage in regular income-earning activities in the formal or informal sector, and (ii) the benefit from the improved teaching and learning environments created by the project will increase their annual earnings by 10%, then, based on an annual average income of $1,500 (slightly above the poverty line), the estimated economic benefit of the project would amount to $2.25 million per year. 6. Cost Effectiveness. The project adopted a community-driven development (CDD) approach. To determine the investment needs to be financed by block grants, the management of madrasahs, teachers, and community members jointly prepared individual madrasah development plans (MDPs), which were based on comprehensive self-assessments and included detailed investment plans with prioritized upgrading and training needs. Simple construction works to build and rehabilitate classrooms, laboratories, sanitation facilities, and basic madrasah clinics were mainly carried out by community members. Procurement of equipment, books, and teaching aids was done by the madrasah committees with the help of facilitators. Funds to implement activities in an approved MDP were transferred directly to the madrasah account. The participatory approach to develop the MDPs, jointly carry out upgrading measures, and oversee the use of the project funds at the madrasah level, created a strong sense of ownership and contributed to the sustainable maintenance, and even further development, of the upgraded facilities. 7. CDD approaches provide good-quality community infrastructure at significantly lower costs than technically comparable projects that use traditional (non-CDD) modes of government contracting. Evaluations of CDD infrastructure projects in Indonesia using local government unit costs and local contractor costs demonstrated that the investment costs of CDD-provided infrastructure were about 25% lower on average. Given that the project spent about $38 million on infrastructure investments to upgrade and rehabilitate project madrasahs mainly using the CDD approach, and considering that these investments cost 75% of conventional contracting costs for similar infrastructure, the savings are estimated at $10.0 million.1 The participatory involvement of communities also ensured that of transparent procurement and disbursement mechanisms were established, which contributed to cost effectiveness and mitigated potential risks of corruption. 8. Financial sustainability. The project is considered to be financially sustainable, as the bulk of the expenditures do not continue beyond the end of the project. Recurrent expenditures, such as on maintenance of facilities and equipment, will be absorbed using additional revenue from government funding, and community contributions. Observations during the project completion mission confirmed that contributions to private madrasahs from private foundations, parents, and other community members have increased due to the positive perception of the improved learning and teaching environments at the project madrasahs. The Ministry of Religious Affairs also confirmed that it would provide sufficient budget to maintain the investments at the project madrasahs and has allocated additional funding for further training measures to continuous professional development of madrasah teachers, which will benefit teachers from public and private madrasahs. Overall, the government is committed to fulfilling its constitutional obligation to direct 20% of the public expenditure to education, thus the education budget of the Ministry of Religious Affairs will be secure in the future.

1 The total amount of $38 million consisted of about $35 million from loan proceeds and $3.1 million from cash and in-

kind contributions from the communities.