implementation completion and results …...non-formal approach to training education and jobs in...

Document of

The World Bank FOR OFFICIAL USE ONLY

Report No: ICR00004588

IMPLEMENTATION COMPLETION AND RESULTS REPORT

TF-16354

ON A

GRANT

IN THE AMOUNT OF US$ 15 MILLION

TO THE

ISLAMIC REPUBLIC OF AFGHANISTAN

FOR THE

NON-FORMAL APPROACH TO TRAINING EDUCATION AND JOBS IN AFGHANISTAN PROJECT

April 30, 2019

Education Global Practice South Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS

(Exchange Rate Effective Jan 24, 2019)

Currency Unit = Afghani

Afghani 75.90 = US$1

US$ 1.30 = SDR 1

FISCAL YEAR December 21 – December 20

Regional Vice President: Hartwig Schafer

Country Director: Shubham Chaudhuri

Senior Global Practice Director: Jaime Saavedra Chanduvi

Practice Manager: Mario Cristian Aedo Inostroza

Task Team Leader(s): Nathalie Lahire, Palwasha Mirbacha

ICR Main Contributor: Gerard Peart

ABBREVIATIONS AND ACRONYMS

AFN Afghani (Currency) AREDP Afghanistan Rural Enterprise Development Project ASDP Afghanistan Skills Development Project CMU Country Management Unit CPF Country Partnership Framework DG Director General DGSD Directorate General of Skills Development DPs Development Partners ESCs Employment Service Centers ESMF Environment and Social Management Framework FM Financial Management GDP Gross Domestic Product GIRoA Government of the Islamic Republic of Afghanistan GIZ German Agency for Technical Cooperation GMF Grant Management Firm GRM Grievance Redressal Mechanism HR Human Resources IADC Italian Agency for Development Cooperation IC International Consultant ICR Implementation Completion and Result Report IEs Impact Evaluations IFR Interim Financial Report IRR Internal Rate of Return ISN Interim Strategy Note LMIS Labor Market Information System LOU Letter of Understanding M&E Monitoring and Evaluation MoF Ministry of Finance MOLSAMD Ministry of Labor, Social Affairs and Martyred and Disabled MTR Mid-Term Review NC National Consultant NGOs Non-governmental Organizations NOSS National Occupational Skills Standards NPP1 National Priority Program 1 NPV Net Present Value NSDP National Skills Development Program ODA Official Development Assistance OP/ BP Operational Policy/Bank Procedure ORAF Operational Risk Assessment Framework PAD Project Appraisal Documents PDO Project Development Objective PIM Project Implementation Manual

SEA Safeguards and Environmental Assessment SGs Savings Groups SMEs Small and Medium Enterprises STEP Systematic Tracking of Exchanges in Procurement TA Technical Assistance TOC Theory of Change TPs Training Providers TTTI Technical Teachers Training Institute TVET Technical Vocational Education and Training UNESCO United Nations Educational, Scientific and Cultural Organization USAID US Agency for International Development WB World Bank

Table of Contents DATA SHEET .......................................................................................................................... 1

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5

A. CONTEXT AT APPRAISAL .........................................................................................................5

Context .................................................................................................................................................... 5

Theory of Change (Results Chain) ............................................................................................................. 5

Project Development Objectives (PDOs) .................................................................................................. 7

Key Expected Outcomes and Outcome Indicators .................................................................................... 7

Components ............................................................................................................................................. 7

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .......................................9

Revised PDOs and Outcome Targets; PDO indicators; and Components .................................................. 9

Other Changes ......................................................................................................................................... 9

Rationale for Changes and Their Implication on the Original Theory of Change ....................................... 9

II. OUTCOME .................................................................................................................... 10

A. RELEVANCE OF PDOs ............................................................................................................ 10 Assessment of Relevance of PDOs and Rating ........................................................................................ 10

B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 10 Assessment of Achievement of Each Objective/Outcome ...................................................................... 10

Justification of Overall Efficacy Rating .................................................................................................... 11

C. EFFICIENCY ........................................................................................................................... 12

Assessment of Efficiency and Rating ...................................................................................................... 12

D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 13

E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 13 Gender ................................................................................................................................................... 13

Institutional Strengthening .................................................................................................................... 13

Poverty Reduction and Shared Prosperity .............................................................................................. 14

Other Unintended Outcomes and Impacts ............................................................................................. 14

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 14

A. KEY FACTORS DURING PREPARATION ................................................................................... 14

B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 16

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 20

A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 20 M&E Design ........................................................................................................................................... 20

M&E Implementation ............................................................................................................................ 21

M&E Utilization ...................................................................................................................................... 22

Justification of Overall Rating of Quality of M&E ................................................................................... 22

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 22

C. BANK PERFORMANCE ........................................................................................................... 23

Quality at Entry ...................................................................................................................................... 23

Quality of Supervision ............................................................................................................................ 24

Justification of Overall Rating of Bank Performance............................................................................... 25

V. LESSONS AND RECOMMENDATIONS ............................................................................. 25

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 28

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 35

ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 37

ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 38

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 41

ANNEX 7. RESTRUCTURING AGREED AT THE MID-TERM REVIEW .......................................... 42

ANNEX 8. RELEVANCE OF PDO ............................................................................................. 44

ANNEX 9. THE BUSINESS GRANT SCHEME (SUB-COMPONENT 3.2) ........................................ 45

ANNEX 10. COMPONENT-WISE ACTIVITIES AND RESULTS ..................................................... 49

ANNEX 11. IMPLEMENTATION AGRANGEMENTS .................................................................. 51

ANNEX 12. PROCUREMENT OF MAJOR TA PACKAGES .......................................................... 55

ANNEX 13. THE NATEJA PROJECT FUNDED BY IADC .............................................................. 58

ANNEX 14. MONITORING AND EVALUATION ........................................................................ 60

The World Bank Non-Formal Approach to Training Education and Jobs in Afghanistan Project (P146015)

Page 1 of 60

DATA SHEET

BASIC INFORMATION Product Information Project ID Project Name

P146015 Non-Formal Approach to Training Education and Jobs in Afghanistan Project

Country Financing Instrument

Afghanistan Investment Project Financing

Original EA Category Revised EA Category

Partial Assessment (B) Partial Assessment (B)

Organizations

Borrower Implementing Agency

Islamic Republic of Afghanistan Ministry of Labor and Social Affairs

Project Development Objective (PDO)

Original PDO The Project Development Objective is to increase the potential for employment and higher earnings of targeted young Afghan women and men in rural and semi-urban areas through non-formal skills training.


Page 2 of 60

FINANCING

Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing TF-16354 15,000,000 5,529,046 5,529,046

Total 15,000,000 5,529,046 5,529,046

Non-World Bank Financing 0 0 0

Borrower/Recipient 3,000,000 0 0

Total 3,000,000 0 0

Total Project Cost 18,000,000 5,529,046 5,529,046

KEY DATES

Approval Effectiveness MTR Review Original Closing Actual Closing 24-Mar-2014 11-Apr-2014 24-Nov-2016 30-Dec-2018 31-Oct-2018

RESTRUCTURING AND/OR ADDITIONAL FINANCING

Date(s) Amount Disbursed (US$M) Key Revisions 15-Oct-2018 6.01 Change in Loan Closing Date(s)

KEY RATINGS

Outcome Bank Performance M&E Quality

Unsatisfactory Unsatisfactory Modest

RATINGS OF PROJECT PERFORMANCE IN ISRs

No. Date ISR Archived DO Rating IP Rating Actual

Disbursements (US$M)

01 28-Jun-2014 Satisfactory Satisfactory 1.00

02 24-Dec-2014 Satisfactory Satisfactory 1.09


Page 3 of 60

03 23-Jun-2015 Moderately Satisfactory Moderately Satisfactory 1.70

04 13-Jan-2016 Moderately Satisfactory Moderately Satisfactory 1.94

05 11-Aug-2016 Moderately Satisfactory Moderately Satisfactory 2.50

06 07-Feb-2017 Moderately Unsatisfactory Moderately Unsatisfactory 3.88

07 30-Aug-2017 Moderately Unsatisfactory Moderately Unsatisfactory 5.59

08 09-Apr-2018 Unsatisfactory Unsatisfactory 6.01

09 24-Oct-2018 Unsatisfactory Unsatisfactory 6.01

SECTORS AND THEMES

Sectors Major Sector/Sector (%)

Education 100

Public Administration - Education 19 Workforce Development and Vocational Education 81

Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 100

Jobs 100

Human Development and Gender 102

Education 34 Access to Education 17

Education Financing 17

Labor Market Policy and Programs 68

Labor Market Institutions 34

Active Labor Market Programs 34


Page 4 of 60

ADM STAFF

Role At Approval At ICR

Regional Vice President: Philippe H. Le Houerou Hartwig Schafer

Country Director: Robert J. Saum Shubham Chaudhuri

Senior Global Practice Director: Jesko S. Hentschel Jaime Saavedra Chanduvi

Practice Manager: Amit Dar Mario Cristian Aedo Inostroza

Task Team Leader(s): Leopold Remi Sarr Nathalie Lahire, Palwasha Mirbacha

ICR Contributing Author: Gerard Peart


Page 5 of 60

I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES

A. CONTEXT AT APPRAISAL

Context 1. At the time of Project appraisal (2013), Afghanistan had 12 million young people aged 18-35 years, constituting 35-40 percent of the total population. The real GDP growth rate for the period 2003-12 had been 9 percent annually. The country had experienced years of significant post-conflict political instability and insecurity that had adversely affected its economy and society. There were concerns that instability and insecurity might heighten once international military forces left in 2014 (as they were then scheduled to do); and that the global financial crisis might negatively affect economic growth. The economy’s growth potential was thought to be constrained by the low levels of literacy and technical skills particularly among the young. Average real GDP growth was anticipated to be 4-6 per cent annually for the period 2013-18. 2. Over 80 percent of Afghan workers were employed in the non-formal economy; the vast majority of them lacked basic technical and literacy competencies, as suggested by employer surveys. Each year, approximately 400,000 young men and women entered the labour market with few qualifications. As a result, the bulk of young people were unemployed or underemployed and unable to break out of a low income, low productivity trap. At the same time, the Project Preparation Facility Grant conducted surveys indicating there was substantial demand for literate unskilled/semi-skilled labor; though employers had few or no links with Training Providers (TPs). 3. Demand for non-formal training was thought to be robust, and tracer studies conducted by National Skills Development Program (NSDP) in 2012 in the provinces of Badakhshan, Bamiyan, Samangan, Kandahar and Khost found high levels of graduate placement. There were concerns about the quality of non-formal technical and vocational training, as well as a mismatch between course offerings and market requirements. In the non-formal sector, apprenticeships were the predominant form of skills training, as well as a major supply channel for semi-skilled and skilled labor. However, apprentices were found to lack basic literacy and numeracy competencies; to have inadequate training in their fields; and to work in suboptimal conditions. There was low implementation capacity at the Ministry of Labor, Social Affairs and Martyred and Disabled (MOLSAMD), with inadequate staffing on skills development and the lack of a credible labor market information system. 4. To address these issues, the Government of the Islamic Republic of Afghanistan (GIRoA) had policies emphasizing skills development. These included the Afghanistan National Development Strategy (ANDs 2009-13), the MoLSAMD Strategic Plan (2013-15), and the National Priority Program 1 (NPP1) which all included measures to strengthen skills development and employment policies with a view to promoting job-rich growth. Theory of Change (Results Chain) 5. The Theory of Change (TOC) has been constructed from the results framework and project description. The project planned a set of activities pertaining to skills training and business development that in turn would increase work- and business-related knowledge, skills and resources of youth. In Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees), training providers would be incentivized to ensure that their students


Page 6 of 60

are certificated, find work or start a business. In Component 3 (Piloting Entrepreneurship among Unskilled and Illiterate Young Afghans), savings groups members would be trained and receive business development advice; eligible youth would receive business grants, training and support; and the informal apprenticeship system would be analyzed with a view to providing training. In parallel, under Component 2 (Project Management, Capacity Building of MoLSAMD and M&E), the capacities of the National Skills Development Program (NSDP) and MoLSAMD would be built in various functions including fiduciary, labor information systems, and monitoring and evaluation. (Details on the components are in the next section). These activities would increase NSDP/MoLSAMD capacities in project management, M&E and labor market analysis. Together, the outputs of all the components would increase the potential for employment and higher earnings among young women and men in areas targeted by the project. These outcomes would in turn contribute to longer-term outcomes pertaining to increased human capacity, economic activity and gender equality. The ToC has also been captured in the diagram below.


Page 7 of 60

Project Development Objectives (PDOs) 6. The PDO was to increase the potential for employment and higher earnings of targeted young Afghan women and men in rural and semi-urban areas through non-formal skills training.

Key Expected Outcomes and Outcome Indicators 7. The PDO was to be achieved through two expected objectives, which had four PDO indicators.

Objective 1: Increased potential for employment among targeted young Afghan women and men in rural and semi-urban areas. Three linked indicators:

• 1.1: Share of trainees who find employment within 6 months of graduation, with respect to control groups

(to be increased from a baseline of 0 percent to 15 percent) • 1.2: The amount of incentive(s) disbursed to training providers (to be increased from zero to USD 4M) • 1.3: Number of beneficiaries and share of female beneficiaries (to be increased from zero to 44,500; female

share to be increased from 20 percent to 30 percent)

Objective 2: Increased potential for higher earnings among targeted groups. One linked indicator:

• 2.1: Increase in trainees’ earning, with respect to control groups (to be increased from 0 to 15 percent) 8. According to the PAD, the indicators 1.1, 1.2 and 2.1 pertain only to results of Components 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes) and Sub-Component 3.1 (Encouraging Rural Employment); while the indicator 1.3 also covers Component 3.2 (Promoting Entrepreneurship among Afghan Youth).

Components 9. Component 1: Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees. A total cost of US$ 7.7 million was estimated at appraisal; US$ 1.05 million were actually disbursed. 10. The component was designed to induce non-formal training providers to improve the quality of their training and the placement of their graduates. An incentive of US$ 100 was to be granted to non-formal TPs for each graduate who was certified or found a job within six months of graduation or started a business.1 TPs were to be selected from across the country, with some preference given to those who offered courses exclusively for women or had a high percentage of women in their regular training programs. Teachers were to be trained at the Technical Teachers’ Training Institute (TTTI) and contracted to provide tailored short-term courses at some of the non-formal TPs.2 Courses were to be aligned with the National Occupational Skills Standards (NOSS), which were to be used for graduates’ certification.3

1 PAD, pp. 7 and 24. A timeframe for starting the business was not indicated in the PAD. The design of the incentive scheme was modified in the Project Implementation Manual (PIM), with USD 50 being allocated upon certification (passing the NOSS skills test) and USD 50 upon employment or self-employment. See paragraphs 136 and 137. 2 The TTTI was to be established under a Bank-financed project, the Afghanistan Second Skills Development Project (ASDP II). 3 The NOSS were being developed by NSDP. They are publicly available competency standards for occupations, and should guide the


Page 8 of 60

11. Component 2: Project Management, Capacity Building of MOLSAMD and M&E. A total cost of US$3.6 million was estimated at appraisal; US$ 2.04 million were actually disbursed. 12. This component was designed to build MOLSAMD capacities in procurement, financial management, Labor Market Information System (LMIS) and monitoring. Capacities were also to be built at the NSDP,4 a national program responsible for skills training with multiple projects (see Annex 11), including the Bank-supported Afghanistan Second Skills Development Project (ASDP II). Its overall management capacity was to be built (including in the areas of channeling financial incentives, fiduciary, and safeguards); as were its capacities in monitoring and evaluation, including the sustained monitoring of project indicators, the conduct of third-party validation exercises, and the design and implementation of randomized experiments for evaluating impact. The component also covered project management costs including salaries, incremental operating costs and expenses for monitoring and evaluation. 13. Component 3: Piloting Entrepreneurship among Unskilled and Illiterate Young Afghans. A total cost of US$ 3.7 million was estimated at appraisal; US$ 2.43 million were actually disbursed. 14. There were three Sub-Components. The first Sub-Component 3.1 (Encouraging Rural Employment) was designed to provide customized training in hard and/or soft skills as well as functional literacy to members of Savings Groups (SGs);5 the training to be customized based on the findings of a baseline survey. The possibility was also held out that interested SG members would receive guidance on establishing small businesses and links to finance programs. The TPs (providing the customized training) were to receive US$ 100 for each trainee who was certified or found a job within six months of graduation or started a business. 15. The second Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) was designed to provide grants of US$ 500 on a competitive basis to 1,890 individuals of three provinces to help them start their own business, and to 630 individuals to expand an existing business.6 The grants were targeted at females and males aged 18-35 years with an entrepreneurial mindset who were mostly illiterate and unemployed/underemployed. The applicants were to develop business plans with assistance from MOLSAMD. The Project was to provide successful candidates with counselling and mentoring support during implementation of their plans, as needed.

development of training materials so that courses provided by any TP enable students to demonstrate their ability to meet the standards. The certification was to be conducted by an agency to be created under the ASDP II. 4 As per NATEJA Grant Agreement, MOLSMD was to maintain throughout the period of the project the NSDP within MOLSAMD and it will be directly responsible for implementation of the Project, coordination of implementation arrangements among departments, agencies, ministries and development partners involved in the Project and provide the technical and capacity building support to Project stakeholders and beneficiaries. 5 The SGs were set up in five provinces as part of a Bank-financed project, the Afghanistan Rural Enterprise Development Project (AREDP). 6 Proposed three provinces were Kabul, Nangarhar and Balkh.

The National Skills Development Program

At project preparation, the NSDP was one of the National Priority Programs still operating that had been created by a former President of the GIRoA in 2004. It provided technical assistance to the TVET system as well as vocational education trainings nationwide with the support of MOF and Development Partners including the World Bank. It functioned as an independent structure and separate program at MoLSAMD, with project staff hired for fixed periods to carry out program activities; its functions were technical and fiduciary, giving it substantial control over program design and execution.


Page 9 of 60

16. The third Sub-Component 3.3 (Feasibility Study on Apprenticeship Program) was to carry out a feasibility study to assess whether there was a need for short-term practical apprenticeship training and how such training might be piloted using various options. If the study found that piloting was viable, then the most promising option was to be finalized and implemented in selected provinces.

B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)

Revised PDOs and Outcome Targets; PDO indicators; and Components

There were no significant changes

Other Changes 17. There was an agreement between the GIRoA and the Bank at the mid-term review (November 2016) to restructure the Project, to address poor project implementation and unsatisfactory progress towards the intermediate and PDO indicators’ targets. The restructuring would have led to a revision of the PDO and PDO indicators including targets; a redesign of certain features of Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees); the cancellation of Sub-Components 3.1 (Encouraging Rural Employment) and 3.3 (Feasibility Study on Apprenticeship Program) and the re-allocation of unspent funds from these two sub-components to Component 1.7 However, subsequent discussions between and within the Bank Country Management Unit and senior officials of the GIRoA focused on the relative merits of restructuring as per the MTR recommendations versus an early closing of the Project altogether and a cancellation of the remaining funds. During the year and a half following the MTR, it was consequently unclear whether the Project would be restructured or closed. In that time, progress was made on Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) and Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), but little was achieved on the other components which all had certain design constraints that were to be addressed through a restructuring based on the MTR recommendations. Finally, in July 2018, the MoF formally requested the Bank to close the Project early and cancel unspent funds. A restructuring paper to this effect was prepared to enable an amendment to the Grant Agreement, and the Project closed two months early on October 31st, 2018. See also Annex 7.

Rationale for Changes and Their Implication on the Original Theory of Change

18. The early closure and cancellation of unspent funds were undertaken due to the Project’s poor implementation rating, high overall risk, and failure to make adequate progress towards its objectives. At the same time, both the GIRoA and the Bank were motivated to engage in a broader discussion on Afghan Youth as a new emerging priority of the country.

7 See Section III.B and Annex 7 for an explanation of why it was proposed that sub-components 3.1 and 3.3 be cancelled, and that the bulk of their activities were not implemented.


Page 10 of 60

II. OUTCOME A. RELEVANCE OF PDOs

Assessment of Relevance of PDOs and Rating

19. Relevance of objectives is rated high. At closing (as at preparation), the PDOs remain relevant to the Bank’s Country Partnership Framework (CPF - FY 17-20). That is, in their focus on improved employment and earnings through targeted vocational training, the PDOs directly support CPF Pillar 3 (Social Inclusion), Objective 3.1 to improve human development including through improved access to inter alia skills training. The intended outcomes by the end of the CPF in education include an improvement in the quality and employability of graduates from skills training institutes. The Project was explicit in its intention that the components would improve access to skills training (including technical, business and literacy skills); while also improving the quality of the training itself through strengthened curricular inputs and capacity building of vocational trainers. The PDOs also support the WBG’s gender strategy in Afghanistan, which involves inter alia empowering women economically in key markets, enhancing their roles as producers in the rural economy, and increasing opportunities for employment and entrepreneurial activities. The PDO and intermediate indicators, as well as the component activities, were designed to ensure that the Project worked to empower women in non-formal and rural economy, by providing them with the skills and finance required to enhance their potential to develop businesses and find skilled employment. The PDOs are also aligned with the 2017-2020 Afghanistan National Peace and Development Framework (ANPDF). See Annex 8.

B. ACHIEVEMENT OF PDOs (EFFICACY)

Assessment of Achievement of Each Objective/Outcome 20. The assessment is organized around the two key outcomes. With respect to the first outcome - increased potential for employment among targeted young Afghan women and men in rural and semi-urban areas - using the three PDO indicators:

• The share of trainees under Component 1 (Improving the Quality of Non-formal Training and Labor Market

Outcomes) and Sub-Component 3.1 (Encouraging Rural Employment) who found employment within 6 months of graduation, with respect to control groups, remained at the baseline value of 0 percent (as against a target of 15 percent) because no TPs/trainees were approved for inclusion in the Project under those components. However, this indicator does not consider employment gains under Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth). As amplified in Section IV.A, given the obvious alignment between the employment promise of the PDO and the employment gains intended and achieved under Sub-Component 3.2, these gains must be considered. As a result of the business development grant, training and mentoring, the 2,482 Sub-Component 3.2 beneficiaries (of whom 30% female) achieved an employment rate of 88 percent, a statistically significant 13 percentage points higher than the control group. The gains for women were even more impressive, with a statistically significant 33 percentage points higher than the control group.

• The amount of incentive(s) disbursed to TPs remained at the baseline value of zero (as against a target of USD 4M), again because no TPs/trainees were approved to receive incentives under the Project.


Page 11 of 60

• The number of beneficiaries increased from zero at baseline to 2,482 (as against a target of 44,500); while the share of female beneficiaries at closure was 30 percent, meeting the end target. The beneficiaries were solely business grant recipients under Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth). Given the Afghan context, the achievement of the gender target on this basis is significant.

• Given that there was some progress against the outcome as measured by employment and the number of beneficiaries, and further that a substantial portion of them were women (thereby meeting the 30% gender target), the rating for achievement of this outcome is modest.

21. With respect to the second outcome - increased potential for higher earnings among targeted groups- using the PDO indicator:

• No TPs/trainees were approved for inclusion under Components 1 and 3.1 of the Project. That is, there was no increase in trainees’ earning (as against a target increase of 15 percent) as measured by the PDO indicator. However, as argued above, gains accruing to Component 3.2 beneficiaries must be considered. With respect to earnings, the 2,482 beneficiaries of Component 3.2 earned approximately USD 11.70/month more in net income than the control group as a result of the Grant Scheme; a statistically significant amount equivalent to 17 percent of their pre-Scheme monthly earnings. Thus, the rating for achievement of this outcome is also modest.

22. Three of the four indicators - pertaining to employment (indicator 1.1), incentives (1.2), and income (2.1) - are not fully satisfactory measures of the Project’s development outcome, in that they did not capture the employment and income resulting from the incentives and training provided under Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth). The ICR team is of the view that the outcomes achieved under Sub-Component 3.2 contribute to the PDO achievement, albeit on a modest scale in relation to the overall outcomes intended by the PDO. Further, on the strength of the design and results of the Scheme, the Italian Agency for Development Cooperation (IADC) is funding a similar intervention with 2,000 beneficiaries; while the MoLSAMD has included in its budget request for the fiscal year 1398 funds to enable such grants for graduates of TVET programs. That is, having looked beyond the Results Framework to other available data and evidence and found positive evidence of achievement against the PDO outcome, the ICR Team judges that the result is sufficient to warrant an efficacy rating of modest. See also Annexes 9 and 10. 23. While the ICR focuses on the extent to which the project’s objectives were achieved, there is evidence that the project contributed albeit modestly to higher-level outcomes as laid out in the TOC. The impact evaluation for Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) found that grant recipients (of whom 30 percent were female) established or expanded businesses that – in line with ANDS Pillar 3 and its cross-cutting gender goal - increased economic activity in response to increased human capacity and skills sets and business services; increased employment among the poor and vulnerable; and increased gender equality.

Justification of Overall Efficacy Rating

24. Overall Efficacy is rated modest because there was some progress towards the targets for the first and second outcomes. As no one underwent training under Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) or Sub-Component 3.1 (Encouraging Rural Employment), there was no possibility to award incentives to the TPs or for trainees to find employment and increase their income; as a result, there was no progress on


Page 12 of 60

any of the three key PDO indicators insofar as beneficiaries of these two components might have been concerned.8 However, there were 2,482 persons (of whom 30 percent female) under Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) who received a grant to start/expand a business, as well as training and follow-up support. This resulted in higher levels of employment and income vis-à- vis a control group, and contributed somewhat to higher-level outcomes of the National Development Strategy. The PDO indicators pertaining to employment and income should have been phrased to capture the results of the Sub-Component 3.2, in the same manner as the (core) PDO indicator pertaining to beneficiaries. The positive outcomes from Sub-Component 3.2 on employment and income are modest but solid because they are based on difference in difference measures using randomized treatment and control groups. . Further, the strength of the Business Grant Scheme led to a similar IADC-funded initiative with 2,000 beneficiaries; while the MoLSAMD is budgeting for the replication of the Scheme. Together, these results warrant assigning a modest rating to efficacy.

C. EFFICIENCY

Assessment of Efficiency and Rating 25. Efficiency of successfully implemented activities is rated negligible. The net present value (NPV) and the internal rate of return (IRR) for the project were calculated for that part of the operation (Sub-Component 3.2 - Promoting Entrepreneurship among Afghan Youth or the Business Grant Scheme) that led to measurable outcomes; operational dimensions of efficiency were also analyzed for all components. The NPV is USD 1.7 million and the IRR is 12.6 percent under the base case, ranging from 8.4 percent to 16.4 percent using a sensitivity analysis; stringent low case assumptions indicate that the positive result is robust. Given the pilot character of the Sub-Component 3.2 activities, these returns are in line with sectoral expectations.9 The costs used to calculate the NPV/IRR included all Project expenditures for Sub-Component 3.2 and a portion of Component 2 (Project Management, Capacity Building of MoLSAMD and M&E); together, these accounted for 48 percent of all Project disbursements by closing. The returns on the remaining expenditures could not be calculated, in that as per Project design, they should have been set against the returns generated by Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) and Sub-Component 3.1 (Encouraging Rural Employment); neither of which successfully completed activities. In the absence of such successes, the returns on Component 1 are likely to have been negligible and on Component 2 to be indeterminate. If one were to set all Project expenditures against the returns of Sub-Component 3.2, then the Project has a negligible, below-discount IRR (1.7 percent) and negative NPV. 26. In terms of operational efficiencies, there were substantial implementation delays and a failure to successfully complete that affected Component 1 and Sub-Component 3.1. These were associated with an overall funds utilization rate of 37 percent, as well as substantial turnover in project personnel. Component 2 activities were completed within the overall project schedule, though with some initial delays; this component might have been better designed to consider the change in implementation arrangements agreed during negotiations. Sub-Component 3.2 experienced delays at the outset (to recruit the Grant Management Firm), though the Project team took advantage of this to strengthen the design of the Scheme by running a small pilot; this laid the foundation for the Sub-Component effectiveness and substantial efficiency gains during the second, larger phase. (See Annex 9). In general, costs were contained within budgets though there were cases of unnecessary over-runs.

8 The funds expended on Component 1 were used to map, screen and recommend TPs; draft and adopt NOSS- and competency-based curricula; and produce written instruction materials. See Annex 10. 9 As explained in Annex 4, Sub-Component 3.2 activities entailed relatively high learning and overhead costs. Were the grant scheme to be scaled up, the IRR would likely be in the region of 26 percent (base case).


Page 13 of 60

D. JUSTIFICATION OF OVERALL OUTCOME RATING

27. The overall outcome rating is rated unsatisfactory because the PDO relevance is rated high, efficacy is rated modest, and efficiency is rated negligible. This rating corresponds to best judgement based on information received for the ICR, in that:

• The PDOs remain relevant to the Country Partnership Framework (FY 2017-2020), as they support one of

current CPF Pillars as well as WBG’s gender strategy in Afghanistan; • There was only modest progress towards achieving the two development outcomes. The business grants

led to positive results in terms of employment and income, which contributed in fact to the PDO. As mentioned earlier, the results were not captured by the PDO indicators because of shortcomings in the results framework. Despite the positive outcomes under the business grant scheme, these results were modest in relation to the Project’s overall targets.

• While those parts of the operation that generated an outcome have an IRR of 12.6 percent, the overall Project IRR was negligible.

E. OTHER OUTCOMES AND IMPACTS (IF ANY)

Gender

28. Under Sub-Component 3.2 (Business Grant Scheme), the 30 percent of the 2,482 beneficiaries who were female benefited from the business grant, training (including a module specifically focused on gender issues in business) and counselling; and as a result, had significantly higher levels of employment and income as compared to a control group.

Institutional Strengthening

29. The NSDP was integrated into MoLSAMD as part of the Project. Nearly all NSDP functions were transferred to the relevant units/departments of MoLSAMD, including technical, HR, fiduciary and M&E functions; in many cases, budgeted posts were created to absorb the functions provided by NSDP staff. The capacities of the relevant units/departments were built through a combination of strengthened operating procedures, skills transfer by NSDP staff (who transferred to the relevant unit/department), and training. The transferred functions and capacity strengthening are discussed in Annex 11. As amplified below, the integration created implementation difficulties for NSDP that were not foreseen during preparation; and further strengthening measures are still required to enable MoLSAMD to implement projects. 30. The Project strengthened training programs, which may have benefited non-beneficiaries. The production and dissemination of 162 NOSS and 45 competency-based curricula, and written instruction materials for 7 trades; the adoption by 70 non-formal TPs of NOSS for the courses in which they train; and the training of 673 non-formal TP vocational trainers: all may have contributed to an improvement in the quality and effectiveness of trainings at non-formal TPs around the country. This in turn may have led to improved learning and employment outcomes for students.


Page 14 of 60

Poverty Reduction and Shared Prosperity

31. The Sub-Component 3.2 (Business Grant Scheme) beneficiaries had higher levels of employment and income than the control group. The treatment group had an employment rate of 88 percent, 13 percentage points higher than the control group; and earned approximately USD 11.60/month more in net income than the control group. The beneficiaries were all small business operators and given the characteristics of the target group are among the semi-urban and rural poor. The beneficiaries also contributed more to employment in their community, as compared to the control group; thus, contributing to shared prosperity.10

Other Unintended Outcomes and Impacts

32. Enrolment of beneficiaries in Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), the Business Grant Scheme required that they open a bank account and present a government-issued ID (Tazkira). Most of the beneficiaries had never opened a bank account before, and this led to a form of ‘financial inclusion’ that conferred banking benefits beyond receiving the incentive grant itself. Many of the beneficiaries – including a large proportion of the women – did not have a Tazkira and were required to get one in order to receive the grant. Aside from accessing the grant, having the Tazkira allows beneficiaries to vote and is a form of ‘civil inclusion’. 33. The Italian Agency for Development Cooperation (IADC) funded a project using the same design as the Bank-supported NATEJA; it is on a smaller scale, covers different geographic zones for training and grant activities, and supports Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) activities that are complementary to those of the Bank-supported NATEJA. While it is too early to quantify its outcomes/impacts, its activities, if effective, should result in improvements in employment and incomes for beneficiaries.11

III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION

34. The preparation phase benefited from several factors. The sector was quite well understood, with defined challenges for the Project to address. This can be attributed to a body of research and experience that underlay the development of the ANDS (2008-13) and its associated NPP1, as well as the MOLSAMD Strategic Plan (2013-15); to all of which the Project was aligned. More recent research on SMEs, TVET and apprenticeships was available and consulted by the preparation team.12

35. The team built on the experience of the Bank, particularly the Bank-financed ASDP I and II (both focused on skills development) and AREDP; as well as of other development partners, notably the GIZ, USAID and UNESCO. Indeed lessons learned from the ASDP I project as well as the 2013 Institutional Analysis commissioned by the Bank guided the design

10 “While businesses in both groups tended to be small and a majority of both groups had no employees, of those who did employ individuals other than themselves beneficiaries were more than twice as likely to employ at least one person and similarly more than twice as likely as non-beneficiaries to employ more than one person. Importantly, while most employees for both groups were members of the beneficiaries’ household, it was more common among beneficiary business to employ individuals outside of their own household.’ Altai Consulting 2018 (v.10), p. 4. 11 See Annex 13 (The NATEJA Project funded by IADC). 12 The 2012 USAID survey of SMEs in six cities and the 2013 GIZ survey of Master Trainers and apprentices.


Page 15 of 60

of the NATEJA project, particularly that the demand for non-formal training was high, that non-formal trainings could be effective in terms of generating higher levels of employment and income (as demonstrated by tracer studies), that the quality of non-formal trainings could be usefully strengthened, and that capacities in management, monitoring and evaluation at MoLSAMD (including NSDP) also required strengthening: all elements that were targeted in the project. Similarly, the AREDP project had learned that one of the constraints SGs members faced in creating more value out of their savings was a lack of literacy and hard/soft skills; again, this was to be addressed by the Project.

36. The project design was relatively straightforward, with a clear operational logic. The results framework was for the most part aligned with operational objectives, had a relatively straightforward mechanism to generate data, and included Third Party Verifications and IEs. The IEs for Sub-Components of Component 3 were designed in detail and laid a foundation for operational rigor that benefited activity implementation. A level of readiness existed on the part of the NSDP, which was responsible for implementation, in that it was already established, staffed and operating, with experience using Bank procedures. Further, a detailed PIM was in place at the time of effectiveness.

37. Certain factors during the preparation phase laid the foundation for subsequent challenges. There were important dimensions of uncertainty and instability in the political and security spheres, of which one may highlight here impending elections and systemic governance weaknesses, as well as the drawdown of international security forces and the recurrence of security incidents. These were articulated in the Operational Risk Assessment Framework (ORAF), and mitigation measures were proposed.13

38. There was an important yet last-minute change to implementation arrangements. Whereas in the PAD the NSDP was identified as being responsible for implementation, and it was assumed that the NSDP would preserve its then-current autonomous status and mode of operation, during negotiations the MoF successfully advocated for integrating the NSDP into the Directorate General of Skills Development (DGSD) in MoLSAMD and this was entered into the Grant Agreement. The change created uncertainty among the NSDP staff responsible for Project implementation. Further, it was a substantial modification of the implementation arrangements, a modification that had not been appraised. While the project design included capacity building of MoLSAMD (including NSDP), it was predicated upon keeping NSDP’s then-current status and receiving financial contributions in support of Project operations from MOF under the discretionary budget. The new arrangements could have reasonably been expected by the Bank to constitute a significant challenge requiring more planning and capacity building to take into account effects on implementation procedures and capacities that would need to be transferred as part of integration, based on the findings it had received from an institutional analysis of MoLSAMD that had been commissioned by the Bank in the previous year and completed in early 2013.14 At the same time, the new arrangements were in line with a general move within the Bank towards institutionalizing implementation responsibilities within permanent structures of the civil service, in keeping with the Bank’s Interim Strategy Note for 2012-14 in force at that time, which highlighted a concern about how foreign aid had created a ‘second civil service’. Weaknesses in planning, implementation and monitoring capacity in MOLSAMD and NSDP were acknowledged as risk factors in the ORAF; though the only mitigation measure identified was to recruit international advisors. See also Annex 11.

39. The principal component (Component 1 – Improving the Quality of Non-formal Training and Labor Market

13 Some of the measures were vague or undefined, though perhaps it was not reasonable to expect more. For instance, in response to security incidents the mitigation measures were ‘NSDP to tailor and adapt implementation arrangements to the security situation in the targeted provinces.’ 14 Institutional Analysis of Ministry of Labor, Social Affairs, Martyrs and Disabled. January 2013.


Page 16 of 60

Outcomes of Trainees) included an experimental dimension that introduced an element of risk that was not recognized, and moreover did not explicitly identify priority, eligible trades. The incentive was tied to graduate certification, employment or starting a business; and was predicated upon sufficient graduates being employed with their fees being paid either by other donors or themselves. In the end, the Project had difficulty finding a sufficient number of TPs enrolling a sufficient number of graduates across a wide range of trades.15

40. Project design relied on the AREDP to facilitate working with the SGs (under Sub-Component 3.1 - Encouraging Rural Employment); and on the ASDP II, to establish the TTTI, which would train instructors at the TPs, and to develop a certification mechanism, which would be used to certify trainee graduates for both Component 1 and Sub-Component 3.1. Given that the ASDP II cancelled the certification mechanism, the MTR recommended that the certification requirement under NATEJA be dropped. B. KEY FACTORS DURING IMPLEMENTATION

41. The political process. Not long after effectiveness, elections were held, the results of which were delayed; and when the new government was formed, there were delays in appointing Ministers. The new Minister for MoLSAMD took up her post in mid-2015, and naturally it took some time for her to become familiar with the project. These events made it difficult for the project to get decisions approved during the opening 12-18 months, including for the procurement of large TA packages that were vital to implementation, as well as to make progress on integrating NSDP into DGSD in MoLSAMD. Further, the new President decreed a freeze on hiring given the GIRoA’s unexpectedly large fiscal deficit, encouraging MOLSAMD to use NSDP staff for non-Project tasks. Partly as a result, there were relatively few activities during the first year of the Project, and no progress on any of the intermediate indicators.

42. NSDP integration. The decision at negotiations to integrate the NSDP into DGSD/MoLSAMD contributed to the implementation difficulties that the project experienced. First, it took up substantial time from NSDP senior management. Second, the new lines of authority meant that NSDP staff working in technical, fiduciary and M&E functions no longer answered directly to the NSDP Director, but rather to the Heads of the relevant MoLSAMD directorate or unit. Whereas it was originally thought the NSDP functions would be integrated into the DGSD such that all the implementing functions would come under one Directorate, in the end the functions were integrated across multiple Directorates. This in itself complicated the tasks for senior NSDP management, as they had less authority to take decisions or otherwise act to ensure the timely completion of work in these areas; and had to liaise with multiple heads of Directorates, each with equal authority and answering to two different Deputy Ministers and (in the case of M&E) to the Office of the Minister. In practice, NSDP staff often needed to consult with the Deputy Ministers and the Office of the Minister to ensure coordination and decision-making. Third, some of the procedures followed by the MoLSAMD units/directorates were reported to be more cumbersome and time-consuming. In some cases, they did not conform to conventional project management procedures. For example, NSDP senior management was not able to sign off on contract deliverables, which sometimes needed to be approved by the M&E unit in the General Planning Directorate. Fourth, persons occupying posts

15 By the time of the MTR, 78 TPs had been short-listed as the first batch to sign LoUs with MoLSAMD; they enrolled approximately 6,300 students, as against the Project’s final target exceeding 40,000. Further, approximately 2,700 of these students were to be enrolled in courses that MoLSAMD was concerned would not lead to employment, viz. project management, secondary academic degrees, IT skills, and English language. (See Annex 13 for further details.) Various factors likely contributed to the relatively restricted number of TP/enrollees and range of trades, including TPs withdrawing from consideration after delays and increased demands on the part of MoLSAMD; cutbacks in donor funding for vocational training; and a reduced field presence of donors including NGOs providing training. These factors are examined in the next section (III.B).


Page 17 of 60

for the NATEJA project were tasked with building capacities of the units/Directorates to which they were attached. While this is laudable, the ICR Team was informed that ‘skills transfer’ – both prior to and after integration – was often taken to mean ‘do work of the Ministry beyond that pertaining to the NATEJA project.’ Finally, the uncertainties and implications of integration (including lower salaries) may have contributed to turnover among NSDP staff and persons occupying posts tasked with NATEJA implementation, which naturally resulted in further delays.16 A case in point is the Program Director post, which was vacant from mid-2016 until closure. In that period the project was managed by an Acting Director who was more technical than managerial, and who had to seek approval from the Deputy Minister for decisions, creating delays.

43. Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) had been designed to build technical capacity within the Ministry. However, the design focused on building capacities in areas that were not directly involved with the selection, operation and incentivizing of TPs and trainees; that is they focused rather on procurement, financial management and LMIS and monitoring. Progress was made in building capacity in these areas, and indeed the Project exceeded the quantitative target of the relevant intermediate indicator (150 MOLSAMD/NSDP staff trained as against a target of 100). However, as Project implementation and NSDP integration went on, the number of staff eligible burgeoned well beyond initial expectations and it became clear that capacity building needs greatly exceeded Component resources. As a result, the capacity building was spread too thinly and the persons occupying the posts that had originally been in mind when the Project was designed did not get the in-depth strengthening of knowledge and skills required to ensure quality in Project implementation.17 See also Annex 11.

44. Recruitment of TA. Implementation arrangements foresaw the placement of TA to assist in key areas of Project implementation, and the ORAF noted that international advisors were a risk management measure to address weak MoLSAMD capacity. For instance, Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) provided for an International Advisor for Management of Training Providers, whose terms of reference included ‘assisting NSDP senior management team in managing and supervising the firms contracted for implementation’ and ‘ensuring that the NATEJA project components, and particularly Component 1…are delivered on time …’18 This Advisor was not recruited. It is perhaps significant that the components and activities with good success (e.g. Sub-Component 3.2 - Promoting Entrepreneurship among Afghan Youth) and the LMIS under Component 2 were fortunate in having successfully recruited a full-time TA (international) responsible for ensuring implementation. The failure to recruit TA, particularly International Consultants (ICs), seems to have been due in part to a MoLSAMD preference for National Consultants (NCs) and in part to the security situation (discussed below).19 There were also significant difficulties associated with recruiting TA firms; this is discussed below.

45. The progressive tightening of eligibility criteria. This affected Component One (Improving the Quality of Non-Formal Training and Labor Market Outcomes of Trainees) in particular. After the Inspection Agency had been recruited and started its screening of TPs, the Ministry questioned the Agency’s procedures for hiring field workers and introduced new and complex TPs screening and registration procedures. While the Ministry was motivated by a concern to ensure that the TPs were of good quality, the increasing informational and regulatory demands upon the TPs (including the need

16 For example, there were two NATEJA Program Directors throughout project implementation. The first stayed in post for approximately 1.5 years; the second for 9 months. There was no Director from July 31, 2016 until closure. The second-in-command post (Program Coordinator) was occupied until end July 2016, and vacant thereafter until closure. 17 See also ISR dated October 24, 2018. 18 Project Implementation Manual, p. 97. Italics added. 19 With respect to the International Advisor for Management of Training Providers, the government insisted on recruiting a National Consultant instead. Discussions between the government and the Bank on this issue took time and eventually became moot in light of the effective suspension of Component 1 activities after the MTR; and this critical post was never filled.


Page 18 of 60

to officially register, and therefore also pay the AFN 30,000 registration fee) encouraged many of them to withdraw their application. The Ministry was also concerned about the preponderance of non-technical skills courses and introduced new restrictions (such as excluding IT and English language). There was also a legitimate concern that the TPs were largely semi-urban, effectively closing off access to a segment of the population targeted by the Project.

46. Altogether, these contributed to delays in signing agreements with TPs and some of the TPs dropping out. In preparation of a first batch of participating TPs, around November 2015, the Inspection Agency initially mapped 250 TPs; inspected 175; and invited 134 to apply to participate in the incentives scheme. Faced with new compliance demands, only 88 TPs submitted a formal application, of which 78 were recommended for approval by the responsible MoLSAMD committee. 70 of these TPs actually signed a LoU (though not MoLSAMD) in May 2016. Yet MoLSAMD then introduced more compliance demands, creating more delays. This led more TPs to withdraw, and it took until December 2017 to finalize a shortlist of 24 TPs enrolling approximately 2,000 trainees who were willing to re-sign the LoU as the first batch of TPs. At this point, it was too late to complete activities on time envisaged in the project, and the decision was taken 7 months later to cease Component 1 activities and close the Project early.

47. Post-MTR uncertainty. The Project implementation and Bank supervision teams were fully cognizant of the design and implementation challenges facing the project, and at the mid-term review (MTR) MoLSAMD and the WB agreed on a restructuring. However, despite MoLSAMD’s initial agreement to restructure at the MTR, there ensued a lengthy period of discussions – not resolved until December 2017 and formally confirmed until July 2018 – within and between WB, MoF and MoLSAMD as to whether the Project should be restructured or closed. (See Section I.B and Annex 7 for details). During this extended period of uncertainty (January 2017- December 2017) as to the Project’s fate, the decision to restructure seems to have been taken and then overturned twice. The agreement to restructure was captured in the Mid-Term Review Aide Memoire (November 22-26, 2016), the Management Letter from the Midterm Review dated January 17, 2017. During the Bank mission in February 12-14, 2017, the Advisor to the Deputy Minister at MoLSAMD overturned the MTR recommendations and ‘conveyed to the mission the desire to terminate the NATEJA project’ (as per the mission’s Back-To-Office-Report). Discussions continued within the Bank between the CMU which advocated early closure and the GP which disagreed. Despite the communication during the February mission, the Bank received a letter from the Minister of Finance dated August 16, 2017 formally requesting the restructuring based on the recommendations of the MTR ‘fully endorsed by both MoLSAMD and the MoF. The discussions continued between the GIRoA and the CMU, which culminated in an agreement in December 2017 to close the Project. It took until July 2018 for MoLSAMD to request MoF to close the Project which was quickly followed up by a formal MoF request to the Bank to close the Project early and cancel unspent funds. A restructuring paper to this effect was prepared to enable an amendment to the Grant Agreement, the Project closed on October 31st, 2018. 48. There was little impetus to implement the Project’s main component (Component 1 - Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees); all parties agreed that it could no longer be implemented as originally designed, although views differed between the CMU which judged that MOLSAMD had not demonstrated ability or willingness to implement the Project effectively and the Education GP which believed restructuring could turn around the implementation performance. This uncertainty also made it difficult to keep project positions filled (see also above).

49. Fiduciary factors. There were shortcomings in procurement that negatively impacted the project’s implementation. The procurement plan had nine major consultancy packages, eight of which should have been awarded and disbursing within the first year. However, despite having the support of a national and international procurement advisor, fifteen months into the implementation period, only one contract had been awarded (for the Social and


Page 19 of 60

Environmental Assessment); resulting in a downgrading of the rating from Satisfactory to Moderately Satisfactory. After thirty months, only two further major consultancies had been recruited – the Inspection Agency for the TPs of Component 1 (Improving the Quality of Non-Formal Training and Labor Market Outcomes of Trainees) and the Grant Management Firm for Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) – and these alone had required more than 300 days between technical proposal and contract award. 20 Only one further TA package was subsequently procured (for the Sub-Component 3.2 Impact Evaluation). This, along with other procurement shortcomings,21 led to the rating being further downgraded to Moderately Unsatisfactory in August 2016, where it remained until closure. The delays in initiating the procurement for the Training Provider for Sub-Component 3.1 (Encouraging Rural Employment), followed by the challenges in identifying a suitable and qualified service provider despite multiple re-advertisements, were such that by the MTR no trainings had taken place; this led to the MTR recommendation that this Sub-Component be dropped. See Annex 7.

50. Non-fiduciary factors contributed to the procurement difficulties. In particular, the procurement function was hampered by the non-execution of predicate activities, for example the TA firms to conduct the Special Purpose Review and the Feasibility Study. Further, the security situation makes international firms and ICs reluctant to bid on Afghan contracts. See Annex 12.

51. The FM performance rating was downgraded from Satisfactory to Moderately Satisfactory in December 2014 (where it remained until closure), due to low budget credibility and weak fixed assets management. However, there were no FM issues that undermined Project performance.

52. Recipient financing. The government committed to provide USD 3.0 million to cover salaries of technical teacher trainees and governmental support staff. In the event, roughly USD 1.5 million were allocated through the discretionary budget to the NSDP during the project period; but none of the funds were used for salaries related to the Project or for support staff. As noted in the ISR of February 2017 (in which Counterpart Funding was downgraded to Moderately Unsatisfactory from Satisfactory), the lack of recipient financing resulted in a disproportionate expenditure on project operating costs and overstretched the NATEJA team, leading to implementation delays.

53. Technical capacity and strong design. The preceding discussion must not obscure the fact that when the Ministry supported Project implementation as per design, there were strong results. In particular, Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) very nearly reached the target written in the project description, providing grants and training/counselling to 2,482 individuals to start or expand a small business.22 Had the project been closed soon after the MTR, this sub-component would not have achieved its results.

Factors subject to World Bank control

54. The Project benefited from proactive and relatively stable Bank supervision. The Bank worked closely to support senior NSDP management. It intervened for example to facilitate the work of TA firms (e.g. the Inspection Agent, the GMF and the Impact Evaluation firm), to ensure they understood project requirements, were able to navigate issues with MoLSAMD, and kept reasonably well to their deliverables schedule; and regularly provided them with recommendations to improve the quality of their work and outputs. The Bank’s supervisory role is discussed further in Section IV.C.

20 The fourth major TA contract - to conduct the Impact Evaluation (for Component 3.2 only) - was signed roughly on time. 21 See Section IV.B. 22 As against the 2,582 target written in the project description (but not included in the results framework).


Page 20 of 60

55. There were five supervision missions throughout the Project’s 4.5 year life (of which one virtual). These took place during the first three years, when the bulk of implementation activities took place, as well as at the mid-term review (MTR). The Bank was proactive in supporting the MTR, and followed up quickly with the government on how to proceed with restructuring. (The subsequent delay is discussed below). The missions were supplemented with email and other communications as well as the regular supervision and support provided by the Bank technical, fiduciary and safeguards staff based in Kabul. 56. The security situation severely restricted the Bank’s ability to supervise, with meetings and visits beyond the Ministry being very difficult to organize. The Bank mitigated these restrictions as it could, for example by organizing MTR meetings with beneficiaries at the Bank’s premises.

Factors outside the control of government and/or implementing entities

57. The security situation. This deteriorated after effectiveness and remained fraught throughout the implementation period. This led inter alia to the resignation of some international TA; made it difficult to recruit international TA firms and ICs; and impacted the mobility of Project staff, hampering their ability to implement and supervise activities.

58. Changing sectoral landscape. In the post-preparation period, Official Development Assistance (ODA) disbursements to Afghanistan for vocational training declined each year from USD 32.7 million in 2013 to USD 24.5 million in 2017.23 The number of international armed forces was reduced after 2014. At the same, there was an increased demand from government to provide ODA through the budget. Together, these factors encouraged the donor community (including NGOs) to reduce their presence in the field, and to cut back their funding of non-formal training providers. This in turn reduced the number of students and the range of courses offered. This may have contributed to the difficulties encountered in Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) to find a sufficient number of TPs enrolling sufficient students across a range of technical trades. With respect to Sub-Component 3.3 (Feasibility Study on Apprenticeship Program), the research by GIZ and its subsequent decision to introduce an informal apprenticeship program at a large scale meant that there was no longer a strong rationale for implementing this component’s activities.24

IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E)

M&E Design 59. The results framework did not encompass project outputs and outcomes in one important respect: there was no intermediate indicator for Sub-Component 3.2,25 while the phrasing of the PDO indicators was such that they measured

23 OECD International Development Statistics, https://stats.oecd.org/qwids. Accessed January 10, 2019. 24 As noted above, had restructuring proceeded this component would have been dropped from the Project. 25 The intermediate result indicator one (‘Number of new micro and small-scale business (sic) established and are running’) might have covered C3.2 activities, but the final target of 70 – in contrast to the 2,520 businesses targeted as stated in the project description for C3.2 – and the indicator description (‘New micro and small-scale business that are established and running post-graduation from components 1 and 3’, emphasis added) make it clear that only C1 and C3.1 activities were included here.

https://stats.oecd.org/qwids


Page 21 of 60

only increases in the potential for employment and income generated by Component 1 and Sub-Component 3.1. That is, the PDO indicators restricted increases in potential to those generated as a result of Component 1 and Sub-Component 3.1 training activities; whereas Sub-Component 3.2 gave grants, training and support to persons to start or expand businesses, and in that way increase their potential for employment and income. This particularity of the results framework was a weakness in that it did not capture Sub-Component 3.2, which was the most successful of the Project’s components with employment/income-related activities, and contributed to positive outcomes in terms of employment and income. 60. Having said that, the Results Framework indicators captured much of the Project’s outputs and outcomes; the indicators were generally well phrased;26 and the data sources, methodologies and definitions were for the most part clear and adequate. Special monitoring exercises were well articulated, including third-party monitoring (to assess quality of TPs selection and transparency of beneficiary selection process), an independent assessment of trainee competencies (to monitor the quality of training), and special purpose reviews (to monitor the cash incentives transactions). The design of impact evaluations were clearly described in the PAD for Sub-Component 3.1 (Encouraging Rural Employment) and Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth).27

M&E Implementation 61. An elaborate software-based Management Information System (MIS) for the Project was developed. It was formally housed at the M&E unit of MoLSAMD’s General Planning Directorate, and could have been used to monitor Project activities and other activities of MoLSAMD and the ESCs; with web-based applications for data entry being feasible for TPs, ESCs and other actors.28 However, the MIS was never operationalized, due in part to the fact that the main activities for which it was required were not implemented (see Annex 14); the results framework indicator target was consequently not met. This did not seriously hamper implementation monitoring of those activities that were executed. While the MIS was being developed, the NSDP staff and TA working on Component 1 (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees), Component 2 (Project Management, Capacity Building of MoLSAMD and M&E) and Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth) built their own monitoring databases and mechanisms that were regularly consulted and used. The M&E Unit of the Planning Directorate also monitored Sub-Component 3.2 activities. An evaluation of Sub-Component 3.2 was successfully conducted using a randomized selection mechanism, an impressive accomplishment in any case but particularly so in the Afghan context. Some planned M&E activities were not conducted, mainly due to the non-execution of predicate activities (Annex 14). 62. More generally, the Project supported MoLSAMD’s monitoring functions as per design. In particular, a Labor Market Information System (LMIS) was developed with project support. It is fed internally by the MoLSAMD using data pertaining to ESCs operation and the registration of work permits for Afghan nationals and for foreigners; only the work permits data are currently being entered, and only one ESC (in Kabul) is currently operational. It is to be fed externally from five other ministries, using for example data on graduates and the labor market; though this aspect is not yet operational. The LMIS also incorporates household and enterprises’ survey data. In this regard, the Project supported the Human Resource Demand and Supply Survey (2016), which provided data for decision-making.29

26 Some of the indicators could have been strengthened. For instance, intermediate result indicator four (number of MoLSAMD/NSDP staff trained) is an activity indicator, and might have been replaced with a higher-level indicator of strengthened capacity. 27 Cf. pp. 34-37 of the PAD. 28 The applications would have enabled the tracking of registered TPs, trainees and grant recipients. 29 The supply side gathered data from households and TVET institutions. The demand side gathered data from formal and informal


Page 22 of 60

M&E Utilization

63. While the software-based Project MIS was not operationalized, other monitoring and evaluation mechanisms were utilized that enabled activities to be monitored and decisions taken. For instance,

• Monitoring of Component 1 led MoLSAMD to reconsider selection criteria for TPs. • Sub-Component 3.2 activities were closely monitored for quality, leading to improvements and timeliness during the second stage of implementation. Further, the IE design for Sub-Component 3.2 was closely followed during the post-applications selection phase; and this contributed to the rigor of the selection process. • The LMIS is being used to register work permits, while results from the labor demand and supply survey were used for example by MoLSAMD to decide on priority areas for training and by the Ministry of the Economy to design new private sector development programs. Justification of Overall Rating of Quality of M&E

64. The ICR assigns a modest rating to the quality of M&E for the following reasons. First, the results and indicators identified for the M&E framework captured the TOC’s key results with mechanisms designed for the most part to generate valid, reliable and accurate information on a timely basis. However, there was a significant design shortcoming in that the PDO outcome indicators should have been phrased to capture the outcomes of Sub-Component 3.2 (in addition to Component 1 and Sub-Component 3.1). Second, there were some shortcomings in implementation, particularly in regard to operationalizing the project MIS and to carrying out certain M&E activities. This was due in part to the fact that the project did not implement its primary activities in Component 1 and Sub-Component 3.1. These shortcomings did not, however, substantially hinder the project’s capacity to monitor and evaluate those activities that were implemented. Third, monitoring data generated by the Project were used to introduce course corrections and control quality in the execution of activities, particularly for Sub-Component 3.2.

B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE

65. The following safeguard policy was triggered at appraisal: (a) Environmental Assessment (OP/BP 4.01). The project developed an Environmental and Social Management Framework to lay out the specific requirements, processes, and responsibilities for ensuring that the activities implemented by the Project will not have negative impacts on the environment and people. As per the provisions of the ESMF, the project was required to undertake a Social and Environment Assessment prior to commencement of any activities proposed under the project. An international firm with a local joint venture was hired to conduct the required assessment. The firm produced a draft SEA report, which was shared with the World Bank for review. The Bank team reverted with comments to address specific shortcomings of the report. However, the report was never shared again for the Bank’s review, and as a result was not considered finalized and cleared by the Bank’s safeguards team, as the comments were not addressed by the firm. The report was approved by the project TTL for the purpose of due payments to the firm. The project was also required to prepare a safety manual. However, there is no evidence of its completion as it was not shared with the World Bank for review.

businesses including SMEs. Data were collected manually and later entered into the databases. The LMIA plans to periodically update the data using its own surveys, with direct data entry using a web-based application; as well as findings from Central Statistical Office (CSO) surveys.


Page 23 of 60

Institutional arrangements for a Grievance Redress Mechanism were established under the project, mainly pertaining to Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), however the Bank Safeguards Team was not provided with evidence as to its functionality, operation and effectiveness.

66. During the MTR, which took place in November 2016, the Bank’s safeguards team did not take part in the meetings. The client reported that a dedicated Environment Specialist and a Social & Gender Specialist were hired for overseeing and implementation of the safeguards compliance under the project. The contracts for the mentioned staff were valid until June 2018. However, there are no records of any reports that have been shared by the client on the monitoring and supervision of the safeguards compliance. 67. The project maintained FM arrangements throughout most of the implementation period, except for the internal audit report which was not conducted regularly nor shared with the Bank on a timely basis. The Bank otherwise received all FM reports on a timely basis. There were no ineligible expenditures, nor are there any overdue IFRs or external audit reports. All the fixed assets purchased from grant proceeds are to be transferred to MoLSAMD before the end of December, 2018. 68. There were significant shortcomings in procurement that affected project performance. These pertained in the main to delays in the procurement, or the non-procurement, of TA packages; see Section III. B and Annex 12. Other procurement shortcomings included the plan not being regularly updated, the procurement of luxury goods, and a failure to reflect the plan in the STEP tool. The rating was consequently further downgraded to Moderately Unsatisfactory in August 2016, where it remained until closure.

C. BANK PERFORMANCE

Quality at Entry

69. The Bank team worked well to identify, facilitate the preparation of, and appraise the operation, thus laying a foundation for the Project to achieve its planned development outcomes and acting in a manner consistent with the Bank’s fiduciary role. The project content and strategic approach were well defined and integrated into the government’s sectoral and national strategies. The project was for the most part technically well designed. The activities built on national good practices, leveraged the experience of other projects, and targeted a gap in the sector that promised to benefit an important segment of the poor and vulnerable population. 70. The project was perhaps overly innovative in that its centerpiece incentives scheme – to reward TPs if their graduates were certified, found employment or started a business – was predicated upon there being sufficient students enrolled in trades with good employment prospects; and this in turn relied upon the students’ fees being paid either by themselves or other DPs. This left the project open to a risk that was not explicitly recognized but nonetheless seems to have been realized, namely that a downturn in the economy or DPs funding, or an increased security-related reluctance on the part of NGOs to operate in the field, would reduce the number of students training at non-formal TPs. Further, while the project targeted the semi-urban and rural poor, it did not formally identify priority trades. This resulted in some TPs being identified by the Inspection Agency that offered courses that would not (according to MoLSAMD) lead to employment and many of which were not actually skills training. Finally, the project design was predicated upon the successful implementation of other projects, leaving it vulnerable should these encounter difficulties. These factors created a substantial challenge for the component most critical to the Project’s development outcome (Component 1 -


Page 24 of 60

Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees). Whereas the Project intended to enroll over 40,000 students under Component 1, by the time of the MTR the Project had managed to finalize a first-batch list of 78 TPs enrolling approximately 6,300 students, predominantly in semi-urban areas. 44 percent of these TPs (enrolling approximately 2,700 students) offered courses in areas that MoLSAMD was reluctant to finance.30 Further, the project had no means to certify TP trainees, as this element of the ASDP II did not materialize. (See also Annex 13). 71. The Project was satisfactorily appraised from both the economic and financial perspectives. The fiduciary arrangements within NSDP and MOLSAMD as set out in the PAD were thoroughly appraised, with risks identified and mitigating measures identified. 72. The implementation arrangements as appraised in the PAD built upon experience previously accumulated by the NSDP. However, at negotiations it was agreed that the NSDP would be formally integrated into the DGSD/MoLSAMD. This was a significant change to the implementation arrangements, and complicated the implementation task for NSDP (see Section III.A and Annex 11). The implementation arrangements arguably should have been re-appraised prior to completing negotiations, and more capacity building under Component 2 should have been incorporated with clear targets pertaining to expected results. 73. The design of the M&E arrangements was generally satisfactory, though arguably the PDO and its indicators should not have been restricted to outcomes deriving only from non-formal skills training, thus excluding similar benefits from the Business Grants Scheme. The risk assessments were suitable (and later proven correct), as were mitigation measures; though there are limits to which mitigation is feasible, particularly with regard to security. Finally, the proposed Bank inputs and processes were adequate, and included regular supervision missions and reporting using a team with the full range of relevant expertise.

Quality of Supervision 74. The Bank carried out five supervision missions (of which one was virtual) up to the immediate post-MTR period; this was one mission short of the bi-annual missions schedule foreseen in the PAD. Thereafter, implementation was largely suspended (except for Sub-Component 3.2). Throughout implementation, the Bank augmented its missions through Kabul-based staff conducting technical and fiduciary supervision, as well as regular email and other communications. Team leadership was quite stable, with two TTLs from the preparation stage to closure, and one co-TTL based in Kabul who was a constant team member since the beginning of Project preparation. The task team included the necessary range of skills to supervise the Project in terms of technical quality and implementation and fiduciary arrangements. (See Annex 2). 75. The Bank’s Supervision Team concentrated throughout on the two development outcomes of the PDO, as can be seen by the focus of the AMs on linking implementation performance (of activities) to their impact on the PDO, and the continued dialogue with government to ensure that deadlines were met. The Team intervened regularly to support the implementing agency on technical matters, as can be seen for example on the detailed guidance provided on the work of the Inspection Agent (Component 1), the design of the first stage of Sub-Component 3.2, and ensuring quality sampling for the IE. Supervision mission findings were communicated clearly to Bank management and to the GIRoA, the latter both in writing and through meetings with senior GIRoA officials. Agreed actions points were included in the AMs with deadlines and responsibilities, and these were subsequently monitored and followed up for implementation by the Team.

30 Such as project management, secondary academic degrees, IT, and English language.


Page 25 of 60

The performance reporting was generally of satisfactory quality, with some exceptions noted below. 76. The fiduciary aspects were generally well supervised. The Team identified early on certain shortcomings in fiduciary performance and procedures, and provided feedback and guidance on remedies to the client; with the exception of the Aide Memoire dated December 13-17, 2015 in which the FM and procurement paragraphs are blank. The supervising of safeguards might have been stronger. The safeguards team did not approve the SEA, did not follow up the non-receipt of a safety manual, did not have input to the GRM (which was only established in mid-2017), and has no records of client reports on safeguards compliance; nor did it participate in the MTR. 77. At the MTR, the Bank was proactive in getting stakeholders to analyze implementation progress, identify lessons learned, and propose recommendations for restructuring. It followed up quickly with a mission to clarify with the government how best to implement the MTR recommendations. A protracted period ensued in which the GIRoA and the CMU discussed both internally and with one another how to act on the MTR recommendations, vacillating between acceptance and rejection, during which the Bank continued to supervise locally ongoing activities. This extended period of discussion and decisions being taken, reversed and then again reaffirmed and reversed was unproductive if not paralyzing. (See Annex 7). It would have been better for the GIRoA and CMU to act decisively on the MTR recommendations, either by implementing them (as initially agreed) or execute legal remedies to either close or partially suspend the project which was performing unsatisfactorily.31

Justification of Overall Rating of Bank Performance 78. The ICR rates the overall Bank performance as unsatisfactory. At entry the Bank supported the government to achieve a project design albeit with major shortcomings in its technical aspects and implementation arrangements. During implementation, the Bank provided supportive results-focused supervision despite a very challenging implementation environment, with moderate shortcomings in supervision regularity and significant shortcomings in supervising safeguards; and was party to an extended post-MTR discussion that unproductively delayed a decision on the MTR recommendations.

D. RISK TO DEVELOPMENT OUTCOME

79. The risks to the employment and income gains experienced by beneficiaries of Component 3.2 (Promoting Entrepreneurship among Afghan Youth) are primarily economic and social in nature. Any deterioration of the economy may hinder the beneficiaries’ capacities to maintain employment and generate income through their business. Further, the female beneficiaries who started up or expanded an existing business require a certain degree of social mobility for their continued success. Any move towards social/religious norms that restrict women’s mobility will likely negatively impact the female beneficiaries’ ability to maintain their levels of business-related income and employment. With respect to MoLSAMD’s budgeted replication of the Business Grant Scheme for TVET graduates, there is the risk that government ownership of the replication might waver, or the Ministry will not have the necessary capacity to sustain the Scheme successfully; these would be compounded by any budget cutbacks resulting from an economic downturn and/or further fiscal tightening.

V. LESSONS AND RECOMMENDATIONS

31 A partial suspension would have allowed Component 3.2 to complete. As noted above, this component yielded positive results.


Page 26 of 60

80. Skills development should continue to be a priority area for cooperation between the GoA, the Bank and Development Partners. The sectoral challenges remain; the factors subject to government control identified in the ICR that impeded Project performance can be resolved; and sectoral interventions can be effective, as demonstrated by Sub-Component 3.2 and other Development Partner-supported projects (e.g. those supported by GIZ), not to mention prior Bank-supported operations (e.g. ASDP). Any future operations should respect certain technical restrictions. First, priority trades should be identified at the outset (a lesson already incorporated into the ASDP II). Second, given the challenging context, the design should be conservative. Innovations (such as Component 1, Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) should be first done on a pilot basis. The inter-linking of projects - such that the implementation of one depends on the successful implementation of another - should be minimized.

81. MoLSAMD’s capacities to implement projects still need to be strengthened such that, first, the departments/units carrying out implementation functions for a project – whether technical, fiduciary or M&E – answer to one overall authority below the level of Minister with powers of coordination and final decision-making. Second, the Ministry requires augmented capacity to implement large-scale activities and manage TA firms, and should hire capable TA accordingly even if it requires recruiting internationally. Third, the power to sign off on contract deliverables should be given to the head of the department/unit managing the contract, consulting with the M&E unit. Fourth, the M&E function should be further strengthened, ensuring at a minimum that it reports systematically on its monitoring of project activities to the overall project authority and the relevant technical departments/units; that it develops a system to work with other departments/units to monitor any trainings for logistics, quality and effectiveness; and that it is able to technically manage TA contracted to undertake evaluations. The government and the Bank may also consider project implementation solutions that are sometimes used in challenging contexts, such as a Project Coordination Unit embedded within the structure of MoLSAMD. 32

82. The Project implementation arrangements were significantly modified at negotiations in response to concerns expressed by MoF, with deleterious consequences. These concerns should have been addressed earlier in the preparation process, to have sufficient time to design a more robust set of implementation arrangements and capacity building package. Concerns of all stakeholder Ministries should be explicitly addressed during the preparation process, to ensure there is broad government ownership at the outset.

83. The Business Grants Scheme shows evidence for a cost-effective way to create higher levels of employment and income, including for female entrepreneurs. Provided the rigorous conditions of the Business Grants Scheme can be replicated, this scheme should be scaled up. Among the conditions contributing to success were: effective implementation arrangements, in this case a full-time international consultant working on behalf of the Ministry, with both technical and operational responsibilities including managing the TA firm; an openness and flexibility to improve the model based on lessons learned during an initial, small run; substantial field-level staff available to support the firm’s local activities; a gender dimension for training and support; and impact evaluation parameters designed at the outset that imposed a certain rigor and quality on activities implementation.33 Further, a similar scheme could be usefully

32 Other countries in the region have used (and in some cases continue to use) education/training-sector PCUs despite having far fewer challenges in terms of security, political instability and management capacity. For example, Tajikistan used a PIU from the mid-1990s until 2011, at which point it introduced split implementation arrangements (half implemented by a PIU, the other by a PCU with TA-augmented units of the Ministry) followed later by full implementation by the Ministry with TA-augmented units working as a PCU. In Kyrgyzstan, a PIU was used until 2013 after which it began to function as a PCU and place its TA in units of the Ministry, while still keeping an independent PCU Director. 33 For instance, the need to have a relatively large number of similar persons in the treatment and control groups led to rigor in the scoring and sorting of applicants. This in turn helped to identify beneficiaries who were genuinely interested to start up or expand a


Page 27 of 60

combined with technical/vocational training (whereby eligible graduates receive a business grant),34 tested on a limited scale, and scaled up if found to be relatively cost-effective. Indeed, MoLSAMD has included in its budget request for the fiscal year 1398 (2019) funds to enable such grants for graduates of the TVET program.

84. There is substantial value to cooperation between DPs within an overall strategic context. The Project would have been able – had the restructuring recommended at the MTR been implemented – to benefit from GIZ’s work on apprenticeships; while the IADC-supported NATEJA benefited from the Project’s design, lessons learned during Project implementation, as well as the strengthened capacities and procedures created by the Project. The projects supported by the Bank, GIZ and the IADC all work within the same strategic context. Such strategically-bound cooperation should continue, and would further benefit from a strengthened policy and strategy framework that covers all aspects of the TVET landscape.

85. It took a year and a half before a decision was made to finally close early the project and cancel the remaining allocation. This prolonged period of uncertainty hampered staff morale and did not alleviate implementation bottlenecks. The project partners should take advantage of World Bank guidelines to change course when it becomes clear that some project design aspects need to be modified. Within the Bank, the Global Practice and CMU should clearly and consistently convey a joint position when communicating with the client.

86. The results framework failed to include PDO indicators that adequately captured the PDOs. This indicates the need to pay careful attention during the concept, preparation and appraisal stages to the PDO and the wording of the indicators to capture all project activities contributing to similar outcomes.

.

business. See Annex 9 (C3.2 Business Grant Scheme). 34 This was in effect one of the recommended restructuring changes agreed upon at the MTR, which proposed that C1 graduates be eligible for the C3.2 grant. See Annex 7 (Restructuring Agreed at the Mid-Term Review).


Page 28 of 60

ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS

A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: The project focused on two key outcomes. The first outcome was to increase potential for employment among targeted young Afghan women and men in rural and semi-urban areas.

Indicator Name Unit of Measure Baseline Original Target Formally Revised Target

Actual Achieved at Completion

Share of trainees who find employment within 6 months of graduation, with respect to control groups

Percentage 0.00 15.00 0.00 13.00

31-May-2014 31-Dec-2018 31-Oct-2018

Comments (achievements against targets): There was no progress against the indicator for employment and incentive to the training providers as component one (Improving the quality of non-formal Training and Labor Market Outcomes of Trainees) was not implemented. However, the results of the Business Grant Scheme (Sub-Component 3.2: Promoting Entrepreneurship Among Young Afghans) is justified to be captured as outcome for this indicator. Therefore, employment for the beneficiaries increased 13 percentages in respect to that of control group. Objective/Outcome: The second outcome for this project was to increase potential of higher earnings among targeted groups.



Increase in trainees’ earnings, with respect to control

Percentage 0.00 15.00 17.00

31-May-2014 31-Dec-2018 31-Oct-2018


Page 29 of 60

groups

Comments (achievements against targets): The increased in earnings of the trainees with respect to control group is calculated from component 3.2 of Grant Scheme and results are presented in endline report of the impact evaluation, showing 17 percent of increased monthly earning with statistic significance. Exceeding the original target by 2 percentage points. The beneficiaries acknowledge 55% increase as an average revenue for last month before the survey was conducted. Unlinked Indicators



The amount of incentive(s) disbursed to training providers

Amount(USD) 0.00 4000000.00 0.00

31-May-2014 31-Dec-2018 31-Oct-2018

Comments (achievements against targets): Since component 1 of the project did not proceed as planned and no TPs were hired under the project, this indicator remain at the baseline value of zero against the target.



Direct project beneficiaries Number 0.00 44500.00 2482.00

31-May-2014 31-Dec-2018 31-Oct-2018

Female beneficiaries Percentage 20.00 30.00 30.00

31-May-2014 31-Dec-2018 31-Oct-2018


Page 30 of 60

Comments (achievements against targets): The direct project beneficiaries include beneficiaries sub-component 3.2 the Business Grant Scheme.

A.2 Intermediate Results Indicators

Component: Component One: Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees.



Number of teachers from selected non-formal training providers who receive training from TTTI/DMTVET or any other accredited teacher training center

Number 0.00 50.00 0.00

31-Mar-2014 31-Dec-2018 31-Oct-2017

Comments (achievements against targets):



Number of selected non-formal training providers that adopt NOSS for improvement of their teaching quality

Number 0.00 70.00 0.00 0.00

30-May-2014 31-Dec-2018 31-Oct-2018



Page 31 of 60

Component: Component Two: Project Management, Capacity Building of MOLSAMD and M&E



MoLAMD/NSDP staff trained under NATEJA

Number 0.00 100.00 150.00

31-Mar-2014 31-Dec-2018 31-Oct-2018




MIS system established and functional

Text Existing MIS system Fully functional MIS system.

A Labour Market Information System (LMIS) was developed with the Project support along with biometric staff attendance system.

31-Mar-2014 31-Dec-2018 31-Oct-2018

Comments (achievements against targets): An elaborate software-based MIS was developed for M&E in General Directorate of Planning for project activity monitoring, labor market assessment, employment center services with addition of web-based application for Training Providers. Component: Component Three: Piloting Entrepreneurship among Unskilled and Illiterate Young Afghans


Page 32 of 60



Number of new micro and small scale business established and are running

Number 0.00 70.00 1871.00

31-Mar-2014 31-Dec-2018 31-Oct-2018

Comments (achievements against targets): The Grant Scheme endline survey completed results show that an estimate of 75% of small micro and small scale businesses established and running by the closing of the project.


Page 33 of 60

B. KEY OUTPUTS BY COMPONENT

Objective/Outcome 1 Increased potential for employment among targeted young Afghan women and men in rural and semi-urban areas.

Outcome Indicators

1. Share of trainees who find employment within 6 months of graduation, with respect to control groups (to be increased from a baseline of 0 to 15 percent). 2. The amount of incentive(s) disbursed to training providers (to be increased from 0 to USD 4 million). 3. Number of beneficiaries and share of female beneficiaries (to be increased from 0 to 44,500; female share to be increased from 20 percent to 30 percent).

Intermediate Results Indicators

1. Number of teachers from selected non-formal training providers who receive training from TTTI/DMTVET or any other accredited teacher training center. 2. number of selected non-formal training providers that adopt NOSS for improvement of their teaching quality. 3. MOLSAMD/NSDP Staff trained under NATEJA.

Key Outputs by Component (linked to the achievement of the Objective/Outcome 1)

1. There was no progress against the indicator for employment and incentive to the training providers as component one (Improving the quality of non-formal Training and Labor Market Outcomes of Trainees) was not implemented. However, the results of the Business Grant Scheme (Sub-Component 3.2: Promoting Entrepreneurship Among Young Afghans) is justified to be captured as outcome for this indicator. Therefore, employment increased 13 percentages (against a target increase of 15 percent). 2. The number of beneficiaries increased from zero at baseline to 2,482 (as against a target of 44,500); while the share of female beneficiaries at closure was 30 percent meeting the end target. The beneficiaries were business grant recipients under C3.2 (Promoting Entrepreneurship among Afghan Youth).


Page 34 of 60

Objective/Outcome 2 Increased potential for higher earnings among targeted groups (to be increased from 0 to 15 percent)

Outcome Indicators 1. Increase in trainees’ earning, in respect to control groups (to be increased from 0 to 15 percent)

Intermediate Results Indicators 1. Number of new micro and small scale business established and are running.

Key Outputs by Component (linked to the achievement of the Objective/Outcome 2)

1. There was no TPs/trainees were approved for inclusion under Project Component One (Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees) and Sub Component 3.1 (Encouraging Rural Employment). However, the results of the Business Grant Scheme (Sub-Component 3.2: Promoting Entrepreneurship Among Young Afghans) is justified to be captured as outcome for this indicator. Therefore, beneficiaries’ earnings increased 17 percentages (against a target increase of 15 percent). 2. 1871 new micro and small-scale business were established and were verified running by firm responsible to collect assessment data for Sub-Component 3.2 (Promoting Entrepreneurship Among Young Afghans).


Page 35 of 60

ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION

A. TASK TEAM MEMBERS

Name Role

Preparation

Supervision/ICR

Nathalie Lahire, Palwasha Mirbacha Task Team Leader(s)

Rahimullah Wardak, Muhammad Abbass Rahimi Procurement Specialist(s)

Syed Waseem Abbas Kazmi Financial Management Specialist

Najla Sabri Social Specialist

Mabruk Kabir Team Member

Sayed Mujtaba Shobair Environmental Specialist

Ahmad Khalid Afridi Team Member

Tariq Ashraf Social Specialist

B. STAFF TIME AND COST

Stage of Project Cycle Staff Time and Cost

No. of staff weeks US$ (including travel and consultant costs)

Preparation FY14 23.865 116,866.84

FY15 0 292.18

FY17 1.525 7,928.17

Total 25.39 125,087.19

Supervision/ICR

FY14 2.025 17,652.28

FY15 28.300 96,805.89

FY16 36.000 114,496.87


Page 36 of 60

FY17 17.662 69,431.44

FY18 16.772 84,839.69

FY19 9.612 38,344.43

Total 110.37 421,570.60


Page 37 of 60

ANNEX 3. PROJECT COST BY COMPONENT

Components Amount at Approval (US$M)

Actual at Project Closing (US$M)

Percentage of Approval (US$M)

Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees

7.70 1.05 13.6%

Project Management, Capacity Building of MOLSAMD and M&E

3.60 2.04 56.6%

Piloting Entrepreneurship among Unskilled and Illiterate Young Afghans

3.70 2.43 65.6%

Total 15.00 5.52 36.80 percent


Page 38 of 60

ANNEX 4. EFFICIENCY ANALYSIS

1. This annex presents a cost-benefit analysis of the NATEJA using the net present value (NPV) and the internal rate of return (IRR); and analyses certain dimensions of operational efficiencies. The NPV/IRR analysis focuses on the costs and returns of Sub-Component 3.2 (Promoting Entrepreneurship among Afghan Youth), the only component that achieved a clear outcome and has data on returns. 2. The following assumptions and data sources were used to estimate the net present value: 3. The benefit is the incremental income earned by beneficiaries as a result of the Grant. This was determined by first calculating the difference in monthly net income between the treatment and control group individuals at baseline (AFN 630 greater for treatment) and at endline (AFN 1,500 greater), and then subtracting the former from the latter (AFN 870, or USD 11.60). This showed that in relation to the control group, individuals in the treatment group had increased their monthly net income by USD 11.60 on average, at the time of the impact evaluation (IE).35 4. The incremental income generated by the Grant at the time of the IE is assumed to be constant (in real terms) for a period of 15 years, in the base case. The costs are the Project expenditures for all of Component 3 (there were virtually no expenditures on either Sub-Component 3.1 or 3.3) and for a proportion (20 percent) of Component 2. These costs constituted 48 percent of all Project disbursements. The discount rate is 5 percent and the rate of inflation is 2.9 percent.36 5. Using the above, the NPV is USD 1.7 M and the IRR is 12.6 percent. Under the alternative cases examined in the sensitivity analysis (see table below), the IRR ranges from 8.4 percent to 16.4 percent. In the low case, the increased income earned by the beneficiaries is assumed to last only 12 years, declining in value each year by 2 percent; and the discount rate is 6 percent. In the high case, the increased income is assumed to last 20 years, increasing in value each year by 2 percent; and the discount rate is 4 percent. That the harsh assumptions of the low case still yield an IRR above discount indicates that the positive result is robust.

Sensitivity Analysis Low case: income gains

erode 2 percent annually for 12 years; discount rate 6 percent

Base case: income gains constant in real terms for 15 years; discount rate 5 percent

High case: income gains increase 2 percent annually for 20 years; discount rate 4 percent

NPV (USD M) 0.4 1.7 4.6 IRR (percent) 8.4 12.6 16.4

6. The real IRR of the Business Grant Scheme is likely much higher. The calculations above are based on what was in effect a pilot scheme with relatively few beneficiaries, and therefore had learning costs

35 Altai Consulting, 2018. Endline Assessment of Promoting Entrepreneurship Among Afghan Youth, Final Report (v. 10, p. 16). 36 The inflation rate is the average for the previous four years.


Page 39 of 60

and high overheads in relation to the number of grant recipients. The number of beneficiaries could have been scaled up with a disproportionately small effect on operational costs. (This can be seen by the fact that Phase 2 took three months to complete, while Phase 1 – the learning phase – took over a year, with far fewer beneficiaries). If one were to implement the scheme such that overheads were limited to a 12 percent surcharge (a common target for project overheads) on the value of the grant, then the IRR would be 26.7 percent in the base case. 7. Project efficiency can also be analyzed from the perspective of operational efficiency, particularly timing against schedule, efficiency of design, and costs against budget. Component 1 implementation was inefficient in that the recruitment of the Inspection Agency took 15 months. Then, despite mapping 250 TPs, the Agency and Ministry could not agree to sign any letters of understanding between MoLSAMD and a TP, and no training took place. As a result, the component expenditures yielded no returns in terms of the outcomes defined by the Project, viz. certification, employment or starting a business consequent to training. There were nonetheless some training-related benefits pertaining to the development of NOSS, curricula and training materials, as well as the training of TPs’ trainers (see Annex 10); though there are no data available to calculate any returns these activities may have ultimately generated for students affected beyond the Project. 8. The implementation of Component 2 activities was relatively efficient in that most of the capacity building activities were completed within the overall planned schedule. However, the design of the capacity building turned out not to be fully adequate in that it was originally predicated upon NSDP having the main implementing responsibility; whereas at negotiations it was agreed that NSDP would be integrated into MoLSAMD, thereby effectively making various units/departments of MoLSAMD responsible for implementation. This would have required a larger program of capacity building. (See Section III and Annex 11). 9. Sub-Component 3.2 implementation was overall efficient. Activities began late particularly due to significant delays in recruiting the Grant Management Firm (GMF), whose contract was signed only in December 2015. The effect of this delay was alleviated by the Project team at the Ministry (led by the IC hired for Sub-Component 3.2), which worked for nearly a year prior to GMF hiring to prepare and initiate a first phase of activities. (See Annex 9). The design and implementation of the first phase including the placement and strengthening of the provincial ESCs, the selection of a first round of beneficiaries (775), and the disbursement of grants took approximately 15 months; whereas the second round (which included the majority of beneficiaries, or 1,707) took roughly three months. The initial lengthy period of design and implementation was due in part to the fact that such a grant scheme had not been previously implemented by NSDP, as well as to GMF recruitment delay. 10. In general, actual costs were in line with budget forecasts, though there instances of unwarranted excess (e.g. the luxury procurement of desks, see Section IV.B and Annex 12. This annex does not undertake a cost-benefit analysis of Component 1 or (for the most part) Component 2. With respect to Component 1, the costs had no obvious returns in that the activities did not result in graduates being certified, employed or starting a business. One might argue that the NOSS, curricula and training provided to certain TPs resulted in a return, but there are no data one might use to test this hypothesis. With respect to Component 2, the capacity building of MoLSAMD arguably resulted in returns, but again there are no data one can use to test this hypothesis; as per Project design, the returns were to be found in


Page 40 of 60

Component 1 and Sub-Component 3.1, but neither of these achieved any results. Nonetheless, a proportion (20%) of Component 2 expenditures were attributed as costs for the purpose of doing the Cost Based Analysis for Sub-Component 3.2, as arguably a proportion of the capacity building did benefit Sub-Component 3.2 activities. If one were to use all Project expenditures as costs to be set against the returns of Sub-Component 3.2 (and without attributing any further returns to the costs beyond those of Sub-Component 3.2), then the Project IRR is a neglible 1.7% with a negative NPV.


Page 41 of 60

ANNEX 6. SUPPORTING DOCUMENTS (IF ANY)

Altai Consulting, 2018 (September). Endline Assessment of Promoting Entrepreneurship Among Afghan Youth. Final Report. Kabul: Altai Consulting. Islamic Republic of Afghanistan, MoLSAMD, 2018 (July). Implementation Completion and Results Report for NATEJA. Kabul: MoLSAMD Pearl Horizon, 2017 (October). NATEJA Final Report of Grant Management Firm (C3.2). Kabul: Pearl Horizon. World Bank (various). Implementation Status and Results Reports. Archived: 28 June, 2014; 24 December, 2014; 23 June, 2015; 13 January, 2016; 11 August, 2016; 7 February, 2017; 30 August, 2017; 9 April, 2018; 24 October, 2018.


Page 42 of 60

ANNEX 7. RESTRUCTURING AGREED AT THE MID-TERM REVIEW

1. Given the constraints experienced during the first 2.5 years of implementation, the GIRoA and the Bank agreed at the MTR to restructure the Project. The ICR Team found the recommendations to be sound. The main agreements included:

• Component One: (i) deploying additional NGOs directly by MoLSAMD to deliver skills training, (ii) focusing on rural areas and inclusion of unemployed, underemployed, illiterate youth (15-35 year olds), including marginalized populations (IDPs, refugees and returnees), (iii) revising the incentive structure for TPs to cover the cost of training up to US$ 300 per trainee inclusive of administration cost, in addition to the provision of a job placement incentive of US$ 100, (iv) the verification of employment outcomes by a third party after six months, (v) opening the entrepreneurship grant to skills training beneficiaries under a revised modality. It was further agreed that the target number of trainees would be reduced.

• Component Three: Sub-Component 3.1 to be dropped because of limited implementation, and the mandate to be transferred to the AREDP program. (By the time of the MTR, only the following activities had been completed: signing of MoU with AREDP, preparation of a database of Savings Groups, and selection of a treatment group from existing baseline information.) There had been significant delays in initiating procurement of the TP. Subsequently, the Project was not able to identify a suitable and qualified service provider despite multiple re-advertisements. Sub-Component 3.3 to be dropped because GIZ had executed a feasibility study and was piloting an apprenticeship program. Under Sub-Component 3.2, the Business Grant Scheme was to be extended to Component 1 beneficiaries.

2. The agreement to restructure was captured in the Mid-Term Review Aide Memoire (November 22-26, 2016), the Management Letter from the Midterm Review dated January 17, 2017, and finally the letter from the Minister of Finance dated August 16, 2017 formally requesting the restructuring based on the recommendations of the MTR ‘fully endorsed by both MoLSAMD and the MoF’. However, despite MoLSAMD’s initial agreement to restructure at the MTR, there ensued a lengthy period of discussions – not resolved until December 2017 and formally confirmed until July 2018 – within and between WB, MoF and MoLSAMD as to whether the Project should be restructured or closed. During this extended period of uncertainty (January 2017- December 2017) as to the Project’s fate, the decision to restructure seems to have been taken and then overturned twice. In between the January Management Letter and the August MoF letter there was a Bank mission (February 12-14, 2017), during which the Advisor to the Deputy Minister at MoLSAMD overturned the MTR recommendations and ‘conveyed to the mission the desire to terminate the NATEJA project’ (as per the mission’s Back-To-Office-Report). Discussions continued within the Bank between the CMU which advocated early closure and the GP which disagreed. Despite the communication during the February mission, the Bank received a letter from the Minister of Finance dated August 16, 2017 formally requesting the restructuring based on the recommendations of the MTR ‘fully endorsed by both MoLSAMD and the MoF’. 3. The discussions continued between the GIRoA and the CMU, which culminated in an agreement in December 2017 to close the Project. It took until July 2018 for MoLSAMD to request MoF to close the


Page 43 of 60

Project,37 which was quickly followed up by a formal MoF request to the Bank to close the Project early and cancel unspent funds.38 A restructuring paper to this effect was prepared to enable an amendment to the Grant Agreement, the Project closed on October 31st, 2018.

37 In a letter to MoF, dated July 24, 2018 (No. 333/252). 38 The request was made in a letter from the Minister of Finance dated July 31, 2018. The letter further refers to a ‘discussion between the World Bank and MoLSAMD on December 19, 2017’ during which an ‘agreement was reached to complete the ongoing activities and close the project, with canceling of the remaining grant.’


Page 44 of 60

ANNEX 8. RELEVANCE OF PDO

1. Beyond the linkages with the CPF, the PDOs have been well integrated into national policies and strategies. At the time of appraisal, the PDOs supported the Afghan National Development Strategy (ANDS, 2008-13) and its associated National Priority Program 1 (Sustainable Decent Work Through Skills Development and Employment Policies for Job-Rich Growth, 2012) as well as the MOLSAMD Strategic Plan (2013-15). The PDOs:

• Directly supported ANDS Pillar 3 (Economic and Social Development), particularly strategies pertaining to vocational training, skills development and job creation, as well as the cross-cutting strategy to empower women;

• Indirectly supported ANDS Pillar 2 (Governance, Rule of Law and Human Rights), particularly strategies aimed at strengthening public sector institutions and governance and administrative capabilities;

• Were aligned with the objectives of Sub-Component A.2 of the NPP1 (to develop a sustainable LMIS), which foresaw building MOLSAMD capacities to design, manage and analyze national surveys and their results; Sub-Component B.2 (to increase access to improved occupational literacy among employed, unemployed and underemployed populations), which aimed to increase access to occupational literacy training based on an improved curriculum; and Sub-Component C.1 (to expand and strengthen MOLSAMD’s General Skills Development Directorate and the National Skills Development Program), which aimed inter alia to increase access to improved informal skills training, in particular for targeted vulnerable groups.

2. At the time of writing (November 2018), the PDOs continue to support the operative national strategy, the 2017-2020 Afghanistan National Peace and Development Framework (ANPDF). In particular, the PDOs support the following ANPDF strategies:

• Creating jobs (in the agricultural sector), increasing yields and opening markets for farmers (indirectly).

• Increasing labor productivity and investing in human capital. Investments in this sector are to better align education with private sector requirements and increase the current skills base. This will be achieved by expanding vocational and technical education, and investing in women’s education and market employment.

• Reforming the public sector, including building a responsible and effective public administration system (again, indirectly). This will be achieved inter alia by establishing, maintaining and improving professionalism, expanding the use of performance-based management, and improving accountability and transparency.

• Empowering women, using a gender strategy that includes ensuring full access to education, and advancing women in business.)


Page 45 of 60

ANNEX 9. THE BUSINESS GRANT SCHEME (Sub-Component 3.2)

1. This Scheme provided a USD 500 grant to 2,482 persons to start up a new small business, or to expand an existing small business. 30 percent of beneficiaries were women. The Scheme achieved two notable outcomes: 2. Beneficiaries earned a monthly net profit of USD 11.60 greater than a control group of non-beneficiaries.39 3. The unemployment/underemployment rate of beneficiaries was 12 percent vs 25 percent for non-beneficiaries.40 4. To achieve this, Sub-Component 3.2 successfully completed several activities.

• An awareness-raising campaign attracted 17,000 applications with business plans, in three provinces (Kabul, Balkh and Nangrahar).

• After screening, 2,482 individuals (of which 30 percent female) were awarded an incentive grant of USD 500 to start or expand an existing small business, just short of the target of 2,582 written in the project description.41

• Beneficiaries also received business training, which for the women included a module on gender aspects of business; as well as post-training counselling and mentoring.

• A well-designed impact evaluation was conducted that included two rounds of data collection from beneficiaries and a control group; it found that beneficiaries had higher levels of employment and income as compared to the control group. The levels exceeded the targets of the related PDO indicators.42

5. The outcomes suggest that the scheme is worth scaling up in-country and replicating elsewhere. The ICR Team met and corresponded with the major actors involved with the design, implementation and

39 That is, the monthly net profit in the month prior to the post-intervention interview was USD 11.6 greater for the treatment group, after deducting the differential that existed between them at baseline. In brief, this amount represents the income benefit to the beneficiaries of the grant, and is the equivalent of a 138% increase in income. It is equivalent to an annual rate of return of 27.8% on the investment, presuming the month prior to the post-intervention interview was representative of the months between the grant being disbursed and the post-intervention interview. The interviews took place roughly 18 months after grant disbursement. See Altai Consulting, 2018 (v.10) p. 15. The parameters of the evaluation were designed during Project preparation (PAD, pp. 36 ff.), enabling the careful construction of treatment/control samples during implementation, as well as the gathering of baseline and endline data. 40 Altai Consulting 2018 (v.10), p. 14. The following activities were taken as proxies for unemployment/ underemployment: casual labor (4% for treatment; 15% for control) and household chores including fetching water (8%/10%). The employment rate for the treatment group was 88% vs. 75% for the control, which represents a difference of either 13 percentage points or 17.3 percent. 41 There was no intermediate indicator associated with this component. See Section IV.A. 42 See Annex 9 (The Business Grant Scheme (C3.2)).


Page 46 of 60

evaluation in order to capture salient aspects of the design and the implementation arrangements, for the benefit of those who might be interested in scaling up and replication 6. The sequence of main activities was as follows:

• Awareness raising • Applications (including business plan) and uploading data • Screening step one: scoring applications and filtering out non-qualifiers • Scoring and ranking of applications to identify sufficient number of interviewees • Screening step two: interviews and selection of those eligible to be in lottery • Lottery to allocate eligible applicants into two groups: beneficiaries (treatment) and non-

beneficiaries (control). A certain number of persons were also identified to be kept in reserve, in case of drop-out prior to grant receipt.

• Further confirmation of selected beneficiaries, and replacement as needed. • Baseline data collection • Signing of Letter of Understanding between MoLSAMD and beneficiary. • Transfer of funds to recipient’s bank account. (In many cases, this required the beneficiary to

open a bank account, something the Project did not help with). • Initiation/expansion of business activities • Training • Counselling and mentoring • Endline data collection

7. The Ministry hired a particularly able long-term IC who was responsible for the Scheme implementation, including the vital task of managing the TA firm hired to execute activities. The success of the Scheme all testify to his technical capacities, team management, and sound judgement; endurance, persistence and flexibility in the face of challenging and changing circumstances, taking full responsibility for the Scheme from inception through to its conclusion; ability and willingness to spend time in the field; and having the right ‘human touch’ and skills in managing large numbers of people with competing agendas and interests. 8. The IC had ample time prior to the start of the work of the Grant Management Firm (GMF), which he used to work out design and implementation details, working from parameters of the PAD and PIM; was able to facilitate the hiring of a small team of people – and subsequently manage frequent turnover - in each targeted province to help with awareness-raising and the screening of applications; and began to recruit a small, first (or pilot) batch of beneficiaries. This enabled the GMF to hit the ground running, as it were, when it came on board; and moreover, enabled the Ministry and the GMF to acquire the necessary capacities and proper design in preparation for the second, much larger batch. The learning curve and benefits of this ‘pilot’ can be seen in the timelines; while the processing of the first batch took approximately one year from start to finish, the second batch was completed in three months. 9. The Project was able to deploy sufficient numbers of field-level people to assist in the implementation of activities, as and when needed. This took two forms. First, the team mentioned in the previous point were in fact staff of the provincial Employment Service Centres, which nominally were responsible for certain functions pertaining inter alia to the registration of job openings and job-seekers


Page 47 of 60

and matching job-seekers with employers; they also performed valuable service for the Scheme in terms of raising awareness and scoring applications. Second, the GMF itself hired staff. In this regard, the capacity of the GMF to adjust staff levels with relatively little friction was instrumental to the Scheme success. The Scheme activities required varying levels of staff according to the intensity of activities (e.g. vetting of applications and interviewing of applicants were labour intensive, whereas post-disbursement activities were not). The GMF was able to manage such staff fluctuations relatively smoothly to ensure timely and efficient implementation, whereas the Ministry’s comparatively cumbersome hiring procedures would have entailed delays and/or under/over-staffing. 10. Keeping control of the message during awareness raising proved difficult, leading to some misunderstandings that in turn required time to dispel. Better quality staff and a tighter discipline in crafting, delivering and monitoring the communication of key messages to potential applicants might have alleviated these difficulties. 11. The impact evaluation (IE) was designed at the outset of the Project, prior to any implementation activities, so that it was clear what minimum numbers were required in terms of beneficiaries and controls. Further, aside from the intrinsic value of having a well-designed IE, the need for substantially similar control and sample populations imposed a certain rigor on the scoring and sorting of applicants. 12. All applications were scored, and all applicants who got through this stage were interviewed in order to further screen their application and determine their suitability as a candidate. The screening of applications required some work in order to get the process right, as it was difficult to achieve standardization across individuals responsible for scoring. One of the main purposes of the interview was to identify the degree to which applicants were serious in their intent to start/expand a business, and to gauge their reliability as an investor. Given the number of applicants, the interview process was time-consuming. However, the implementing team is convinced that this was instrumental in weeding out individuals who would not have used the grant for their stated purposes, as well as individuals who were not particularly well suited to taking on the responsibilities of running a business. The screening process was a substantial task, yet it seems the MoLMSAMD/GMF treated it seriously and were subsequently rewarded with the outcomes mentioned above. There were over 18,000 applications, and approximately 5,000 individuals were identified through the interview process as being eligible for the lottery which selected grant recipients. (Those not winning the lottery became the control). 13. The LoU was a means of imparting to the beneficiaries the seriousness of their commitment. The official character of the letter, the explaining to the signatory of the clauses, including penalties, and so on – as the IC explained, these were all commitment reinforcers. 14. All participants received training on business skills, with women receiving a supplementary module on gender issues in business.

• A great deal of thought went into the training and support received by the participants, particularly to account for their time constraints and illiteracy. Therefore, it was decided that the training would be short, lasting only one afternoon; to be supplemented by phone counselling and, in some cases, mentoring visits. Further, the training materials were designed to convey as much information pictorially as possible; included practical forms, such as checklists; and were


Page 48 of 60

given to the participants as a handbook. Theory was minimized, focusing on more practical aspects of business. Finally, pedagogy was in part story-based, as it was found that participants responded well to learning from story.

• The training was followed up by phone counselling, during which counsellors would contact the grant recipients to enquire about progress and information needs. The IC noted that the moral support provided by these phone calls was as important as any advice given, and that the recipients appreciated knowing that someone was following up who was sympathetic to the struggles they were going through. On a pilot basis, a sub-group of beneficiaries (based in Kabul) received visits from mentors operating in a similar line of business.

• It was planned to deliver the training at the moment the grant was received, as per usual practice.

However, due to implementation slippages, the training was delivered a few months after the grant was disbursed. This may have been a blessing in disguise, in that the training was now deeply relevant to the participants; they were able to relate the course contents to their business experiences since receiving the grant. In this way, it is possible they were able to extract more utility from the training than if they had received it at the same time as receiving the grant.

15. The ratio of mentees to mentor was roughly 5. The mentors were identified through a local small business association. Each mentor received an honorarium, perhaps just enough to cover their expenses associated with visiting the mentee on-site. 16. Further detailed lessons and recommendations can be found in the GMF’s final report. (Pearl Horizon, Final Report, October 1, 2017).


Page 49 of 60

ANNEX 10. COMPONENT-WISE ACTIVITIES AND RESULTS

1. This Annex presents some of the activities carried out under the Project. Many of them yielded valuable outputs, even though they did not result in PDO-level outcomes. 2. Component 1: Improving the Quality of Non-formal Training and Labor Market Outcomes of Trainees, completed valuable activities, though these fell short of an outcome.

• An Inspection Agent was contracted to map, vet and recommend non-formal TPs. It mapped 250 TPs; inspected 175; and invited 134 to apply to participate in the incentives scheme. 88 TPs submitted a formal application, of which 78 were recommended for approval by the Inspection Agency. (The number selected would later drop to 24, for reasons examined in Section III.B.i).

• Two general guidelines were drafted and adopted on the development of NOSS and curricula; 162 National Occupational Skills Standards (NOSS)-based curricula were either developed or revised; 45 competency-based curricula were developed; and written instructional materials (WIM) were produced for 7 trades.

3. Component Two aimed to strengthen MoLSAMD capacities that were required to facilitate the implementation of Component 1 and Component 3 activities, which were required to achieved the PDO outcomes. In this regard:

• 150 MOLSAMD/NSDP staff were trained in financial management, procurement, curriculum development, labor market analysis, and entrepreneurship. This exceeded the target value for the intermediate indicator (100).

• The NSDP (including NATEJA operations) was integrated into the General Directorate Skills Development (DGSD), and capacities were built across various units of MoLSAMD that assumed the integrated NSDP functions. However, this integration was not part of the Project as appraised in the PAD and was included at negotiations at the behest of MoF.43 The last-minute decision to integrate the NSDP constituted an extra activity of some importance and a significant modification to implementation arrangements, which affected implementation performance. These issues are discussed in detail in Annex 11.

• Three provincial Employment Service Centers (ESCs) were strengthened, each with 5 posts. They provided valuable support to Sub-Component 3.1. However, it is not clear that the ESCs ever functioned effectively in terms of their core mission, viz. registering job openings and job-seekers, and matching the two. Further, the posts were paid for under the Project, and were never formally established and budgeted under the local government authority (taskheel). Hence they are no longer operational.

• A Labor Market Information System (LMIS) was developed with Project support. It is fed internally by the MoLSAMD using data pertaining to ESCs operation and the registration of work permits for Afghan nationals and for foreigners; only the work permits data are currently being entered, and

43 A legal covenant pertaining to the development and implementation of an integration action plan was included in the Grant Agreement. In the event, the plan was not quite fully executed, and the covenant was considered to have been partially complied with at closure.


Page 50 of 60

only one ESC (in Kabul) is currently operational.44 It is to be fed externally from five other ministries, using for example data on graduates and the labor economy; though this aspect is not yet operational. Further, the Project supported the Human Resource Demand and Supply Survey (2016), which provided valuable data for decision-making.45

• 70 non-formal TPs adopted the National Occupational Skills Standards for their courses, meeting the intermediate indicator target.46

• 673 Vocational Trainers working at non-formal TPs were trained on NOSS and curricula content.47

• The capacity building of MoLSAMD’s M&E function is discussed in Section IV.A, particularly as it relates to the Project.

4. Component 3.1 completed some preparatory activities, including a MoU with AREDP to train members of their SGs; and the updating of the SGs membership database as well as the identification of a control group for impact evaluation purposes. However, a suitable TP was never selected despite multiple re-advertisements.48 It was recommended at the mid-term review (November 2016) that the component would be dropped, any activities to be continued would be transferred to the AREDP, and the unused funds would be re-allocated to C1.49 5. Component 3.3 was not implemented. During the second year of the Project, it was agreed that the GIZ study on apprenticeships obviated the need for the component’s planned feasibility study. There were some discussions to use the GIZ findings to design and implement a pilot program, but GIZ’s subsequent work to support the development of an apprenticeships program led to a recommendation at the mid-term review to cease activities under this component and use the unallocated funds elsewhere.50

44 Further, there are two ESCs currently being supported by the IADC-financed NATEJA project; they will also be unsustainable once the project completes, unless the government changes its policy about taksheel financing of the ESCs. None of the ESCs are currently providing data to the LMIS. The LMIS has the capacity through web-based applications to populate databases for job-seekers, employers registration, job-seekers training and capacity building, and job-placement. The ICR Team was not able to receive security clearance to visit the site where the LMIS is operational. 45 The supply side gathered data from households and TVET institutions. The demand side gathered data from formal and informal businesses including SMEs. Data were collected manually and later entered into the databases. The LMIA plans to periodically update the data using its own surveys, with direct data entry using a web-based application; as well as findings from Central Statistical Office (CSO) surveys. 46 The value for the intermediate indicator ‘Number of selected non-formal TPs that adopt NOSS for improvement of their teaching quality’ was registered as 70 in the ISRs from August 2016 to August 2018. However, the final ISR of October 2018 registered a value of 24, presumably to reflect the final number of TPs shortlisted by the Inspection Agent. 47 The value for the intermediate indicator ‘Number of teachers from selected non-formal TPs who receive training from TTTI/DMTVET or any other accredited teacher training center’ remained at zero for all ISRs, because no TPs were formally selected. The trainers used by the Project were NOSS experts on contract with MoLSAMD. The original Project intent was for the training to be conducted by the TTTI that was to be established under the ASDP II, but this component dropped from the ASDP II at restructuring. It is not possible to determine if any outcome resulted from the trainings, which were not monitored for effectiveness. 48 See Annex 12. 49 This was to be part of a restructuring that in fact never took place. 50 Again, through a restructuring that never took place.


Page 51 of 60

ANNEX 11. IMPLEMENTATION AGRANGEMENTS

1. The Project implementation arrangements as laid out in the PAD differed in some important respects from those stipulated in the Grant Agreement. This Annex describes these differences and analyzes their implications. 2. During Project preparation and appraisal, the implementation arrangements identified the NSDP as being responsible for implementation, including fiduciary and monitoring and evaluation functions. It was also stated that the NSDP would ‘strive to build capacity among their counterparts in the civil service.’ 3. However, during negotiations, the MoF objected to these arrangements and advocated for greater implementation responsibility to be given to MoLSAMD. The MoF’s underlying concerns derived from its own evaluation, which were summarized by a consultant hired by the Bank to do an institutional analysis of MoLSAMD as follows: ‘The evaluation report questioned transparency of the bidding procedures and contract award process.’ The analysis also noted that ‘the NSDP methodology for impact monitoring has been highly doubted by MoF … Concern was expressed about the NSDP monitoring and outcomes, particularly in relation to tracer studies and findings that around 86% of vocational training graduates have found employment shortly after completion of training… the lack of third-party evaluation appears to have undermined credibility of these tracer studies.’51 On the basis of these concerns, MoF had ‘refused to disburse required amounts for covering of NSDP’s operating costs.52 4. As a result, it was formally agreed (through the Grant Agreement) that an action plan would be designed and implemented ‘designed at integrating NSDP with the Directorate General for Vocational and Educational Training within MoLSAMD.’53 (This Directorate later became the General Directorate for Skills Development, DGSD). Further, the Grant Agreement contained a legal covenant to the effect that the action plan would be prepared and implemented. 5. A plan was subsequently developed and agreed, with some delay.54 The steps are provided in 51 These same studies were used in the PAD to justify project design. 52 All quotes in the paragraph are from ‘Institutional Analysis of Ministry of Labor, Social Affairs, Martyrs and Disabled.’ World Bank, January 2013. Pp. 40-41. 53 Afghan Reconstruction Trust Fund Grant Agreement of April 11, 2014, p. 5 (I.A.1c). 54 The Grant Agreement stipulated that the action plan would be prepared and furnished to the Bank within six months of Grant

The National Skills Development Program

At Project preparation, the NSDP was one of the National Priority Programs still operating that had been created by the former President of the GIRoA in 2004. It provided technical assistance to the TVET system as well as vocational education trainings across the country with the support of MOF and DPs including the WB. It functioned as a parallel structure and separate program at MoLSAMD, with project staff hired for fixed periods to carry out program activities; its functions were technical and fiduciary, giving it substantial control over program design and execution. It had implemented previous WB-supported operations (e.g. the ADSP), with satisfactory ratings. It had four regional sub-offices, and in the years leading up to 2015 averaged 70-100 staff.


Page 52 of 60

the table below and include remarks about the extent to which the step was implemented.55

Step NSDP unit. To be integrated into

By Actual and remarks

1 TPs registration

DGSD TPs registration unit

31 March, 2015

NSDP unit disbanded, with no posts transferred. All NSDP documents and procedures for registration were handed over. MoLSAMD registration fees remain at 30,000 AFN/year, with only international NGOs registering.

2 Curriculum development and standards

DGSD – curriculum development department

31 December, 2015

NSDP still retain 7 staff, and works closely with DGSD department.

3 M&E DGSD - M&E General Planning Directorate (MoLSAMD)

30 August, 2015 31 April, 2016

NSDP unit disbanded. DGSD has no M&E unit, so 5 NSDP posts transferred to General Planning Directorate (of which one was vacant at time of writing). No training provided to DGSD, though persons occupying the transferred posts did transfer their skills to the department through their regular work.

4 Procurement DGSD – General Administration Directorate of Procurement (MoLSAMD)

30 April, 2015 31 May, 2016

NSDP unit disbanded after November 2016, with two posts transferred. There is no procurement function within DGSD, so both NSDP posts were transferred (with loss of grade and salary) to Directorate of Procurement. Both persons occupying NSDP posts received procurement training through the World Bank, and have worked to transfer skills to other colleagues within Directorate. Procedures of Directorate are somewhat similar to what they were under NSDP, though there are concerns that they take longer to execute and that persons nominated to bids evaluation committee may not always have the requisite technical capacities. Further, government procedures do not conform to those approved for World Bank-supported operations, and transferred staff needed to explain these differences to the Directorate. Note that of the nine major TA packages, only one remained to be procured (the impact evaluation) after the integration process.

5 Finance DGSD – General Administration Finance and Accounts Directorate

30 December, 2015 31 May, 2016

NDSP unit disbanded. Function was integrated, with similar issues identified elsewhere (more time-consuming, concerns about technical proficiency).

Agreement signing (April 11, 2014); to be thereafter implemented within a period of around 12 months (or any other later date agreed in writing with the Bank) after the date of the Grant Agreement. In the event, the parties agreed to develop the plan by 19 February 2015, with a view to it being fully implemented by May 2016. 55 The remarks are based in part on interviews with NSDP staff and MoLSAMD staff who were previously with NSDP.


Page 53 of 60

(MoLSAMD) 6 HR DGSD – General

Administration HR Directorate (MoLSAMD)

31 August, 2015 31 May, 2016

NSDP unit was disbanded. Integrated into Directorate, as DGSD has no HR function. Roughly similar procedures as before, though concerns were expressed that they are more time consuming and that recruitment panels not as technically proficient. Procedures different in an important respect: recruitment is done against a step 1 level, to be possibly renegotiated subsequently; rather than recruiting to a higher-level step given candidate’s experience.

7 Others Skills Support Directorate, DGSD (to be newly established)

31 May, 2016 The new Directorate was not established. Therefore certain functions remain within NSDP, including project development and cooperation/coordination with DPs. NSDP’s IT unit was disbanded and functions integrated with ICT Directorate of MoLSAMD.

6. Given the uneven implementation of the action plan, the legal covenant was considered to have been partially complied with at closing.56 7. The new implementation arrangements had two potent characteristics. First, it created work that had not been identified in the PAD and that moreover created uncertainty among the NSDP staff responsible for the Project implementation. 8. Second, it was a substantial modification of the implementation arrangements that had been appraised. Stated differently, the new implementation arrangements were not appraised. The new arrangements could have reasonably been expected by the Bank to constitute a significant challenge. An institutional analysis of MoLSAMD had been commissioned by the Bank in the previous year and completed in early 2013.57 It found that inter alia ‘the management of the Ministry remains weak. Business processes are complex and involve multiple decision-makers. The senior managers tend to micromanage and make both small, operational and strategic decisions. The MoLSAMD’s structure is complex, large and inefficient. Several directorates across the Ministry and employment service centres have not produced any outputs for years, signaling a lack of commitment and accountability, especially at directorate’s and unit’s levels.’58 The same report concluded that ‘the rationale for the Bank’s continued support to NSDP is also strong’, and provided various recommendations to strengthen the NSDP, none of which entailed integration within MoLSAMD.59 In this light, arguably the Bank should have insisted at negotiations to take further time to discuss and design the implementation arrangements with MoF and MoLSAMD, and to re-appraise them as needed including identifying suitable mitigation measures. 9. The new implementation arrangements likely contributed to some of the implementation delays and difficulties that the Project experienced. First, it took up substantial time from NSDP senior 56 As per the final ISR dated October 24, 2018. In prior ISRs, the Project was considered to be in compliance with the covenant because the final date for plan implementation had not yet been reached. 57 Islamic Republic of Afghanistan. Institutional Analysis of Ministry of Labor, Social Affairs, Martyrs and Disabled. January 2013. 58 Ibid, p. 5 59 Ibid, pp. 45 ff.


Page 54 of 60

management. Second, the new lines of authority meant that persons working in technical, fiduciary and M&E functions no longer answered directly to the NSDP Director, but rather to the Head of the relevant MoLSAMD directorate or unit. This in itself complicated tasks for senior NSDP management, as they no longer had the same authority to take decisions or otherwise act to ensure the timely completion work in these areas. The issue was further complicated by the fact that integration did not proceed fully as initially planned, whereby the departments/units with implementing functions would come under the authority of the DG of the DGSD; in this scenario, all final decision-making authority would have resided in one post. In the event it was decided that certain functions would be transferred to departments/units that answered to different DGs (e.g. HR, finance, procurement and M&E) who in turn did not all answer to one Deputy Minister (DMs) but rather to two DMs and (in the case of M&E) the head of the Office of the Minister. This meant that the NSDP Director had to liaise with several DGs, and in practice often needed to consult with different Deputy Ministers and the Office of the Minister to ensure coordination and decision-making. Third, the procedures followed by the MoLSAMD units/directorates were reported to be more cumbersome and time-consuming. In some cases, they did not conform to normal project management procedures; for example, NSDP senior management was not able to sign off on contract deliverables, which sometimes needed to be approved by the M&E unit in the General Planning Directorate. Fourth, persons occupying posts for the NATEJA project were tasked with building capacities of the units/Directorates to which they were attached. The ICR Team was informed that ‘skills transfer’ – both prior to and after integration – was often taken to mean ‘do work beyond that pertaining to the NATEJA project.’ Finally, the uncertainties and implications of integration (including lower salaries) may have contributed to turnover among NSDP staff and persons occupying posts tasked with NATEJA implementation, which naturally resulted in further delays. 10. The ICR Team has recommended that further activity be conducted in this sector. However, the extent to which how well NSDP integration has enabled MoLSAMD to implement projects must first be analyzed thoroughly, perhaps as part of appraisal of any future operations. As more than one NSDP staff member noted to the ICR Team, the new arrangements have resulted in the ‘disintegration’ of NSDP rather than its integration into MoLSAMD. Given NATEJA’s implementation performance, there is a prima facie case to be made that future implementation arrangements would need to be strengthened. 11. The above notwithstanding, one must be careful not to overstate the negative effect of integration upon implementation timeliness. For example, delays in procurement led the Bank to downgrade the procurement rating from Satisfactory to Moderately Satisfactory to Moderately Unsatisfactory, all by August 2016 – prior to the transfer of the NSDP procurement posts to MoLSAMD’s Directorate of Procurement.


Page 55 of 60

ANNEX 12. PROCUREMENT OF MAJOR TA PACKAGES

1. There were nine major TA packages (firms) in the procurement plan at effectiveness. The failure to recruit the bulk of these packages led to repeated down gradings of the procurement rating in the ISRs, and is part of the explanation as to why the Project performed poorly. This Annex describes the packages, attempts to reconstruct the reasons for why each package was or was not successfully recruited, and identifies lessons to be learned.

No./ Comp.

Contract description Remarks

1/C1 Inspection Agent for TPs

Contracted (OSDLR) with significant delay, in September 2015.

2/C2 Institutional Capacity Building

Was pending during first year of implementation due to integration of NSDP into DGSD. Thereafter, the main ToRs for this package included guidelines (SOP) for TVET course implementation; updating the NSDP strategy; and manuals on HR and finance procedures. In the event, the NSDP revised its SOP and strategy with GIZ support, and manuals were developed with support from Bank-financed TA. Upon recommendation of the Bank, MoLSAMD dropped the package.

3/C3.1 Training of SGs There were significant delays in recruitment. Firms expressing interest did not meet the criteria, resulting in multiple re-advertisements. Eventually a short-list was established, and RFP responses were received in December 2015. A firm was recommended by Evaluation Committee. The WB rejected Committee’s recommendation based on technical reasons pertaining to qualifications of Committee membership. At this point, the MTR took place and government/WB agreed there was no longer sufficient time to recruit the TA, conduct the trainings and complete the component activities. The MTR recommended the component be dropped as part of a restructuring, and hence no further action was taken to recruit a firm.

4/C1, C3.1, C3.2

Awareness Campaign

Firm was selected, with significant delay. In the meantime, the materials and awareness-raising activities were done by NSDP/MoLSAMD and Component 1 TA Firm for Component 1; and by ESCs (MoLSAMD) for Sub-Component 3.2. No progress was made in Sub-Component 3.1, and so no awareness-raising activities were required. Hence there was no need to hire a firm, and package was dropped.

5/C3.2 Grant Management Contracted December, 2015 (Pearl Horizon). Work was to be completed by June 2017, but four-month extension (without extra charge) was granted to complete tasks added to the contract.

6/C3.2 Impact Evaluation Contract with TA firm signed December, 2017 (Hawak Vision) and terminated March 2018 due to poor performance on first deliverable. New contract signed June 2018 (Altai Consultancy Services), and final payments to be made before end of 2018 once final draft is submitted.

7/C3.3 Feasibility Study Package was re-advertized in December 2015 as no firms who had


Page 56 of 60

expressed interest met qualifications, at which time changes were introduced to budget and scope of work. At MTR it was recommended to drop Sub-Component 3.3, the activities of which were being implemented by GIZ. Therefore, the package was dropped.

8/ C1, C3.1, C3.2

Special Purpose Review

The Special Purpose Review was intended to track (on a sample basis) the disbursement of grants to beneficiaries: TPs in Component 1 and Sub-Component 3.1, and small business operators in Sub-Component 3.2. First round of procurement in 2015 cancelled by Bank due to no actual progress made on the ground by Project (ISR of June, 2015). Subsequent initiation of recruitment did not begin until mid-2016 (ISR of August 2016). However, because little or no progress was made in Component 1 and Sub-Component 3.1, and it was eventually decided not to recruit the firm and the package was dropped.

9/all Social and Environmental Assessment

Contract signed in November, 2014 and completed in May of 2015.

2. The various delays referenced in the table above led to the ISR rating to be downgraded from Satisfactory to Moderately Satisfactory in June 2015 (at which point only one package had been procured, for the SEA). It was further downgraded to Moderately Unsatisfactory in August 2016, when the ISR noted that only three TA packages had been contracted (SEA, Inspection Agency for Component 1 and GMF for Sub-Component 3.2), there had been no progress in contracting the TA for Sub-Component 3.1 (TP for the SG members), and the recruitment process for the Special Purpose Review had only recently been initiated. In February 2017, the ISR rated was again Moderately Unsatisfactory because of extensive delays in initiating and finalizing major contracts, the procurement plan not being updated on a regular basis, and the plan not being reflected in the STEP tool; and it remained Moderately Unsatisfactory until Project closure. 3. Though executed on a somewhat more timely basis, the procurement of certain goods was still extensively delayed.60 The Bank also noted other shortcomings such as the procurement of luxury goods in some cases.61 One can also note that some of the equipment procured was to be used to facilitate Project implementation (e.g. vehicles, furniture and IT equipment) yet many of the Project activities were not implemented, which would suggest that the goods did not yield their full value. 4. The ICR Team was informed by the Bank in-country fiduciary team that MoLSAMD is considered to have good procurement capacities. They were one of the first ministries to get accreditation from the national procurement authority to procure on their own, and their accreditation has been regularly renewed without incident. Despite this, its performance on NATEJA was rated Moderately Unsatisfactory at closure for reasons provided above, with the main recurring reason being delays. There are several underlying reasons, only some of which are strictly speaking related to fiduciary capacity. First, there were shortcomings in the proper execution of procurement tasks, as evidenced for example by delays, 60 E.g. Aide Memoire, November 22-26, 2016. 61 Aide Memoire, June 8-15, 2015. For example, 12 desks each costing AFN 24,000 or approximately USD 330 at then-current rates.


Page 57 of 60

failure to regularly update the procurement plan, the procurement of luxury goods, and not using the STEP tool. Second, the procurement function was hampered by the non-execution of predicate activities, for example the TA firms to conduct the Special Purpose Review and the Feasibility Study. Third, the security situation makes international firms (and ICs)62 reluctant to bid on Afghan contracts.

62 For instance, C3.1 foresaw the recruitment of an IC. Despite multiple attempts at recruitment, in the end an IC was not found.


Page 58 of 60

ANNEX 13. THE NATEJA PROJECT FUNDED BY IADC

1. The IADC is funding a project that is in all respects identical to the Bank-financed NATEJA Project (indeed it has the same name), except that it does not include Sub-Component 3.3 (the activities of which are being ably implemented through a GIZ-supported project). That is, the IADC used the PAD as its template, and financed Component 1 and Component 3 activities in geographic zones not covered by the Bank-financed NATEJA Project; while supporting Component 2 activities in complementary fashion to the Component 2 activities financed through the Bank-supported Project. It uses the same implementation arrangements as those contained in the Grant Agreement (though with different project staff, of course), viz. an integrated NSDP is jointly responsible with relevant units of the MOLSAMD for implementation. At the time of writing (December 2018), the IADC NATEJA project is in its third year of implementation, and is proceeding satisfactorily according to IADC representatives; there are no plans for restructuring or early closure. 2. The ICR Team met with the IADC team responsible for the project, and discussed the contrasting fates of the Bank-financed and IADC-financed NATEJA projects, particularly with respect to Component 1. The following factors may help to explain the contrast:

• During the first 2.5 years of implementation, the Bank-financed NATEJA Project had to manage both NSDP integration and implementation. The challenges involved with this dual task are discussed in Section III.B. By the time the IADC project became effective in January 2016, some of these challenges had been alleviated, leading to a smoother implementation process.

• Also, during the first couple of years of implementation, there was some instability in the senior management at MoLSAMD. The Minister who was in place during Project preparation was replaced post-elections, with some delay; the new Minister assumed her duties in mid-2015, and naturally it took some time for her to become familiar with the Project. (This Minister was in turn replaced in mid-2017.)

• The IADC-financed project delayed its implementation in order to get extra funding from the Italian government to cover operational costs of the NSDP, given shortfalls in MoF funding on the discretionary budget. The Bank-financed Project, on the other hand, never received the amounts foreseen from the discretionary budget; the funds provided from the discretionary budget were both less than planned, and were used to support non-project activities. As noted in the body of the ICR text, this contributed to implementation delays.

• In brief, the IADC may have benefited from better timing, in that their project effectiveness came after the period of post-election uncertainty and when the Ministry was growing in implementation capacity.

• The IADC encountered difficulties in finding sufficient TPs offering a broad range of trades in which they trained, much as the Bank-financed NATEJA did. In response, they widened their geographic coverage. (The Bank-financed NATEJA also developed responses, as captured in the MTR


Page 59 of 60

restructuring recommendations;63 but these were not implemented). As a result, they were able to enroll TPs in their first phase of training as follows:

o MoUs signed with 60 TPs, of which 39 had started training at the time of writing. o Total enrolment of the 39 TPs is approximately 12,000. Some of the TPs have multiple

training sites. o The following 21 trades are offered: Iron Work, Mobile Repair, Electric Work, Bed

Sheet Making, Leather Bag Making, Shoe Making, Motor Cycle Repair, Backyard Poultry, Carpet Weaving, Beauty Parlor, Tailoring, Khamak Dozi, Mora Dozi, Charma Dozi, Handy Craft Making, Rug Knitting, Food Processing, Ball Making, Carpentry, Jewelry Making and Yarn Spinning.

• This is in contrast with the first-batch TPs with whom the Bank-supported Project was ready to sign MoUs. Of the 78 TPs on the list for signing LoUs at the time of the MTR, the ICR Team was able to view data for 43 TPs on enrollment and courses offered.

o 43 TPs enrolling 3,472 students in the following 24 trades: tailoring, auto mechanic, plumbing, metal work, electrician, ball-making, carpentry, carpet weaving, computer, BCS/BBA/BSF (secondary academic degrees), poultry, hand crafts, IT, embroidery, English language, journalism, charma dozi, food processing, candy making, flower making, leadership, agriculture, curtain sewing, and project management.

o 19 of these TPs enrolling 1,497 were offering courses in areas that MoLSAMD was concerned would not lead to employment, viz. project management, secondary academic degrees, IT skills, and English language.

o If one takes these 43 TPs to be broadly representative of all 78, then this would correspond to first-batch TPs enrolling approximately 6,300 students; with 44 percent of the TPs enrolling approximately 2,700 students in courses that MoLSAMD was concerned would not lead to employment.

63 See Annex 7.


Page 60 of 60

ANNEX 14. Monitoring and Evaluation

3. This annex amplifies some of the M&E activities that were implemented, and others that were not. 4. The M&E Unit of the Planning Directorate monitored the following Sub-Component 3.2 activities.

• It monitored the applications and screening process, and provided feedback to the ESCs and GMF. • It monitored the trainings and provided feedback to the GMF. However, the monitoring was not

based on a standardized methodology, and no formal reports were filed. • It interviewed 10 percent of Sub-Component 3.2 beneficiaries and provided feedback to the GMF.

However, the raw data generated from the questionnaires was not entered into a database and analyzed, and no formal report was produced.

5. A biometric staff attendance registration system was developed and operationalized for NSDP and later scaled up to cover all MoLSAMD employees at the Ministry in Kabul. However, it was no longer operational at the time of the ICR mission (November, 2018). 6. Some of the planned M&E activities were not executed.

• The independent third-party monitoring exercises, the independent assessments of trainee competencies, and the impact evaluations for Component 1 or Sub-Component 3.1. This was due to the fact the underlying activities that were to be monitored were themselves not carried out.

• The special purpose reviews (to audit the disbursement of incentives) were not carried out because it was judged that – in the absence of incentives being disbursed in large quantities under Component 1 - the scale of Sub-Component 3.2 incentives was too small to justify the SPRs expenditure. (See Annex 12).

• The IE workshops to build NSDP capacity to undertake IE analysis. 7. The failure to operationalize the Project MIS had to do with the fact that the individual hired to design and operate the MIS was not able to transfer his capacities to the two persons working at the M&E unit at MoLSAMD. He reported leaving a User’s Manual. Further, a solution was not found as to where to host the server for the MIS. The original intention was to have it hosted by the Ministry of Communications and Technology, but when the MIS was ready to go live the Ministry was encountering cyber security problems and did not want to assume responsibility for the database. The Project Team approached private companies, but MoLSAMD insisted that the NATEJA staff responsible for the MIS personally guarantee the data’s security, which the staff refused to do. At this stage, it was becoming clear that the Project was very possibly going to be closed early, and consequently the main activities for which the MIS was required (under Component 1 and Sub-Component 3.1) were not going to be implemented. Therefore, there was no longer an impetus to resolve these operational issues.