public disclosure authorized implementation ... - world bank

Document of The World Bank

Report No: ICR00003600

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H3430 TF-91653)

ON A

IDA GRANT

IN THE AMOUNT OF SDR 640,000 (US$ 1 MILLION EQUIVALENT)

AND

MULTI-DONOR TRUST FUND (US$ 27.3 MILLION EQUIVALENT)

TO THE

DEMOCRATIC REPUBLIC OF TIMOR-LESTE

FOR A

NATIONAL HEALTH SECTOR STRATEGIC PLAN SUPPORT PROJECT

December 10, 2015

Health, Nutrition and Population Global Practice East Asia and Pacific Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

ii

CURRENCY EQUIVALENTS (Exchange Rate Effective November 5, 2015)

Currency Unit = US$

1 SDR = US$ 0.72 AUD 1 = US$ 0.71 / US$ 1.00 = AUD 1.40

EUR 1 = US$ 1.09 / US$ 1 = EUR 0.92

FISCAL YEAR January 1 – December 31

ABBREVIATIONS AND ACRONYMS

AusAID Australian Agency for International Development

M&E Monitoring and Evaluation

BETF Bank Executed Trust Fund MDGs Millennium Development Goals BSP Basic Service Package MDTF Multi-Donor Trust Fund CAS Country Assistance Strategy MOH Ministry of Health CHC Community Health Center MOF Ministry of Finance CPV Commitment Payment Voucher MTEF Medium Term Expenditure Framework

DFAT Department of Foreign Affairs and Trade (Australia)

MTR Mid Term Review

DPF Department of Planning and Finance of the MOH

NHGV National Hospital of Guido Valadarez

DPM Department for Partnership Management of the MOH

NHSSP National Health Sector Strategic Plan

EU European Union NHSSP-SP

National Health Sector Strategic Plan Support Project

GoTL Government of Timor-Leste PAD Project Appraisal Document

HMIS Health Management Information System

PDO Project Development Objectives

HSP Hospital Service Package PFM Public Financial Management

HSRDP Health Sector Rehabilitation and Development Project

PMT Project Management Team

HSSP Health Sector Strategic Plan PMU Project Management Unit HSSP-SP

Health Sector Strategic Plan Support Project

RETF Recipient Executed Trust Fund

IDA International Development Association

SAMES Autonomous Medical Supply System (Servico Autonomo de Medicamentos e Equipamentos de Saude)

IHS Institute of Health Sciences SIHSIP Support to the Implementation of the Health Sector Investment Programme

IOI Intermediate Outcome Indicator SISCa Servisu Integradu da Saúde Communitária: Integrated Community Health Service Delivery

ISDS Integrated Safeguards Datasheet SWAp Sector Wide Approach KPI Key Performance Indicator TTL Task Team Leader (World Bank)

LMIS Logistics Management Information System

WHO World Health Organization

iii

Global Practice Senior Director: Timothy Grant Evans

Sector Manager: Toomas Palu

Project Team Leader: Eileen Brainne Sullivan

ICR Team Leader: Hui Sin Teo

ICR Author: Hui Sin Teo

iv

DEMOCRATIC REPUBLIC OF TIMOR-LESTE National Health Sector Strategic Plan Support Project

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 12. Key Factors Affecting Implementation and Outcomes .............................................. 73. Assessment of Outcomes .......................................................................................... 154. Assessment of Risk to Development Outcome ......................................................... 275. Assessment of Bank and Borrower Performance ..................................................... 276. Lessons Learned ....................................................................................................... 307. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 31Annex 1. Project Costs and Financing .......................................................................... 31Annex 2. Details of Components and Outputs by Component ..................................... 34Annex 3. Economic and Financial Analysis ................................................................. 43Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 48Annex 5. Beneficiary Survey Results ........................................................................... 51Annex 6. Stakeholder Workshop Report and Results ................................................... 52Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 53Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 57Annex 9. Project Results Framework –Original and Revised ...................................... 58Annex 10. Country and Health Sector Context at Project Appraisal ............................ 63Annex 11. List of Supporting Documents .................................................................... 65MAP

v

A. Basic Information

Country: Timor-Leste Project Name: Health Sector Strategic Plan Support Project

Project ID: P104794 L/C/TF Number(s): IDA-H3430,TF-91653

ICR Date: 11/15/2015 ICR Type: Core ICR

Lending Instrument: SIL MDTF

Borrower: GOVERNMENT OF TIMOR-LESTE

Original Total Commitment:

US$ 20.3M Disbursed Amount: US$ 23.2M

Revised Amount: US$ 28.3M (US$ 1M IDA; US$ 27.3M MDTF)

Environmental Category: C

Implementing Agencies: Timor Leste Ministry of Health

Cofinanciers and Other External Partners: Australia Department of Foreign Affairs and Trade (DFAT); European Union (EU) B. Key Dates

Process Date Process Original Date Revised / Actual Date(s)

Concept Review:

02/12/2007 Effectiveness: 06/17/2008 06/17/2008

Appraisal: 09/11/2007 Restructuring(s): 03/05/2012 02/27/2013 10/31/2014

Approval: 12/13/2007 Mid-term Review:

07/18/2010 07/18/2010

Closing: 06/30/2013 06/15/2015 C. Ratings Summary C.1 Performance Rating by ICR

Outcomes: Moderately Unsatisfactory

Risk to Development Outcome: Substantial

Bank Performance: Moderately Unsatisfactory

Borrower Performance: Moderately Unsatisfactory

vi

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Moderately Unsatisfactory

Government: Moderately Unsatisfactory

Quality of Supervision:

Moderately Satisfactory

Implementing Agency/Agencies:

Moderately Satisfactory

Overall Bank Performance:

Moderately Unsatisfactory

Overall Borrower Performance:


C.3 Quality at Entry and Implementation Performance IndicatorsImplementation Performance

Indicators QAG Assessments (if any)

Rating

Potential Problem Project at any time (Yes/No):

Yes Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:


D. Sector and Theme Codes Original Actual

Sector Code (as % of total Bank financing)

Central government administration 12 12

Health 83 83

Sub-national government administration 5 5

Theme Code (as % of total Bank financing)

Child health 17 17

Health system performance 33 33

Malaria 17 17

Nutrition and food security 17 17

Population and reproductive health 16 16 E. Bank Staff Positions At ICR At Approval

Vice President: Axel van Trotsenburg James W. Adams

Country Director: Franz Drees-Gross Nigel Roberts

vii

Practice Manager: Toomas Palu Fadia M. Saadah

Project Team Leader: Eileen Brainne Sullivan Timothy A. Johnston

ICR Team Leader: Hui Sin Teo --

ICR Primary Author: Hui Sin Teo -- F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) To improve the quality and coverage of preventive and curative health services, particularly for women and children, in order to accelerate progress toward the health (Millennium Development Goals) MDGs. Revised Project Development Objectives (as approved by original approvingauthority) To support the Government of Timor-Leste to get more resources to where they are needed most, to improve the delivery of health services in districts and sub-districts. (a) PDO Indicator(s) The table of PDO indicators contains 17 indicators: 5 original indicators; 4 indicators added during the February 2013 Level 1 restructuring / Additional Financing (AF); 4 indicators added during the October 2014 restructuring, and 4 indicators added during the Implementation Completion Report (ICR). The indicators are marked “Original” if they were from the PAD, “2013 revision” if they were introduced in the February 2013 Level 1 restructuring, “2014 revision” if they were introduced in the October 2014 restructuring, and “ICR” if they were introduced at the ICR stage. Modifications to the phrasing of indicators, formally revised targets, and dropped indicators across the duration of the project are outlined in the Comments sections. A summary of changes to the Results Framework is included in Annex 9.

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

KPI #1 (Original): % of children under 1 year vaccinated with (i) DPT3 and (ii) measles

Value: (i) DPT3: 63% (ii) Measles: 61% HMIS 2006

(i) DPT3: 90% (ii) Measles: 90%

(i) DPT3: 76.8% (ii) Measles: 82% HMIS 2014

Date: December 31, 2006 June 30, 2013 December 31, 2014Comments: Partially achieved. Indicator dropped at February 2013 restructuring.

However, the HMIS continues to track these indicators. They are reported above and will be considered in this ICR.

viii



approval documents)




KPI #2 (Original): % of births attended by skilled health personnel Value: 27%

HMIS 2006 45%

57.7% HMIS 2014

Date: December 31, 2006 June 30, 2013 December 31, 2014Comments: Achieved (target surpassed). Indicator dropped at February 2013

restructuring. However, the country’s HMIS continues to track this indicator, and it is reported above and will be considered in this ICR.

KPI #3 (Original): % of pregnant women receiving at least four antenatal visits Value: 36%

HMIS 2006 55%

42% HMIS 2014

Date: December 31, 2006 June 30, 2013 December 31, 2014Comments: Not achieved. Indicator dropped at February 2013 restructuring.

However, the country’s HMIS continues to track this indicator, and it is reported above and will be considered in this ICR.

KPI #4 (Original): % of children (6-59 months) receiving vitamin A supplements Value: 36%

DHS 2003 80%

66% HMIS 2014

Date: November 14, 2007 June 30, 2013 December 31, 2014Comments: Partially achieved. Indicator dropped at February 2013 restructuring.

However, the country’s HMIS continues to track this indicator, and it is reported above and will be considered in this ICR.

KPI #5 (Original): % married or cohabitating women using modern contraceptives (contraceptive prevalence rate)

Value: 10% DHS 2003

25% 49.9% HMIS 2014

Date: November 14, 2007 June 30, 2013 December 31, 2014

Comments:

Achieved (target surpassed). Indicator dropped at February 2013 restructuring. However, the country’s HMIS continues to track this indicator, and it is reported above and will be considered in this ICR. The ICR mission was informed that measurement of this indicator is likely to be inaccurate. It is more likely to reflect contraceptive issuance at outpatient visits to women of reproductive age, rather than actual usage.

KPI #6 (2013 revision): Number of districts that receive recurrent expenditures consistent (+/- 20%) with the NHSSP MTEF and approved budget Value: N/A 8 0

Date: December 31, 2012 June 15, 2015 May 31, 2014 Source: MOH

ix



approval documents)




Comments: Not achieved. New indicator adopted at February 2013 restructuring. Dropped in October 2014 restructuring.

KPI #7 (2013 revision): Number of districts that report expenditure against the approved plan/budget on time Value: N/A 8 0

Date: Dec 31, 2012 June 15, 2015 May 8, 2014 Source: MOH

Comments: Not achieved. New indicator adopted at February 2013 restructuring. Dropped in October 2014 restructuring.

KPI #8 (2013 revision): Percentage of stock-outs of tracer essential drugs at (i) Autonomous Medical Supply System (Servico Autonomo de Medicamentos e Equipamentos de Saude) SAMES, (ii) health facilities.

Value: (i) 22% (ii) No baseline

(i) 2% (ii) 5%1

(i) 15% (ii) 30%

(i) 20% Source: SAMES (ii) 43% Source: MOH

Date: December 31, 2012 June 15, 2015 June 15, 2015

December 31, 2014

Comments:

Not achieved. This is a revised version of an original IOI (see IOI #7). The IOI was revised in Jan 2013 to the following: “% of stock outs of tracer essential drugs at health facilities”. It was brought up to the PDO Indicator level. At the same time, “% of stock outs of tracer essential drugs at SAMES” remained as an IOI. This revised PDO indicator was then dropped in the October 2014 restructuring; and KPI #11 listed below in this same table was introduced as an alternative. However, stock outs continue to be measured and are reported here and will be used in the ICR.KPI #11 is reported on separately, below. Baseline for (ii) was set during the February 2013 restructuring, at 40% (proxy data from 2011 survey).

KPI #9 (2013 revision): Joint annual sector review and planning meetings held as planned (at least 1/year) Value: No Yes Yes


Comments: Achieved. New indicator adopted at February 2013 restructuring. Not changed in October 2014 restructuring. Valid at project closing.

1 These target figures are inferred from the PAD. This indicator was an IOI in the PAD: “Availability of tracer essential drugs at (i) SAMES; (ii) health facilities”, with targets at 98% and 95% respectively. As such, the target for stock outs would be the reciprocal: 2% and 5% respectively.

x



approval documents)




KPI #10 (2014 revision): Number of requests from budget divisions receiving advances per quarter and which are acquitted within 30 days following end of calendar quarter (National Hospital not included) Value: 9.5 15 18

Date: June 30, 2014 June 15, 2015 May 22, 2015 Source: MOH

Comments: Achieved. New indicator adopted in October 2014 restructuring. KPI #11 (2014 revision): Management of expiry dates leads to reduction of expired medicine at warehouse. Value: 7.2% 10% 4.6%

Date: July 1, 2014 June 15, 2015 May 22, 2015 Source: MOH

Comments:

Unknown. Target is higher than baseline. However, definition of indicator suggests that a reduction, rather than increase, would indicate improvement. This could be due to unknown or inaccurate baseline data at the time of adopting the indicator. There are also different figures reported: 4.6% in May 2015 ISR, versus 20% in an update from the medicines warehouse (SAMES) for 2014. Therefore, final achievement of this indicator is marked unknown.

KPI #12 (2014 revision): Number (percentage) of facilities having taken the readiness assessment survey and who score at least 80%

Value: 54% 80% 2% of Health Posts; 17% of Community Health Centers

Date: June 30, 2014 June 15, 2015 December 31, 2014Source: MOH

Comments: Not achieved. New indicator adopted in October 2014 restructuring. KPI #13 (2014 revision): Number of facilities (districts) that have undertaken a facility readiness survey Value: 3 13 13

Date: June 30, 2014 June 15, 2015 May 22, 2015 Source: MOH

Comments:

Achieved. New indicator adopted in October 2014 restructuring. All 13 districts undertook the facility readiness survey with support from MOH in 2014. Further, four districts undertook the survey independently in 2015.

KPI #14 (ICR): Antenatal care dropout rate: Percentage point difference between % of women receiving four or more antenatal care visits, and % of women receiving just one antenatal care visit Value: 22 percentage 10 percentage 23 percentage

xi



approval documents)




points (ANC4 = 35%; ANC1 = 57%) HMIS 2008

point decrease points (ANC4 = 42%; ANC1 = 65%) HMIS 2014

Date: 2008 June 15, 2015 2014 Source: HMIS

Comments: Not achieved. Indicator introduced at ICR to evaluate the quality aspect of the original PDO. Target inferred from target for KPI #3 (ANC4 to reach 55%), and taking into consideration further improvements in ANC1.

KPI #15 (ICR): % of SISCa posts that function according to Category A standard

Value: 31% Source: MOH

60% 44% Source: MOH

Date: 2009 June 15, 2015 2014

Comments:

Not achieved. Indicator introduced at ICR to evaluate the quality aspect of the original PDO. Target inferred from KPI #12: Significant (approx. 30 percentage point) improvement in functionality and readiness of health service delivery post. This should be reasonable given the longer lead time for improvements (2009 – 2014) assumed, as compared to the shorter duration for KPI #12 (one year).

KPI #16 (ICR): Average number of outpatient visits to health facilities, per capita

Value: 1.9 HMIS 2009

5 2.9 HMIS 2014

Date: 2009 June 15, 2015 2014 Source: HMIS

Comments: Not achieved. Indicator introduced at ICR to evaluate improvement of health services in the revised PDO. Target set based on WHO guidelines / benchmark.

KPI #17 (ICR): % of health posts with resident (i) midwives, (ii) nurses

Value: N/A (i) 65% (ii) 90%

(i) 43% (ii) 54%

Date: -- June 15, 2015 2014 Source: MOH

Comments:

Not achieved. Indicator introduced at ICR to evaluate improvement of health services in the revised PDO. Target set based on actual figures of total number of midwives and nurses in Timor in 2014, and appropriate distribution of health professionals across facilities.

xii

(b) Intermediate Outcome Indicator(s) The following table includes 26 indicators: 14 original indicators, 8 new indicators added during the February 2013 restructuring / AF, and 4 new indicators added during the October 2014 restructuring.



approval documents)




IOI #1 (Original): % of health clinics providing the comprehensive Basic Service Package (BSP) Value: 0% 80% No data Date: December 31, 2006 June 30, 2013 -- Comments: Unknown. Indicator dropped at February 2013 restructuring. The ICR

mission found out that there was no mechanism established and agreed on for measuring this indicator. This is discussed in the M&E section of the ICR. Further, the ICR mission was informed that KPI #12 (see above) was developed in order to measure implementation of the BSP. In practice, KPI#12 replaced IOI #1.

IOI #2 (Original): % of pregnant women receiving tetanus toxoid (TT) injection Value: N/A N/A 50% Date: December 31, 2006 June 30, 2013 December 31, 2014Comments: Unknown. Indicator dropped at February 2013 restructuring. Unable to

assess achievement due to absence of baseline and target. IOI #3 (Original): Cesarean section rate Value: N/A N/A 1.3% Date: December 31, 2006 June 30, 2013 June 2012 Comments: Unknown. Indicator dropped at February 2013 restructuring. Unable to

assess achievement due to absence of baseline and target. IOI #4 (Original): Number (%) of children (6 mo – 3 yrs) participating in integrated community nutrition programs Value: TBD 60% No data Date: December 31, 2006 June 30, 2013 -- Comments: Unknown. Indicator dropped at February 2013 restructuring. Activity

dropped from project. IOI #5 (Original): Number of referral hospitals implementing the Hospital Service Package (HSP)

xiii



approval documents)




Value: 0 6 No data Date: December 31, 2006 June 30, 2013 --

Comments: Unknown. Indicator dropped in February 2013 restructuring. Activity dropped from project.

IOI #6 (Original): Number of referral hospitals implementing guidelines for hospital management and clinical quality, including (i) monthly clinical review meetings; (ii) maternal death audits Value: 0 6 No data Date: December 31, 2006 June 30, 2013 -- Comments: Unknown. Indicator dropped in February 2013 restructuring. IOI #7 (Original): Availability of tracer essential drugs at (i) SAMES; (ii) health facilities

Value: (i) 88%; (ii) N/A (i) 98%; (ii) 95%See comments

See comments

Date: June 30, 2008 June 30, 2013 --

Comments: N/A. This IOI was revised during the February 2013 restructuring. See comments under PDO KPI #8.

IOI #8 (Original): Percentage of biomedical equipment in hospitals that is out of order (non-functional) Value: N/A 10% No data Date: December 31, 2006 June 30, 2013 --

Comments: Unknown. Indicator dropped in February 2013 restructuring. Unable to assess achievement of indicator as there was no baseline and no data collected on it.

IOI #9 (Original): (i) Total health staff trained with Project financing; (ii) % of district and CHC managers that have received management training Value: 0 (i) TBD; (ii) 70% No data Date: December 31, 2006 June 30 ,2013 -- Comments: Unknown. Indicator dropped in February 2013 restructuring. IOI #10 (Original): % of health facilities submitting completed HMIS monitoring reports 1 month after end of each quarter. Value: TBD 90% 100% Date: December 31, 2006 June 30, 2013 June 15, 2015

Comments: Achieved. Indicator dropped in February 2013 restructuring. The ICR mission was informed that all reports are received on time, but that data quality is a concern.

IOI #11 (Original): (i) % recurrent expenditures on hospitals (<40%); (ii) % State Budget allocated to health; (iii) % execution of health budget

Value: (i) 38% (ii) No baseline

(i) <40% (ii) No target set

(i) No data (ii) 5.2% (for 2015)

xiv



approval documents)




(iii) No baseline (iii) No target set (iii) 89% (for 2014)

Date: December 31, 2006 June 30, 2013 September 2015 Source: MOF

Comments: Unknown. Indicator dropped in February 2013 restructuring. However, data is available for (ii) and (iii). Baselines were not set in the PAD. However, actual figures were: (ii) 4.3% in 2009; (iii) 94% in 2011.

IOI #12 (Original): Number of partners submitting to MOH a fully costed work program for forthcoming year Value: 3 8 No data Date: December 31, 2006 June 30, 2013 -- Comments: Unknown. Indicator dropped in February 2013 restructuring. IOI #13 (Original): Number of pilot initiatives (i) financed by Project and implemented; (ii) evaluated with lessons disseminated; (iii) scaled up and/or influenced policy

Value: 0 (i) 12; (ii) 10; (iii) 5

No data

Date: December 31, 2006 June 30, 2013 -- Comments: Unknown. Indicator dropped in February 2013 restructuring. IOI #14 (Original): Number of formal MOH-private sector / NGO partnership contracts signed Value: 1 10 6 Date: December 31, 2006 June 30, 2013 June 30, 2011

Comments: Not achieved. Indicator dropped in February 2013 restructuring. Nov 2012 ISR reports that all 6 contracts that were signed ended in June 2012.

IOI #15 (2013 revision): Health MTEF prepared based on costing of strategic plan

Value: MTEF based on new strategic plan no available

Available and updated

No

Date: December 31, 2012 June 15, 2015 May 30, 2014

Comments: Not achieved. Indicator adopted in February 2013 restructuring. Indicator dropped in October 2014 restructuring. Activity cancelled from project.

IOI #16 (2013 revision): Number of districts that submit annual district plan/budget to the Central MoH for approval on time Value: 0 9 13 Date: December 31, 2012 June 15, 2015 May 30, 2014

Comments:

Achieved. Indicator adopted in February 2013 restructuring. Indicator dropped in October 2014 restructuring, but was achieved for 2014. However, the final ISR (May 2014) that records this indicator notes that the process of planning/budgeting needs to be improved.

IOI #17 (2013 revision): Percentage of targeted staff who pass competency test in financial management, procurement, and/or management

xv



approval documents)




Value: 0% 60% 50% 0%

Date: July 31, 2014 June 15, 2015 June 15, 2015

May 22, 2015 Source: MOH

Comments:

Not achieved. Indicator adopted in February 2013 restructuring. Revised in October 2014 restructuring to the following: “Percentage of targeted staff who pass competency test in 7 Public Financial Management (PFM) areas organized by the MOF”, with a revised target of 50%

IOI #18 (2013 revision): Percentage of SAMES’ procurement activities using direct contracting (emergency order, in $ value) Value: 47% 30% 0% No data

Date: December 31, 2012 June 15, 2015 June 15, 2015

May 22, 2015

Comments:

Unknown. Indicator adopted in February 2013 restructuring. Revised in October 2014 restructuring to: “Percentage of SAMES’ procurement activities using direct contracting (emergency order, in $ value) from Nov 2011 to Oct 2013”. The May 2015 ISR records the actual value as 0%, and provides a comment that “SAMES does not procure medical drugs and supplies”. The revised indicator was not understood. 0% would have been an accurate reflection of the fact that SAMES no longer procured medical drugs and supplies from Oct 2013 onwards. However, the revised indicator was asking for retrospective reporting on SAMES procurement from Nov 2011 to Oct 2013. As the indicator was not understood and measured accurately, its result is marked as unknown.

IOI #19 (2013 revision): Number of targeted places (13 districts and 6 hospitals) with functional National Logistics Management Information System (LMIS) Value: 0 11 15 0

Date: Dec 31, 2012 June 15, 2015 June 15, 2015

May 22, 2015 Source: MOH

Comments: Not achieved. Indicator adopted in February 2013 restructuring. Revised in October 2014 restructuring to: “Number of districts and hospitals with functional National LMIS”

IOI #20 (2013 revision): A computerized drug registration system available

Value: No computerized system

Functional system available

No functional system available

Date: Dec 31, 2012 June 15, 2015 May 22, 2015

Comments: Not achieved. Indicator adopted in February 2013 restructuring. No change in October 2014 restructuring. Indicator valid at project closing.

IOI #21 (2013 revision): Number of structured sector coordination meetings held to progress toward one plan, one budget, and one sector M&E framework Value: 0 2 2

xvi



approval documents)





Comments: Achieved. Indicator adopted in February 2013 restructuring. No change in October 2014 restructuring. Indicator valid at project closing.

IOI #22 (2013 revision): Percentage of research projects approved by the research cabinet that responds to the national research priorities defined by the MoH Value: N/A 60% 0% Date: December 31, 2012 June 15, 2015 May 30, 2014

Comments: Not achieved. Indicator adopted in February 2013 restructuring. Indicator dropped in October 2014 restructuring.

IOI #23 (2014 revision): Number of pilots with function mSupply / mobile mSupply by 2015 Value: 0 3 3 Date: July 31, 2014 June 15, 2015 May 22, 2015 Comments: Achieved. Indicator adopted in October 2014 restructuring. IOI #24 (2014 revision): Percentage of rejected Commitment Payment Vouchers on a quarterly basis

Value: 7% 5% 7% Source: MOF

Date: August 31, 2014 June 15, 2015 December 31, 2014Comments: Not achieved. Indicator adopted in October 2014 restructuring. IOI #25 (2014 revision): Number of districts implementing supportive supervision Value: 3 13 13 Date: July 31, 2014 June 15, 2015 May 22, 2015 Comments: Achieved. Indicator adopted in October 2014 restructuring. IOI #26 (2014 revision): Amount of Department of Pharmacy procurement activities using direct contracting (sole source) in $ value from October 2013 to date Value: US$ 182,845 Not specified No data Date: June 30, 2014 June 15, 2015 May 22, 2015

Comments: Unknown. New indicator adopted in October 2014 restructuring. No target set, and information not shared with the Bank.

xvii

G. Ratings of Project Performance in ISRs

No. Date ISR Archived

DO IP Actual Disbursements (US$ millions)2

1 09/18/2008 Satisfactory Satisfactory 0.07 2 03/02/2009 Satisfactory Moderately Satisfactory 0.07 3 11/09/2009 Satisfactory Moderately Satisfactory 1.48 4 04/12/2010 Moderately SatisfactoryUnsatisfactory 2.63

5 02/12/2011 Satisfactory Moderately Unsatisfactory

7.16

6 01/22/2012 Satisfactory Moderately Unsatisfactory

13.67

7 06/06/2012 Moderately Unsatisfactory


14.66

8 11/20/2012 Moderately SatisfactoryModerately Unsatisfactory

15.53


Unsatisfactory 18.12


Unsatisfactory 18.12



19.98



21.34



23.01



23.01

H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in

US$ millions

Reason for Restructuring & Key

Changes Made DO IP

03/05/2012 -- S MU 13.67 Amend Grant Agreement to increase allocation for RETF project funds;

2 Actual disbursements for ISR 1-4 derived from Client Connections disbursement portal, as Trust Fund disbursements were not recorded in these ISRs.

xviii

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in

US$ millions

Reason for Restructuring & Key

Changes Made DO IP

Include operating costs as eligible expenditure

02/27/2013 Yes MS MU 16.24

Amend project name, extend Closing Date, revised PDO and RF, change of project scope and activities, Additional Financing

10/31/2014 -- MU MU 21.93

Reduce project scope, amend RF, reduce contribution from co-financer Australian Dept. of Foreign Affairs and Trade

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal 1. Timor Leste is a relatively young country, having fully gained independence only in May 2002, following a campaign of violence after a 1999 referendum for autonomy from Indonesia. The early years of independence had focused on restoring peace and stability in the country, supported by United Nations missions. Yet, at project appraisal, Timor Leste had just endured a period of political crisis and violence in 2006. With the help of international peacekeeping forces, a measure of stability was restored. Presidential and parliamentary elections had been held just 5 months prior to project appraisal, resulting in the installation of a new coalition government. A relative calm was restored following elections, and the country was entering an uncertain yet new phase of stability. Thankfully, the Government of Timor Leste (GoTL) had a strong fiscal position as a result of growing petroleum revenues in recent years. However, weak capacity to execute the budget was a major constraint in its ability to effectively utilize these resources. Poverty levels remained the highest in the region, unemployment among youth was high and contributed to social tensions, and an estimated 70,000 people were still in Internally Displaced Persons’ camps in the capital Dili and surrounding districts. 2. In this context, it was extremely important for the government to demonstrate to its population that it could effectively deliver services. Much of Timor Leste’s physical infrastructure had been left in ruins by the late 1990s, including nearly 80 percent of health centers. Most Indonesian citizens had departed, resulting in a severe shortage of qualified and experienced professionals, and leaving Timor-Leste with very thin human resource and institutional capacity, including in the health sector. The early years of independence thus focused on reestablishing basic infrastructure and services. The Ministry of Health (MOH) continued to operate through the political crisis and unrest of 2006, ensuring that health services remained functional. While there had been progress in improving health outcomes in the early 2000s, the country nonetheless had a significant way to go in rebuilding functional and good quality health services, and institutional capacity for managing the health sector. Key challenges that remained included persistently poor health status particularly for women and children and in communicable diseases; inadequate quality and low utilization of health services; weak systems for planning and monitoring and evaluation; and challenges in efficient budget execution and proper expenditure management. Strengthening both supply and demand sides of health service delivery and tackling governance and planning that underpinned service delivery were the continued challenges at the time of appraisal. More detailed descriptions of the country and health sector contexts at appraisal can be found in Annex 10.

3. The World Bank’s (hereafter “the Bank”) Country Assistance Strategy (CAS) at that time (2005) focused on three pillars: (i) delivering sustainable services, (ii) creating productive employment, and (iii) strengthening governance. It aimed to support Timor Leste in consolidating early progress in its post-independence years, and move from a focus on post-conflict issues to creating conditions for growth and poverty reduction. In

2

the health sector, the government’s Health Sector Strategic Plan (HSSP) 2008-2012 laid out three key objectives: (i) Improved accessibility to and demand for quality health services, (ii) Strengthened management and support systems, and (iii) Strengthened coordination, planning, and monitoring. The HSSP provided the overall framework for this project: the HSSP Support Project (HSSP-SP). The project was designed to be fully aligned with the objectives of the HSSP. The project also supported pillars (i) and (iii) of the CAS: to deliver sustainable health services and to strengthen health sector governance.

4. The Bank has played a leadership role in health sector dialogue and coordination in Timor Leste since the country’s independence. HSSP-SP was preceded by two Bank-supported projects in the health sector. The first Health Sector Rehabilitation and Development Project (HSRDP1) (June 2000 – Dec 2005) was an emergency relief grant which aimed to restore access to a basic package of health services, and lay the foundations for health policies and health system development. HSRDP2 (June 2001 –July 20083) aimed to rehabilitate and develop a cost-effective and financially sustainable health system in Timor Leste, to be responsive to the immediate basic health needs of the population and, within a well-integrated and sustainable health policy framework, to prepare the health system to meet future needs. HSRDP 1 and 2 closed with Outcome ratings of Satisfactory and Moderately Satisfactory, respectively. This project aimed to draw heavily on the lessons learned in the previous two projects.

5. In this context, both GoTL and development partners wanted the Bank to maintain its engagement in the health sector, despite the Bank’s limited grant resources. Simultaneously, the Australian Department of Foreign Affairs and Trade (DFAT; then known as AusAID), who had been an active partner in the health sector, wished to increase its financial contribution and engagement in policy dialogue, but wanted to do so jointly with the Bank. There was thus an agreement for a joint Bank-AusAID project, with combined funding from IDA and AusAID under a Multi-Donor Trust Fund (MDTF). A joint project would also facilitate donor coordination and dialogue, and create a framework to further reduce the burden of donor requirements on the government.

1.2 Original Project Development Objectives (PDOs) and Key Indicators 6. The Project Development Objective (PDO), as listed in the Financing Agreement dated April 16, 2008 and in the Project Appraisal Document (PAD) was: To improve the quality and coverage of preventive and curative health services, particularly for women and children, in order to accelerate progress toward the health Millennium Development Goals (MDGs), inter alia, through (a) financial support using the Health Sector Strategic Plan and MTEF framework, including (i) improvement of accessibility, demand and quality of health services; (ii) strengthening of support services, human resource development, and management; and (iii) strengthen of coordination, planning and monitoring; and (b) addressing challenges and innovations needed for the health sector to be prepared for the next generation of issues. This PDO will hereafter be referred to as

3 The HSSP-SP PAD states that HSRDP2 would be extended until June 30, 2008. However, the ICR for HSRDP2 and the IEG Review of the ICR state that the project closed on July 31, 2008.

3

the original PDO. Given that the specific objectives under the PDO actually describe project component objectives, for evaluation purposes only the main part of the PDO will be used for the evaluation of this project (in italics above). 7. The Key Performance Indicators (KPIs) corresponding to parts (a) and (b) above are as follows: Part (a): The KPIs included sector-wide health service PDO indicators such as vaccination coverage, contraceptive prevalence, and trained birth attendance. Intermediate indicators linked with activities and outputs supported by the project corresponded to the specific objectives and included service delivery indicators (e.g. number of children participating in community nutrition health programs); support service, human resource, and management indicators (e.g. percentage of health center and district managers receiving management training); and indicators relating to coordination, planning, and monitoring (e.g. number of partners submitting to MOH a fully costed work program for forthcoming year). Part (b): KPIs included (i) the number of pilot initiatives that were financed by the project, monitored and evaluated, with findings and lessons disseminated, and scaled up or influencing national policies; and (ii) the number of formal partnership contracts signed between MOH and NGOs or private providers. A full list of the KPIs (PDO indicators and Intermediate Outcome Indicators - IOIs) is presented in the ICR Datasheet.

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 8. The PDO was revised in the Level 1 restructuring / Additional Financing (AF) – in the amount of US$17.7 million4 – in February 2013, to the following: “To support Government of Timor-Leste to get more resources to where they are needed most, to improve the delivery of health services in districts and sub-districts”. This PDO will hereafter be referred to as the revised PDO. 9. The original PDO aimed to reflect the initial intention of pursuing a Sector Wide Approach (SWAp)5, rather than establishing a stand-alone project that would finance a narrow set of priorities. However, the project proved to be too broad and complex, with multiple specific objectives, components and sub-components, and a wide range of

4 Total AF was US$21m. Out of approximately US$ 20.2 million after 1% central unit cost recovery and 2.92% management unit cost recovery, US$ 17.7 million was used to finance the project, through the Recipient Executed Trust Fund (RETF). US$2.5m was used to finance two Bank Executed Trust Funds (BETFs): (i) US$1m for analytical and advisory activities; and US$1.5m for implementation support. 5 A SWAp is an approach whereby all significant funding for the sector supports a single sector policy and expenditure program, under government leadership, adopting common approaches across the sector, and progressing towards relying on government procedures to disburse and account for all funds.

4

activities. As such, the PDO was revised to provide more targeted support to priority areas. In particular, the restructured project would support the MOH’s roadmap for “one sector plan, one budget and one monitoring and evaluation (M&E) framework”. Activities would focus on health system and resource management issues and aim to address key bottlenecks to health service delivery. The project’s KPIs were substantially revised during the February 2013 restructuring / AF to be in line with the revised PDO. Reasons for these changes are explained in Section 2.3 on M&E; new and revised KPIs are outlined in the datasheet.

10. The project’s KPIs were further revised during the October 2014 restructuring (see Section 2.3), to align with a reduced scope of work to be undertaken in the final year of the project, and to measure progress on key activities that the project was supporting. The PDO was not revised during this restructuring. A summary of changes to the Results Framework is outlined in Annex 9.

1.4 Main Beneficiaries 11. The original PDO aimed to focus on women and children, and original PDO indicators were all on maternal and child health. As part of achieving this, the project would support the rollout of MOH’s Basic Service Package (BSP), which included interventions to reduce child and maternal mortality. The Level 1 restructuring / AF set a target for the total number of beneficiaries the project aimed to reach, and within this aggregate target, the number of female beneficiaries to be reached. However, the project also included many other activities and components such as improvements in support services and coordination and monitoring, which would have population-wide benefits. The economic analysis in the PAD thus calculated expected benefits of the project based on a population-wide figure.

1.5 Original Components (as approved) 12. The project had four components which corresponded to the specific objectives outlined in the PDO, and in line with the HSSP: Component 1: Health service delivery (US$12m), including primary care and

hospital care packages, community nutrition services, and improvements in quality of care.

Component 2: Support services, human resource development, and management (US$4m), including overall governance, management, and human resource development; procurement and supply chain of medicines and supplies; fiduciary and planning support functions; and strengthening training capacity.

Component 3: Coordination, planning and monitoring (US$2m), including coordination of donor programs, planning and budgeting processes at national and district levels, and strengthening M&E systems.

Component 4: Innovation and program development (US$2.3m), to promote community demand for health services, provide incentives to service providers,

5

establish public-private partnership options, and pilot rapid results initiatives to build local implementation capacity and strengthen the focus on results.

1.6 Revised Components 13. Project components were revised twice. All four components were revised in the February 2013 Level 1 restructuring / AF. The revised project components were (see Annex 2 for details): Component 1: Improving Public Financial Management (PFM) and Fund

Flows for Service Delivery (total cost USS12.2m), to strengthen MOH’s planning and budgeting systems, advocate for an increase in health budget allocation, and provide flexible and innovative financing for service delivery.

Component 2: Strengthening Pharmaceutical and Medical Supplies Management (total cost US$1.6m), including clinical and logistics management capacity, regulatory capacity, and quality control.

Component 3: Improving Evidence-Informed Decision Making and Health Sector Coordination (total cost US$3.9m), including the development and implementation of an M&E framework, institutionalizing health sector coordination meetings and joint annual sector reviews, and strengthening research capacity.

14. Additional Financing of US$17.7m was allocated to the three revised components, with estimated costs for each component in parentheses above. Funds from the original project financing continued to be allocated to the original four components. 6 The February 2013 Level 1 restructuring / AF brought the total project cost to US$38.7m. 15. In the October 2014 Level 2 restructuring, there were revisions to the components to reduce the scope of the project, so as to concentrate on areas with increased momentum and improve project status. A realistic assessment of what could be achieved by the project closing date led to scaling down and cancellation of activities that were not performing satisfactorily. US$6.9m of project funding was cancelled in this restructuring. This was due to the Government of Australia’s decision in May 2014 to reduce its contribution to the project from AUD 33.7m to AUD 24.5m. Based on currency exchange at that time, this was equivalent to a reduction of US$6.9m. This brought the total project cost to US$28.3m.7 The Bank project team’s assessment at the time of restructuring was

6 There was a reallocation of project costs across components in an earlier Level 2 restructuring, approved March 5, 2012. See Annex 1 for details. 7 The February 2013 restructuring / AF brought the total project cost to US$38.7m. In order to manage exchange rate risks, out of US$17.7m of AF, the initial RETF allocation was US$14.2m. Total project cost was thus effectively US$35.2m, until an adjustment was made for the full tranche of AF to be allocated to the RETF. However, this adjustment was never made during the project. When US$6.9m of Australian contribution was cancelled in the Sept 2014 restructuring, the total project cost was cited as US$35.2m (rather than US$38.7m), comprising US$1m from IDA and US$34.2m from the MDTF. This figure of US$34.2m is confirmed in the Amended MDTF Grant Agreement signed by the Vice Minister of Finance on April 25, 2013. With this cancellation of funds, the revised project cost then became US$28.3m,

6

that the reduced envelope of project financing would not negatively impact achievement of the PDO. The reduction in scope of work and revised costs were as follows: Component 1: (Total cost revised from USD$12.2m to US$3.5m). Sub-

component (ii) on advocating for an increased overall health budget allocation in line with the development of an MTEF was cancelled, as the MTEF would be postponed to post-project in 2017.

Component 2: (Total cost revised from US$1.6m to US$1.5m). Only sub-component 1 on management systems was retained. This included planning and forecasting, and supply chain systems development. Work on regulatory capacity and quality control was cancelled as it was deemed to be too ambitious to achieve results in these areas in the remaining timeframe.

Component 3: (Total cost revised from US$3.9m to US$1.3m). This would focus on strengthening M&E, with continued efforts on institutionalizing sector coordination. Sub-component 3.3 (strengthening research capacity) was cancelled due to weak capacity and slow implementation.

1.7 Other significant changes 16. In addition to the revisions outlined above, the following changes were made during the project. They are listed chronologically, corresponding to the three project restructurings: Level 2 Restructuring (restructuring note dated February 28, 2012; approved

March 5, 2012). This was a minor restructuring to: i. Amend the Grant Agreement (GA) to enable the project to access funds that

became available due to exchange rate gains. To manage exchange rate risks, the initial allocation to the project through the Recipient Executed Trust Fund (RETF)8 was set at US$15.7 million in the GA. With the rise of the Australian Dollar over the period when tranche payments had been received, the amount available to the RETF increased to US$20 million. Combined with the US$1m from IDA, total fund allocation in the GA was revised to US$21 million.

ii. Include operating costs as eligible expenditure in the Financing and Grant Agreements, in order to be able to support key activities in service delivery.

iii. Reallocate funds across project components. Details can be found in Annex 1. Level 1 Restructuring and Additional Financing (Project Paper dated January

24, 2013; approved February 27, 2013). The amendments included:

comprising US$1m from IDA and US$27.3m equivalent (34.2 minus 6.9). These final figures are confirmed in the letter on the Amendment to the MDTF Grant Agreement, signed by the Minister for Finance on April 22, 2015. This explanation is included here because of inconsistencies in these figures in the October 2014 restructuring paper. 8 From the parent MDTF, an RETF was established for project funds, to be executed by MOH.

7

i. Change of project name. The government had replaced the HSSP 2008-2012 with the National HSSP (NHSSP) 2011-2030. The project name accordingly changed to NHSSP-Support Project (NHSSP-SP).

ii. Amendment of the PDO, described in Section 1.3 iii. Revised project components and scope of work, described in Section 1.6. iv. Amendments to KPIs, to be discussed in Section 2.3 v. Adjustment to implementation arrangements, to be discussed in Section 2.2 vi. Extension of project Closing Date from June 30, 2013 to June 15, 2015, to

allow time for implementation of revised project activities vii. AF of US$17.7m to the project, channeled through the RETF, comprising an

increase in contribution of from the Government of Australia and the entry of the European Union (EU) into the MDTF. The project financing plan was amended due to allocation of the AF to revised project components.9

Level 2 Restructuring (Project paper dated September 30, 2014; approved

October 31, 2014) to: i. Reduce the scope of work to focus on priority areas and allow for completion

of activities within the limited timeframe remaining (see Section 1.6) ii. Revise the Results Framework (RF) to better monitor and report on PDO

achievement (see Section 2.3) iii. Revise the financial envelope to the MDTF, given the cancellation of

US$6.9m contribution from the Government of Australia, as outlined above (Section 1.6).

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry 17. Timing of preparation: Project preparation spanned the period of unrest and violence in 2006, but aside from minor delays, there is no evidence to suggest that this adversely affected the adequacy of project preparation. The decision to pursue a SWAp was taken at an early stage of project preparation, and despite the challenging context, MOH maintained its focus on the longer term objective of strengthening health sector performance. It is important to note, however, that institutional instability affected the appropriateness of the project design by the time it became effective. The better part of project preparation was done under the leadership of Minister for Health Dr. Rui Maria de Araujo. Dr. Araujo left this position in August 2007, just ahead of project appraisal and approval. The new Minister for Health, Dr. Nelson Martins, was on board from September 2007, and by late 2008 there were new Directorates and teams of staff at MOH due to reorganization. This is significant as the government’s HSSP was developed

9 An exception to Bank policy was approved by the Regional Vice President on the requirement for “Satisfactory” project performance in order to apply AF procedures under OP 13.20. Although the project rating was “moderately unsatisfactory”, the exception was granted in view of mitigating actions undertaken during the restructuring process – such as hiring of advisors, plans to put in place a Project Management Team – to improve the pace and quality of implementation.

8

under a technical team comprising Directors and Heads of Departments at MOH and institutes who were no longer in those positions shortly after the project became effective in June 2008. With this change, the MOH’s ownership of the government’s own HSSP declined considerably, and the level of understanding of the SWAp also diminished.

18. Assessment of project design: The Bank’s project team built on lessons from HSRDP 1 and 2 and sought to consolidate gains from these projects.10 As mentioned in the PAD, two key lessons included: (i) the importance of keeping project design simple, flexible and focused on a manageable set of high priority objectives, especially in the context of fragile states, and (ii) providing a flexible design framework which would allow resources to be accessed and allocated in line with the government’s annual rolling plans. These lessons proved to be in conflict with one another in this project. The crux of the project design was its flexibility, which turned out to be a double-edged sword. Two examples here – one positive and one negative – illustrate the point. The upside of the project’s flexibility was shown in its ability to fund a new strategic priority at MOH: Integrated Community Health Service Delivery (Servisu Integradu da Saúde Communitária, or SISCa). SISCa was not part of the original project design and activities. However, it became clear that it would contribute to the PDO, in particular maternal and child health. It was thus adopted under the project to be funded starting from 2010. On the downside, the project activities read like a “shopping list” which was extensive, varied, and lacked focus. The ICR mission found that in the early phases of the project, requests to fund regular MOH functions were submitted to the project because it was perceived that it was easier to access project funds than the domestic budget (due to late release of the domestic budget in the first quarter of the fiscal year). This contradicted the project’s intent of strengthening governance and financial management in the health sector, and the project began to lose focus due to the broad range of requests for it to fund various MOH activities. In hindsight, the project was not able to be sufficiently focused on a manageable set of high priority objectives, while at the same time being flexible and responding to MOH’s annual plans. 19. Assessment of implementation arrangements: Implementation arrangements could likely have benefitted from more thorough appraisal in two areas. First, as specified in the PAD, project implementation would rest with MOH departments, and an explicit decision was made not to have a dedicated Project Management Unit (PMU). However, there was no proper assessment of MOH’s capacity for implementation and sector coordination of a SWAp. Further, management of the project would sit within a newly created Directorate of Planning and Financing and further subsumed under the Department for Partnership Management (DPM). This new department was untested, and was also under-staffed for a year into project implementation. Second, the project was financed through a RETF and IDA grant. This meant that the Recipient (MOH) would need to understand and be able to comply with Bank requirements, such as using project funds only for eligible expenditures, procurement standards and procedures, and fiduciary responsibilities. While MOH had some experience in managing RETFs, capacity for fund

10 HSRDP1 (June 2000 – Dec 2005) and HSRDP2 (June 2001 –July 2008) closed with outcome ratings of Satisfactory and Moderately Satisfactory, respectively.

9

execution continued to be low, and the absence of a PMU which had existed in former projects compounded the problem. 20. Limited capacity for planning and implementation had been highlighted as a Substantial project-level risk. A mitigation measure mentioned in the PAD was to minimize the complexity of program design and reduce transaction costs. However, the fact that the project had so many components, sub-components, and a broad and varied range of activities was in hindsight unwise, and made the project unnecessarily complex to implement. Further, as a SWAp, good implementation required effective sector coordination led by MOH, so as to avoid duplication and make the most of all resources available to the health sector. The PAD outlined partnership and coordination arrangements for the SWAp, including joint donor missions and the establishment of Joint Annual Sector Reviews and Planning Summits. In practice, in Phase 1 (from project effectiveness until the February 2013 restructuring / AF) of the project, joint sector meetings were planned but were often postponed with no notice or rescheduling. MOH did not have adequate capacity to manage this process, and the recruitment of a partnership management specialist had to be included as one of the conditions and legal covenants in the February 2013 restructuring. It was not until Phase 2 (February 2013 until project closing on June 15, 2015) of the project that a “code of conduct” for stakeholder engagement and coordination was developed, and Joint Annual Sector Reviews were conducted. In hindsight, stronger support for strengthening MOH capacity for implementation and coordination would have greatly benefited the project.

2.2 Implementation 21. This section will summarize key issues which affected project implementation during the two project Phases. 22. Phase 1 faced significant implementation challenges and from the outset there was a significant delay in project start up. This was for several reasons. First, an assassination attempt on the President and Prime Minister on February 11, 2008 led to a politically charged environment nationwide, and the curfew which was imposed for two months hampered project implementation. Second, it took longer than expected for new Directorates at MOH to become fully functional after reorganization, and a new round of “socialization”11 was required to familiarize new staff with the project. Third, there were delays in recruiting key advisors to support project implementation, including a procurement advisor and an international advisor to DPM. Finally, two ongoing projects by the EC – the Health Support Program (HSP) and Support to the Implementation of the Health Sector Investment Programme (SIHSIP) – were running in parallel with the HSSP-SP, and impacted implementation of this project. The already limited bandwidth at both MOH and the Bank were mostly expended on HSP and SIHSIP activities. Both projects were implemented through RETF grants managed by MOH and overseen by the Bank, which detracted staff time and attention from this project.

11 Socialization is roughly understood as a process of engagement, information dissemination, and gaining buy-in from relevant parties

10

23. The Mid-Term Review (MTR) was conducted in July 2010. This was a highly participatory process, with 3 consultation workshops held around the country and all districts invited to provide input. Relevant MOH and development partner representatives were also present. As of March 2010, the project was rated Moderately Satisfactory for PDO achievement and Unsatisfactory for implementation progress. The key finding and recommendation from the MTR was that project design and scope of work needed to be amended substantially, as the project was overly complex and ambitious. The revised scope of work would focus on: i. District service delivery, in particular the flow of funds and supplies to health

facilities. ii. Institutional strengthening would be continued, with a focus on human resources,

logistics support, pharmaceuticals, other MOH support services (e.g. financial management and procurement), and M&E.

iii. Improving coordination of all development assistance to the health sector.

24. The MTR was proactive in identifying key challenges and bottlenecks in implementation at an early enough stage in the project. However, it took two years before these recommendations from the MTR were adopted in the February 2013 restructuring. During this period, there were substantial developments in content of the proposed restructuring, as well as in government. On the proposed restructuring, most significantly, the project PDO was revised, even though the MTR noted that the original PDO was still relevant.12 Revisions to project components also underwent at least three iterations in this period. There was then a change in government in mid-2012 which stalled the restructuring process, and through which a new Minister for Health and staff were put in place. Another full round of re-consultation was needed in order to “socialize” the new leadership and staff with the proposed restructuring. 25. Phase 2 of the project began after the February 2013 restructuring. Similar to Phase 1, implementation in Phase 2 got off to a slow start. MOH underwent a significant re-organization during the first half of 2013, which left several directorate leadership positions vacant. However, once the project got off to a good start, there were significant improvements in two areas in particular: i. Improved capacity for implementation due to changes in implementation

arrangements. A Project Management Team (PMT) was established in July/August 2013, with a dedicated Project Manager and 11 staff hired. A Project Steering Committee was also established and met on a quarterly basis.

12 It is unclear when and how the revised PDO which was eventually adopted was formally proposed and discussed with MOH. While there are statements as early as the May 2011 AM which sound similar to the revised PDO, even project documents just preceding the official restructuring (e.g. AM of September 2012 and ISR of November 2012) do not clearly mention the intent to revise the PDO.

11

ii. Improvements in financial and procurement procedures. Related to the establishment of a PMT, MOH hired one procurement specialist and one project accountant, each solely responsible for procurement and accounts under NHSSP-SP. This led to demonstrable improvement in project procurement and accounting (see Section 2.4 for details).

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 26. Design: The Results Framework (RF) in the original project was fully in line with MOH’s M&E framework, with indicators drawn from the HSSP. The PAD contained 5 PDO indicators on health service coverage. However, it was not clear that project activities would directly support achievement of these indicators e.g. immunization coverage, which would largely be achieved through the Expanded Program for Immunization under the WHO. However, as the HSSP-SP was designed as a sector-wide program, these health outcome indicators were deemed to be appropriate. Section 3 on Assessment of Outcomes explains how it turned out to be challenging to attribute achievement of health sector outcomes to the project. In terms of IOIs, the PAD contained 14 indicators which were linked to the four project components. 7 out of 14 IOIs had no baselines, two of which had no targets as well. This proved to be problematic in assessing the achievement of these IOIs. 27. One of the project’s objectives (in Component 3) was to strengthen the M&E at MOH. While the Health Management Information System (HMIS) was already functional, the PAD acknowledged that there needed to be further strengthening of timeliness, quality, and use of data. Even by 2011, for example, the project reported significant differences between HMIS data and data collected through other means, particularly the Demographic Health Survey completed in 2010. The project intended to provide technical assistance and support for hardware and software related to M&E. Facility and household surveys, program evaluation and operational research, and community-level monitoring was planned as part of the project. 28. Implementation: Due to the abovementioned reasons, implementation of M&E was patchy in Phase 1. Further, with the change in project scope in the February 2013 restructuring / AF, the RF was substantially revised (see ICR datasheet for details): All 5 PDO indicators were dropped as they were beyond the scope of the revised

project. 4 new PDO indicators were introduced. 13 out of 14 of the original IOIs were dropped as they were beyond the scope of the

revised project. 8 new IOIs were introduced. 29. Further changes were made to the indicators during the October 2014 restructuring to align with a reduced scope of work: 3 PDO indicators from the 2013 restructuring / AF were dropped to better reflect

data that MOH was able to collect and report on, and to measure achievement of selected project activities.

12

Two PDO indicators were introduced to measure activities that would be undertaken to assess quality of health services, and reflect implementation of M&E tools.

4 new IOIs were introduced to better reflect key activities that the project was supporting in PFM, medicines supply chain, and M&E. Of the 9 IOIs as of the February 2013 restructuring, 3 were revised, 3 were maintained, and 3 were dropped.

30. In terms of data collection and reporting, one advantage of having the project’s original PDO indicators aligned with the MOH M&E framework was that despite poor M&E systems under the project, all original PDO indicators were tracked in the HMIS until project closing. However, with indicators that were not already in the HMIS, a key problem with data collection under the project was an absence of definitions and tools for M&E. One example is the indicator for “percentage of health clinics providing the comprehensive BSP”. A tool for monitoring progress and a definition of “comprehensive” was not developed until Phase 2 of the project. This hindered measurement of progress of the BSP for the first 5 years of the project. Routine reporting by MOH departments was also often delayed. Overall, absence of definitions, baselines and targets, and a weak culture of reporting resulted in 10 out of 14 original IOIs returning “unknown” achievement by the time they were dropped or at project closing. In addition, with two substantial rounds of revisions to the RF, monitoring of progress on indicators had to be revised accordingly. The round of revisions in October 2014, in particular, allowed for less than one year (before project closing) for improvement and reporting on progress. There was a sense among key stakeholders of “moving the goalpost”: activities and KPIs were changed too often for the project to settle into a good momentum of implementation and monitoring of progress. 31. Nonetheless, under Phase 2 of the project, MOH’s M&E capacity – and thus M&E for the project – improved. This was to a large extent due to a committed M&E Officer at MOH and an excellent M&E advisor on board from end 2012, and good collaborative effort among development partners in providing support on M&E. A holistic review of M&E at MOH was conducted, and realistic and user-friendly indicators and scorecards were developed to be used at the facility level. A new tool to measure service readiness was put in place and became the practical implementation of measuring achievement of the BSP (IOI #1), which was never measured in Phase 1. In collaboration with other development partners, a system of “supportive supervision” was instituted to measure the availability and readiness of health facilities for delivering care. 32. Utilization: There were initial inroads in the utilization of data for decision-making. While there was an overall increase in awareness of the importance of M&E at MOH, a critical challenge was that the Ministry’s annual plans and budget framework continued to be disconnected from data reported in the M&E framework. At the time of writing the ICR, there was a concerted effort to use data from the M&E framework to inform the budget for Fiscal Year 2016, which will hopefully lead to better budget planning. At the district level, one positive development was that the system of supportive supervision led to growing interest among District Health Administrators, who were keen to benchmark the quality of their health facilities against those in other districts, and engage in discussions on how to improve service readiness in their district.

13

In addition, one positive externality from the project has been that the culture of M&E has spread beyond the health sector. The Ministry of Agriculture, for example, requested a presentation on the MOH’s M&E framework, so as to adopt it in the agriculture sector. The Prime Minister’s Office also took notice of MOH’s M&E framework and provided comments for specific indicators to be incorporated into the health sector’s national scorecard. A culture of M&E was thus beginning to be built over the last two years of the project, and will hopefully be sustained.

2.4 Safeguard and Fiduciary Compliance

33. One safeguard policy (Environmental Assessment (OP/BP 4.01)) was triggered in the PAD, as the project would finance the rehabilitation of facilities and procurement of drugs and medical equipment. Civil works including maintenance and upgrading of some health posts were undertaken in Phase 1 of the project. Environmental impacts were anticipated to be minimal and temporary, such as noise, air emissions, and generation of construction waste. Healthcare waste management was also evaluated with regard to disposal of hazardous materials. A Healthcare Waste Management Plan was disclosed in 2007, prior to project appraisal. The environmental rating for the project was thus Category C. As part of the Level 1 restructuring / AF in February 2013, an updated Integrated Safeguards Datasheet was prepared and approved by the Sector Manager on November 19, 2012. The updated ISDS noted that the proposed modifications in the PDO and its components would not change the overall environmental category of the project, nor would they trigger new safeguard policies. No additional environmental impacts were anticipated. However, the trigger for the Environmental Assessment remained due to civil works and procurement of drugs and medical equipment under the original project scope. Civil works had been completed by the time the updated ISDS was approved, while distribution of medical equipment was pending. Phase 2 of the project continued work on healthcare waste management. 34. To address the Environmental safeguard that was triggered, the project supported the use of clinical waste gasifiers to manage healthcare waste, including maintenance and training. The most recent environmental mission in September 2014 conducted training at two regional hospitals, in addition to earlier training sessions in August. Continuing challenges on the collection, transportation, and disposal of biohazardous materials were highlighted. Post-project, the work on healthcare waste management will be continued by WHO. Guidelines on healthcare waste management will be developed, and monitoring of healthcare waste management will begin at two hospitals and two CHCs.

35. Fiduciary compliance: The project’s rating for financial management was Moderately Unsatisfactory for the majority of the duration of the project, including the most recent rating provided in the Bank’s Financial Management Implementation Review Report in March 2015. There were persistent shortcomings particularly in Phase 1 of the project. These included: unreconciled advances and a lack of action on outstanding advances; inadequate registers for consultant contracts and financial commitments; delays in fund flows and errors in claims resulting from a lack of timely and regular submission of Withdrawal Applications; payments made for potential ineligible

14

expenditures with no proper documentation; and delays in submission of annual audits. Improvements were noted in Phase 2 of the project when project finances were managed by the PMT, with assistance from a project accountant (international consultant). Notably, a Finance Manual was adopted in 2013 as part of the Project Operations Manual, which included provisions on internal controls. FM reviews found these provisions to have adequate segregation of incompatible duties and responsibilities, and designated levels of review and approval provided reasonable assurance that policies and procedures for recognizing and recording assets, revenues and expenses would be complied with. Another persistent finding was the inability to use the Freebalance system – the government’s accounting software – for project accounts, as nominated in the legal agreement. The project made substantial efforts to move onto Freebalance, but despite support from MOF to migrate to this system, the project remained on an Excel accounting system. This impacted the timeliness and reliability of financial information. 36. Procurement: Project procurement started off Moderately Satisfactory in the first year, but declined to a consistent Unsatisfactory / Moderately Unsatisfactory rating from 2010-2013. When procurement got underway in 2009/10, there were difficulties due to limited understanding of Bank procurement procedures, and weak planning and follow up actions. Procurement packages submitted to the Bank for clearance consistently had major shortcomings, and the Bank had to issue “objection” letters several times, requiring revisions to bid evaluation reports or other documentation. Procurement processes were greatly improved in Phase 2 of the project, due to project procurement being handled by the PMT, separate from MOH procurement, and with full-time assistance from a procurement specialist. Greater clarity was established on procurement processes with the development of a manual on NHSSP-SP Procurement Management. All procurement of goods, works and consulting services under Phase 2 were deemed to have been carried out in accordance with provisions of the Project Operations Manual, Bank procurement guidelines, and MOH procedures. Procurement ratings in the last four ISRs were Moderately Satisfactory. Nonetheless, timely execution of the project’s procurement plan remained a challenge. This was due to inadequate submissions from technical departments in MOH, such as poor justifications and budget estimates for new project activities, and lack of detailed technical specifications, TORs, and other relevant documentation that were necessary for commencement of procurement processes.

2.5 Post-completion Operation/Next Phase 37. At project closing, a balance of approximately US$5m remained in the RETF. With agreement from co-financers (Government of Australia and EC), this amount will be transferred to the Bank Executed Trust Fund (BETF) that was already part of the parent MDTF, and will be used for technical assistance and advisory services to GoTL / MOH. The topics and areas for analysis will follow closely the revised project components under the project, namely: Improving PFM and fund flows for health service delivery, Strengthening pharmaceuticals and medical supplies management, and Improving evidence-informed decision making and health sector coordination.

15

38. However, sustainability of key activities which were funded under the project will greatly depend on MOH, and funding from the domestic health budget. At the time of writing, planning and budgeting for FY2016 was underway. Key risks to sustainability and further improvement on aspects that were supported under the project are outlined in Section 4 on assessment of risks to development outcomes, to the Bank’s best knowledge at the time of the ICR mission.

3. Assessment of Outcomes 39. The assessment of project outcomes will consider two phases of the project, before and after the February 2013 restructuring / AF.

3.1 Relevance of Objectives, Design and Implementation 1. Objectives: The project development objectives in Phase 1 were consistent with and directly relevant to HSSP 2008-2012. They remain relevant to the current NHSSP 2011-2030, which aims to ensure that health care services are available, accessible, and affordable for all Timorese people. Achieving health MDGs continues to be a priority for GoTL: maternal mortality and immunization coverage are still lagging behind their targets, and improvement in coverage of four antenatal care visits for pregnant women has not been sustained over the last decade. While coverage of maternal and child care has improved, there has been little improvement in quality. These will be discussed in the section on Achievement of PDO in Section 3.2. The objectives under Phase 2 also remain relevant: the NHSSP’s objectives include improving health system management and support services, including the stewardship role of MOH, health service delivery, and efficient deployment of human resources for health. These were all core parts of Phase 2 of the project. The PDO was revised to focus on health systems strengthening and management, in order to tackle the underlying bottlenecks to service delivery. Objectives of both Phase 1 and 2 are also relevant to the World Bank Group’s Country Partnership Strategy FY2013-2017, which aims to continue investing in the quality of human capital and improving health status; and strengthening institutions, spending, and governance. Relevance of objectives is thus rated as High for each of the two phases. 40. Design: The project design in Phase 1 was appropriate as a sector support project. Flexible funding enabled the project to adapt to MOH’s changing needs. However, this breadth and flexibility in project activities resulted in inability to attribute achievement of results to the project. The RF which followed the M&E framework of the HSSP was thus not ideal for assessing project effectiveness. Phase 1 design is rated as Modest. In Phase 2, project design was modified to focus on bottlenecks in health service delivery. The project improved in its responsiveness to the needs of MOH and based on feedback from districts, and revisions to project activities were in line with the revised PDO. That said, there remained a slight disconnect between the RF and the PDO: achievement of the PDO would require evidence that each process flow (of money, supplies, and staff) was complete, all the way from the central level to health facilities. However, many of the revised KPIs in the RF only speak to one portion of each process. Achievement of specific indicators – especially IOIs – would thus not necessarily imply achievement of

16

the PDO. This results chain will be explained in Section 3.2. Design in Phase 2 is thus rated as Substantial, taking into account the responsiveness of changes in project design, but also shortcomings in the link between the RF and PDO. 41. Implementation: In Phase 1, the approach of managing the project through Directorates at MOH, without a dedicated PMU, caused significant delays and challenges in implementation due to weak capacity for executing the RETF. There was also limited capacity at MOH for effective sector coordination of this SWAp. This approach was not adjusted in a timely enough manner: it was not until 2013 that implementation arrangements were modified. However, at that time this approach was deemed appropriate by the government, in particular in view of Minister for Health Dr. Araujo’s vision of flexible funding in a sector wide approach, and the subsequent Minister Dr. Martins’ comment as recorded in the May 2010 AM that he was “committed to the [sector wide approach] and it [was] not his desire for HSSP-SP to default into project mode”. Recognizing the challenges inherent in the implementation approach yet with respect to MOH’s ambition and commitment, relevance of the implementation approach in Phase 1 is rated as Modest. Relevance of the implementation approach in Phase 2 is Substantial. The project took a much more hands-on approach by installing a PMT, consciously improving financial and procurement procedures within the PMT, and improving its monitoring of the work of advisors to various Directorates. A partnership management specialist was also hired to assist in improving sector coordination. Overall, there was a much more focused and concerted effort in tackling specific problems and bottlenecks faced in the health system, as compared to the much more scattered approach in Phase 1. 42. The summary of the overall relevance rating for each Phase is as follows: Table 1: Ratings for relevance of objectives, design, and implementation

Relevance of: Phase 1 Phase 2 Objectives High High Design Modest Substantial Implementation Modest Substantial Relevance rating Modest Substantial

3.2 Achievement of Project Development Objectives 43. The achievement of PDOs is based on the original and revised PDOs. This corresponds with Phases 1 and 2 of the project. 44. The original PDO aimed to improve the quality and coverage of preventive and curative health services, particularly for women and children, in order to accelerate progress toward the health MDGs. In terms of the first sub-objective of improving quality of health services, achievement was Negligible. At SISCa posts, there was no improvement in quality in Phase 1. At the end of Phase 1 of the project (measured at end of 2012), the proportion of SISCa posts functioning according to “Category A” standard

17

had remained stagnant at 31% since 2009.13 (KPI #15) “Category A” grading is given to posts that have human resources, equipment, and funding as set out in the SISCa guidelines, whereas Categories B through D would imply deficiencies in available resources, and hence negatively impact the quality of service delivery. At health posts and Community Health Centers (CHCs), there was also no improvement in the quality of health services. At project closing, just 2% of health posts and 17% of CHCs met the benchmark of service readiness in the facility readiness assessment – far short of the target of 80%, and even below the baseline of 54% that was set in June 30, 2014.14 (KPI #12) 45. A final indicator on quality of health services, specifically for pregnant women, is the dropout rate in antenatal care, measured as the difference between % of women receiving four or more antenatal care visits, and % of women receiving just one antenatal care visit (KPI #14). From a baseline dropout rate of 22% in 2008, the completeness of antenatal care – as a measure of quality – in fact got worse: the dropout rate had increased to 29% by the end of Phase 1 in 2012. At project closing, the dropout rate was slightly higher than the baseline: 23%, based on 2014 data (see Figure 1). Figure 1: Dropout rate for antenatal care (1 visit vs. 4 or more visits)

Source: Timor Leste HMIS 46. On the second sub-objective of improving coverage of health services, achievement in Phase 1 of the project was Substantial. Based on HMIS data available in June 2012, KPI #2 had surpassed its target, KPI #3 had been achieved, and KPI #1 and #5

13 While there was no improvement in the % of SISCa posts graded “Category A” in Phase 1, this indicator improved in Phase 2 of the project. This will be discussed in the section on Achievement of the revised PDO. 14 It is important to note that facility readiness was not measured in Phase 1 of the project, due to absence of measurement methodology and framework (see comments for IOI #1 in datasheet). This indicator was measured solely in Phase 2 of the project. It is used as evidence in Phase 1 for two reasons: First, this indicator of facility readiness was, in practice, the operationalization of the original IOI #1 for which data was never collected in Phase 1. It thus fully reflects the intent of measuring quality of services at health facilities. Second, the results from this indicator show that improvements in quality were not observed despite the longer duration (until project closing) and further investments made for such improvements to happen. These further investments in Phase 2 will be discussed under Achievement of the revised PDO.

57

68 64 65

78 7465

3545 42 43

4946 42

0

10

20

30

40

50

60

70

80

90

100

2008 2009 2010 2011 2012 2013 2014

% of pregnan

t women

ANC 1

ANC 4

18

were partially achieved. By project closure, 2 KPIs had surpassed their targets, and 2 had been partially achieved. However, there was a notable deterioration in KPI #3 which had previously achieved its target in 2012. Status of the indicators at the time of being dropped and results at project closing are presented in Table 2: Table 2: Achievement of PDO indicators, June 2012 and at project closing

Indicator Baseline

(from PAD)

Target (from PAD)

Status just prior to indicator

being dropped (June 2012 data)

End line (data available at

project closing June 15, 2015)

KPI #1: % of children under 1 year vaccinated with (i) DPT3 and (ii) measles

(i) 63% (ii) 61%

(i) 90% (ii) 90%

(i) 73% (ii) 67% Partially achieved

(i) 76.8% (ii) 82% Partially achieved

KPI #2: % of births attended by skilled health personnel

27% 45% 59% Achieved (target surpassed)

57.7% Achieved (target surpassed)

KPI #3: % of pregnant women receiving at least four antenatal visits15

36% 55% 58% Achieved

42% Not achieved

KPI # 4: % of children (6-59 months) receiving vitamin A supplements

36% 80% 48% Not achieved

66% Partially achieved

KPI #5: Contraceptive prevalence rate (% married or cohabiting women using modern contraceptives

10% 25% 22% Partially achieved

49.9% Achieved (target surpassed)

Source: Timor Leste HMIS 47. Overall, achievement of the original PDO was Modest. The negligible improvements in quality of health services and substantial improvements in coverage were in spite of significant efforts in the project to these ends. The activities in Phase 1 of the project aimed to support the original PDO in three main ways: (i) Direct delivery of health services, (ii) Investments in inputs at health facilities, and (iii) Improvements at the system level which would contribute to better service delivery. Many activities done and outputs produced were less than optimally delivered or not fully utilized. Annex 2 provides a list of outputs; salient examples of activities and their achievements or shortcomings are outlined here for illustrative purposes: i. On direct delivery of health services, the project supported SISCa for community

health outreach, to bring preventive and curative services to beneficiaries at the suco

15 Measurement and/or achievement of this indicator has been uneven. The 2012 Multi-Donor Trust Fund Annual Report for the HSSP-SP noted that this PDO indicator had been achieved with a result of 58%, exceeding its target of 55%. However, the report also noted that the Timor-Leste Demographic Health Survey had a different result of 55% for this indicator. The dip to 41.6% in 2014 could either indicate deterioration in coverage of antenatal care, inaccuracies or inconsistencies in data collection across the years, or both.

19

(village) level. In 2012, when SISCa activities had ramped up, it supported 475 outreach events were supported across 442 sucos, servicing more than 300,000 participants, 93% of whom were pregnant women and children.16 5 NGO contracts (IOI #14) were established to support SISCa activities. While the project supported operating costs for SISCa at US$35 per event, this was not based on proper costing and was thus inadequate to fund the necessary transportation and equipment. Another constraint was the lack of qualified health professionals. A rapid assessment of SISCa in 2012 found large variations in the staff mix across SISCa, which implied variations in service availability and quality. Outcomes from SISCa in Phase 1 thus illustrate decent coverage but inadequate quality of care.

ii. Investments in inputs at health facilities were made to support the rollout of the BSP. This included the purchase of 13 vehicles and 60 motorbikes to enable access to community health programs, procurement of $1.2m worth of medical equipment, installation of 40 solar panels at health posts for improved electrification, training of 21 midwives (all women) and 77 nurses (50 men and 27 women), and support for 119 scholarships in various disciplines including pharmacy, nursing, dental care, public health, health policy and administration etc. While training and deployment of personnel was successful, other areas faced difficulties, for example: suspected misuse was reported in the distribution of some vehicles, a delay in distribution of medical equipment for over one year, and inappropriate specifications such as equipment that did not fit into health facilities.17

iii. Efforts on improvements at the system level included technical assistance and

capacity development on financial management, procurement, and HR management to address capacity gaps in MOH departments and among district health management teams. Reviews found that TA was only moderately successful because activities did not reflect the needs of MOH and district health management staff, and/or roles of technical advisors were not well communicated to the districts.

48. The problem of attribution of PDO achievements to project activities was raised many times in Phase 1. The HSSP-SP 2011 Annual Report rated the contribution of project activities to reported improvements in overall PDO indicators as modest. This was in view of the much larger proportion of resources provided by GoTL through its domestic budget for the health sector. Across the life of the project, other development partners also provided valuable and substantial support to MOH. For 2011 and 2012 combined, this project’s expenditure was just 16% of actual donor expenditure on health in Timor Leste18. More critically, from 2008-2012, 30% of donor funding on health was spent on family and child health and health policy and systems development19: areas that were similar to this project and likely overlapping.

16 HSSP-SP MDTF Annual Report 2012 17 Data on proportion of functioning biomedical equipment and total number of staff trained in project financing and management (IOIs #8 and 9) not available. 18 Data only available for these two years within Phase 1 19 Timor Leste Medium Term Expenditure Pressure Study, World Bank Group

20

49. The overarching intent of the project was to contribute to the progress of health MDGs. This will not be evaluated as a specific objective of the Project, as achievement of the MDGs is a higher level national objective. Nonetheless, progress on selected health MDGs that relate to health services supported by the project are presented in Table 3. KPIs 1, 2, 3, and 5 from the project are also MDG indicators, and will not be repeated. Table 3: Progress on selected health MDGs20

50. The revised PDO was to support the Government of Timor-Leste to get more resources to where they are needed most, to improve the delivery of health services in districts and sub-districts. The first sub-objective addresses three types of resources: money, staff, and supplies.21 The definition of “where they are needed most” is inferred from project activities that were planned under Phase 2. Priority areas included: (i) the flow of resources from the central government to District Health Services, and (ii) improving systems and management at central and district levels. The following paragraphs assess the flow of money, staff, and supplies in turn. 51. Financial resources (money): The aim was to strengthen planning and budgeting processes within MOH and at districts, to improve the flow of funds from central government to District Health Services (DHS), and to improve budget execution. Activities and outputs included the development of (i) a PFM Roadmap and Working Group, (ii) a Financial Instructions Manual, (iii) Standard Operating Procedures for budget implementation, and (iv) training on financial internal control, among others. 52. These activities and outputs did not translate into improved outcomes. As an indicator of whether proper PFM procedures had been adhered to and carried out, the rejection rate of Commitment Payment Vouchers on a quarterly basis was tracked (IOI #24). This stagnated at 7%, missing the 5% target, implying no improvement in PFM procedures. Rejected CPVs and thus a delay in payment meant that financial resources were not getting to districts in a timely manner. Further, the overall budget execution rate for MOH (IOI #11(iii)) declined from 94% in 2011 to 89% in 2014.

20 Sources: World Development Indicators (for 2007); Timor Leste MDG Goals Report 2014 (for national targets); The State of the World’s Children 2015, UNICEF (for 2013 estimates) 21 This interpretation is supported by details in the HSSP-SP 2011 Annual Report and ISRs from 2011/12, which note that there had been “continued difficulties in getting adequate resources (money, staff, and supplies) to the districts to support improved service delivery” (author’s emphasis).

MDG indicator At project inception

(2007)

National target for

2015

At project closing (2013

estimates) Remarks

Infant Mortality Rate (per 1000 live births)

61 53 46 Surpassed

Under-5 Mortality Rate (per 1000 live births)

74 96 55 Surpassed

Maternal Mortality Rate (per 100,000)

425 252 270 Partially achieved

21

53. The lack of improvement in fund flow and PFM could be for several reasons. First, it is acknowledged that systemic changes such as PFM reform takes time. The project was likely too ambitious in trying to achieve this in two years, from the time of restructuring in February 2013. Second, while in the medium term the activities may have adequately addressed discrete components of fund flow and improved specific processes at MOH, there were other significant factors not entirely within the control of MOH that impeded fund flow. For example, the ICR mission found that in 2015, there was late disbursement of advances to districts in the first quarter (Q1) of the fiscal year, due to overall late release of the domestic budget. Q1 advances were disbursed almost a full quarter late, in mid-March 2015. This had a snowball effect, as by the end of Q2 some districts remained unable to reconcile their accounts for the previous quarter. Thus, while the project achieved its indicator (KPI #10) of timely acquittal of advances, a substantial delay in the initial advance meant that in practice, funds were still not getting to districts on time. Identical problems with fund flow were raised in a Health Resource Tracking Study conducted in 2013.22 This suggests that timeliness of budget disbursements did not improve over the 2014/15 budget cycles. 54. Physical resources (supplies): This section looks specifically at the management of the supply of medicines, as the procurement and distribution of medicines and medical equipment that the project undertook in Phase 1 was discontinued in Phase 2. Key activities included implementation support for a pharmaceutical management system (mSupply) at the central medicines warehouse run by SAMES. mSupply was also extended to the National Hospital, Baucau Hospital, and Baucau DHS23, meeting the target of 3 sites with successful pilots of the system (IOI #23). Development of a Logistics Management Information System (LMIS) was also undertaken to improve quantification and inventory management for medicines (IOI #19), but was not achieved. A paper-based LMIS had been developed by project closing but not rolled out yet, and its late implementation contributed to continuation of poor supply chain management for medicines. A computerized drug registration system was also intended to be developed, but was not (IOI #20). Overall, while discrete functions supported by the project were improved, other parts of the supply chain continued to be bottlenecks. These included a lack of strategic procurement and proper quantification, leading to over-supply of some drugs and stock-outs of others; tardy annual procurement processes and late delivery of medicines; and a slow feedback loop between MOH’s Department of Pharmacy and health facilities on stock levels. As a result, stock-outs of tracer essential drugs at SAMES and health facilities were 20% and 43% respectively, not meeting their targets of 15% and 30% (KPI #8).24

22 Nixon, Hamish and Caryn Bredenkamp. “Timor-Leste Health Resource Tracking Study”, September 2014, World Bank Group. 23 Information from interview with SAMES representative 24 This indicator and its target were formally revised during the February 2013 restructuring. Neither the original nor revised target was met.

22

55. Human resources (staff): Phase 2 did not contribute directly to getting staff to health facilities in order to improve availability of inputs (presence of staff). However, funding for health professionals training in Phase 1 may have had a sustained effect with staff deployments lasting into Phase 2. Even then, achievements are modest: at project closing, only 43% and 54% of health posts had a resident midwife and resident nurse, respectively.25 (KPI #17) This is likely due to poor distribution, rather than a shortage in absolute numbers of health professionals.26 Indeed, outside of the project, there were significant changes to human resources for health in Timor Leste, with an influx of Cuban-trained doctors deployed to health facilities in 2013 and 2014. The project supported development of a plan for the reintegration of these doctors into the Timorese health system, but training was by and large overseen by the Cuban Brigade.

56. The key activity in Phase 2 relating to human resources was a functional analysis of human resources at the central level (MOH), selected districts, and the National Hospital of Guido Valaderes (HNGV). This analysis aimed to assess the internal organization of MOH and National Hospital, and serve as the basis for developing a human resources strategy, post-project. At the time of project closing, MOH was just starting to take into account the recommendations from this functional analysis in order to inform its human resource strategy for improved service delivery. The view of key government stakeholders interviewed during the ICR mission was that much more work needed to be done to develop a full HR strategy and policy, in particular to address the mal-distribution of midwives and nurses mentioned above. Altogether, the human resources work that was commissioned and completed under Phase 2 of the project was thus of moderate benefit, and at best laid the foundation for a much longer process of HR reform. At the time of writing, the inputs and activities on human resources for health in Phase 2 had not yielded any demonstrable results that could be linked to the PDO. 57. Overall, the achievement of sub-objective 1 of the revised PDO is Modest. This recognizes that many activities and outputs in Phase 2 aimed to contribute to and may have led to better resource allocation. However, their contribution was partial and the timeframe for achievement too short, thus leading to limited achievement of the PDO. Revised PDO sub-objective 2: To improve the delivery of health services in districts and sub-districts 58. As mentioned in the evaluation of the original PDO, the project continued to aim to improve delivery of health services. Sub-objective 2 of the revised PDO is interpreted to be a continuation of the original PDO, with a particular focus on district service delivery as highlighted in the MTR. The phrase “to improve the delivery of health services” in the revised PDO is not defined. Indeed, the KPIs and IOIs introduced in Phase 2 of the project focused on measuring the efficacy of revised PDO sub-objective 1;

25 Joint Annual Health Sector Review, Ministry of Health Timor Leste, December 2014. 26 Data from the Supportive Supervision exercise in 2014 shows that 8 health posts had 2 midwives, and 16 health posts had 2 or more nurses. Absence of these staff at certain health posts is thus a problem of allocation, rather than absolute shortage.

23

2009 2010 2011 2012 2013 2014

Cat A 31 32 29.1 31 43.2 44.1

Cat C & D 4 8.2 8.4 7 3.3 3.7

05

101520253035404550

% of total SISCAs

Figure 2: % SISCa posts that are functioning according to classification category

no new metrics for sub-objective 2 were introduced. It is inferred that sub-objective 2 of the revised PDO encompassed a similar definition to that of the original PDO, and thus the metrics for the original PDO on quality and coverage of health services also apply to the evaluation of sub-objective 2 of the revised PDO. 59. Some activities from Phase 1 were continued in Phase 2. In August 2013 the project initiated additional financial support of SISCa by covering the full operating cost of US$120 per SISCa, an increase from US$35. This correlated with an improvement in quality of SISCa posts (KPI #15), with an uptick in the percentage of “Category A” SISCa in 2013 and 2014 (see Figure 2). Thus, continuation and improvement of SISCa in Phase 2 contributed to the quality aspect of the original PDO, which had been dropped. Had the original PDO been kept, there would be more positive outcomes. 60. Health service delivery also encompasses other elements. These include availability, access, and utilization. 27 Availability includes the physical presence or delivery of services, as well as their inputs. Availability of inputs (resources) was evaluated under sub-objective 1 of the revised PDO. Access is a broad term with varied dimensions, including physical, economic, and socio-psychological aspects of people’s ability to make use of health services. Utilization is the quantity of health care services used, and assesses demand-side issues in service delivery. 61. This section evaluates physical access to health services. In this respect, the project managed to substantially increase health outreach services to communities through SISCa.28 The 2003 Demographic Health Survey found that 25% of households were more than two hours away from their usual health care provider. This did not improve much by 2009: a study on health care seeking behavior found that travel time to a health facility was over two hours for 29% of households, and over five hours for 5%. SISCa guidelines state that there is to be one SISCa post per 2 hour distance from the nearest health facility, thus expanding the network of health services closer into

27 “Monitoring the building blocks of health systems: A handbook of indicators and their measurement strategies”, Chapter 1: Health Service Delivery. http://www.who.int/healthinfo/systems/WHO_MBHSS_2010_full_web.pdf, accessed October 28, 2015. 28 This is evaluated qualitatively, as there is no primary data on physical access.

24

Figure 3: Average number of outpatient visit per person per year

1.9 22.3

22.2

2.9

0.70.51

0.67 0.59 0.57 0.65

0

0.5

1

1.5

2

2.5

3

3.5

2009 2010 2011 2012 2013 2014Average

no of Visit per person

Outpatient Visit Rate Average Visit to SISCa

communities. There is no available data on actual distribution and distance to SISCa posts. Further, SISCa posts may not be considered “usual” / regular health facilities as outreach events are held once per month. Nonetheless, a process evaluation of SISCa found that communities generally found SISCa to be substantially helpful in improving physical access to health services, in particular for those who live in remote, mountainous areas. Indeed, the counterfactual – of no SISCa – would be that CHCs and health posts continue to be the closest point of contact for Timorese who live in remote areas. 62. In terms of utilization, the project likely contributed to the slight increase in the average number of outpatient visits (KPI #16). SISCa visits per capita remained relatively constant across the years and raised the average rate of contact of the population with health services, and there was an increase in outpatient care visits at health facilities from 2 to 2.9 during Phase 2 (Figure 3). Utilization nonetheless remains on the low side, far short of the WHO benchmark: in the OECD countries, the average number of physician consultations per person per year is about 6, and the proposed benchmark according to the Service Availability and Readiness Assessment is 5 visits per person per year. 63. Achievement of sub-objective 2 of the revised PDO is Substantial, given its contribution to quality, access, and utilization. The overall rating for the revised PDO is Modest: while there were improvements in health service delivery, the intent of addressing underlying system and management issues in the health sector was not met.

3.3 Efficiency 64. A benefit analysis and cost-effectiveness analysis was carried out for this ICR, modelled after the economic analysis in the PAD. The analysis found overall cost-effectiveness of this project to be Substantial. A separate analysis of efficiency specifically for Phase 1 is not possible due to difficulty of attribution. Given that Phase 1 comprised the majority of the project duration (approx. 5 years, vs. 2 years of Phase 2), the rating of Substantial from the overall cost-effectiveness analysis will be assigned to Phase 1. This is based on quantitative analysis which estimates the cost per Disability Adjusted Life Year averted at US$251, which compares very favourably to GDP per capita (constant local currency unit) of US$1,070 in 2014. The evaluation also takes into consideration the project’s contribution to higher-level MDG health outcomes and broader improvements to the health system which are difficult to quantify. Details of this analysis are in Annex 3. A proxy indicator for efficiency in health spending was also

25

assessed. This replicates the economic analysis that was done for Phase 2 of the project. Using updated data, the analysis shows that between 2007 and 2013, Timor Leste did not improve in its efficiency of spending, measured by infant mortality rate relative to health spending per capita, vis-à-vis selected comparator countries (see Annex 3). Overall, Phase 2’s overarching objective of more efficient allocation and use of resources was not evident in this proxy measure. For Phase 2, efficiency is rated as Modest.

65. Efficiency of implementation is also rated Modest. This was due to the delay in project start up, the two year period it took from the MTR until restructuring in 2013, and again delays in putting in place the new mode of implementation in Phase 2. Staff shortages, delays in filling key positions, multiple rounds of socialization of the project among MOH staff, and absence of consistent in-country Bank supervision at the beginning of the project all contributed to inefficiencies in project implementation.

3.4 Justification of Overall Outcome Rating 66. Ratings for relevance, efficacy, and efficiency are presented in the table below: Table 5: Project outcome ratings

Phase 1 Phase 2 Relevance Modest Substantial Efficacy Modest Modest Efficiency Substantial Modest Outcome rating Moderately Unsatisfactory Moderately Unsatisfactory 67. The overall assessment of project outcomes is Moderately Unsatisfactory. This rating is also supported by analysis of weighted ratings against project disbursements: Table 6: Project outcome ratings weighted by disbursements

Original PDO

(Phase 1) Revised PDO

(Phase 2) Overall

Rating Moderately

Unsatisfactory Moderately

Unsatisfactory Moderately

Unsatisfactory Rating value 3 3 -- Total disbursements (US$ million)

16.24 6.96 23.2

Weight (disbursement for this phase as share of final disbursement of US$ 23.2 million)

70% 30% 100%

Weight value (Rating x Weight)

2.1 0.9 3

Final rating -- -- Moderately

Unsatisfactory Rating scale: HU=1; U=2; MU=3; MS=4; S=5; HS=6

3.5 Overarching Themes, Other Outcomes and Impacts

26

N/A (a) Poverty Impacts, Gender Aspects, and Social Development 68. The project included gender aspects and equity. First, there was a focus on maternal health, including improving quality and coverage of antenatal care visits, skilled births, and family planning. At SISCa events, on average approximately two-thirds of attendees were female. The project intended to track the number of female beneficiaries reached; however, the ICR mission found that this data was not available. Second, training and scholarships in Phase 1 of the project had a reasonable gender balance. As noted in Section 3.2, midwives who were supported for training were all women, and 50 men and 27 women were supported for nurse training. Overall, there is no evidence of unfair gender bias in funding support for training under the project. 69. The incidence of catastrophic out-of-pocket payments for healthcare has declined over the course of the project. In 2007, 14.2 percent of households spent at least 40 percent of their non-food budget on healthcare; this declined to just 2.7 percent by 2011. 29 While unfortunately the poverty rate in Timor Leste – proportion of the population living below the poverty line of US$1.25 per day – has increased over the last decade, healthcare expenditure has not been a driver of this change. In other words, households have not fallen into poverty due to involuntary, catastrophic expenditure on healthcare. While this change may not be directly attributable to the project, it nonetheless signifies that household spending on healthcare is not exacerbating poverty. (b) Institutional Change/Strengthening 70. Improvements in capacity are detailed in Section 2.2 on procurement and financial management, and Section 2.3 on the wide and important audience that gained interest in the M&E framework. The project made some headway in building institutional capacity at MOH and among district health administrators. In particular, financial internal control was strengthened through training of district health administrators, and is likely to be sustained, post-project. Conversely, limited improvement has been observed in procurement. Procurement under the project improved in Phase 2 with a procurement specialist on board, but there was limited capacity building for staff in the MOH procurement department. (c) Other Unintended Outcomes and Impacts (positive or negative) N/A

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable, as this a core ICR

4. Assessment of Risk to Development Outcome Rating: Substantial

29 Health Equity and Financial Protection Report, Timor Leste. World Bank Group, 2014.

27

71. The risk to development outcomes is Substantial. With project closure, the function of the PMT has come to an end, and the status of many activities that were being managed under the project was being discussed by MOH at the time of writing. For example, while there has been strong commitment from GoTL for the continuation of SISCa, there have been concurrent discussions on folding SISCa into the overall Primary Healthcare Package. The status of these plans was still being discussed internally at MOH, and the budget for SISCa for FY2016 was unknown. That said, the work under the project – especially on PFM reform and M&E – have laid a strong foundation, and indications are that this work will continue to be priority areas for MOH, as well as the development of an HR strategy, as recommended in the HR functional analysis conducted under the project. One key risk is financial sustainability: MOH’s budget has been reduced by a substantial 18% from US$67m in FY2015 to US$55m for FY2016. The ICR mission found that there may be significant reductions in the budget for medicines. While this may spur efficiency gains in medicine procurement, this also puts the availability of essential medicines at further risk, from an already high stock out rate of 43% at health facilities. A careful consideration of how to incorporate Ministry priorities and project activities into this reduced budget envelope will be required in order to sustain development outcomes.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 72. The PAD was very detailed and well-considered, articulating the rationale for adopting a SWAp. This was in line with the government’s overarching strategy and direction and other development partners’ concurrence that this was an appropriate approach. The Bank team drew lessons from HSRDP 1 and 2, and intended be in line with and pick up from where the EC’s HSP and SIHSIP left off. The approach of flexible funding which would respond to government needs and MOH’s annual plans was commensurate with a sector support strategy. 73. However, there was insufficient attention and analysis given to implementation arrangements, despite the fact that low capacity for implementation was highlighted as a substantial project-level risk. Section 2.2 discusses the challenges with implementation that arose due to institutional instability, and insufficient staff bandwidth to properly implement the project. In addition, the use of a RETF required the implementing agency to have full understanding of how the financing instrument was meant to be used, including familiarity with Bank rules e.g. on eligible expenditures and procurement, and the implementing agency’s responsibility for executing the funds.

74. In hindsight, flexibility in the project design, complexity of project components, and capacity and knowhow that was required of the implementing agency in order to execute the RETF proved to be an inappropriate combination of factors. In the context of

28

a low capacity setting, this would have necessitated better support and supervision from the Bank. The capacity of MOH as an implementer and the amount of Bank support that would be needed were both severely underestimated at project entry. (b) Quality of Supervision Rating: Moderately Satisfactory 75. Bank supervision across the duration of the project was uneven, but improved over time. The critical challenge, especially in the first 2 to 3 years of the project, was the absence of an experienced, in-country Bank team dedicated full time to managing the project. From project preparation to effectiveness, overall responsibility was with a TTL who was not based in-country, and management responsibility was with an internationally-recruited Operations Officer who oversaw both health and education portfolios. In hindsight, the lack of strong support from the Bank in the early stages of the project was ill-suited to the context. Much more support and explanation of Bank procedures and rules – especially on the allowable uses and procedures required under the RETF – was required in order for MOH to execute the funds and implement the project effectively. Given the high turnover of leadership and staff in government, it was all the more important for a strong and consistent Bank team to maintain good communications with MOH and provide support to facilitate key processes. This situation improved from 2012 onwards, with much stronger oversight from the Bank’s project TTL based in Dili, actively monitoring project management. Support to MOH on project procurement and financial management was also stepped up significantly through recruitment of specialists (see Section 2.4 for details).

76. It should also be recognized that thorough reviews of the project were conducted, starting early on in the MTR in July 2010, and there were proactive discussion on restructuring the project to improve its effectiveness. The Bank team was honest very early on that achievements of the PDOs may not be attributable to the project. The MTR and discussions leading up to the February 2013 restructuring critically assessed key bottlenecks in health service delivery and refocused the project’s activities around it. Even though these were the most challenging issues to be tackled (e.g. PFM, supply chain management), the intent was to continue supporting MOH in health system strengthening for the long term. (c) Justification of Rating for Overall Bank Performance Rating: Moderately Unsatisfactory 77. Quality at entry is rated Moderately Unsatisfactory given that the combination of flexible project design and implementation arrangements turned out to be ill-suited to the Timorese context and hindered implementation. Quality of supervision is rated Moderately Satisfactory, to recognize the improvements in Bank supervision over time, and the proactive nature of discussions on shortcomings and possibilities for restructuring. Unfortunately, despite modifications in project design and supervision to try to improve project effectiveness, this was still insufficient to improve overall project performance. Overall Bank performance is thus rated Moderately Unsatisfactory.

29

5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 78. GoTL and MOH have consistently been committed to improving health sector

outcomes in a holistic manner. This has been consistently articulated, first in the HSSP (2008-2012), and its successor NHSSP (2011-2030). The ambition for adopting a sector-wide approach and a single strategy / direction has also been clear, with each successive Minister for Health articulating a cohesive vision which development partners’ efforts would be aligned with. 79. The key challenge with GoTL has been the lack of institutional stability, and with that, high staff turnover. Over the 8 years of the project, there were 4 Ministers for Health and 3 re-organizations at MOH. Each change in leadership brought with it a slowdown in decision making, due both to the uncertainty in a time of political change, as well as the highly centralized nature of decision making in GoTL. Each re-organization meant that about 5 to 6 months of implementation time would be lost, as new positions took time to fill and another round of “socialization” would be needed to familiarize new directors and staff with the project. A 2011 review by the Bank’s Independent Evaluation Group (IEG)30 commented that the very limited implementing capacity in Timor Leste was not adequately taken into account in the design and timetable of project implementation.

(b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 80. Phase 1 of the project was characterized by limited MOH staff bandwidth and knowhow for project implementation. With no dedicated PMU, implementation responsibility rested with relevant directors at MOH, none of whom were tasked with overall coordination and responsibility for project delivery. With the establishment of a PMT in mid-2013, staffing for project management improved substantially, and the PMT held overall responsibility for project coordination and implementation. Critical functions such as procurement and financial management of the project were conducted outside of the MOH structure. Overall, however, this separation of responsibilities led to an improvement in project implementation. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Unsatisfactory 81. The overall rating of Moderately Unsatisfactory for Borrower performance reflects the substantial challenges in project start up, delays in implementation throughout the project, and low capacity at MOH. However, the Ministry’s strong commitment to health system strengthening through a SWAp is acknowledged, and improvements in project implementation in Phase 2 are recognized.

30 Evaluation of World Bank Group Program, Timor-Leste Country Program Evaluation, 2000–2010. Independent Evaluation Group.

30

6. Lessons Learned 82. Key lessons from the project are:

i. Lessons from positive outcomes of the project: As a SWAp, the project was able to significantly leverage IDA funds, with US$1m of

IDA funds crowding in substantial donor funding. Good collaborative effort with counterparts is key to success, as seen in the

collaboration among partners to develop the M&E framework at MOH. Positive externalities generated by the project include an increased awareness among

other government agencies on good outcomes from the project (e.g. M&E), and awareness on the importance of auxiliary services that are often neglected (e.g. healthcare waste management).

ii. Design

Project design needs to take into account political and institutional realities in the country. In particular, the level of ambition of project achievement and timeframe for implementation need to be realistically assessed, and project objectives should be set in accordance with these realities. With a SWAp, the role of government is critical, and in an environment of weak government capacity, much more robust support for key functions in the Ministry would be required to enable the project to function effectively. Considerations include: (i) the potential for delays in project start up and delays with each modification in project design, (ii) institutional stability and attrition at counterpart agencies, (iii) baseline readiness and level of development of institutions, as substantial amount of work is likely to have to be done to build fundamental capacity and capabilities, before project outputs and outcomes can be met. Institutional reform can take a substantial amount of time. The level of ambition of project achievement may need to be tempered to account for this, and support stepped up to assist the government in meeting these challenges.

There is a need to strike a balance between breadth and depth of project activities and objectives, even in a SWAp. Specifically: - Even if the project is intended to provide broad, sector-wide support, it is essential

to be clear on what outcomes the project will directly support. This is to mitigate the risk of sector outcomes being achieved but not being attributed to the project, thereby resulting in the PDO returning a less than satisfactory achievement.

- It would be preferable for the project to select specific elements of the SWAp it will focus on and aim to do fewer things, better.

Participation in and consultation on project design are critical to ensure project ownership. This was done well in the early stages of project preparation, and again in the Mid Term Review. However, a shortfall in re-engagement and further socialization limited ownership of the project among the new leadership and cadre of staff (note: related to above lesson on keeping in mind institutional stability and attrition).

31

iii. Implementation

In settings where there is low implementation capacity, this capacity needs to be properly assessed during project preparation, and sufficient support provided to build stronger implementation capacity along the way. In particular: - This underscores the importance of having experienced, consistent, in-country

resources (including from the Bank) to support counterparts if and where implementation capacity has been identified as a critical risk.

- Even if the ambition is to have the project managed through country systems instead of in parallel with existing Ministry structures, it could help to establish or maintain a dedicated project management unit / team at least at the beginning of the project to kick-start project start up and provide initial guidance.

- Capacity building needs to be coupled with sufficient staff strength, so that staff have time for both “learning” (training) and “doing”. Related to this is a key lesson that when internationally-recruited consultants are hired to help implement the project and provide capacity building simultaneously, this often leads to substitution instead of true capacity building.

Financing instruments that place the onus on the client for executing funds in line with Bank requirements need to be coupled with robust support to the government. In particular: - Substantial support and training on procurement should be embedded in

government, so that this capacity is developed and retained. - Better explanation and training on financing rules and requirements is necessary

in the context of complex project designs and/or flexible financing arrangements. This would mitigate the risk of submission of ineligible expenditures and improve the quality of internal control and financial reporting.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies The project assessment conducted by the GoTL is provided in Annex 7. (b) Cofinanciers Comments from the European Union, one of the project’s co-financers, are in Annex 8. These comments have been taken into consideration and incorporated into the report. (c) Other partners and stakeholders N/A

32

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in US$ Million equivalent)

Components Appraisal Estimate

(Sept 2007)

Revised costs after Level 2

Restructuring (Mar 2012)

Additional Financing (Jan 2013)

Revised costs after Additional Financing (Jan

2013)

Revised costs after Level 2

Restructuring (Sep 2014)

Actual/Latest Estimate31

Percentage of Appraisal

Original components 1. Health service delivery 12.0 9.0 0.0 9.0 9.0 4.0 33% 2. Support services, human

resources, and management 4.0 9.0 0.0 9.0 9.0 10.4 260%

3. Coordination, planning, and monitoring

2.0 1.7 0.0 1.7 1.7 1.7 85%

4. Innovation and program development

2.3 1.3 0.0 1.3 1.3 1.3 57%

Sub-total 20.3 21.0 0.0 21.0 21.0 17.4 86% Restructured components (“percentage of appraisal” is calculated as % of estimated amount at AF, rather than of appraisal estimate in Sept 2007) 1. Improving public financial

management and fund flows for service delivery

0.0 0.0 12.2 12.2 3.5 2.2 18%

2. Strengthening pharmaceutical and medical supplies management

0.0 0.0 1.6 1.6 1.5 0.6 38%

3. Improving evidence-informed decision making and health sector coordination

0.0 0.0 3.9 3.9 1.3 2.0 51%

31 Based on information received from Project Management Team (PMT), Oct 21, 2015. Total disbursements based on Client Connection (CC) data, correct as of Nov 3, 2015 are US$23.3m, but this is not disaggregated by project component. Difference in PMT vs. CC figures is just 0.1m.

33

Carry over from HSSP activities (uncategorized)

-- -- -- -- -- 0.9 --

Advances to be cleared (as of Oct 21, 2015)

-- -- -- -- -- 0.1 --

Sub-total 0.0 0.0 17.7 17.7 6.3 5.8 33%

Total 20.3 21.0 17.7 38.7 27.332 23.2

115% (of appraisal estimate) 71% (of AF total)

(b) Financing

Source of Funds Type of

Cofinancing

Appraisal Estimate

(US$ millions)

Additional Financing

(Jan 2013)33 (US$ millions)

Revision to financing

(Sept 2014)

Actual/Latest Estimate

(US$ millions)

Percentage of Appraisal

AUSTRALIA: Australian Agency for International Development / Department of Foreign Affairs and Trade

Multi Donor Trust Fund Grant

19.3 11.00 Reduction of

US$ 6.9m 22.7 --

European Union Multi Donor Trust Fund Grant

0.00 10.00 No change 9.30 --

Borrower 0.00 0.00 0.00 0.00 -- IDA Grant 1.00 0.00 0.00 1.00 --

32 This figure and the allocation of funds to revised components are taken from the October 2014 Level 2 restructuring paper datasheet. There is no other document which provides this breakdown by component. However, total project cost should be US$28.3m, instead of US$27.3m (see footnote 9). It is not known why there is an unaccounted for US$1m. This is possibly an error in the restructuring paper. 33 Refer to Footnote 4

34

Annex 2. Details of Components and Outputs by Component Original components Component 1 details: Health service delivery (approximately US$ 12m), This component would support the following areas: (i) BSP for primary health care, including improved outreach services to remote areas; (ii) community nutrition and health services; (iii) district level planning and management capacity to support service delivery; (iv) hospital care and the referral system; (v) quality of care throughout the health system, including improvement of infrastructure.

Component 1 outputs: Sub-component (i): Basic Service Package for primary health care, including improved outreach services to remote areas.

13 vehicles and 60 motorbikes procured for districts for supervision of outreach services. [Note: 3 vehicles ended up being distributed to central MOH to facilitate implementation of project activities]. Of the 60 motorcycles, 51 were distributed to districts, 7 to regional hospitals, and as of March 2012 no information was currently available on the location of 2 motorcycles.

40 solar panels procured and installed at district health posts. Medical equipment procured. Significant delays in distribution.

Sub-component (ii): Community nutrition and health services 475 SISCa per month currently supported by the project through provision of

small incentives for volunteers and village chief (Chefes de Suco), at $35 per SISCa per month

5 NGOs contracted to provide support to SISCA activities. They are: Oxfam, Child Fund, Medicos do Mundo, Healthnet, and Care Int’l, in Oecussi, Manatutu, Viqueque, Ainaro, Covalima, and Bobonaro respectively.

Sub-component (iii): District level planning and management capacity to support service delivery

MOH developed PFM roadmap for both central and district teams, with support from Central PFM advisor. Implementation of the roadmap initiated with ongoing annual budget preparation.

District advisors supported District Health Management Teams (DHMT) in completing annual action plan and budgeting, harmonizing planning process with partners in target districts, establishing regular meetings, quarterly financial acquittal, compiling progress of DHMTs

Collection and analysis of SISCa operational cost conducted at MOH, with support from District advisors

IT systems: Deployment of new technologies (e.g. network storage devices), data center and services; establishment of IT work-order system, drafting of IT policies.

35

Sub-component (iv): Hospital care and the referral system Essential Drug List and Standard Treatment Guidelines developed Pediatrician consultant provided support to improve hospital management for

child health

Sub-component (v): Quality of care throughout the health system, including improvement of infrastructure

Renovation of health posts: Completion of Cassa HP/Ainaro, Laga (Sagadate) HP/Baucau, and Asmano HP/Liquica. Cassa HP provides BSP covering 1 suco/5 aldais; Laga HP covering 1 suco and 1 aldai from Baucau and 1 suco from Quielikai

Renovation of INS building completed providing space for pre-service and in-service training

Extension of SAMES warehouse and office space; additional storage space for drugs and inventory management constructed (e.g. receiving sections, offices). Further work needed to address temperature control in warehouse as this was not adequately incorporated into the design

Construction of temporary office space at MOH for Director of Planning & Finance and Director of Logistics & Administration

Component 2 details: Support services, human resource development, and management (approximately US$ 4m) This aimed to: (i) Improve overall governance and management of the health sector; human resource development and management practices; procurement, distribution and management of essential drugs and supplies; core health sector fiduciary and support functions including planning and supervision of civil works; logistics and maintenance of infrastructure and equipment; financial management; procurement; and information and communication technologies, and (ii) Strengthen the Institute of Health Sciences (IHS); and training for health staff and managers. Component 2 outputs: Sub-component (i): Improve overall governance and management of the health sector; human resource development and management practices in the health sector; procurement, distribution and management of essential drugs and supplies; and core health sector fiduciary and support functions, including planning and supervision of civil works; logistics and maintenance of infrastructure and equipment; financial management; procurement; and information and communication technologies

Overall governance and management practices: Limited reporting on achievements. Trust Fund Annual Reports noted that overall, MOH was not getting best value out of consultants. Long term TAs have time-based contracts without clear deliverables. Payments are often made without looking at the quality of work completed and progress against ToRs. Consultants engaged as ‘advisors’ do not always have clearly identified counterparts to work with and help build skills. Capacity development is not undertaken in a systematic and assessed way.

36

Human resource development: Development of district HR workforce plans against the BSP, and a workforce plan for hospitals. However, it was noted that deliverables were not user-friendly enough, and there was very limited HR management capacity supported by the HR advisor

Procurement, distribution, and management of essential drugs and supplies: Pharmaceutical Management Information System (mSupply) installed and

SAMES currently using it for day to day operation; mSupply to be refined to accommodate the local context

Quantification team set up for annual requirement preparations; quantification exercise initiated

With partial support from HSSP-SP, SAMES has developed various manuals (accounting, procurement, standard operational procedures, HR and administration manuals), but they have not been fully utilized for day-to-day operation due to limited capacity, even after providing several trainings for staff including supply chain management, quality control, procurement management.

Quality control lab (MiniLab), supported by Global Pharma Health Fund, operational through training provided under the project

Sub-component (ii): Strengthen the Institute of Health Sciences; and training for health staff and managers.

In 2012, INS was in the process of recruiting individual consultants to develop 7 modules including clinical practicum of midwifery, community nutrition, ethics for health professionals, and provide training to INS trainers

119 scholarships provided. Component 3 details: Coordination, planning and monitoring (approximately US$ 2m) This component would (i) Strengthen coordination and monitoring of donor programs, including through the establishment of a Department for Partnership Management (DPM) within MOH; and strengthen policy development and operational research capacity, and (ii) Support health sector program planning and budgeting procedures at national and district levels; project monitoring and evaluation system including the Health Management Information System (HMIS), and support for surveys and evaluations. Component 3 outputs: Sub-component (i): Strengthen coordination and monitoring of donor programs, including through establishment of a Department for Partnership Management within the MOH; and strengthen policy development and operational research capacity.

Strengthening coordination and monitoring of donor programs: Joint planning/review meetings were organized in 2008 and 2009.

Strengthening policy development and operational research capacity: Health Financing Note completed in 2011 and used to inform some discussions on budget preparation and implementation, as well as to inform development of

37

NHSPP-SP. A normative costing of the MNCH components of the BSP was also completed in 2011.

INS Research Cabinet working with Menzies under the twining agreement to improve research capacity in TL

Situation analysis of the health research environment and existing data review in TL conducted

Research methodologies and epidemiology training provided for TL stakeholders based on training needs assessment

Sub-component (ii): Support: health sector program planning and budgeting procedures at national and district levels; project monitoring and evaluation system, including the Health Management Information System, and support for health sector surveys and evaluations

Program planning and budgeting procedures: Refer to Component 1(iii) Monitoring & evaluation: HMIS advisor for one year provided some trainings to

central and district staff and worked with MOH to simplify HMIS reporting forms; however, HMIS remains overly complex. Further effort to streamline reporting and data collection points is needed, as well as support for analysis of data.

Component 4 details: Innovation and program development (approximately US$ 2.3m) This would promote community demand for health services, provide incentives to service providers, establish effective public-private partnership options, and pilot rapid results initiatives to build local implementation capacity and strengthen the focus on results. Component 4 outputs: To promote community demand for health services, provide incentives to service providers, establish effective public-private partnership options, and pilot rapid results initiatives, to build local implementation capacity and strengthen the focus on results.

Promote community demand for health services: See Component 1(ii) on SISCa Incentives to service providers: Some sector discussion on results based financing

during 2010 but agreement made that initial focus needed to be on establishing some basic systems and accountability for resource use (particularly PFM and monitoring and evaluation).

Public-private partnership options: See Component 1(ii) on NGO contracts, and Component 3(ii) on twinning with Menzies

Revised components Component 1 details: Improving Public Financial Management (PFM) and Fund Flows for Service Delivery (approximately US$ 12.2m) To support MOH to assist central and district health teams to use the resources available to them more efficiently, equitably and effectively by: (i) strengthening MOH’s central and district planning and budgeting in line with the “one plan, one budget, one sector

38

M&E framework” concept through the preparation of annual activity plans and budgets that are focused on improved health service delivery, (ii) advocating for an increased overall health budget allocation that has adequate provision for recurrent costs for essential health service delivery across the country, and (iii) increasing flexible and innovative financing, such as grants to districts, for district service delivery. Component 1 outputs: Sub-component (i): On Public Financial Management Leadership and processes:

Public Financial Management Working Group established in October 2014. A series of meetings the SAMES management team took place, and basic

accounting sessions conducted. SAMES management has been assisted in its activity-based planning and

budgeting for 2015 based on MoH developed planning guidelines and tools. Key Performance Indicators developed for procurement, warehouse management,

quality assurance and finance and administration and discussed with SAMES management team.

Revised chart of accounts for SAMES and various supports for preparation of annual financial statement and bank reconciliations, and auditing support.

Financial Instructional Manual completed. Financial Standard Operation Procedures for the management of the Ministry of

Health’s financial processes and procedures. 62 job descriptions and performance evaluation for each finance related roles for

National Directorate of Finance Management and Procurement and all Central Directorates, District Health Services, Referral Hospitals autonomous/semi-autonomous agencies completed.

Directive for Opening of District Health Service Bank Accounts approved and Bank accounts for DHSs and 5 referral hospitals completed.

Training need assessment for Public Financial Management for finance officer at Central Services of MoH and District Finance Officers conducted in November 2014.

Updated and enhanced SAMES finance and accounting system manual that incorporate several proposed finance and accounting policies/guidelines and implementation procedures.

Competency based assessment for Central Services conducted on 24-25 of June 2015.

20 individuals completed Training-of-Trainers training On financial internal control:

Tools for Financial Assessment developed. District Financial Standard Operating Procedures and Tools developed Banking policy for district level developed.

39

Operating procedures for Budget Management and Internal Control (BMIC) and Monitoring checklists completed.

Budget Divisions ability to report, develop and orient procedures for Budget Divisions to track commitments of procurement plan supported.

Developed finance management internal control tools, finance management performance rating guideline for the national finance controllers.

One session of capacity building for auditors of GIFA is done through one week training on auditing flow chart and report writing to GIFA staff’s completed.

3-Year Strategic Plan for GIFA completed. Internal Audit Manual for GIFA developed. High level advice on various audits/investigations provided.

Sub-component (ii): Advocating for an increased overall health budget allocation that has adequate provision for recurrent costs for essential health service delivery across the country

This sub-component was cancelled due to time constraints for preparation and implementation of associated activities.

Sub-component (iii): Increasing flexible and innovative financing, such as grants to districts, for district service delivery.

475 SISCa services delivered to 442 villages (Suco) in the country every month since 2008. Participation of community leaders and community in the implementation of SISCa improved.

Flexible funding (performance-based) not implemented in the project due to technical issues and implementation constraints

Component 2 details: Strengthening Pharmaceutical and Medical Supplies Management (approximately US$ 1.6m) This component would support MOH in ensuring adequate essential pharmaceuticals and supply availability to improve health service delivery by: (i) strengthening the clinical and logistics management capacity of the health sector; (ii) strengthening regulatory capacity of the health sector; and (iii) improving quality control. Component 2 outputs: Sub-component (i): Strengthening the clinical and logistics management capacity of the health sector

National Drug and Therapeutics Committee for the Ministry of Health established. Revision of Essential Medicines List (EMLTL) completed and released in May

2015. LMIS tools and accompanying job aids/ tools developed in April 2014 and printed

training for DoP staffs and 275 health facilities at all levels conducted. Organogram and realistic work plans for Department of Pharmacy (DOP). Duties and responsibilities were developed for all the seven units plus the DOP.

40

Equipped DOP, 6 hospitals and 13 District health Services with computers and accessories.

9 staff from the DOP, districts and hospitals sent for short training course on pharmacy management

Drafted STG for primary level and socialised in two rounds of reviews Five volumes of the hospital clinical guidelines completed.

At SAMES warehouse:

Upgraded mSupply including the latest version; prepared mSupply interface into Tetun; assessed oxygen cylinder system use of mSupply.

Work plan with deadlines for each sub-activities developed; Review of SAME/MOH current implementation/use of mSupply completed; Reviewed and updated SOPs for mSupply core processes updated; Basic KPIs developed and configured system and management reporting

developed; Conducted user competency assessment; Remote technical support as required; addressed technical issues at pilot sites

provided. Assessment on ICT and staff capacity in remaining hospitals and districts;

developed staff training and implementation plans for mSupply in “feasible” sites conducted.

Installation of Mobile mSupply to limited sites: DHS Baucau and Baucau Referral Hospital and DHS Dili. Each site is supplied with an iPad and mSupply Mobile installed in the devices. Users from all piloted facilities trained.

National Hospital Guido Valadares (NHGV) is supplied with 4 computers, a server and label printers and mSupply Dispensing module and Pharmacy Store installed. Users trained and dispensing medicines in Outpatient and Inpatient Dispensaries using the system.

Pharmacists have been able to use mSupply system albeit limited skills in operating the computer.

Dispensary room (NHGV) has been rehabilitated to facilitate smooth movement of pharmacy staffs in providing services to patients as well as data entry into the mSupply system.

Improved quality of processes and practices around picking and dispatching of stocks to ensure that the integrity of system data is maintained and that service level to health facilities are at an acceptable standard;

A set of dashboard report via internet tailored to DoP, SAMES and HNGV requirements prepared.

MSupply systems in SAMES set up and HNGV into single integrated Warehouse Management System to enable direct stock transfers of medicines and consumables from SAMES to HNGV and immediate visibility of stocks in HNGV from SAMES’ mSupply system.

Two pilot health facilities the Customer Module (online ordering) to ensure more efficient and streamlined ordering and approval process conducted.

41

The Customer Module has been configured for Bairopite Clinic and Clinica Café Timor.

Baucau DHS and Hospital have been processing transaction using the system. Commitment of officers to use the system noted, and future support and monitoring is required. DHS Dili has ceased using the system because the trained staff left and no replacement.

SAMES staffs are trained on receiving orders from mSupply Mobile sites, including SOPs for handling this work flow and stock transfer to sites.

SAMES staff trained on creating and updating use logon credentials for mSupply mobiles sites.

SAMES staffs are trained on issuing stock to mSupply mobile stores using the stock transfer system and write an SOP for this work.

A set of standard reports for HNGV pharmacy management report defined. Sub-component (ii): Strengthening regulatory capacity of the health sector;

Pharmaceutical Law reviewed by WHO. Marketing Authorization Guidelines as part of drug registration and quality

assurance developed. These guidelines were proposed to be included in the decree law, however is pending until finalization of the up-coming revised decree law.

This sub-component was cancelled in the Sept 2014 restructuring Sub-component (iii): Improving quality control.

This sub-component was cancelled in the Sept 2014 restructuring. Component 3 details: Improving Evidence-Informed Decision Making and Health Sector Coordination (approximately US$ 3.9m) This component aimed to build the capacity of central and district health teams to make evidence-informed decisions and coordinate the health sector by: (i) developing and using an agreed sector M&E framework to assess performance across the health sector, (ii) institutionalizing health sector coordination group meetings and joint annual sector reviews, and (iii) strengthening the capacity to plan, conduct, approve, oversee and disseminate research in the health sector. Component 3 outputs: Sub-component (i): Developing and using an agreed sector monitoring and evaluation framework to assess performance across the health sector

Approval of Monitoring and Evaluation Framework and launched on the 22 of December 2014 and is being implemented. Handbook of M&E and Epidemiological terms are completed and printed and being distributed

Priority data collection indicators for MCH, CDC and NCDC program and completed in July 2014.

M &E road map endorsement held and M & E working group ToR developed and formed.

42

Department of M&E developed Supportive Supervision Guidelines translated into Tetum.

Supportive supervision completed for all 13 districts/municipals in 2014. All municipals are using M & E progress cards.

The first three districts namely Ermera, Manatuto and Oecussi were able to conduct quarterly supportive supervisions at HP and CHC independently.

Progress card for Municipals and Hospital completed and is being implemented. M & E capacity development plan (Included in M & E guidebook and M & E

strategic plan 2015-20) developed. In coordination with HMIS and other relevant departments, provided advice on

development of M & E data base and the need for strong IT in database management.

Capacity of relevant staff to establish M&E system prepared and trained. A total of 476 MoH staffs were trained during the last two and half years. Few Master trainers were identified to train district level staffs. Topics covered were basics of M & E, Supportive Supervision, Basics of Excel, Planning and refresher trainings. Basics of DHIS 2 HMIS induction training. One M &E staff was trained in STATA for data management.

Support provided to MoH to monitor the implementation of M & E systems, HMIS and Surveillance system and ensure availability of reliable and updated data are available and accessible at National and District level

Linkage between M & E, HMIS and Surveillance established. Sub-component (ii): Institutionalizing health sector coordination group meetings and joint annual sector (planning and performance) reviews

Ministry of Health conducted two health sector coordination meetings in 2014: on 18 August 2014, and 20-22 December 2014, in tandem with the Joint Annual Health Sector Review.

Health Sector Development Partner Coordination and Management Procedures (‘Health Sector Partnerships: Procedures Manual’), including Code of Conduct. Approval from Council of Director pending

Joint Semesteral Review workshop organized 29 September - 1 October 2015. Sub-component (iii): Strengthening the capacity to plan, conduct, approve, oversee and disseminate research in the health sector.

This sub-component was cancelled in the Sept 2014 restructuring.

43

Annex 3. Economic and Financial Analysis The economic analysis in the PAD qualitatively describes the benefits to be gained from improvements in both the primary health care system and better hospital care and referrals. These benefits would be achieved with support from the project through implementation of the Basic Service Package which targeted health MDGs, in particular maternal and child health and infectious diseases, and investments in the Hospital Service Package. It was noted that substantial components of the project on health systems strengthening would contribute to these benefits, and thus it would not be possible to attribute benefits to health outcomes to specific components of the project. Benefit analysis in the PAD presented “pessimistic” and “optimistic” scenarios of the project, and computed the estimated total benefits to health outcomes population-wide (Disability Adjusted Life Years34 averted), total cost averted, and cost per DALY averted. These benefits of the project were considered to be substantial and within a reasonable range of cost-effectiveness, compared to selected benchmarks: Table 1: Estimated project benefits and cost-effectiveness at appraisal

A similar approach is undertaken here to estimate actual benefits at project closing, but with several differences with regard to key parameters used: Table 2: Assumptions and key parameters used in economic analysis

34 DALY is a measure of the additional years of life gained by an intervention, and also the improvement in health gained as a consequence. 35 This analysis will use different DALY figures than that used in the PAD. The original source of data used in the PAD cannot be found at the time of writing the ICR. For comparability across the PAD and ICR,

Pessimistic Optimistic Benchmark

Total DALYs averted 24,324 39,912 --

Total "cost" averted US$ 18.2m US$ 30m Project cost of US$ 20m

Cost per DALY averted US$ 822 US$ 501 Per capita income of US$ 750

PAD ICR

Year of project closure 2012 2015

Population size 1,200,000 1,160,000

Project cost $20m $28.3m

Age-standardized DALYs per 100,00035

18098 (to be recomputed using more recent estimates - see below)

To be computed using more recent estimates (see below)

Assumption of baseline improvements in DALYs

Calculated based on assumed decline of 5%, 4%, 3%, 2% from 2003-2006 respectively, from 2002 figure

2.5% annual decline, which is the average annual % decline between 2000 and 2012, from more recent WHO estimates

44

Notably, a more recent estimate of DALYs is used in the analysis. This is in order to ensure comparability between the analysis in the PAD and the ICR. A computation of DALYs averted is done using more recent WHO estimates of DALYs for Timor Leste: Table 3: Scenarios of effectiveness in reducing DALYs by end of project

As the use of new data significantly alters the estimate of DALYs averted, the following table re-computes benefits and cost-effectiveness as they would have appeared in the PAD, had it been based on this data. This is then compared to project end line results, computed using revised parameters (population size, project cost). Table 4: Scenarios and approximate benefits and cost-effectiveness in PAD vs. ICR

The “optimistic” scenario of the project’s end line is relatively close to estimates put out by WHO for 2012 (see Table 3: WHO estimates for DALYs averted is 9466 per 100,000. This is relatively close to the “optimistic” scenario computed, of 9705 DALYs averted per 100,000). This analysis thus compares the ICR Optimistic scenario with original scenarios in the PAD. As Table 4 shows, the total benefits – as measured in terms of total

more recent estimates by the WHO will be used. These estimates include a comparison of DALYs in 2000 and 2012. Data available at WHO Global Health Observatory Data Repository, DALYs (age standardized, per 100,000): http://apps.who.int/gho/data/node.main.DALYCTRY?lang=en, accessed October 19, 2015.

Age-standardized

estimated DALYs per 100,000 by

cause in 2000

Baseline in 2007 (calculated

based on 2.5% annual decline,

for 6 years)

Pessimistic (CDs reduce by

15%; NCDs reduce by 5%; Injuries reduce

by 10%)

Optimistic (CDs reduce by 25%; NCDs reduce by

8%; Injuries reduce by 15%)

2012 estimates (WHO)

Communicable, maternal, perinatal, nutritional conditions

33,016

27,654

23,506

20,741 19667

NCDs

30,618

25,645

24,363

23,593 24935

Injuries

5,892

4,935

4,442

4,195 4166

Total

69,526

58,234

52,310

48,529

48,768 DALYs averted per 100,000 population -- --

5,924

9,705

9,466

PAD ICR

Pessimistic Optimistic Pessimistic Optimistic

Total DALYs averted 71,088 116,460 68,717 112,582

Total "cost" averted US$ 53.3m US$ 87.3m US$ 51.5m US$ 84.4m

Cost per DALY averted $281 $172 $412 $251

45

DALYs averted – are within the range of what the PAD anticipated. Likewise, the cost per DALY averted is $251, within the range estimated in the PAD. Benchmarking cost-effectiveness of health interventions is extremely tricky. The cost per DALY averted for different sets of health interventions has shown to vary widely across countries and regions. Best estimates, based on evaluations of different proposed packages of care that would improve the coverage and/or the quality of routine maternal care, found a range of US$77 to US$104 per DALY averted in Sub-Saharan Africa, to US$150 per DALY averted in South Asia. The range for the cost of family planning programs is US$30 to US$60 per DALY averted, after including other health impacts along with averted deaths. The cost per maternal death averted was found to be between US$5,000 and US$35,000, and US$1,300 and US$5,000 per infant death averted.36 It was noted that cost-effectiveness within regions also varies by as much as two orders of magnitude because of differences in fertility rates, risks of mortality, and existing contraceptive prevalence rates. It is thus difficult to assess the cost-effectiveness of this project against international or regional benchmarks for specific health interventions. At $251 per DALY averted, its cost is higher than the combined cost of maternal care and family planning packages combined, as outlined above (approx. $107 to $164). Benefits that were anticipated to be accrued from the Hospital Service Package were also likely not realized, as the project dropped this activity, and at project closing there is no available data to assess hospital performance. However, the reduction in mortality rates (MDG indicators) which this project likely contributed to can also be said to be an achievement. The cost of $251 per DALY averted measures very favourably when benchmarked against GDP per capita of $1070 in 2014 (proxy measure of human productivity as a “benefit” to the country). Finally, this computation only takes into account direct benefits, as indirect benefits from broader improvements to the health system are not quantifiable. Overall, the cost-effectiveness of this project is thus rated as Substantial. A separate analysis of efficiency specifically for Phase 1 is not possible due to difficulty of attribution. Given that Phase 1 was the majority of the project duration, the rating of Substantial will be assigned to Phase 1. Economic analysis for Phase 2 of the project was based on the World Bank’s 2011 Health Financing Note for Timor Leste. The note presented proxy indicators for measures of efficiency in health spending. One of these measures showed the relationship between health expenditure per capita and infant mortality rate (IMR) for selected countries, benchmarking Timor Leste against other island states. The comparator countries are: Indonesia, Maldives, Philippines, Papua New Guinea, Solomon Islands, Seychelles, Samoa, Marshall Islands, Mauritius, and Vanuatu. Timor Leste’s position relative to these countries is presented in red in Figure 1 below:

36 Jamison DT, Breman JG, Measham AR, et al., editors. Priorities in Health. Washington (DC): World Bank; 2006. Chapter 4, Cost-Effective Strategies for the Excess Burden of Disease in Developing Countries. Available from: http://www.ncbi.nlm.nih.gov/books/NBK10258/

47

Figure 1: IMR versus Health Expenditure, 2007 vs. 2013, Timor-Leste vs. Island States

Source: World Development Indicators Unfortunately, the analysis shows that between 2007 and 2013, Timor Leste did not improve in its standing vis-à-vis these countries. While there were improvements in IMR (61 per 1000 live births in 2007, to 48 in 2013), IMR remained high compared to these countries, while health spending remained low. Given that IMR is one of the MDGs that Timor Leste surpassed its target on by 2015 – while others such as maternal mortality have not fared as well – it is reasonable to use this as a proxy measure (albeit a partial one) of the best improvement in health outcomes, relative to efficiency in health spending. While Phase 2 of the project may not have directly affected IMR progress, its overarching objective of more efficient and effective allocation and use of resources, in order to improve health service delivery, is not evident in this proxy measure of efficiency. As such, efficiency for Phase 2 is rated as Modest. The project also had other aspects of inefficiency which can be qualitatively described. These included the delay in project start up, the two year period it took from the MTR until its recommendations were officially adopted in the February 2013 restructuring, and again delays in putting in place the new mode of implementation in Phase 2 of the project. Staff shortages, delays in filling key positions, multiple rounds of “socialization” of the project among MOH staff, and absence of consistent in-country Bank supervision at the beginning of the project all contributed to inefficiencies in project implementation. Efficiency of project implementation is thus rated Modest.

0

10

20

30

40

50

60

70

0 100 200 300 400 500 600 700

Infant mortality rate (2007)

Health expenditure per capita (2007, current USD)

IMR versus health expenditure per capita 2007,Timor Leste vs. island states

0

10

20

30

40

50

0 200 400 600 800

Infant mortality rate (2013)

Health expenditure per capita (2013, current USD)

IMR versus health expenditure per capita 2013,Timor Leste vs. island states

48

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members Names Title Unit Lending Rosario Aristorenas Senior Program Assistant GEDDR Natasha Beschorner Senior ICT Policy Specialist GTIDR David Michael Chandler Sr Financial Management Specialist EAPDE Junxue Chu Division Manager WFAFO Olivio Euclides Dos Santos Program Assistant EACDF-DIV Joao Jose Augusto Gomes Procurement Specialist EASRP-HIS Bisma Husen Senior Procurement Specialist EASRP-HIS Nurul Alam Senior Procurement Specialist EAPCO

Timothy Johnston Senior Health Specialist / Team Leader

EASHD

Roch Levesque Senior Counsel LEGAM James Orehmie Monday Senior Environmental Engineer GENDR Cristiano Costa e Silva Nunes Senior Procurement Specialist GGODR Kishor Uprety Senior Counsel LEGES Hope C. Phillips Volker Consultant GHNDR Isono Sadako Social Development Specialist EASIS Ninin Dewey Social Development Specialist EASIS Josefo Tuyor Environmental Specialist EASRE Christopher Scarf Hospital Management Consultant Consultant Nigel Wakeham Architect / Consultant Consultant Supervision/ICR Rosario Aristorenas Senior Program Assistant GEDDR Natasha Beschorner Senior ICT Policy Specialist GTIDR Aissatou Diack Senior Health Specialist GHNDR Olivio Euclides Dos Santos Program Assistant EACDF-DIV Elke Frieda Ender E T Consultant EASHD - HIS

Robert Gilfoyle Senior Financial Management Specialist

GGODR

Joao Jose Augusto Gomes Procurement Specialist EASRP-HIS Jeremias Baptista Gomes E T Temporary EASHH - HIS Stephen Paul Hartung Financial Management Specialist GGODR

Sarbani Chakraborty Senior Health Specialist – Team Leader

EASHD

Susan Lynette Ivatts Senior Health Specialist GHNDR Puti Marzoeki Senior Health Specialist GHNDR Evelyn Villatoro Senior Procurement Specialist GGODR

49

David Whitehead Financial Management Specialist GGODR Tanya Dawn Wells-Brown E T Consultant EASHH - HIS Marjorie Mpundu Senior Counsel LEGES Sebele Berhanu Legal Analyst LEGES Caroline Cheng Finance Analyst WFALN Arip Syaman Sholeh Financial Management Consultant Consultant Unggul Suprayitno Sr. Financial Management Specialist GGODR Virza S. Sasmitawidjaja Environmental Consultant Consultant Sara Pereira Operations Analyst Consultant

Yi-Kyoung Lee Senior Health Specialist – Team Leader

GHNDR

Manuela da Cruz Procurement Specialist GGODR Maria Isabel Alda da Silva Operations Analyst EACDF Cornelio Quintao de Carvalho Team Assistant EACDF

Eileen Brainne Sullivan Senior Operations Officer – Team Leader

GHNDR

Hui Sin Teo Health Specialist GHNDR Sabrina Terry Program Assistant GHNDR

50

(b) Staff Time and Cost

Stage of Project Cycle Staff Time and Cost (Bank Budget Only)

No. of staff weeks US$ Thousands (including travel and consultant costs)

Lending FY07 2.81 16.39 FY08 18.25 94.02

FY09 0.15 4.17 Total: 21.21 114.58

Supervision/ICR FY09 6.92 27.16 FY10 35.98 12.75 FY11 10.50 79.20 FY12 4.72 108.29 FY13 46.23 177.39 FY14 59.46 118.00 FY15 7.53 114.12 FY16 37.40 112.21

Total: 208.74 749.12

51

Annex 5. Beneficiary Survey Results

Not applicable for Core ICRs

52

Annex 6. Stakeholder Workshop Report and Results Not applicable for Core ICRs

53

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR Summary of Borrower’s ICR:

1. Strengthening NHSSP-SP Project Management The management of the project improved during NHSSP-SP. This was due to improvements in the management approach through the implementation of a Project Operations Manual, adequate staffing and resources made available to the Project Management Team (PMT), and improved commitment from implementing directorates, departments, and agencies.

2. Relevance of project design to National Health Sector Strategic Plan A health system is made up of several components such as health policy and planning, human resources, financing, information, drugs, service delivery and monitoring and evaluation. This is why strengthening a health system is very complex, needing detailed analysis of the dynamics and constraints in all areas. The project played an important role in developing these areas. The development of the project was influenced by the Health Sector Strategic Plan 2008-2012 and responded to the sector priorities in service delivery and human resources development.

3. Implementation and operational experience (success and challenges) There are many important lessons learnt from the management side of the project. This ranges from financial management, procurement, human resource management, to contract management. Financial Management Payment process for consultants took long periods to complete. The average time was two weeks, but the longest process noted was one to two months. This was due to lengthy processes from approval of timesheets and progress reports from Direct Supervisors in implementing directorates/departments, followed by review and approval from the PMT, approval from Ministry of Health and Commitment Payment Voucher development and approval by Ministry of Health, and endorsement by Ministry of Finance and transfer of payment. A system of payment must be developed to address the lengthy process with regular monitoring of the payment from authorities to avoid payment mistakes. One example provided to us was the World Bank internal payment system. Unless it has interface with the Government system (FreeBalance), the system has weaknesses. The weakness of this mode is that there is no direct control from Ministry of Finance and Ministry of Health.

54

The use of Excel for transaction registration has resulted in difficulties in obtaining immediate information on the project’s financial position. The migration to FreeBalance system was not successful and the migration process had to be suspended. It is recommended that a standard accounting software is used while migration to any government accounting system in the future. One important financial aspect that needs to be addressed in the future projects is the funding for a small team during grace period—a short term period at the end of the closing down of the project. The small closure team suffered from no financial support during this period. It is recommended that a small petty cash fund is set aside from the grant to enable the team to operate well at the end of the project. The funding can come from state budget that is clearly earmarked for its objectives or a small petty cash fund from the Bank Executed Trust Fund. This must be clearly stated in the Grant and Financing Agreement to avoid confusion of roles between the recipients of the trust fund. Monitoring of Consultants and contract management It is important in future that the Project Office keeps track of consultants’ deliverables and ensures that it checks to see that the consultant is on track as per the agreed upon work plan. Further, the PMT needs to increase the number of available translators. This will in turn help consultants to more productively use their available time. Managing expectations has been another challenging role of the PMT. This is to do with demands on PMT staff time from project counterparts and consultants. Holding regular meetings in large groups does not work well, but one-to-one meeting to listen to challenges from consultants and departments worked well. However, the PMT was over-stretched by applying this approach.

4. Evaluation of the Bank and any co-financiers There has been a good collaborative efforts and support from the Bank as Administrator of the Trust Funds and its co-financiers in during the life of the project. Support to the project has been indicated by regular support to decision making through Project Steering committee meetings. There is some degree of synergy in the effort brought to strengthen the Ministry of Health by aligning activities with Ministry’s priorities and avoidance of overlapping activities regardless of the non-functioning of the health sector coordination committee. The following are noted challenges and opportunities for future improvement of support to Ministry of Health:

Support to and compliance with Paris Declaration and country system by strengthening and using Government system and direct budget support.

Continuous support to achieve outputs through any modality of execution of the trust fund.

55

5. Borrower’s own performance Implementing the NHSSP-SP, which was the descendent of the HSSP-SP with unsatisfactory performance status, was a challenging task for Ministry of Health. However, with improved collaboration with the implementing directorates, departments, and agencies, the World Bank and Donors during the project cycle under the leadership of Honorary Minister and Hon. Vice-Minister for Health, achievement of the outcomes and outputs were possible. Overall, there was improvement in project management, though noted some minor weaknesses in monitoring of financial management. Further improvement is also recorded in the commitment from implementing Directorates, Department, and agencies in monitoring and evaluation of the work of consultants. The sense of ownership of activities improved, as indicated by improved engagement of implementing departments, directorates, and agencies during the implementation process. All of these culminated in the achievement of outputs from consultants.

6. Lessons for future operations Much has been learned through the life cycle of the project implementation. Lessons learnt for future operation are:

Improvement of project design through a well-coordinated consultation with all key stakeholders.

Introduce SMART indicators for Result framework indicators and avoid frequent introduction or changing of the indicators.

Separate indicator for project management and indicators related to NHSSP is required.

The leadership of PFM working group must be maintained by National Directorate for Finance Management and Procurement.

Endorsement of manuals developed by consultants, particularly Financial Instruction Manual and Internal Audit Manual.

Future evaluation on updated and enhanced management system and accounting system at SAMES is required to assessed whether or not the provided assistance have been maintained in SAMES operation and financial management system.

Build on financial control capacities to improve financial management in health sector in the future.

Implementation of the GIFA’s strategic plan and evaluation of the implementing is important for improvement in the future.

Prioritize implementation of recommendations from the functional analysis of human resources in National Hospital, Central Services of MoH and other facilities;

Department of Pharmacy is heavily understaffed, it is recommended that Ministry addressed this by recruiting professional staffs to run the DOP effectively;

When necessary review and amend the law on drug marketing authorization;

56

Improve capacity to independently operate the mSupply via trainings and identification of national trainers;

Gradually extend the utilization of mSupply to other District Health Services and other facilities based on the assessed ITC feasibilities;

Improve basic storekeeping practices and other related conditions such as inventory data at health facilities at all level to enable successful implementation of mSupply;

Maintain availability of pharmacists at health facilities; Establish a strong Biomedical Equipment team to provide regular maintenance to

biomedical equipment. Repatriation of biosafety unit has not been done. Re-asses the repatriation of

biosafety units to other facilities with suitable rooms or adjust laboratory sites for the installation of uninstalled biosafety units is required.

Grace period funding must be provided to the project management team to allow funding for operational cost, i.e. filling system, fuel, and bank transfers fees.

Opening an Escrow Account and closing of Designated Account have been two new experienced for Project Management Team, of Ministry of Health. There has been lack of clear guidelines on these two processes. It is recommended that for the future, project Operation Manual must include these in a special section. Topics covered in financial management section must include clear steps/guide on opening of an Escrow Account, transfer fees for both transfers of funding from Designated Account (DA) to Escrow Account and changes of refund of any remaining balance from DA to the World Bank Account.

There has not been a clear guideline on the evaluation of project by Ministry of Health. The evaluation prepared has been based on the concept of Monitoring and Evaluation by linking inputs, to outs and outcomes. It is recommended that future Project Operations Manual must add a section on the closing down of the project that addresses the issues of report, evaluation and financial management during Grace Period.

57

Annex 8. Comments of Co-financiers and Other Partners/Stakeholders Comments from European Union: Objectives of the MDTF have been always over-ambitious, and this has been demonstrated by the need for continuous reduction through the years of the MDTF intervention areas and adjustment of its objectives. In other words, the MDTF has mostly remained at theoretical level without taking into consideration the realistic absorption capacity of the health system in Timor Leste, especially at peripheral levels and in relation to available infrastructure, equipment and human resources. More specifically, human resources – both in quantity and quality – are remaining a relevant bottleneck for services delivery. The MDTF spent most of the financial and human resources in Technical Assistance for the definition of most perfect sector strategies and policies, without paying enough attention to their implementation at district/municipality, Community Health Center, and Health Post levels, which resulted in the MDTF’s overall moderate unsatisfactory performance. During the different phases of its implementation, the MTDF management has been poor and mainly in the hands of the MDTF Project Management Team, which in addition has not been able to create a conductive environment between development partners, stakeholders and MoH. Further, according to our experience the "MDTF steering committee" spent a considerable amount of energy on the program's restructuring, instead of focusing on planned activities. It seems that MOH never took real ownership of this programme and did not integrate it as a sector support programme supporting the implementation of its Health Strategy objectives and Annual Action Plans. Recorded implementation delays were surely related to the political instability at the beginning of the intervention, and later to the slow and highly centralized decision making process, the latest partially affected by rapid turnover of human resources and poor or unavailable data. Looking at the initial objectives/indicators and the subsequent downsizing of the programme, the level of performance is particularly unsatisfactory, especially compared to amount of funds invested. It was supposed to be a sector support programme, but eventually it only focused on very limited areas of intervention. In addition, it was supposed to pool resources from several partners to avoid duplication and reduce transaction costs for the Ministry, but in the end DFAT reduced its contribution to the project (due to poor performance) and started providing parallel assistance. Overall although some of the selected indicators have shown an improvement in the delivery of health services and the reduction of mortality indicators, the EU agrees on the MDTF’s moderately unsatisfactory outcomes highlighted by this report.

58

Annex 9. Project Results Framework – Original and Revised

1. Revisions during February 2013 Level 1 Restructuring / Additional Financing:

Revisions to the Results Framework Comments/

Rationale for Change

PDO

Original (PAD) Revised To improve the quality and coverage of preventive and curative health services, particularly for women and children, in order to accelerate progress toward the health millennium development goals (MDGs)

To support the Government of Timor-Leste to get more resources to where they are needed most to improve the delivery of health services in districts and sub-districts

The original PDO reflected the initial intention of using the MDTF as a sector financing instrument under a sector wide approach. However, the initial design was too complex. The restructured project would target system and resource management issues that underpin improved health service delivery at the district level.

PDO indicators

Original (PAD) Revised % of children under 1 year vaccinated with (i) DPT3 (ii) Measles

Indicator dropped as it was beyond the project scope

% of births attended by skilled health personnel


% of pregnant women receiving at least four antenatal visits


% of children (6-59 months) receiving vitamin A supplements


% married or cohabitating women using modern contraceptives


Number of districts that receive recurrent expenditures consistent (+/- 20%) with the NHSSP MTEF and approved budget

New indicator for the restructured project and revised scope of work

Number of districts that report expenditure against the approved plan/budget on time


Percentage of stock-outs of tracer essential drugs at health facilities

Intermediate result indicator revised to correctly reflect the level of result

Joint annual sector review and planning meetings held as planned (at least 1/year)


Intermediate Results indicators

Original (PAD) Revised % of health clinics providing the comprehensive Basic Service Package (BSP)*


% of pregnant women receiving tetanus toxoid (TT) injection

Indicator dropped as it was beyond the project scope; HMIS did not provide this kind of data

59


Rationale for Change Cesarean section rate Indicator dropped as it was beyond

the project scope; HMIS did not provide this kind of data

Number (%) of children (6 mo – 3 yrs) participating in integrated community nutrition programs

Indicator dropped as it was beyond the project scope; HMIS did not provide this kind of data

Number of referral hospitals implementing the Hospital Service Package*

Indicator dropped as Hospital Service Package was dropped

Number of referral hospitals implementing guidelines* for hospital management and clinical quality, including (i) monthly clinical review meetings; (ii) maternal death audits


Availability of tracer essential drugs at (i) SAMES;

(ii) health facilities

Percentage of stock-outs of tracer essential drugs at SAMES

Intermediate result indicator revised to correctly reflect the level of result

Percentage of biomedical equipment in hospitals that is out of order (non-functional)


(i) total health staff trained with project financing; (ii) Percentage of district and CHC managers that have received management training


% of health facilities/ submitting completed HMIS monitoring reports 1 month after end of each quarter


(i) % recurrent expenditures on hospitals (<40%); (ii) % State Budget allocated to health; (iii) % execution of health budget


Number of partners submitting to MOH a fully costed work program for forthcoming year


Number of pilot initiatives (i) financed by project and implemented; (ii) evaluated with lessons disseminated; (iii) scaled up and/or influenced policy


Number of formal MOH- private sector/NGO partnership contracts signed


Health MTEF prepared based on costing of strategic plan


Number of districts that submit annual district plan/budget to the Central MoH for approval on time


Percentage of targeted staff who pass competency test in financial management, procurement, and/or management


60


Rationale for Change Percentage of SAMES’ procurement

activities using direct contracting (emergency order, in $ value)


Number of targeted places (13 districts and 6 hospitals) with functional National Logistics Management Information System


A computerized drug registration system available


Number of structured sector coordination meetings held to progress toward one plan, one budget, and one sector M&E framework


Percentage of research projects approved by the research cabinet that responds to the national research priorities defined by the MoH


2. Revisions during October 2014 Level 2 Restructuring :


Rationale for Change

PDO

Current (from Feb 2013 restructuring / AF)

Revised

To support the Government of Timor-Leste to get more resources to where they are needed most to improve the delivery of health services in districts and sub-districts

No change --

PDO indicators


Revised

Number of districts that receive recurrent expenditures consistent (+/- 20%) with the NHSSP MTEF and approved budget


Number of districts that report expenditure against the approved plan/budget on time

Indicator dropped as it was beyond the project scope (Reporting done centrally instead of by districts, therefore indicator not appropriate)

Percentage of stock-outs of tracer essential drugs at health facilities

Management of expiry dates leads to reduction of expired medicine at warehouse

Revised to reflect appropriate level of project activity (at warehouse, not at health facilities)

Joint annual sector review and planning meetings held as planned (at least 1/year)

No change

61


Rationale for Change Number of requests from budget

divisions receiving advances per quarter and which are acquitted within 30 days following end of calendar quarter (National Hospital not included)

New indicator to reflect revised scope of work

Number (percentage) of facilities having taken the readiness assessment survey and who score at least 80%


Number of facilities (districts) that have undertaken a facility readiness survey


Intermediate Results indicators


Revised

Percentage of stock-outs of tracer essential drugs at SAMES

No change

Health MTEF prepared based on costing of strategic plan


Number of districts that submit annual district plan/budget to the Central MoH for approval on time


Percentage of targeted staff who pass competency test in financial management, procurement, and/or management

Percentage of targeted staff who pass competency test in 7 PFM areas organized by the MOF

Indicator revised to reflect revised scope of work

Percentage of SAMES’ procurement activities using direct contracting (emergency order, in $ value)

Percentage of SAMES’ procurement activities using direct contracting (emergency order, in $ value) from Nov 2011 to Oct 2013

Indicator revised to reflect institutional changes at SAMES

Number of targeted places (13 districts and 6 hospitals) with functional National Logistics Management Information System

Number of districts and hospitals with functional National LMIS

Minor change in phrasing of indicator

A computerized drug registration system available

No change

Number of structured sector coordination meetings held to progress toward one plan, one budget, and one sector M&E framework

No change

Percentage of research projects approved by the research cabinet that responds to the national research priorities defined by the MoH

Indicator dropped as it was beyond the reduced project scope

Number of pilots with functional mSupply / mobile mSupply by 2015


Percentage of rejected Commitment Payment Vouchers on a quarterly basis

New indicator to reflect revised scope of work and data that are available

62


Rationale for Change Number of districts implementing

supportive supervision New indicator to reflect revised scope of work

Amount of Department of Pharmacy procurement activities using direct (sole source) in $ value from October 2013 to date

New indicator to reflect institutional changes at Department of Pharmacy

63

Annex 10. Country and Health Sector Context at Project Appraisal Country Context Occupying the mountainous eastern half of the island of Timor, Timor-Leste was a Portuguese colony for 450 years. Following a brief civil war, the country unilaterally declared independence on November 28, 1975. Within days it was invaded and illegally occupied by Indonesia, though for 24 years the population of Timor-Leste maintained a staunch resistance. Following the fall of President Suharto in 1998, a referendum on autonomy for the territory was held, and an overwhelming majority of the Timorese preferred independence. However, the outcome of the referendum was accompanied by a campaign of violence which left over 1,000 people dead and the majority of the population displaced. Following the intervention of a multilateral peacekeeping force, the United Nations Transitional Administration in East Timor (UNTAET) was established. In 2001, elections were held for a Constituent Assembly, followed by the adoption of the Constitution and presidential elections in April 2002. The Democratic Republic of Timor-Leste fully restored its independence on May 20, 2002. The Constituent Assembly became the National Parliament, and the current Government of Timor-Leste (GoTL) was sworn in. A 900-person United Nations Mission of Support in East Timor (UNMEET) was created, and was succeeded in May 2005 by the United Nations Office in Timor-Leste (UNOTIL). However, political tensions and growing discontent among urban youth contributed to periodic unrest. Beginning April 2006, however, a major political crisis led to violence between the police and armed forces and exacerbated regional tensions. Over 40 people were killed initially, and subsequently thousands of homes were burned in a cycle of violence and retribution that lasted several months. An estimated 150,000 people - out of a population of just over a million - fled their homes and settled in Internally Displaced Person (IDP) camps. The return of international peacekeeping forces helped restore a measure of stability. Presidential and parliamentary elections were held in April 2007 and May 2007, respectively, and resulted in the installation of a new coalition government. A relative calm returned following the elections, but tensions remained. An estimated 70,000 are still in IDP camps. The government’s fiscal position had improved dramatically by 2007 as a result of growing petroleum revenues, but capacity to execute the budget emerged as a major constraint. Poverty levels remained among the highest in the sub-region, and unemployment among youth, particularly in urban areas, continued to contribute to social tensions. In this context, it was extremely important for the government to demonstrate that it could effectively deliver services to its population. Improving the quality and coverage of health services was of vital importance for the stability of the country as well as the health of the population.

64

Health sector context Much of Timor Leste’s physical infrastructure was left in ruins by the late 1990s, including nearly 80 percent of health centers. Most Indonesian citizens had departed, resulting in a severe shortage of qualified and experienced professionals and leaving Timor-Leste with very thin human resource and institutional capacity. In the early years of independence and through the 2006/7 crisis, the Ministry of Health (MOH) continued to function. Many central MOH staff relocated with their families to IDP camps on the grounds of the national hospital in Dili. Senior MOH officials held daily meetings and maintained regular telephone and radio contact with district health team, to ensure that health service remained functional in both Dili and in rural areas. The MOH also sought to ensure coordination of the international response to the crisis, including coordinating NGOs providing health services to IDP camps. While the health sector had made significant progress since independence in reestablishing basic infrastructure and services, it was nonetheless starting from a relatively low base in terms of basic functionality of services and human resource capacity. Key challenges included: i. Persistent poor health status and outcomes: Child and maternal mortality, child malnutrition, and fertility indicators for Timor-Leste remained among the highest in the region. In 2003 (most up to date information at project appraisal), the child mortality rate was 83 per 1,000 children, and nearly two-thirds of children under five were underweight. Fertility rates were among the highest in the world, with women having an average of 7.7 children. The maternal mortality rate was also very high, estimated at 660 per 100,000 live births. Communicable diseases such as diarrhea, TB, and malaria were also prevalent. ii. Utilization and quality of health services: Despite progress in reestablishing basic services and infrastructure, problems persisted with inadequate quality and low utilization of health services, particularly for preventive services such as vaccination and prenatal care. Vaccination rates had improved, but only 18 percent of children were fully immunized in 2003, and only 18 percent of births were attended by skilled personnel. Low utilization was a consequence of both supply and demand factors. Quality of health services was compromised by continued shortages of key inputs such as essential drugs, laboratory supplies, or fuel for field outreach services, as well as inadequate training, motivation, and supervision of health personnel.

iii. Human resource development: There had been an influx of over 300 Cuban doctors in the mid-2000s, which helped reduce shortages of medical personnel. Over 600 Timorese had also been sent to Cuba for medical training. The Cuban doctors had played an important role in maintaining health services following the April 2006 political crisis. However, it created new challenges in human resource management, as doctors were now stationed in health clinics originally designed to be staffed by nurses, and were prescribing a much wider range of drugs. However, access to qualified health personnel remained difficult in rural areas, and inadequate training, supervision, and motivation of health workers represented an important constraint on the quality of care.

65

Annex 11. List of Supporting Documents

1. Project Appraisal Document, November 14, 2007 2. Financing and Grant Agreements / Amendment Letters:

i. Financing Agreement between Democratic Republic of Timor Leste and International Development Association, signed April 16, 2008

ii. Health Sector Strategic Plan Support Project Multi-Donor Trust Fund Grant Agreement (TF091653) Amendment Letter, signed April 25, 2013

iii. National Health Sector Strategic Plan Support Project, Amendment to the Agreement Providing for the Amendment and Restatement of the Financing Agreement and Amendment to the Multi-Donor Trust Fund Grant Agreement, signed April 22, 2015

3. Restructuring Papers: i. Level 2 restructuring, dated February 28, 2012, approved March 5, 2012 ii. Level 1 restructuring / AF, dated January 24, 2013, approved February 27,

2013 iii. Level 2 restructuring, dated September 30, 2014, approved October 31,

2014 4. Implementation Status and Results Reports 5. Progress reports and MDTF Annual Reports 6. Timor Leste Health Financing Note, 2011. World Bank Group

66

MAP