presentation of final report the results of the thematic evaluation on maternal health (2000-2011)...

Presentation of Final Report

The Results of the Thematic Evaluation on Maternal Health (2000-2011)

United Nations Population FundEvaluation Branch

Division for Oversight Services

New York, 07.11.2012

Overview of proceedings10:30 Introduction:

• Objectives, scope of evaluation

• Methodology used

Louis Charpentier, Evaluation Manager, Evaluation Branch / DOS (UNFPA)

10:40 Presentation of key findings and conclusions of thematic evaluation

Martin Steinmeyer, Team Leader (AGEG)Isabelle Cazottes, Co-Team leader (AGEG)

11:20 Open discussion

11:40 Presentation of the recommendations of the evaluation

Martin Steinmeyer

12:00 Use of evaluation results Laura Laski, Chief – Sexual and Reproductive Health Branch, Technical Division (UNFPA)

12:30 Open discussion

12:50 Next steps Louis Charpentier

2 Evaluation Branch, Division for Oversight Services

Evaluation ManagerChief, Evaluation Branch,

Division for Oversight Services

Introduction: Objectives, Scope and Methodology of the Thematic Evaluation


Objective, users, rationale of maternal health thematic evaluation

Objective

• Assess relevance, effectiveness, efficiency and sustainable of UNFPA in contributing to improvement of maternal health

Evaluation Users

• UNFPA country offices, Programme and Technical Divisions, including MHTF staff

• Partner Governments, member states, donors, civil society

Rationale &

Purpose

• Inform decision-making on policy / project management • Establish accountability and oversight of fund management• Promote lesson-learning culture• Provide guidance on strengthening partnerships


Thematic & temporal scopeMYFF 2000 - 2003 MYFF 2004 - 2007 Strategic Plan 2008 - 11

2000 2010/11

Capacity Development / Human Resources for Health

SRH Services (Family Planning / EmONC)

Results & evidence orientation

Harmonization & coordination of support

Coherence of SRH, Population & Development, Gender

Relevance of UNFPA support

Integrating MH into national policies & frameworks

Coherence between country, regional & global programmes

Visibility of UNFPA in maternal health

MH support in humanitarian contexts


Geographic scope of evaluation

1st stage sampling (“Universe”)55 programme countries with MMR

greater than 300 death / 100,000 life births

2nd sample22 countries (range of MH

performance, GNP, quality of public admin, HIV prevalence)

3rd sample (country case studies)

Burkina Faso, Cambodia, DRC, Ethiopia, Ghana, Kenya, Lao

PDR, Madagascar, Sudan and Zambia


Analysis and reconstruction of UNFPA’s own

intervention logic (Core fund (& MHTF))

Analysis of the context of UNFPA support to maternal

health

Identification of critical aspects (Core Fund & MHTF)

DAC Criteria (Relevance,

Effectiveness, Efficiency,

Sustainability)2 sets of Evaluation Questions:

1. Maternal Health Thematic Evaluation (12 EQs)

2. MHTF Mid-Term Evaluation (8 EQs)

UNFPA results frameworks

(MYFFs, Strategic Plan, etc.)

MHTF Business Plan, etc.

Analysis of strategies of other

UN agencies & donors

Academic and “gray” literature on

reproductive & maternal health

ToR / Input from Reference

Group

Main Topics / Issues / Purpose identified in ToR;

specified by Reference Group

Methodology – Definition of Scope


Methodology – Data Collection

Data were collected to answer 12 Evaluation Question of Thematic Evaluation

Data collected by following means: Desk review of existing evaluations, reviews and other

documents; Ten country case studies (Burkina Faso, Cambodia, Ethiopia,

Ghana, Lao PDR, Madagascar, Sudan (North) and Zambia); focus groups, interviews with UNFPA staff, Government, development partners, civil society, beneficiaries

Online survey of 55 country offices on technical support to country offices & on organizational capacity

Face-to-face and telephone interviews with UNFPA staff members (headquarters, regional and sub-regional offices), other external partners.


Methodology – Number of People Interviewed

InstitutionsNumber of People

Interviewed

UNFPA 23 (global), 6 (regional), 65 (country)

Central Government 90

Local Authorities 50

Development Partners 8 (global) 65 (country)

Civil Society Organisations 55

Training Institutions 30

Health Service Providers 40

Final Beneficiaries (focus group discussions)

18 discussions (10-15 people each)


Martin Steinmeyer, team leader (AGEG)Isabelle Cazottes, co-team leader (AGEG)

Key Findings and Conclusions: The UNFPA contributions to changes in maternal health

outcomes


SELECTED KEY FINDINGSAnswers to the Evaluation Questions


Evaluation Question 1: Relevance

Resources not allocated proportional to severity of maternal health needs

Resource allocation system has not fully considered relative need levels in each country

Resources distributed based on criteria such as “degree of political support for ICPD”, “absorptive capacity”

No clear, operational definition of maternal health-related vulnerability

Common practice: targeting MH support geographically (regions / districts with high maternal mortality prevalence)

Less common: targeting systemic socio-economic barriers to MH services (cost, distance, transportation)


Evaluation Questions 4: Capacity Development – Human Resources for Health Training support thematic priority of UNFPA

2000 – 2005: in-service trainings; Challenges: Low alignment with national HRH systems; Little discernible sustained effect on quality of service

2005 – 2010: pre-service trainings, support of nat. training oversight agencies (curricula review, regulatory frameworks); Advantages: alignment w. nat. structures, sustained effects on SRH training systems

Challenge: Inappropriate deployment & placement of trained staff / high mobility

Linked to weak nat. HRH systems (planning, implementation, monitoring) – beyond SRH / MH

UNFPA offices struggled to approach cross-cutting challenges Some partnerships w. HRH offices in health ministries (e.g.,

Cambodia) 13 Evaluation Branch, Division for Oversight Services

Evaluation Question 7 – Emergency Obstetric and Newborn Care (EmONC) EmONC support prior to 2008, but MHTF has

intensified UNFPA EmONC support Technical assistances, finances sped up EmONC needs

assessments; basis for scale-up plans (Lao PDR)

But: Availability of; access to EmONC affected by admin. gaps, cost, weak referral systems

UNFPA offices struggled to approach cross-cutting challenges (barriers) (UNFPA mandate? Capacity?)

Positive exceptions: 1) Analysis of barriers; 2) Long-term Partnerships in- & outside SRH; 3) Advocacy

Burkina Faso: EmONC subsidy, emerged out of long-term UNFPA advocacy, partnership w. Direction de La Famille (MoH), Parliamentarians; increase in facility-based deliveries (28% overall (2003 - 2010))14 Evaluation Branch, Division for Oversight Services

Overview of key findings for other evaluation questions (selection)

Questions Key Findings

EQ2: Harmonization / Coordination

• Long-term partnerships prerequisites for UNFPA effect on evidence-based harmonized MH support

EQ3: Communities & MH demand

• MH awareness raising to be coupled with addressing financial barriers to be successful

EQ6: Family planning

• Good results / partnerships in commodity security (GPRHCS); less focus on FP demand / delivery

EQ8: Results / Evidence

• Weak monitoring and evaluation of MH interventions; low results-focus

EQ9: MH policy frameworks

• Success depended on combining data & research, advocacy, technical assistance, partnerships

EQ10: SRH, Gender, P&D

• Country offices commonly lacked management mechanisms for integrated programming (strategy!)

EQ12: Visibility • Visible advocate for MH; extent depended on CO capacity to bring technical knowledge to bear


Performance patterns of country-level support – Example HRH

CO Management

P&

D

SR

H

Ge

nd

er

RH policy arena

Health policy arena (& other policy arenas)

SBA assessm. TA SBA Policy

advocacy

SBA plan; midwifery

training policy

HRH Policies / Programmes

Long-term strategy

HR

H

Dep

. /

MoH

MH

C

en

tre

/

Nu

rsin

g

Co

un

cils

E.g, p

oor

depl

oym

ent

E.g

, W

HO

Oht

er

dono

rs

Strengthening

deployment

system

Curricu

lum

revie

ws;

Traini

ng


Performance patterns of country-level support - positive scenario

Management

P&

D

SR

H

Ge

nd

er

RH policy arena


Data / research

Technical Assistance

Policy advocacy

MH policies & programmes

Health Policies / Programmes

Long-term strategy

Gov

. pa

rtne

r

Gov

. pa

rtne

r

MH B

arrie

rs

Dev

. pa

rtne

r

Dev

. pa

rtne

r

System

Strengthening

Imple

men

t.

Suppo

rt

Country Office17 Evaluation Branch, Division for Oversight Services

Performance patterns of country-level support – opportunities for improvement

18

CO Management

P&

D

SR

H

Ge

nd

er

RH policy arena


Data / research

Technical Assistance

Policy advocacy

MH policies & programmes

Health Policies / Programmes

Short-term planning (project by project)

Gov

. pa

rtne

r

Gov

. pa

rtne

rMH B

arriers

Dev

. pa

rtne

r

Dev

. pa

rtne

r

Curricu

lum

revie

ws;

Traini

ng

Country Office


Conclusions

Conclusions - Appropriateness of UNFPA maternal health strategy

C1: UNFPA maternal health support in programme countries has not been sufficiently based on country-specific medium or long-term strategies

Maternal health support has been more effective when country offices based interventions on multi-annual strategic vision

Current templates not conducive for strategic planning; No requirement to develop multi-annual operational plan.

Country offices without multi-annual perspective more likely to manage interventions separately from each other;

Fewer incentives for staff working on different interventions to coordinate work; pool financial and organizational resources from different funding sources


Conclusions - Appropriateness of UNFPA maternal health strategy (2)

C2: In its approach to support maternal health, UNFPA has not sufficiently responded to its mandate to focus on addressing the root causes of poor maternal health of the most vulnerable

HQ has not sufficiently defined operational implications of focusing on needs of “most vulnerable”

Country offices did not sufficiently analyse risks and barriers keeping women and girls from accessing MH services (instead targeted geographical pockets of high maternal mortality)

COs did not address weakness of health systems that made specific groups more vulnerable to poor maternal health (e.g., deployment challenges, in particular for rural areas; deficiencies in supervision and service quality control)


Conclusions - Appropriateness of UNFPA maternal health strategy (3)

C3: UNFPA support of maternal health services at sub-national level has not consistently reflected comparative strengths of UNFPA as primarily knowledge- and evidence-based organization

Significant portion of small SRH budget allocated to sub-national level (sub-national offices, staff, interventions); often one of several organisations active there

Country offices have not consistently used presence at sub-national level to generate data, lessons and knowledge to further national MH policy agenda


Conclusions – role & capacity of UNFPA country offices

C4: Insufficient staff capacity and skill gaps have negatively affected ability of country offices to act as brokers of maternal health expertise and as facilitator of national and international maternal health commitments and partnerships

Small numbers of RH staff made it difficult to be present in national technical working groups or policy forums

Annual planning based on inadequate templates (AWPs, CPAPs) and limited experience in strategic planning made it difficult to develop multi-annual strategies that combined resources and skills from different areas to facilitate sustained changes in health systems


Conclusions – role & capacity of UNFPA country offices (2)

C5: Country offices have not received sufficient technical support from regional offices and headquarters to fulfill central role delivering maternal health support

Availability of technical support has been limited in human resources for health, EmONC and midwifery (pre-MHTF), strategic planning, results-based management and monitoring and evaluation

Maternal Health Thematic Fund (MHTF) has been valuable instrument to make available additional resources for technical support


Conclusions – partnerships with donors, governments and other stakeholders

C6: Varying capacity of country offices to establish partnerships with government, donors and civil society in and outside of reproductive health has reduced ability to address service access barriers and to strengthen MH systems

Long-term, proven working partnerships allow UNFPA to extend reach beyond sexual and reproductive health Partnerships allow UNFPA to tie maternal health HRH

support to larger governmental efforts to strengthen national health systems

Pre-conditions: UNFPA to repeatedly prove worth as reliable partner, gain trust and collaboration, consistent leadership through country representatives


Conclusions – Use of Evidence; Monitoring and Evaluation

C7: Lack of appropriate monitoring and evaluation mechanisms has affected capacity of UNFPA to assess results of maternal health support and to optimize corporate and country-level MH strategies over time.

Monitoring has focused on activities or higher level societal changes relating to maternal health,

Monitoring system thus not providing information on UNFPA contribution to these changes (which mirrors deficiencies in UNFPA planning)

UNFPA implementing partners w/o required technical capacity to fulfil M&E responsibilities; UNFPA country offices w/o sufficient skilled M&E staff to support implementing partners


Conclusions – Added value of the Maternal Health Thematic Fund (MHTF)

C8: Maternal Health Thematic Fund (MHTF) has helped to provide much needed financial and staff resources to UNFPA country offices and headquarters in the short and

medium-term.

MHTF-financed staff positions bolstered staff capacity of country offices, allowing them to intensify engagement in in EmONC, midwifery, obstetric fistula

Partnerships made available additional technical support (e.g., Averting Maternal Death and Disability (AMDD), International Confederation of Midwives (ICM))

Helped to raise UNFPA profile and visibility globally & in countries


Conclusions – Added value of the Maternal Health Thematic Fund (MHTF)

C9: The MHTF has not been sufficiently integrated into organizational structure of UNFPA and the overall planning process at country level to ensure sustainability of its interventions

MHTF has proved instrumental in facilitating EmONC assessments, midwifery needs assessments, which constituted basis for governments to launch MH-relevant reforms

MHTF not sufficiently integrated into UNFPA planning to ensure timely resource mobilization from within UNFPA and partners for continued support of reforms


QUESTIONS?COMMENTS?


Recommendations

Recommendations: Reviewing UNFPA maternal health strategy and support

R1: UNFPA should revise internal procedures, tools and templates for strategic planning. New approach should require country offices to develop maternal health support strategies for medium to long-term, and to detail how resources from different sub-programmes will be used to implement these strategies. (based on conclusions C1, C7)

Present detailed analyses of specific political, administrative, socio-economic challenges in revised CPDs and CPAPs and corresponding strategy to address them

Put in place multi-annual plans that detail theory of change, risks & assumptions and monitoring indicators for strategy


Recommendations: Reviewing UNFPA maternal health strategy and support (2)

R2: UNFPA needs to better define operational implications of targeting the needs of the “most vulnerable”. Concept is relevant part of maternal health strategy, yet it is too vague in its current form to guide maternal health programming at country level. (based on conclusion C2)

Develop typology of MH-relevant barriers for different types of services and contexts

Prepare operational and programming guide to a) offer clear definition of MH vulnerability and the contributing social, economic, political variables; b) identify policy sectors that influence MH vulnerability; c) lay out options to influence determinants of vulnerability; d) define UNFPA role in contributing to donor response; e) discuss use of partnerships


Recommendations: Reviewing UNFPA maternal health strategy and support

R3: UNFPA needs to increase focus on knowledge generation and learning, to make use of lessons from sub-national service delivery support (and other interventions) to inform evidence-based policy advocacy at core of UNFPA mandate. (based on conclusion C3, C7)

Service-delivery support needs to contribute to body of knowledge on maternal health support in programme country and beyond

Strengthen provisions for monitoring, evaluation, learning, in particular for interventions at sub-national level


Recommendation: Improving capacity of country offices

R4: UNFPA needs to better align capacity and skill mix of staff and managers to ensure that country offices can fulfill their role as knowledge brokers and facilitators of evidence-based approaches to improve maternal health. (based on conclusions C1, C4, C7)

Country offices to develop resource plan (part of CPAP) to allocate staff time to strategy components

Strengthen skills related to health policy and management, public health ; also project management, M&E

Strengthen accountability of country representative for strategic positioning of country offices for partnerships, advocacy to complement UNFPA interventions


Recommendation: Improving capacity of country offices

R5: Planning process for regional-level technical support to be better aligned with long-term strategic and operational planning for maternal health support at country level. (based on conclusions C4, C5) Long-term strategic planning at country level needs to

be mirrored by long-term planning of technical support at regional level

Regional offices’ planning processes need to address current country-level programming needs and anticipate future support requirements

RPAPs / Country Programmes to be developed jointly with country offices, HQ, MHTF, GPRHCS, including resource plan (similar to country level)


Recommendations: Improving guidance on UNFPA partnerships

R6: Anchor concept of partnerships more firmly in strategic documents, operational guidelines and job descriptions of managers. Strategic documents need to explain importance of different types partnerships to ensure sustainable results. UNFPA managers need to be responsible for setting up required partnerships at country, regional and global levels. (based on conclusions C1, C6) Develop typology of stakeholders and their significance for

addressing root causes of poor maternal health Strengthen capacity of regional offices to support country offices

in establishing partnerships Job descriptions of managers need to emphasize their

responsibility for strategic positioning of UNFPA through partnerships (performance criterion)


Recommendations: Improving UNFPA provisions for monitoring

R7: Strengthen result-oriented monitoring for country offices to measure results and not only activities and inputs. Assign greater responsibilities and offer more guidance to country offices for supporting the set-up of appropriate monitoring mechanisms with implementing partners. (based on conclusions C1, C4, C5, C7)

Develop guidelines for results-oriented monitoring & related training resources

Strengthen responsibilities and capacity of country offices to support set-up of monitoring mechanisms with implementing partners (consider partnerships)


Recommendations: Revise role & integration of MHTF

R8: Strengthen MHTF as catalytic tool to facilitate implementation of evidence-based maternal health interventions. Clarify that MHTF is not only responsible for launching initiatives (EmONC, midwifery), but also for assisting country offices to support these initiatives until completion. (based on conclusions C8, C9 + MHTF-mid-term evaluation) Clarify: MHTF helps launch initiatives; and supports their

implementation (business plan) Update guidance to country office on purpose and role of MHTF

Support to expect from MHTF Responsibilities of country offices when working with MHTF

funds Harmonize MHTF planning with (proposed) multi-annual planning

approach for core funds


Recommendations: Revise role & integration of MHTF

R9: UNFPA should use MHTF funds to carry out pilot interventions on selected core maternal health issues, such as the development of appropriate support strategies to better target populations with high vulnerability to poor maternal health. (based on conclusions C3, C8, C9)

Inventory of MH issues and topics that require pilot-testing Invite country offices to submit proposals for pilots for

maternal health interventions Ensure access of country offices to required resources for

pilots through MHTF Ensure analysis and dissemination of results with MHTF

resources


QUESTIONS?COMMENTS?


NEXT STEPS

- Presentation of the evaluation results and the management response to the Executive Board – January 2013

- Webinar presentation of the management response – February 2013


Maternal Health Thematic Evaluation Report and related products are available at the following link:


http://www.unfpa.org/public/home/ about/Evaluation/EBIER/TE/pid/10094

http://www.unfpa.org/public/home/about/Evaluation/EBIER/TE/pid/10094








THANK YOU FOR YOUR ATTENTION


Any other questions?

Contact us: [email protected]

mailto:[email protected]

presentation of final report the results of the thematic evaluation on maternal health (2000-2011)...

Documents

evaluation branch dos

mhtf midterm evaluation

geographic scope of

evaluation manager chief

sets of evaluation questions

oversight services slide

reproductive health

technical division unfpa