unicef programming in health systems strengthening … · however, unicef is not always perceived...

Final Report

Itad in association with UNICEF

Final Synthesis Report

UNICEF PROGRAMMING IN HEALTH SYSTEMS STRENGTHENING – A FORMATIVE EVALUATION

Volume 1 (see Volume 2 for Annexes) Date: January 2019 Submitted by Itad In association with

Final Report

Itad in association with UNICEF i

Acknowledgements

The evaluation team wishes to thank all of the stakeholders who have contributed to the evaluation over the past two years, including the seventeen country offices that participated in Year 1 of the evaluation and in particular those six countries that participated in Year 2. We would also like to thank the Evaluation Office for its support.

All of the inputs have made a valuable contribution to this report; however, the findings remain the sole responsibility of the evaluation team.

Disclaimer

The views expressed in this report are those of the evaluators. They do not represent those of Itad or UNICEF or of any of the individuals and organizations referred to in the report.

‘Itad’ and the tri-colour triangles icon are a registered trademark of ITAD Limited.

Final Report

Itad in association with UNICEF ii

Contents

List of acronyms iii

Executive summary 1

1. Introduction 6

2. Evaluation approach and methodology 8

2.1. Evaluation approach and design 8

2.2. Scope 9

2.3. Data collection and analysis 9

2.4. Limitations 11

3. Findings 12

3.1. To what extent is UNICEF implementing relevant and effective HSS interventions? 13

3.2. What strategies and factors have enabled or hindered progress? 22

4. Implications for UNICEF 32

4.1. Recommendations 35

4.2. Short-term results Error! Bookmark not defined.

4.3. Longer-term results Error! Bookmark not defined.

Final Report

Itad in association with UNICEF iii

List of acronyms

AIDS Acquired Immune Deficiency Syndrome

CHW Community Health Worker

CO Country Office

CP Country Programme

CPD Country Programme Document

DAC Development Assistance Committee

DIC District Investment Case

DLG District-Level Governance

ECARO Europe and Central Asia Regional Office

EPI Expanded Programme of Immunization

ESARO Eastern and Southern Africa Regional Office

GEM Gender Equality Marker

HQ Headquarters

HF Health Financing

HIV Human Immunodeficiency Virus

HRH Human Resources for Health

HSS Health Systems Strengthening

IMCI Integrated Management of Childhood Illness

LACRO Latin America and Caribbean Regional Office

MICS Multiple Indicator Cluster Survey

MNCH Maternal, Newborn and Child Health

NICU Neonatal Intensive Care Unit

ODA Official Development Assistance

OECD Organisation for Economic Co-operation and Development

PDIB Programme Information Database System

PNS Patronage Nursing System

PSCM Procurement and Supply Chain Management

QoC Quality of Care

RAM Results Assessment Module

RO Regional Office

ROSA Regional Office for South Asia

SDG Sustainable Development Goal

TCS Thematic Case Study

ToC Theory of Change

Final Report

Itad in association with UNICEF iv

ToR Terms of Reference

UHC Universal Health Coverage

UK United Kingdom

UNICEF United Nations Children’s Fund

WASH Water, Sanitation and Hygiene

WCARO West and Central Africa Regional Office

WHO World Health Organization

Final Report

Itad in association with UNICEF 1

Executive summary

Introduction

1. The United Nations Children’s Fund (UNICEF) has identified an important role for the organization in ensuring national systems are strong and resilient, in order to provide sustained, quality services to those who need them, particularly children. Strong systems are important for UNICEF strategy1 2 to achieve its stated goals, and in response to global imperatives such as the Sustainable Development Goals (SDGs), including the target on Universal Health Coverage (UHC). In 2016, UNICEF published its Approach to Health Systems Strengthening as an internal, operational framework to help the agency decide how it can most effectively support health systems strengthening (HSS) in different contexts. For UNICEF, the approach to HSS connects community, sub-national and national levels especially acknowledging the importance of sub-national management capacity and community engagement to the overall performance of national health systems

2. UNICEF has commissioned a formative evaluation of its programming in HSS for the purpose of generating evidence and learning on what is needed for its successful implementation. The main objectives of the evaluation are to 1) assess the relevance, efficiency, effectiveness, equity focus and sustainability of UNICEF’s HSS Approach; 2) assess and document HSS programming at the implementation level; and 3) based on evidence gathered, produce clear conclusions and recommendations for policy and management decisions to further transform HSS in UNICEF.

3. The evaluation is theory-based, centred on the use of an overarching Theory of Change (ToC) for UNICEF’s work in HSS, and addresses 22 evaluation questions, grouped under 7 overarching questions (Table 1). Data collection and analysis took place over two years, during 2017 and 2018, through six in-depth3 and eleven light-touch country studies, four thematic case studies, key informant interviews with global and regional stakeholders and an online survey. We used both the ToC and a framework from a peer-reviewed journal article,4 with four explicit criteria,5 to analyse the extent to which an intervention was supporting or strengthening the health system.

Table 1: Evaluation questions

1. Relevance: How relevant, appropriate and coherent are UNICEF strategies, plans and actions for HSS at global, regional and national levels?

2. Effectiveness: How effective are UNICEF Country Programmes (CPs) in achieving tangible results for HSS?

3. Efficiency: Is UNICEF using the available resources for HSS efficiently to achieve outcomes?

4. Equity and gender: To what extent does UNICEF target issues of equity and gender in its HSS programming?

5. Sustainability and scale-up: Is UNICEF effectively supporting sustainability of HSS programmes and the scale-up of evidence-based approaches?

6. Management/operations: To what extent is UNICEF as an organization set up to deliver on its HSS strategy?

1 UNICEF. 2017. Strategic Plan 2018–2021. E/ICEF/2017/17/Rev.1, p.6. 2 UNICEF. 2015. UNICEF’s Strategy for Health 2016–2030. 3 Bolivia, Ethiopia, Kazakhstan, Myanmar, Nepal, Sierra Leone. 4 G. Chee, N. Pielemeier, C. Connor; Why differentiating between health system support and health system strengthening is needed. Int J Health

Planning and Mgmt 2013; 28: 85–94. 5 The four criteria are set out in the paper by Chee et al include: 1) Do interventions have cross-cutting benefits beyond a single disease? What are

these? 2) Do interventions address policy and organizational constraints or strengthen relationships between the (WHO Health System) building blocks? 3) Will interventions produce permanent systemic impact beyond the term of the project? 4) Are interventions tailored to country-specific constraints and opportunities, with clearly defined roles for country institutions?

Final Report


7. Knowledge and data generation and use: Does UNICEF generate and use knowledge and data to support achievement of its HSS goals?

4. This report is intended primarily for UNICEF internal decision-makers, and presents a synthesis of evaluation findings set out in detail in two primary evaluation outputs.6 7 We use two high-level questions designed to promote accessibility for a range of stakeholders with varying engagement with UNICEF’s work on HSS: To what extent is UNICEF implementing relevant and effective HSS interventions? And what strategies and factors have enabled or hindered progress? Through these questions we address all of the key evaluation questions.

To what extent is UNICEF implementing relevant and effective HSS interventions?

5. UNICEF is valued as a development partner that is well aligned with national policy drivers and identified needs. In many contexts, UNICEF’s ways of working are well embedded in national planning process and at sub-national levels.

6. However, UNICEF is not always perceived to be the most relevant partner for HSS, even though it is responsive. UNICEF has a position of influence in the health sector and has presence at all levels of the health system – but does not always leverage this for HSS. There are clear examples from all countries studied and across the thematic case studies of where UNICEF has focused on implementing programmes in thematic areas, to deliver results for women and children, but has not focused on sustainably strengthening the health system.

7. UNICEF has a comparative advantage in clear areas, including in multi-sectoral approaches, use of data and having a mandate to focus on gender and equity issues. UNICEF also contributes to specific areas that are not included in the HSS Approach Paper. Strengthening the articulation of UNICEF’s contribution in these areas would facilitate a stronger shift to HSS.

8. There is some evidence, albeit limited, that UNICEF has effectively used an HSS approach to achieve development results (but not in all countries). However, evidence of effectiveness is more often in terms of inputs/activities/outputs and less often in terms of outcomes achieved. UNICEF could do more to shift towards a systems strengthening approach, and in some cases it has already done this.

9. There is a mixed picture on sustainability of HSS interventions, because the evaluation found that UNICEF was not consistently using HSS approaches. The evaluation identified mixed results on two aspects related to the question on scale – replication of global models and scaling interventions that have been proven effective at country level. In some cases, UNICEF had scaled ahead of gathering more robust results but with strong political support. This can mean scaling is happening before important operational learning becomes available and results in risks to scaling. While the efficiency of UNICEF HSS interventions could be improved with a stronger focus on sustainability, there is insufficient data to track any efficiency gains at project or intervention level.

10. At country level, momentum is building behind the HSS agenda. Key points for integrating the focus on HSS include the development of Country Programme Documents (CPDs). Commitment to HSS has been driven partly by the HSS training course in Melbourne, but also led by HSS-trained health section chiefs. UNICEF staff broadly recognize the need to transition to HSS.

What factors have enabled or hindered progress?

11. UNICEF has a specific role within the HSS ecosystem based on its position, expertise, mandate and vision, but various factors present obstacles to acting on that comparative advantage. UNICEF has at

6 Itad and HPI. 2018. UNICEF Programming in Health Systems Strengthening – A formative evaluation: Preliminary Findings Report. 7 Itad and HPI. 2019. UNICEF Programming in Health Systems Strengthening – A formative evaluation: Thematic Case Study Reports. Expected to be published in early 2019.

Final Report


its disposal a range of tools and approaches that can be applied to strengthen its work on HSS, to engage partners in a clear vision on HSS. However, there are a range of factors at play that make it difficult for UNICEF to act on the basis of this comparative advantage for HSS, including the tension between HSS and implementation/emergency response and lack of operational clarity around HSS. UNICEF staff consistently highlighted the tension between HSS and UNICEF’s mandate to meet the needs of women and children, which can require interventions that are not consistent with an HSS Approach. There is scope to provide further guidance on how to operationalize HSS. In view of these tensions, the shift towards HSS is not an easy one to make and requires careful consideration in terms of pace and scale of ambition.

12. UNICEF as an organization is continuing to adapt to the new focus on HSS and the new HSS Approach, but this change is slow and incremental. Overall, there appears to be an increase in staff awareness, understanding and capacity to deliver the new approach, and some evidence of increased internal coordination and linkages with other sectors. However, capacity gaps remain and cross-sectoral engagement continues to be challenging. There is also some evidence of increased support for HSS from Headquarters (HQ) and Regional Offices (ROs), but this does not appear to be systematic or universal, and Country Offices (COs) reported that they needed additional support in a range of areas. At the same time, there has been limited progress in adapting corporate systems and structures to support a greater focus on HSS.

13. UNICEF’s contribution to the HSS evidence base, and its use of HSS data for decision-making and course correction, continues to be very limited. Monitoring of HSS results and expenditure is inadequate, and evidence gathered does not indicate that COs have received significant guidance or support from HQ/ROs in any of these areas.

Conclusions

14. Overall, the evaluation concludes that there is further scope for UNICEF to clarify its niche in the area of HSS and to translate this into a core part of UNICEF programming. The HSS Approach Paper is a start, and there are pockets of progress, but the HSS Approach Paper is neither sufficient nor has been sufficiently rolled out to catalyse an organization-wide transition towards HSS. There is a risk that staff will lose interest and the opportunity to make the transition will be missed, if progress is not accelerated.

15. UNICEF’s intention to transition towards HSS is complex and difficult to achieve. In many contexts, funding streams, government demands and UNICEF’s lead position in emergency responses can pull UNICEF into offering direct support versus a systems strengthening approach. This tension is a fundamental obstacle to UNICEF’s transition to HSS, and to overcome it requires a shift in mindsets, capacities and structures. Without a clear transition plan or a clear view of how UNICEF can manage these divergent roles, it will continue to be difficult for UNICEF to make this shift.

16. UNICEF is well positioned to capitalize on its comparative advantage in specific areas, including strengthening sub-national level governance in the era of decentralization and improving data for decision-making. However, UNICEF COs can view their comparative advantage in continuing to implement thematic interventions. UNICEF’s contribution to HSS is to some extent limited by the division of labour with UN organisations and others, such as the World Bank, that often take the lead in HSS. UNICEF’s potential offer and identity as an organization focused on HSS stems from its mandate to advocate for equity for children and vulnerable groups. This means it has a place in many agendas related to HSS, including as an implementation partner, developing equitable packages of care, health financing systems to provide coverage for vulnerable groups and piloting innovative models of care.

17. UNICEF has made limited progress in terms of making changes to structures and systems to support working on HSS, with implications in terms of HSS implementation. Focusing on organizational

Final Report


change to support the transition to HSS is a key priority, including on cross-sector working, guidance and support, staff capacity and systems for monitoring and learning.

18. Limited focus on communication and dissemination around UNICEF’s HSS Approach has impacted on UNICEF’s progress in transitioning to HSS, not least because of implications in terms of securing additional resources for HSS. However, lack of data on HSS effectiveness and insufficient clarity on UNICEF’s role is a limiting factor in more proactive communication.

Recommendations

We have identified a limited number of priority recommendations that will help UNICEF accelerate and strengthen its transition towards HSS. These were discussed with UNICEF during a series of meetings, including with the Evaluation Reference Group8 and Programme Division staff, in New York on 16 January 2019; following which feedback was incorporated and is reflected in the following 7 recommendations. We grouped these under four headings: 1) Clarifying vision and strategy; 2) Support staff to work on HSS; 3) Building the evidence base; and 4) Making the case. They are listed in order of priority and build progressively towards enabling UNICEF to achieve two overarching objectives: UNICEF and other partners understand and value UNICEF contribution on HSS; UNICEF current and future programmes maximise HSS potential within the existing mandate. Section 4 provides a fuller description of Figure 1.

Figure 1: Prioritised outcomes within the organizational domain of the ToC

1. Clarifying vision and strategy

Recommendation 1.1: Clarify vision for UNICEF role in HSS and develop cross-organization strategy to deliver on this vision. This will provide clear direction to UNICEF staff and partners on what UNICEF intends when it talks about transitioning to HSS. It should include:

• Revisions to HSS Approach.

8 Evaluation Office, HSS team, Supply Division, MENA Regional Office.

Final Report


• Development of a cross-organization strategy to deliver the HSSA. Clarification on what success looks like and how HSS can be incorporated into thematic issues, starting with the five issue-specific areas of existing UNICEF capacity and perceived priority9.

• Development of operational guidance on HSS.

2. Support staff to work on HSS

Recommendation 2.1: Ensure UNICEF staff have the capacity ‘to do’ HSS. This concerns ensuring that they have the skills and knowledge to incorporate systems-thinking into their day-to-day work. We recommend three main elements to this focus on staff capacity:

• Ensure staff have relevant skills and knowledge through training and on-the-job mentoring.

• Ensure staff are focused on HSS through recruitment and performance management.

• Ensure all staff have appropriate responsibility and accountability for delivering on HSS objectives.

Recommendation 2.2: Support COs to strengthen their focus on HSS. This builds on recommendations 1 and 2, and on evidence that UNICEF staff have not found it easy to work on HSS and have requested proactive support from ROs and HQ to make this transition. Support could be provided in the following areas:

• Review and revise existing programmes.

• Design new HSS strategies and interventions.

• Incorporate HSS into new Country Programme Documents, Strategy Notes, Mid-term Reviews and Country Programme Evaluations.

• Support exchange of practical experience between UNICEF staff.

• Formalising support to COs by ROs and HQ.

3. Building the evidence base

Recommendation 3.1: Develop and implement a clear strategy for monitoring and evaluation of UNICEF work on HSS. In order to strengthen and institutionalize HSS monitoring, the HSS team, working with relevant parts of the organization, needs to ensure data and evidence is available to track performance on HSS, including through:

• Develop strategy to ensure data and analysis needs of organisation are met.

• Strengthen availability of evidence on effectiveness and efficiency of UNICEF HSS interventions.

Recommendation 3.2: Ensure UNICEF is learning from its work on HSS as an input to continuous improvement planning and to global public goods. UNICEF needs to develop and implement strategies to ensure it is reflecting on and learning from what it is doing on HSS, as a means to improve HSS programming. There are two key aspects to this learning agenda:

• Encouraging replication and adaptation within UNICEF.

• Informing policy development and the wider HSS community.

4. Making the case

Recommendation 4.1: Advocate across UNICEF for an organization-wide change management process to increase UNICEF engagement on HSS and ensure systems in place to strengthen effectiveness on HSS. Whilst there may appear to be overlap with recommendations 1-3 above, the emphasis here is on internal advocacy to ensure that recommendations 1-3 are taken forward. This should include cross-organization

9 See Fig. 2 above. These include: Improving data information systems; Procurement and supply chain management; Social protection and

welfare; Engagement and regulation of the private sector; and Quality of care at community and facility levels

Final Report


communication to encourage staff to participate in HSS training (rec. 2.1), to press for changes to systems to ensuring monitoring & evaluation of HSS (rec. 3.1 and 3.2), and to encourage staff to communicate with external partners (rec. 4.2). Additional objectives should include:

• adequate staff resources at all levels to work on HSS

• systems are in place to better articulate goals for HSS interventions, and support cross-sector engagement

Recommendation 4.2: Develop partnerships with external stakeholders to maximize UNICEF’s comparative advantage in HSS. There is a significant imperative for UNICEF staff to work with external partners to:

• Promote understanding of UNICEF contribution and added value as an HSS partner, and of UNICEF mandate.

• Leverage UNICEF comparative advantage to maximise effectiveness of HSS investments.

• Raise resources to ensure UNICEF’s status as an effective implementation partner is adequately funded.

1. Introduction

The United Nations Children’s Fund (UNICEF), through its health programming, seeks to achieve two central goals: end preventable maternal, newborn and child deaths and promote the health and development of all children. It also has a reputation as an effective, responsive partner in emergency contexts, and has also led interventions to support maternal, newborn and child health (MNCH) in some countries for decades. As part of its strategy to achieve these goals, and in response to global imperatives such as the Sustainable Development Goals (SDGs) and Universal Health Coverage (UHC), UNICEF has identified an important role for the organization in ensuring national systems are strong and resilient enough to provide sustainable, equitable and quality services to those who need them, particularly children.10 11 For UNICEF, the approach to HSS connects community, sub-national and national levels especially acknowledging the importance of sub-national management capacity and community engagement to the overall performance of national health systems.

This focus is also influenced by economic progress in the countries in which UNICEF works, which will mean that many countries have or are projected to become ineligible for official development assistance (ODA), with implications for UNICEF future funding flows. The context (challenges, enablers and constraints) within which UNICEF seeks to strengthen health systems, and the health systems themselves, varies substantially from country to country with implications for UNICEF and other partners approach to systems strengthening.

In 2016, UNICEF published its Approach to Health Systems Strengthening (hereafter called the HSS Approach). This focuses on a range of activities that the organization sees as priority areas for future HSS programming, including three broad functional levels of health systems (community, sub-national and national) and five issue-specific areas that the organization prioritizes (see Figure 2). As stated above, this framework sets out an approach that builds on UNICEF’s mandate, comparative advantage, capacities and priorities. It is intended as an internal, operational framework to help UNICEF decide how it can most effectively support HSS in different country contexts and settings. The HSS Approach was developed by the HSS team within the health section of UNICEF’s headquarters in New York. Following its publication in November 2016, UNICEF staff at all levels and across sections (although mostly in the health section) have been working, to varying degrees, to adapt their work in line with guidance set out in the Approach paper.

10 UNICEF. 2017. Strategic Plan 2018–2021. E/ICEF/2017/17/Rev.1, p.6. 11 UNICEF. 2015. UNICEF’s Strategy for Health 2016–2030.

Final Report


Figure 2: HSS Approach diagram12

UNICEF has commissioned a formative evaluation of its programming in HSS to generate evidence and learning on what is needed for the successful implementation of HSS. As defined in the Terms of Reference (ToR), the main objectives of the evaluation are to 1) assess the relevance, efficiency, effectiveness, equity focus and sustainability of UNICEF’s HSS approach at an organizational level during the transitional period [organizational domain of the evaluation]; 2) assess and document HSS programming at the implementation level in general and in specific focal areas, and assess the evaluability of desired results and the likely sustainability of those results [implementation domain of the evaluation]; 3) based on evidence gathered, produce clear conclusions and recommendations for policy and management decisions to further transform HSS in UNICEF and strengthen its contribution to Country Programme (CP) results within the context of UNICEF’s overall commitment to equity.

As a global evaluation, this report is primarily intended for UNICEF internal decision-makers, to address objective (3) above. It presents a synthesis of findings from two years of evaluation, conducted from 2017 to 2018. Findings from two primary source documents are synthesized: a Preliminary Findings Report (2018),13 which set out the findings from Year 1 of the evaluation, and four thematic case studies (TCS),14 submitted to UNICEF in November 2018. It makes recommendations for UNICEF to consider on how it can accelerate progress and strengthen its work on HSS. It will be fed into HSS planning at local level through UNICEF management response, through dissemination activities supported by the HSS team and

12 Ibid. 13 Itad and HPI. 2018. UNICEF Programming in Health Systems Strengthening – A formative evaluation: Preliminary Findings Report. 14 Itad and HPI. 2019. UNICEF Programming in Health Systems Strengthening – A formative evaluation: Thematic Case Study Reports. Expected to be published in early 2019.

Final Report


Evaluation Office. UNICEF country- and regional-offices will then take forward implementation, adapted as required to meet to fit local contexts.

The report is divided into four sections: an Introduction; the evaluation approach and methodology; findings; and implications for UNICEF. Additional detail and supplementary information can be found in a number of annexes set out in Volume 2 of the report.15

2. Evaluation approach and methodology

This section provides a brief overview of the main features of the evaluation approach and methodology. A fuller description is provided in Volume 2, Annex D.

2.1. Evaluation approach and design

The design for this evaluation is theory-based, centred on the use of an overarching Theory of Change (ToC) for UNICEF’s work in HSS, which elaborates the theoretical causal pathways through which UNICEF intends to effect change. The ToC, presented in Figure X, and in more detail in Volume 2, Annex B, shows the linkages between development of the new HSS Approach, the organizational components (referred to as the ‘organizational domain’) that support its institutionalization and how this might translate into implementation activities that strengthen country health systems (referred to as the ‘implementation domain’). The ToC also helped with identification of key stakeholders, including implementing agency(ies), development partners, primary duty bearers, secondary duty bearers, and rights holders; and in the identification of the specific contributions and roles of key stakeholders (financial or otherwise), including UNICEF.

Figure 3: Overarching theory of change for UNICEF’s approach to health systems strengthening

15 This includes a Supplementary Annex containing the six in-depth country case study reports.

Improved decentralised management capacity for evidence based planning, budgeting, supervision and monitoring

Health systems are strengthened and are resilient

UNICEF 'takes action' at every level to deliver on its HSS

approach

UNICEF programs and plans at every level are

designed to address HSS priorities

The evolving global context

UNICEF's vision is achieved

No child dies from a

preventable cause and all

children

reach their full potential in health and

well being

Country context

1. Consultation and partnership

2. Communication and advocacy

3. Capacity-building

4. Supporting countryimplementation

5. Knowledge generation and dissemination

6. Leveragingresources

Health outcomes for women and children are

improved

The health & developmentof all children is promoted

There are no preventable

maternal, newborn & child deaths

Organizational domain of evaluation Implementation domain of evaluation

Regional and country programs

Contextualize

Implement evaluate,

promote, and disseminate

Assess, prioritize,

design and resource

Global plans and programs

National health policies, strategies, planning, financing and approaches to budgeting are developed, withthe incorporation of an equity focus

Qu

ali

ty o

f ca

re i

mp

rov

ed

at

com

mu

nit

y a

nd

fa

cili

ty le

ve

ls

The work of government and partners in health system strengthening

Health systems that

close the equity gap in MNCH and

related

outcomes, contribute to

UHC and broader child development

and are resilient

UNICEF's Health Strategy 2016-2030 Approaches

UNICEF's approach to HSS is a priority within

UNICEF's Health Strategy 2016-2030

Addressing inequities in health outcomes

Promoting integrated, multi-sectoral policies and

programs

Strengthened community platforms for demand generation,accountability, service delivery, social inclusion and reduction offinancial barriers

Incr

ea

sed

en

ga

ge

me

nt

an

d r

egu

lati

on

of

the

pri

va

te

sect

or

Str

en

gth

en

ed

soci

al p

rote

ctio

n a

nd

we

lfa

re s

yst

em

Str

en

gth

en

ed

pro

cure

me

nt

an

d s

up

ply

ch

ain

m

an

age

me

nt

Imp

rov

ed

da

ta a

nd

info

rma

tio

n s

yst

em

s

Qu

alit

y o

f ca

re im

pro

ve

da

t co

mm

un

ity

an

d f

aci

lity

lev

els

UNICEF as an organisation adapts to deliver on its HSS

approach

Shared values

Leadership

Staff and skills

Structure

Systems and processes

Resourcing

HSS activities at every level are

implemented with

UNICEF's support,

underpinned by the

principles of equity,

sustainability and

scaleability

Final Report


The ToC, as well as the ToR, formed the basis for the development of 22 evaluation questions, grouped under 7 overarching questions (Table 2).16 In the second year of the evaluation, we also incorporated questions based on an explicit framework for making judgements about whether an intervention is ‘supportive’ or ‘strengthening’ in nature. 17 These questions are set out in an evaluation framework that has guided the evaluation (Volume 2, Annex C).

Table 2: Seven key evaluation questions

1. Relevance: How relevant, appropriate and coherent are UNICEF strategies, plans and actions for HSS at global, regional and national levels?

2. Effectiveness: How effective are UNICEF CPs in achieving tangible results for HSS?

3. Efficiency: Is UNICEF using the available resources for HSS efficiently to achieve outcomes?

4. Equity and gender: To what extent does UNICEF target issues of equity and gender in its HSS programming?

5. Sustainability and scale-up: Is UNICEF effectively supporting sustainability of HSS programmes and the scale-up of evidence-based approaches?

6. Management/operations: To what extent is UNICEF as an organization set up to deliver on its HSS strategy?

7. Knowledge and data generation and use: Does UNICEF generate and use knowledge and data to support achievement of its HSS goals?

2.2. Scope

The evaluation scope is global but its limits are defined in a number of dimensions:

▪ Focuses on HSS rather than on other aspects of UNICEF’s Strategy for Health;

▪ Examines UNICEF as an organization and will look across units and levels or the organization;

▪ Primarily assesses the period mid-2016 to mid-2018;

▪ Covers UNICEF programming in countries that reflect the diversity of settings in which UNICEF works and the different levels (community, sub-national and national) in which it engages.18

2.3. Data collection and analysis

The evaluation data collection and analysis took place over two years, during 2017 and 2018. In Year 1 (2017), the ‘organizational domain’19 was explored in detail; in Year 2 (2018), the ‘implementation domain’ was the primary focus. Separating the two domains allowed the evaluation to identify issues of interest to UNICEF as the subject of more in-depth exploration in Year 2.

16 Linked to the Organisation for Economic Co-operation and Development (OECD) Development Assistance Committee (DAC) evaluation criteria. Note that we do not look at the criterion ‘impact’. 17 G. Chee, N. Pielemeier, C. Connor; Why differentiating between health system support and health system strengthening is needed. Int J Health

Planning and Mgmt 2013; 28: 85–94.. 18 The specific countries are listed in Section 3 and the process of selection is described in detail in the Inception Report. 19 The above evaluation objectives translate into two key lines of enquiry within the evaluation design: the organizational and implementation domains. These can be seen in the ToC in Volume 2, Annex B.

Final Report


In Year 1, data collection was undertaken through six in-depth and eleven light-touch country studies,20 key informant interviews and an online survey. In-depth country studies21 included country visits during May–September 2017; ‘light-touch’ studies were undertaken during the period July–September 2017. The evaluation conducted semi-structured interviews with a total of 49 people at global and regional levels, in addition to 333 respondents who were interviewed as part of the 17 country studies. An online survey was also organized to create an opportunity for all Country Offices (COs) (i.e. including those that did not participate in the 17 country studies) to feed into the evaluation. Data collection for this year included an extensive document review, at global and country levels. Data analysis in Year 1 was through cross case study analysis, thematic coding of interviews based on a coding structure linked to the evaluation questions and analysis of quantitative and qualitative survey data through use of descriptive statistics.

Table 3: Thematic case studies and countries

Thematic case study Countries

Quality of care (QoC) Kazakhstan, Bolivia, Nepal, Sierra Leone

Emergency contexts Ethiopia, Sierra Leone, Myanmar

Procurement and supply chain management (PSCM) Ethiopia, Sierra Leone, Myanmar

District-level governance (DLG) Bolivia, Nepal, Sierra Leone

In Year 2, data collection was done through three data sources: four TCS, updates to the in-depth 2017 country reports and a limited number of key informant interviews. Work in Year 2 iteratively built on work completed in Year 1. The primary focus was on the implementation domain, which was explored through four TCS,22 identified during 2017 country visits and in consultation with UNICEF. Each case study was designed to explore the main evaluation questions and used a specific ToC, nested in the overarching ToC for the evaluation. Case study data collection included review of secondary data and documents, key informant interviews, in-country focus group discussions and field visits. Work on the case studies took place between May and October 2018, and was conducted by a national consultant in each country, with guidance and support from a core evaluation team member leading on each TCS.

In addition to TCS, the evaluation team completed visits to the six countries that had been visited in 2017; where possible, visits were undertaken by the same team members to ensure continuity and efficiency in data collection. These visits aimed to identify progress made in the organizational domain, based on a limited number of country-specific issues identified in the 2017 country reports.23 A total of 127 key informant interviews were conducted during the country visits.

Twelve key informant interviews were also undertaken with global and regional stakeholders – 9 UNICEF staff and 3 other stakeholders, including development partners – also with the purpose of picking up progress during 2018.

20 Bolivia, Ethiopia, Kazakhstan, Myanmar, Nepal, Sierra Leone. 21 In addition to their inputs into the Preliminary Findings for Year 1 of the evaluation, the country visits had a secondary focus in terms of selection and evaluability assessment of topics for the thematic case studies planned for Year 2 of the evaluation. These are detailed in the TCS Design document, a separate deliverable. 22 QoC, PSCM, DLC, emergency contexts and HSS. These were selected using the clear criteria, including links to the beams and pillars of the HSS Approach; interest from a global HSS perspective; potential for learning by UNICEF; interest in building experience; and innovative programming. 23 See ‘Changes we might expect to see in Year 2 and issues to follow up on’ in Annex D of country reports (2017) : e.g. UNICEF programming in health systems strengthening – a formative evaluation Ethiopia 2017 visit report (oct 2017), Annex D.

Final Report


Analysis of these three data sources was divided across the team. The focus of analysis was on the country- and thematic case study reports, as well as the Preliminary Findings Report, all of which already reflected substantial primary analysis. These reports were structured to allow robust analysis across the country context, with findings detailed against each of the key evaluation questions. Team members took responsibility for reviewing these sources, and incorporating any relevant findings from global and regional key informant interviews conducted in 2018. Analysis was primarily qualitative - focused on two analytical frameworks: the TCS-specific ToC and a peer-reviewed article by Chee et al.24 on HSS. The team discussed findings from this analysis at a two-day workshop during which key messages for this report were identified and supporting evidence discussed.

The evaluation framework included a specific question on equity and gender (see table 2, EQ4): To what extent does UNICEF target issues of equity and gender in its HSS programming? This included sub-questions on the following four points:

• How adequately does UNICEF address equity concerns in its HSS programming? What methods

have proven effective in reducing inequities?

• How adequate are monitoring and accountability mechanisms to inform program managers on

the effects of HSS programming on marginalized groups?

• To what extent does UNICEF advocate among partners for an equity focus in HSS programming?

How effective are these efforts?

• To what extent does UNICEF support gender-sensitive approaches in its HSS programming

We gathered information to explore this question through the data collection and analysis methods described above, and specifically through reviewing relevant global UNICEF strategies25. We have presented all evaluation outputs with explicit findings on equity and gender. However, there was limited scope for the evaluation data to be disaggregated by gender or on using equity criteria, given the source data and measures that were incorporated in our evaluation framework.

2.4. Limitations

We have identified a limited number of limitations with the evaluation design and implementation, which are important to consider in interpreting the evaluation findings. These are summarized below:

▪ Early stage of implementation of HSS Approach. The evaluation was commissioned at a very early stage of the roll-out of the HSS Approach Paper. There has therefore been limited time in which to see evidence of change based on the paper.

▪ Limited evidence for some evaluation questions. As highlighted in Year 1, and confirmed during Year 2, the evaluation found limited evidence to answer some evaluation question, in particular on effectiveness and efficiency. This is a reflection of UNICEF reporting systems for HSS, and is discussed in more detail in Section 3.3.

24 Based on findings from Year 1, we identified this as a relevant and potentially valuable framework that helped fill a gap identified during the first year of our evaluation – namely. the need for an explicit basis for making judgements about whether an intervention is ‘supportive’ or ‘strengthening’ in nature. G. Chee, N. Pielemeier, C. Connor; Why differentiating between health system support and health system strengthening is needed. Int J Health Planning and Mgmt 2013; 28: 85–94.. This papers sets out four criteria which we used to analyse the extent to which an intervention was supporting or strengthening the system: 1) Do interventions have cross-cutting benefits beyond a single disease? What are these? 2) Do interventions address policy and organizational constraints or strengthen relationships between the (WHO Health System) building blocks? 3) Will interventions produce permanent systemic impact beyond the term of the project? 4) Are interventions tailored to country-specific constraints and opportunities, with clearly defined roles for country institutions? 25 UNICEF Strategy for Health 2016–2030. UNICEF. 2015. UNICEF’s Strategy for Health (2016-2030). UNICEF’s Gender Action Plan which

recognizes the importance of gender-responsive strategies and systems to achieve programmatic results as well as meeting commitments in the UN-SWAP. UNICEF. 2017. Gender Action Plan 2018–21. E/ICEF/2017/16

Final Report


▪ Identification of TCS. The TCS were identified through a process including evaluability assessments, application of clear criteria and discussion with UNICEF staff at country level. This means the TCS are illustrative examples of how a specific type of programming is done in the COs selected for visits. They thus do not necessarily reflect the best or most relevant examples of UNICEF’s work in these areas. They were not designed to be comprehensive, but to provide an opportunity for learning and reflection, which they have done.

Ethics

Consideration of ethical standards is extremely important when conducting health-related evaluations. It is essential that those engaged in and informed by the evaluation are treated appropriately and with respect. Privacy, confidentiality, and data protection are critical, and steps were taken to ensure this remained a priority throughout the evaluation process.

All interview participants were informed of the purpose of the evaluation and were read an informed consent statement, which included information about the evaluation and the elements of informed consent. This included highlighting that participation is entirely voluntary, and that anonymity would be maintained. Interviews were audio-recorded, so there is a comprehensive record of all conversations, but only with the consent of the interviewees. No personally identifying data was recorded or noted. Views have not been attributed to individuals at any point in the report.

The repository for the data collected during the evaluation was a private Dropbox that only members of the evaluation team could access. The Dropbox was organized to clearly segregate types of data being collected and the locations from which the data has been retrieved. All individual identifying information gathered for the KIIs and FGDs will be destroyed by the Project Manager on completion of the evaluation.

The evaluation team were independent from UNICEF and were vetted to ensure there were no conflicts of interest with the subject matter. Steps were taken to triangulate all findings between team members in order to ensure credibility and impartiality at all times. In addition, feedback from analysis was received from stakeholders wherever possible.

Evaluation principles

Given the focus of the evaluation and the sample of respondents being taken from policy makers, funders and implementers of HSS interventions (rather than from direct beneficiaries), we judged that the evaluation design did not need to explicitly use a rights-based framework.

Due to the formative nature of the evaluation, and the internal purpose of the HSS Approach document, key stakeholders were identified as primarily UNICEF staff but also their counterparts in government and other development partners. UNICEF staff were involved in the design of the evaluation, in the preparation for data collection – including providing inputs on country and regional level key informants, in the data collection itself, and in discussion of key findings and recommendations. On the latter, the evaluation team debriefed country office staff at the end of each country visit and shared draft reports from these visits. At the end of the evaluation, the overall synthesis report and its conclusions and recommendations were discussed by UNICEF staff during a series of meetings in New York in January 2019. This included the evaluation reference group that had been constituted by UNICEF’s Evaluation Office to provide oversight of the evaluation process and products.

3. Findings

This section presents findings from analysis of the data collected across both years of the evaluation. We use two high-level questions that aim to promote accessibility for a range of stakeholders with varying engagement with UNICEF’s work on HSS: To what extent is UNICEF implementing relevant and effective HSS interventions (Section 3.1)? And what strategies and factors have enabled or hindered progress

Final Report


(Section 3.2)? Through these questions, we address all of the evaluation questions described in Table 2 above. Key findings below are supported by source reports.26 Section 4 discusses implications for UNICEF.

3.1. To what extent is UNICEF implementing relevant and effective HSS interventions?

This section examines issues concerning relevance, gender and equity and effectiveness, then implications for UNICEF’s effectiveness on HSS programming (including sustainability and efficiency). We first present key findings on the question ‘To what extent is UNICEF implementing relevant and effective HSS interventions?’ and also summarize relevant findings at the beginning of each sub-section.

Key findings

UNICEF is valued as a development partner that is well aligned with national policy drivers and identified needs. In many contexts, UNICEF’s ways of working are well embedded in national planning process and at sub-national levels.

However, UNICEF is not always perceived to be the most relevant partner for HSS, even though it is responsive. UNICEF has a position of influence in the health sector, and has presence at all levels – but does not always leverage that for HSS.

UNICEF has a comparative advantage in clear areas, including in multi-level, multi-sectoral approaches, use of data and having a mandate to focus on gender and equity issues. UNICEF also contributes to specific areas not included in the HSS Approach, such as human resources for health and health financing including costing of innovative models. Strengthening the articulation of UNICEF’s contribution in these areas would facilitate a stronger shift to HSS.

UNICEF is recognized for its mandate and work on gender and equity. While gender and equity are priorities, the implications for HSS programming have not been spelt out clearly, and there has been little change since evaluation findings were presented in 2017.

There is some evidence, albeit limited, that UNICEF has effectively used an HSS approach to achieve development results (but not in all countries). However, evidence of effectiveness is more often in terms of inputs/activities/outputs and less often in terms of outcomes. UNICEF could do more to shift towards a systems strengthening approach, and in some cases has already done this.

Given findings on effectiveness, there is a mixed picture on sustainability of HSS interventions. The evaluation identified mixed results on two aspects related to the question on scale – replication of global models and scaling interventions that have been proven effective; on both, UNICEF needs to be more systematic and follow a clear, agreed, step-wise approach.

While the efficiency of UNICEF HSS interventions could be improved with a stronger focus on sustainability, there is insufficient data to track any efficiency gains at project or intervention level.

There is a building momentum behind the HSS agenda, not least through greater focus on HSS in some Country Programme Documents (CPDs). But there is less evidence of issues highlighted in Year 1 of the evaluation being taken forward by COs.

3.1.1. Relevance

Key findings: Relevance

26 Year 1 Preliminary Findings Report and the four Year 2 TCS Reports.

Final Report


UNICEF is very valued as a relevant development partner that is well aligned to national policy drivers and identified health needs. However, the evaluation found that UNICEF was not always relevant to HSS, or making the most of leveraging a strong position for HSS goals.

UNICEF is valued as a development partner that is well aligned to national policy drivers and identified needs. There is strong evidence that UNICEF is viewed as a trusted development partner in most countries, that is well aligned with current government priorities and supporting the government to put policy into practice. In almost all countries reviewed, UNICEF’s health programming was responsive to national policy drivers, and responsive to the needs identified in child and maternal health. This was clearly identified with the TCS, in which there is evidence that UNICEF has been responsive to policy drivers and needs in QoC, emergency, PCSM and DLG. Specific examples are as follows. In Sierra Leone, the CO was directly involved in developing the Every Newborn Action Plan in collaboration with government; in Bolivia, DLG activities27 are aligned with the implementation of the Plan for Accelerated Reduction of Neonatal and Maternal Deaths, and DLG activities were developed in close coordination with the Ministry of Health at central level; in Myanmar, UNICEF’s support to PSCM is specifically aligned with both the National Health Plan 2017–2021 and more detailed national plans.

In many contexts, UNICEF’s ways of working are well embedded in national planning process and at sub-national levels. In several instances, UNICEF has been undertaking joint work-planning with relevant government ministries, as part of pooled mechanisms, with regular engagement in coordination mechanisms. For instance, in Kazakhstan, UNICEF conducts joint work-planning with the Ministries of Health, Social Protection and Labour, to deliver a programme of work to strengthen social protection. In many instances, UNICEF is well positioned from ‘top to tail’, with a seat at the table for national policy-making and a strong presence at decentralized level. In the context of decentralization, this makes UNICEF well placed to strengthen the devolved governance structures, while supporting feedback to the central level to inform policy-making. There is mixed evidence on how well UNICEF is doing this in practice, as can be seen in the District Level Governance Thematic Case Study, where there was some evidence of effectiveness in building decentralised capacity in Bolivia, Nepal and Sierra Leone but also some concerns about the sustainability of these gains.

However, UNICEF is not always perceived to be the most relevant partner for HSS, even though it is responsive to identified country needs. While UNICEF is well aligned to policy drivers and national priorities, bilateral and multilateral development partners in multiple countries did not perceive UNICEF as having a track record, expertise and capacity to support HSS28 and therefore as the most relevant partner for HSS. In many contexts, UNICEF is perceived as being more linked to sub-national implementation and emergency response. Governments often demand a mix of support and strengthening approaches, particularly in emergency settings (which have affected four of the six countries in which the evaluation worked in depth) where basic health services are not functional, and where UNICEF’s capacity to step into direct support is valued. But health sections in UNICEF COs in all of the countries studied were staffed mostly with technical experts on thematic issues (such as MNCH, HIV/AIDS, immunization) and infrequently with cross-cutting expertise on issues such as health financing (HF), although this is improving.

UNICEF is viewed as being responsive to government, but this is not always viewed positively. While being responsive is seen as a useful approach, including to promote country ownership, some development partners (e.g. in Nepal) reported that UNICEF had responded to government requests after other partners had intentionally decided not to provide support, with the implication that this could undermine efforts to harmonize behind an agreed agenda. There is a risk that being responsive can actually divert UNICEF from HSS efforts, as was seen in Kazakhstan, where political pressure to rapidly

27 Such as improving the quality of care for mothers and children at health facilities, training of health personnel, strengthening data availability and analysis and improving equity. 28 This was in particular on health financing, but in general partners did not see UNICEF as explicitly working on HSS or as playing a lead role on

HSS.

Final Report


scale the Patronage Nursing System (PNS) undermined a more consolidated HSS approach. One of the most significant challenges UNICEF faces is that the capacity and structures required to be responsive to emergency situations is very different to those required to work effectively on HSS.

And the evaluation found some evidence that, while UNICEF responds well to country needs, overall UNICEF could do more to shift towards a systems strengthening approach. In Year 2 of the evaluation, the evaluation used the Chee criteria to evaluate the mix of support versus strengthening in UNICEF programming. It found that, in many contexts, UNICEF’s comparative advantage was in retaining a focus on ‘support’. For instance, in QoC programming for rehabilitation of Primary Health Centres (PHCs), some up-front investment in basic equipment, training and commodities is needed ahead of increasing demand for services. In some cases, there has been a lack of transition planning, which has hindered a shift from support to strengthening. With this said, there were a limited number of examples of UNICEF planning for transition from the countries that we studied – such as the Community-Based IMCI roll-out in Nepal, and planning for scale in the PNS model in Kazakhstan; the UNICEF Kazakhstan office also has developed a ‘sine qua non’ checklist to ensure piloted models generate robust data to inform scale-up and tranisition. UNICEF does undertake joint annual work-plans with government ministries as a means to operationalize CPD implementation, but these efforts are limited in scope in terms of driving towards sustainability. In a few contexts, there was evidence that UNICEF COs had re-evaluated their approaches with ‘health systems thinking’, resulting in a more consolidated HSS approach. Examples include Ethiopia, Myanmar and Nepal, where the development of the CP development plan had resulted in a critical rethink of approach in favour of more systems-based approaches. The evaluation found that, whilst there was not always clear strategy for HSS in place at country level, UNICEF COs have still benefited from developing their own views of where they can contribute

UNICEF has a position of influence in the health sector, and has presence at all levels – but does not always leverage this for HSS. There is very strong evidence, across all sources, that one of UNICEF’s key comparative advantages is its ability to work at all levels – from community through to national level. In many contexts, development partners wanted to see more learning from work at sub-national level feeding into central-level policy-making. In a few cases, there were strong demands for UNICEF support for technical assistance, particularly for use of data for performance management. This would facilitate central-level performance management, particularly in cases of rapid devolution, where central levels are shifting roles to more remote settings and needing to provide oversight. The federalization process in Nepal is a good example of this. Health governance was devolved to municipality level in July 2017. Central-level government was over-burdened with facilitating this change, and UNICEF’s strength at sub-national levels and in feeding back policy and capacity-building needs to central level was valued. As discussed in Section 3.2, UNICEF can do more to prioritize documenting and learning from its implementation activities, in order to benefit HSS efforts.

UNICEF has a comparative advantage in multi-sectoral work. This makes it well placed to respond to demands for integrated service delivery models. There is strong evidence across UNICEF COs of multi-sectoral work being implemented, in particular covering collaboration between health and nutrition, water, sanitation and hygiene (WASH), education and social protection. For example, on social protection, there are clear opportunities for joint work between social policy and health teams: in addition to work in Bolivia, Ethiopia and Nepal,29 there is evidence of further progress in Myanmar (on the Maternal and Child Cash Transfer programme) and Kazakhstan (on the PNS and cash+ model). It is important to note that this working across sectors is not straightforward, either internally within UNICEF (see section 3.2.1.1 for more discussion on this) or externally where it is possible that political commitment and levels of funding for cross-sector working may be important factors, but which were not part of this evaluation.

There is often good evidence of the use of types of analyses, such as political economy analyses, to design approaches to strengthening health systems. A clear view of health system deficits is vital to ensuring relevance of approach. In some countries, there had been good use of health systems deficit 29 As highlighted in Itad’s Preliminary Findings Report for this evaluation (2017).

Final Report


analysis to design a relevant approach to UNICEF HSS programming. There were good examples of UNICEF COs using health systems analysis to drive programming, in the absence of a clear vision for HSS at country level. For example, in Nepal, political economy analysis was used to refine the approach to HSS in a context of complex and rapid change. Health systems analyses are very different from situation analyses, which provide more insight into identified (health and other) needs by socio-demographic group, rather than how systems are performing. And the evaluation saw in several countries how political economy analysis is important, given how the political context has radically shifted as a result of health reform, elections and new governments. In these scenarios, UNICEF needs to be adept at recognizing how political developments can result in shifts in demand for new approaches. The crucial insight here is that UNICEF’s focus on data does not always provide a clear view of health systems deficits or a joint vision of how to address these.

UNICEF is a contributor to specific areas not included in the HSS Approach. There is scope to strengthen the articulation of UNICEF’s contribution in these areas. Many respondents emphasized the importance of government capacity for HSS in terms of finance and human resources for health (HRH). These are areas that UNICEF intentionally did not include in the HSS Approach, given that other organizations are mandated and have capability to take these forward. However, UNICEF staff and external partners expressed clear demand for further clarity on UNICEF’s position on these subjects, not least because UNICEF does engage on HF (e.g. through work on social protection, and vaccine financing) and on HRH (e.g. through substantial capacity-building or support to community health workers (CHWs) and immunization supply chain personnel). Based on experience from the TCS – in particular on QoC and DLG º UNICEF has more to contribute in being part of planning on how UHC will be delivered (with an equity focus). For example, this can include costing promising approaches ahead of scaling, which is part of UNICEF’s sine qua non30 model in Kazakhstan, and strengthening planning and resourcing at sub-national levels, which was the focus of the District Investment Case (DIC) in Nepal. These capacities should be viewed as integral to developing viable innovative models of care.

3.1.2. Gender and equity

Key findings: Gender and equity

UNICEF is recognized for its mandate and work on gender and equity. While gender and equity are priorities, the implications for HSS programming have not been spelt out clearly, and there has been little change since evaluation findings were presented in 2017.

Gender

UNICEF is committed to promoting gender equality across the organization’s work. However, as highlighted in the 2018 Preliminary Findings Report, the HSS Approach Paper has almost no discussion or mention of gender beyond the need to disaggregate data, and this is seen as a gap at country level. Where COs did see the need to integrate gender into HSS programming, they were keen to know about models, examples and success stories. Some teams recognized the need to strengthen integration of gender issues in HSS but had struggled to find models and successful examples of gender-transformative approaches. Where gender is being taken up in relation to systems strengthening, this is often cited as part of a multi-sectoral approach.

30 UNICEF in Kazakhstan has adopted a sine qua non model as a checklist or guide to ensuring pilots are robust and as influential as possible. This includes 11 steps: 1) ToC, 2) equity-based hypothesis to describe the pathways from model to above ToC, 3) expected equity-based overall results formulated as child rights realization, 4) baseline, including equity-increasing impact indicators, 5) sustainability/exit strategy and termination date agreed with partners, 6) monitoring mechanisms, including for process indicators, adequately funded, 7) impact equity-based evaluation clearly scheduled, budgeted for, partner-led; 8) cost–benefit analysis/budget impact analysis and estimated resource for scaling up; 9) dates and budget to document the practice; 10) strategies and budget to disseminate results; 11) total budget for the model.

Final Report


UNICEF is credited with consistently advocating and promoting the collection and analysis of disaggregated data. At both the country and the global level, UNICEF is credited with the promotion and facilitation of robust data to identify and understand gender differences. In the Results Assessment Module (RAM) system, UNICEF has created a gender marker, and using the Gender Equality Marker (GEM)31 tool it has the potential to make an assessment of the extent to which planned activities are potentially transformative. Use of this tool was not referred to by many of those interviewed at country level, although in Ethiopia it had been used as part of a gender review of new strategy notes.

Equity

An equity focus is evident within the UNICEF Strategy for Health 2016–2030 and the HSS Approach Paper, in line with UNICEF’s broader mandate, but there are questions about operationalizing the equity approach in the context of HSS. Equity is part of UNICEF’s Mission Statement, and is one of the three key approaches within the UNICEF Strategy for Health 2016–2030.32 Equity is picked up in the HSS Approach Paper, in which UNICEF’s vision for HSS is articulated as: ‘A health system that closes the gaps in access to quality services and in health and nutrition outcomes, contributes to UHC and the SDGs, and is resilient.’33 However, partners at global level raised questions about operationalizing the equity approach in the context of HSS, in particular in terms of ensuring a shared understanding about what an equity approach means in practice within HSS programming; operationalizing the theory; and being a stronger voice. Questions were raised first about whether there was a common understanding of what an equity approach meant for programming in HSS – that is, is it a general underpinning principle, or a specific focus on starting with the hardest to reach? Internally, some staff questioned why equity was a separate ‘stream’ within the health strategy, and what that meant for programming. Others noted that, while equity is emphasized in the language and tools, there is more work to do to translate this into consistent practice: ‘There is a big bridge to cross from the conceptual to the practical.’ And several partners at a global level felt UNICEF could be pushing more on the equity agenda, using its position with government and strength in data generation.

As explained in the Strategy for Health and the HSS Approach Paper, the use of data to identify the most marginalized groups, then to identify the barriers and bottlenecks to access and finally to monitor progress in addressing the barriers forms a key part of UNICEF’s equity strategy. This was recognized by partners at global level – many of whom noted UNICEF’s role, particularly in the areas of research and data generation (e.g. surveys such as the MICS) and using evidence to bolster the case for an equity-based approach, for example through the recent publication of the Narrowing the Gaps report.34 It is also clear, across the countries studied, that UNICEF is supporting the generation and use of data on equity to inform priorities with government for its programming, ensure data is timely and relevant and, to a certain extent, in advocacy, to inform other areas of programming. For example, in Sierra Leone, UNICEF supported revision of Health Management Information System forms to ensure they were equity-focused and provided disaggregated data by gender and age.

3.1.3. Effectiveness

Key findings: Effectiveness

There is some evidence, albeit limited, that UNICEF has effectively used an HSS approach to achieve development results (but not in all countries). However, evidence of effectiveness is more often in terms of inputs/activities/outputs and less often in terms of outcomes.

31 UNICEF. 2010. Gender Equality Marker Tracking of Resource Allocations and Expenditure for Gender Equality Results. 32 UNICEF. 2015. UNICEF’s Strategy for Health 2016–2030. 33 UNICEF. 2016. The UNICEF Approach to Health Systems Strengthening. 34 UNICEF. 2017. Narrowing the Gaps: The Power of Investing in the Poorest Children.

Final Report


In 2018, the evaluation found that the measurement challenges highlighted in 2017 had continued, and there are important considerations around clearly defining effectiveness. It is important to note that many countries are in the early stages of implementation of current CPDs or in a transition phase to new CPDs; and there is little evidence so far that UNICEF is systematically tracking HSS results. However, our work in 2018, including through the TCS, explored effectiveness in more detail, and this has raised important considerations around what is meant by effectiveness. In most cases, the interventions that made up the TCS targeted results related to health services outputs and outcomes35 (i.e. outputs in terms of availability of vaccines or of maternal health services, and outcomes in terms of improved child or maternal health indicators). But the emphasis of HSS interventions, as highlighted in the HSS Approach definition, is on the sustainable functioning of the health system itself36 – and several UNICEF respondents highlighted that MNCH outcomes could be achieved through vertical interventions and quick wins. The lack of focus on achieving HSS outcomes may reflect when interventions were designed (whether before the HSS Approach or not) and/or lack of clarity around what HSS results or success looks like.

There is some evidence, albeit limited, that UNICEF has effectively used an HSS approach to achieve development results (but not in all countries). For example, on QoC in Bolivia, Continuous Cycles of Quality Improvement (CCQIs) have led to greater compliance with evidence-based standards for quality of care in maternal and newborn health and data on monthly review meetings captured through routine monitoring shows that facilities are largely compliant. This approach has paired CCQIs with over-sight at higher levels to address systems gaps (in the health network, ‘SEDES’), as well as an embedded training facility to ensure staff capacity gaps are addressed through in-service training. In Sierra Leone, there is strong evidence that UNICEF has contributed substantially to the establishment of functioning neonatal intensive care units (NICUs) in four hospitals. Establishing the NICUs has relied on on-the-job mentoring provided by expatriate paediatricians, but with a future plan to transition training to medical training centres. On DLG, in Nepal respondents reported that the DIC had considerably enhanced the capacity of the District Health Management Team, which resulted in improved district-level planning, monitoring and data use; and of UNICEF staff. On PSCM, in Ethiopia UNICEF’s support has increasingly been effective at strengthening government PSCM systems – for example working with the government to improve port of entry processes; develop and expand the cold chain; and successfully transition vaccines, and later child health and malaria commodities, into government warehouse and distribution systems. In Myanmar, as a result of UNICEF’s and other development partners’ more recent HSS-oriented support, the government’s capacity to forecast Expanded Programme of Immunization (EPI) commodity requirements, to manage the cold chain and to finance vaccines has increased dramatically, and vaccine stock-outs have been minimal – with only one stock out, owing to an international supply shortage. Finally, in emergency contexts, while UNICEF cannot move away entirely from direct service delivery and coordination, in Ethiopia it has encouraged government to take on structures and services as these become more established.

However, evidence of effectiveness was more often identified in terms of achieving inputs/activities/outputs and less often in terms of outcomes. There is strong evidence to support this across the TCS. We found that UNICEF had used available inputs to achieve undertake relevant activities and achieve relevant outputs, but that outcomes had not been targeted in terms of HSS outcomes and so we found limited evidence of effectiveness in terms of the higher-level results expressed in the TCS ToCs (outcomes in the interventions studied were more often expressed in terms of health outcomes or lower level results – e.g. in the DLG TCS in terms of changing resource allocations (Nepal), increasing availability of data (Sierra Leone), or establishing an indigenous health network (Bolivia). An illustrative example is from the DIC in Nepal, where opportunities to routinize and institutionalize the relevant processes were not identified. Focusing on institutionalizing DIC processes would have helped ensure the DIC model was

35 In this report we refer to outputs, using the OECD/DAC definition, as ‘The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.’ The OECD/DAC definition of outcomes is ‘The likely or achieved short-term and medium-term effects of an intervention’s outputs.’ UNICEF uses these terms differently in its strategy and planning documentation. 36 ‘Actions that establish sustained improvements in the provision, utilization, quality and efficiency of services delivered through the health system’.

Final Report


systematically taken forward by specific individuals, with the appropriate resources, and was the focus of outputs and outcomes in the DLG ToC. As seen with the DIC, the risk of not taking action at this level is to jeopardize the mid-to-long-term sustainability of interventions and the outcomes that they seek to support. This focus on inputs/activities/outputs was also seen in Sierra Leone and Bolivia (on DLG), and in the Quality of Care thematic case study. In QoC, UNICEF’s contribution was often more strongly focused on one causal pathway, which was strengthening health workers’ skills and capacities to provide, but less on strengthening health systems oversight and management, which arguably would have had more long-term impacts. The key point here is that, in the interventions that we looked at in the TCS, results were more often expressed at the level of activities/outputs (the left hand of the TCS TOCs) and less so at in terms of outcomes (the right hand of the TOC).

3.1.4. Sustainability

Key findings: Sustainability

Given findings on effectiveness, there is a mixed picture on sustainability of HSS interventions. The evaluation identified mixed results on two aspects related to the question on scale – replication of global models and scaling interventions that have been proven effective; on both, UNICEF needs to be more systematic and follow a clear, agreed, step-wise approach.

Given the above findings on effectiveness, there is a mixed picture on sustainability of HSS interventions. On the one hand, there is strong evidence of concerns around sustainability across the TCS. For example, on QoC in some settings, there were consistent concerns about the sustainability of UNICEF approaches, particularly for maintaining clinical centres of excellence; in Sierra Leone, for example, the NICUs relied on expatriate paediatricians to mentor and train heath workers in newborn care. While UNICEF reported that they had a transition plan in place, key stakeholders were concerned that these vital skills would be lost, and that there were still issues with the government’s capacity to cover basic remuneration of health care workers and thus ensure continuity of staff. There were also more basic concerns about the capacity to maintain basic equipment. For example, in terms of DLG in Nepal, some respondents reported limitations relating inter alia to lack of continuous follow-up and to poor leadership in the districts, which seems to have undermined on-going effectiveness of the DIC in some districts. On the other hand, the emergency TCS highlighted that, in two of the interventions studied (Ethiopia and Myanmar), there was a good likelihood of permanent systemic impact beyond the term of the project. In each case, UNICEF is working with government in order to institutionalize the factors that need to be sustained for their impact to continue.

UNICEF’s ability to focus on sustainability is influenced by context, underlining the importance of analysis of health systems data and the political economy within which the health system is developing. In recent years, there has been a clear shift towards more systems strengthening support in Ethiopia and Myanmar, and this has been accompanied by a corresponding decrease in direct implementation and gap-filling support37. However, in Sierra Leone, while there have been attempts at more systems strengthening-oriented work, the overall context (e.g. limited government capacity and a weak health governance infrastructure, as well as limited funding from the government health budget) has made systems strengthening work difficult, and direct implementation and gap-filling still appear to be the primary forms of support provided by UNICEF. It is clear that systems strengthening may be harder to achieve in contexts where systems are nascent, fragile or under threat owing to emergencies; from our

37 It is important to note that in Ethiopia an important driver of this shift in focus has been the Government’s ambition to reach middle-income

(MIC) status by 2025, which would have implications for UNICEF programmes, including funding available to support UNICEF interventions. Perhaps more important though has been the de facto reduction in ECO health budget, from $85 million to $35 million, which drove a reappraisal of their portfolio and led to a decision to prioritise HSS as a strategic intervention to maintain their impact with fewer resources. It also interesting to note that there are differences of opinion with UNICEF about whether using income status or maturity of systems is the best basis on which to categorise countries in terms of HSS support.

Final Report


sample of countries, responding to emergencies has influenced UNICEF’s work in Ethiopia, Myanmar, Nepal and Sierra Leone. Conversely, it is more feasible to expect to focus on strengthening health systems where government capacity is in place and government systems are both mature and stable (e.g. in Bolivia and Kazakhstan and to some extent in Ethiopia and Myanmar,38 as highlighted above), although this does not necessarily mean it is easier to achieve HSS results in these contexts. On this basis, it is appropriate that the HSS Approach Paper does not prescribe how to respond to any context, or provide normative guidance (but there is a demand from COs for clearer, contextualised guidance – as discussed in section 3.2.1.2). The key challenge is to ensure COs are able to ensure UNICEF’s position on HSS is contextually driven, including by using political economy analysis and analysis of health systems deficits – and UNICEF can use available levers to help address these deficits (see section 3.2.3 and Figure 4 for more on these levers). It is feasible that this is linked to the maturity of the systems and the stability of government partners in Ethiopia and Myanmar.

The evaluation identified mixed results on two aspects related to the question on scale – replication of global models39 and scaling interventions that have been proven effective. On replication of global models, there is strong evidence to show that, when sourcing technical solutions that have worked in other contexts, UNICEF is in a good position to bring in learning from within the UNICEF network. However, there are examples of where UNICEF has sought to do this for models that have limited evidence of effectiveness (e.g. Vaxtrax and Comprehensive Centres of Excellence in Nepal). While UNICEF may be replicating ‘models’ that have potential, the introduction of these models needs to be more systematic, and to be initiated with sustainability in mind; this was acknowledged by the Nepal CO and has been behind the development of the sine qua non model in Kazakhstan. On taking interventions to scale, there is limited evidence, from the country and case study sample, of where UNICEF has done this in a systematic way (such as with community-based Integrated Management of Childhood Illness (IMCI) in Nepal); in other cases, UNICEF does not appear to have comprehensively thought about what is needed to scale (e.g. with the PNS in Kazakhstan).

Some UNICEF staff do see the value in being HSS-oriented because they recognize that it can produce the long-term systemic change needed; however, achieving HSS results requires a long-term timeframe. It is important to contextualize these findings, given the limited time that has passed since the HSS Approach was published. There is strong evidence from interviews with UNICEF staff that COs do see a strong value in HSS as a means to produce long-term systemic change which their current approaches do not always deliver. In all contexts, the nature of work required to achieve systemic change requires long-term interventions, not least because activities are often outside UNICEF’s direct control. But global key informants highlighted that it was not straightforward for UNICEF to provide long-term support in the context of biennial funding cycles and where donor funding is often geared towards short-term impact and attributable results. The challenge is to develop a long-term vision and goals, with political commitment to achieve these, and then to make clear how short-term goals contribute to this broader vision. UNICEF can then use this to locate its comparative advantage and contribution.

3.1.5. Efficiency

Key findings: Efficiency

While the efficiency of UNICEF HSS interventions could be improved with a stronger focus on sustainability, there is insufficient data to track any efficiency gains at project or intervention level.

38 While emergency responses have been a feature of the country context that UNICEF has had to respond to in Ethiopia and Myanmar, the underlying political economy context in these countries is relatively mature and stable, which has enabled UNICEF to shift towards HSS. 39 Our working definition of a global model is an intervention that has been designed by HQ or Regional Offices, or for which funding has been

secured at global or regional levels (e.g. through proposal to donor organization), and which may require adaptation to respond to country-specific contexts.

Final Report


While the efficiency of UNICEF HSS interventions could be improved with a stronger focus on sustainability, there is insufficient data to track any efficiency gains at project or intervention level. HSS as an approach is likely to increase the efficiency of UNICEF’s approach. An effective approach to HSS will ensure UNICEF’s investments go further and have more impact. At programme/intervention level, UNICEF could increase efficiency by focusing on the sustainability of some common programme approaches such as training and capacity-building (see Box 1). However, there is no basis to understand whether these potential benefits have been attained because data is generally lacking for any assessment of efficiency at project or intervention level. UNICEF is not well set up to collect information on the efficiency of its work: while it may have expenditure data tracking, it rarely collects comprehensive information on HSS outputs or outcomes. Across the TCS, the evaluation found two interventions that had clear costing data – the PNS and Ethiopia’s Mobile Health Units. The Mobile Health Units also have output/ outcome data that have allowed for the formulation of a unit cost. Apart from these examples, it was hard to judge in any detail the efficiency in UNICEF’s approach at programme/intervention level. This is a disadvantage when it comes to making the case for greater investment in the new HSS Approach.

The more integrated UNICEF’s work becomes within the health system the harder it will be to judge its efficiency. This is because the inputs of UNICEF and government become harder to separate out and measure, and also the impact they have on outcomes/ outputs is harder to follow. This is the case with inputs that UNICEF is making into joint technical processes. A good example is in Ethiopia, where UNICEF has been advising the Ethiopian Health Insurance Agency on criteria for targeting the poor (when subscriptions will be paid by the government). Such work is vital to embedding considerations of equity into a system that is strengthening the heath system, but to assess the ‘efficiency’ of such work would be difficult.

Resources for HSS within UNICEF are constrained, and they are not currently sufficient to achieve a major shift within the activities of the organization. The evaluation found that the funding environment for new initiatives within UNICEF COs was constrained; to programme major new HSS initiatives, UNICEF will need to raise money from other donors. For instance, the Myanmar CO has had success raising funds for HSS projects from the Global Fund. Global key informants highlighted that, in the absence of new funding, it was hard for the HSS approach to be fully implemented, as there was little space within existing country budgets to reallocate resources towards these activities. The ability to report results on HSS efficiency and sustainability would assist UNICEF to raise additional funds for HSS either internally or externally – although, as highlighted above, there is scope to improve HSS reporting generally, including on efficiency and sustainability.

In the context of low resources for the HSS Approach, UNICEF offices are finding cost-effective ways of building in the approach without too many extra resources to bring to implementation. For instance, HSS thinking is being incorporated into areas that it has historically been working in, and it is trying to adapt support to be more HSS-focused. In both Ethiopia and Myanmar, there are examples of this in UNICEF’s work on PSCM. In both countries, UNICEF historically ran parallel procurement and supply chain systems, but is now incrementally handing things over to government, and UNICEF is also supporting both countries to move toward self-reliance in vaccine financing. In addition, in its work on the Electronic

Box 1: Capacity-building UNICEF employs multiple strategies to build capacities of counterpart organizations (including train and equip, training, embedded staff). There is evidence of where this has been effective, including on PSCM in Ethiopia, where UNICEF has provided capacity-building support to effectively strengthen government systems. There are examples that raise concerns about whether skills are transferred using these approaches, and/or whether these capacity-building interventions can be sustained without support from UNICEF. Again on PSCM in Ethiopia, embedded staff have not been shown to strengthen the system; in Nepal, there have been challenges in terms of planning for high turnover of government staff. UNICEF need to strategize about how to ensure capacity-building will contribute to HSS, e.g. through measuring skills transfer.

Final Report


Logistic Management Information System (eLMIS) in Myanmar, UNICEF is working with an EPI focus, but is specifically considering how the eLMIS will link to broader data collection platforms, such as the District Health Information System (DHIS), so that there is one data collection platform for health staff.

3.1.6. Progress over 12 months

There is building momentum behind the UNICEF HSS agenda, owing to a range of internal and external factors. There is clear evidence across the countries studied that UNICEF COs recognize the value of an HSS approach that can deliver long-term systemic change, and this is reflected in some CPDs (e.g. Ethiopia and Nepal; UNICEF staff also highlighted how HSS would be incorporated into the new CPD for Sierra Leone that is currently under development). Global key informants in 2018 highlighted a range of initiatives that UNICEF was implementing to take forward HSS, including leveraging immunization as a key entry point to take forward HSS; convening a global meeting on HSS and PSCM strengthening aiming to catalytically build capacity to strengthen supply chains as part of HSS and programmatic work; making training on HSS more accessible, such as through a blended version of the Melbourne training; and inclusion of three HSS indicators in the UNICEF Strategic Plan (a mid-term review in 2019 will consider whether these indicators are sufficient). These shifts have been influenced by movements towards HSS within the broader development environment, including for example on UHC and the SDGs.

But there is less evidence of issues highlighted in Year 1 of the evaluation being taken forward by COs. The evaluation reviewed the extent to which COs had taken forward issues highlighted in Year 1 country visit reports and in the Preliminary Findings Report. It is important to note that these recommendations were not necessarily due a formal response, and we would not have expected to see wholesale changes in approach to HSS in UNICEF COs; but our focus in Year 2 was on specific points highlighted in the Year 1 country visits, and a number of these would have been actionable even within a relatively tight time-frame. Given this, we found limited evidence of change on the issues highlighted. As noted above, there is more evidence of progress during the past 12 months at the global level, and the key is for UNICEF to ensure that global initiatives on HSS percolate to COs – which is challenging in view of the busy, responsive, thematically focused prevailing conditions that the evaluation observed in all countries visited. Many of the concerns raised during 2017 were in the organizational domain, as this was the focus of the evaluation in Year 1, and this is discussed in more detail below (Section 3.2).

3.2. What strategies and factors have enabled or hindered progress?

This section first examines the organizational systems and structures that underpin UNICEF’s HSS response, as a precursor to discussing UNICEF’s comparative advantage and what the levers and barriers are to UNICEF strengthening its focus on and effectiveness in supporting HSS.

Final Report


Key findings

UNICEF as an organization is continuing to adapt to the new focus on HSS and the new HSS Approach, but this change is slow and incremental. Overall, there appears to be an increase in staff awareness, understanding and capacity to deliver of the new approach, and some evidence of increased internal coordination and linkages with other sectors. However, capacity gaps remain and cross-sectoral engagement continues to be challenging. There is also some evidence of increased support for HSS from Headquarters (HQ) and Regional Offices (ROs), but this does not appear to be systematic or universal, and COs reported that they needed additional support in a range of areas.

UNICEF’s contribution to the HSS Evidence Base, and its use of HSS data for decision-making and course correction, continues to be very limited. Monitoring of HSS results and expenditure has not improved significantly over the past year, and evidence gathered does not indicate that COs have received significant guidance or support from HQ/ROs in any of these areas.

3.2.1. Management and operations

This section examines issues relating to management and operations and how they relate to one of the central hypotheses in the HSS ToC – namely, ‘For the HSS approach to be implemented successfully, UNICEF will have to change the way in which it works and implement a change management process within the organization.’

3.2.1.1. Internal alignment and coordination and integration with other sectors

UNICEF sees internal coordination and integration with other sectors as critical to HSS. Promoting integrated, multi-sectoral policies and programmes is prioritized in UNICEF’s new Strategy for Health 2016–2030,40 and the new HSS Approach states that, ‘The cross-sectoral nature of HSS requires mechanisms for inter-sectoral collaboration within and outside the organization.’41

From data collected in Year 2, there is some evidence of increased internal coordination and linkages with other sectors, but this continues to be challenging. The majority of COs (e.g. Bolivia, Ethiopia, Kazakhstan, Nepal and Sierra Leone) reported that their existing or new CPs had strong cross-sectoral

40 UNICEF. 2015. UNICEF’s Strategy for Health 2016–2030. 41 UNICEF. 2016. The UNICEF Health System Strengthening Approach.

Key findings: management and operations

UNICEF sees internal coordination and integration with other sectors as critical to HSS, and there is some evidence of increased internal coordination and linkages with other sectors, but this continues to be challenging - the mechanisms for inter-sectoral engagement and coordination remain largely the same as in 2017.

There was some evidence of guidance and support for HSS being provided by HQ and ROs during 2018 but this does not appear to be systematized or universal. At present, the majority of leadership and support for HSS appears to be coming from health section chiefs, and there is a perceived need for additional support from HQ and ROs in a range of areas.

UNICEF health staff reported increased awareness and understanding of HSS and the new HSS Approach, COs were also increasing HSS capacity through recruitment of new staff with HSS skills, but capacity gaps remain. However, despite increases in staff capacity, UNICEF continues to be known for its technical skills and its capacity to implement, rather than its HSS competencies.

Final Report


linkages and that cross-sectoral work was increasing, and there have been some efforts at global level (e.g. through a cross-section meeting on Goal 142held in August 2018 which UNICEF staff reported had included a focus on systems strengthening work. But cross-sectoral engagement continues to be complicated and challenging. UNICEF staff reported that it was difficult to identify work that could be done jointly across the office and to see how this was linked to joint or integrated programmes, and that this level of complexity was not reflected in the HSS Approach. The smaller COs (e.g. Bolivia and Kazakhstan) reportedly found this easier than the larger COs (e.g. Ethiopia, Nepal and Sierra Leone), and Nepal reported that this was easier at the sub-national level than at the national level.

The mechanisms for inter-sectoral engagement and coordination remain largely the same as in 2017. Inter-sectoral engagement continues to be promoted through harmonized work planning (e.g. Sierra Leone) and cross-sectoral programme design (e.g. Ethiopia, Kazakhstan, Myanmar), and routine linkages between sections continues to be accomplished through the monthly Programme Management Team Meetings chaired by the deputy representative and attended by section chiefs. Work across sections is facilitated by small working groups or task forces (e.g. Nutrition and Early Childhood Development Task Forces in Nepal; Thematic Working Groups in Sierra Leone; Trans-Sectoral Task Forces on Early Childhood Development and Maternal Newborn Health in Bolivia) and ad hoc meetings between sections. Specific illustrative examples of inter-sectoral interventions were found in Myanmar on the Maternal and Child Cash Transfers (MCCT) programme, in Ethiopia on Poverty Safety Net Programme (PSNP), and in Kazakhstan on piloting a cash+ model.

Overall, there is some evidence of positive change in 2018, but COs reported that inter-sectoral work continued to be challenging. We have not seen any clear evidence of UNICEF having taken specific follow-up action over the past year and there is a need for further action in this area. As found in the Preliminary Findings Report, UNICEF could further integrate HSS throughout the planning cycle. This included reviewing the strategies being used to incentivize inter-sectoral working; determining if they were delivering promising results; and disseminating lessons learned. In order to make further progress in this area, it would be useful for UNICEF to revisit and take action on this recommendation, and analyse and minimize on-going challenges to cross-sectoral engagement.

3.2.1.2. Guidance and support for HSS programming

There was some evidence of guidance and support for HSS being provided by HQ and ROs during 2018 but this does not appear to be systematized or universal. While a comprehensive assessment of HQ and RO support was not undertaken during 2018, some information was collected during key informant interviews with selected HQ, RO and CO staff. UNICEF HQ was supporting HSS capacity-building through roll-out of the HSS course developed with the University of Melbourne,43 hosting webinars on thematic issues’ intersection with HSS (e.g. on immunization as an entry point to HSS), incorporating HSS into global and regional network meetings and supporting country-level Equist training. UNICEF’s National Supply Chain Strengthening Centre was also reported to be supporting the PSCM pillar of the HSS Approach. This centre has developed a national supply chain strengthening strategy and a process guide and toolkit for strengthening public health supply chains, and since early 2018 has been working with 14 COs to agree common language and metrics for supply chain strengthening, and to pilot the use of a supply chain maturity score card. Key informants reported that some ROs had been providing support, but this appeared to vary according to RO size and capacity. The smaller regional offices were reported to be providing limited support (e.g. Latin America and Caribbean Regional Office (LACRO), Europe and Central Asia Regional Office (ECARO)), whereas large ROs with designated HSS teams (e.g. Eastern and Southern Africa Regional Office (ESARO)) were reported to be providing higher levels of support.

42 Goal Area 1in the Strategic Plan (2018-21) – Every child survives and thrives – is composite nature, incorporating health, nutrition, early childhood development and HIV/AIDS. This “reflects the importance of integrated approaches to child survival and development”. https://www.unicef.org/about/execboard/files/2017-17-Rev1-Strategic_Plan_2018-2021-ODS-EN.pdf 43 The HSS course is about to enter its third year and has been taken by more than 200 people from over 60 country offices.

https://www.unicef.org/about/execboard/files/2017-17-Rev1-Strategic_Plan_2018-2021-ODS-EN.pdf

Final Report


The COs reviewed provided some specific examples of HQ or RO support received during 2018. Examples included 1) HQ support to Equist training of implementing partner staff in Myanmar; 2) HQ and RO support to incorporating HSS into maternal and newborn health and immunization during Regional Office for South Asia (ROSA) network meeting; 3) ESARO support to the UNICEF Ethiopia Country Office to redesign its programme following a substantial reduction in available funds (see footnote 32) including multiple visits by the ESARO team to facilitate and support ECO thinking on HSS; 4) a regional HSS presentation during a LACRO meeting with COs; and 5) Sierra Leone requesting West and Central Africa Regional Office (WCARO) staff with HSS experience to be involved in an upcoming strategic moment of reflection workshop.

At present, the majority of leadership and support for HSS appears to be coming from health section chiefs, and there is a perceived need for additional support from HQ and ROs in a range of areas. The majority of COs reviewed in 2018 (e.g. Bolivia, Ethiopia, Myanmar, Nepal, Sierra Leone) reported good HSS leadership from senior managers, particularly health section chiefs; UNICEF supply division also noted that Supply Officers also provide this kind of support in the PSCM space. While this type of support, and the above support from HQ and ROs, is highly valued by COs, CO staff requested additional engagement and support from HQ or ROs in the following areas: 1) distilling HSS and developing an appropriate HSS engagement/support strategy; 2) incorporating HSS into staff recruitment (e.g. developing and sharing model HSS profiles, interview questions and testing tools); 3) increasing contributions to the HSS evidence base; 4) increasing HSS learning inside UNICEF; 5) improving HSS monitoring and evaluation (e.g. supporting COs to develop appropriate HSS outcomes, outputs and indicators and adapting UNICEF’s overall coding and RAM indicators for HSS); and 6) adapting the HSS course (e.g. making it cheaper and bringing it closer to the COs).

Overall, there is some, albeit limited, evidence of positive change in 2018. However, there is no clear evidence of UNICEF taking action to provide systematized, comprehensive guidance and support and there is a need for further action in this area. As highlighted above, relevant action has been taken by some COs, ROs and HQ to provide guidance and support. However, as found in the Preliminary Findings Report, UNICEF could further integrate HSS throughout the planning cycle, including through increasing its capacity to manage change processes required to institutionalize the new HSS Approach. This could include resourcing the changes and providing concrete guidance, support and tools to promote and facilitate the shift towards a strong HSS focus. In order to make further progress in this area, it would be useful for UNICEF to revisit and take action on this recommendation, and to define clear roles and responsibilities for HQ and the ROs in order to deliver on the specific areas where COs are requesting support.

3.2.1.3. Staff capacity

UNICEF health staff reported increased awareness and understanding of HSS and the new HSS Approach. In the majority of COs reviewed in 2018, health staff had received HSS training or participated in HSS-related learning opportunities. For example, the Ethiopia, Myanmar and Nepal COs reported that staff had participated in the new HSS course developed by the University of Melbourne. The Bolivia CO reported that staff had participated in an HSS learning opportunity. Only one CO, Kazakhstan, noted that staff had received no additional HSS training, reportedly because of a lack of financial resources within the CO. A high level of staff understanding and awareness of HSS was also reported as an enabler by four COs (Ethiopia, Nepal, Myanmar and Sierra Leone) and in the DLG TCS.

COs were also increasing HSS capacity through recruitment of new staff with HSS skills, but capacity gaps remain. The majority of COs reviewed in 2018 reported incorporating HSS skills and experience into new staff recruitment; Myanmar and Nepal reported recruiting new international and national staff with HSS skills; and Ethiopia and Sierra Leone reported recruiting new international staff with HSS skills. However, no COs reported receiving any guidance or model HSS profiles or tools from HQ or ROs to support these processes, and half of the COs reviewed reported that they still had gaps in HSS staff

Final Report


capacity (e.g. Bolivia, Kazakhstan, Nepal (particularly among field staff) and Sierra Leone (particularly amongst lower-level staff)). There is also evidence of variable staffing at RO level: for example, there are 2.5 dedicated HSS staff in ESARO compared with no dedicated HSS staff in some ROs and 3 people on the HSS team in HQ. Two of the COs (e.g. Myanmar and Nepal) specifically reported that they could further increase HSS staff capacity if a cheaper version of the HSS course was made available closer to the CO.

However, despite increases in staff capacity, UNICEF continues to be known for its technical skills and its capacity to implement, rather than its HSS competencies. In 2018, one global key informant noted that, ‘UNICEF hasn’t traditionally been an HSS partner. UNICEF is more of an implementer.’ Another noted that, ‘Partners see UNICEF as implementers and as doing emergency response; this is changing slightly, but it is still an issue.’ This is very similar to what was found in 2017 and elaborated in the Preliminary Findings Report, and there does not appear to have been a change in this perception over the past year. This has been influenced (or not) by limited action by UNICEF to communicate its shift towards being an HSS partner consistent with its new approach. For instance, the evaluation found little evidence that the HSS Approach had been used as an advocacy or communications tools, and as part of engaging wider development partners in a conversation about UNICEF’s approach to HSS.

Overall, there is some evidence of increased HSS staff capacity over the past year, but gaps remain and COs continue to need support from HQ and ROs for increasing and institutionalizing HSS staff capacity. As found in the Preliminary Findings Report, UNICEF should continue to develop staff HSS capacity through mechanisms that expand access to training and include more targeted recruitment. It also noted that UNICEF should build on the new HSS course on the basis of course evaluation findings and a comprehensive assessment of CO needs; develop guidance and model HSS profiles; share recruitment experiences; and meet CO demand for guidance on building capacity through recruitment. In order to make further progress in this area, it would be useful for UNICEF to revisit and take action on this recommendation, and to support COs to 1) incorporate HSS competencies into staff profiles when it develops its new staff organogram at the start of a new CP; 2) include HSS competencies in annual staff performance assessments; 3) incorporate HSS into staff recruitment (e.g. through developing model HSS profiles, interview questions, testing tools); and 4) adapt the HSS course based on evaluation findings, feedback and CO needs and, potentially, develop cheaper version(s) of the course that is closer to the COs (regional/online) and potentially linked to an HSS community of practice.

3.2.2. Knowledge and data generation and use

This section examines issues relating to knowledge and data generation and use and how they relate to one of the central hypotheses in the HSS ToC – namely, ‘For the HSS approach to be implemented successfully, UNICEF will have to change the way in which it works and implement a change management process within the organization.’

As an organization, UNICEF prioritizes knowledge management and learning. In its new Strategic Plan 2018–2021, UNICEF recognizes the importance of continuing to invest in knowledge-sharing and information management throughout the organization. The HSS Approach Paper also gives knowledge management a central place, and identifies the need for development and implementation of an agency-wide joint learning agenda, including research, evaluation, knowledge management and advocacy for HSS.

Key findings: knowledge and data generation and use

UNICEF’s contribution to the HSS Evidence Base, and its use of HSS data for decision-making and course correction, continues to be very limited. Monitoring of HSS results and expenditure has not improved significantly over the past year, and there is no evidence of COs receiving significant guidance or support from HQ/ROs in any of these areas.

Final Report


3.2.2.1. Monitoring and evaluation of HSS

Monitoring of HSS results and expenditure is still limited and seen to be difficult. Three COs reviewed in 2018 noted an increased focus on HSS and HSS monitoring in their new CPs (Bolivia, Myanmar, Nepal) and Sierra Leone said HSS was being prioritized in its upcoming CP. However, COs noted that HSS monitoring continued to be difficult. For example, the Kazakhstan CO noted that it had difficulty in selecting appropriate HSS indicators and had received limited support to do so, and the Nepal CO noted that picking indicators from the Global RAM list limited its options. The evaluation also found that it was not possible to accurately measure HSS results and expenditure at country level when undertaking the four TCS.

While the Global RAM indicator list and Programme Information Database System (PIDB) codes were recently updated to be aligned with the new Strategic Plan and the new Global Health Strategy, this does not appear to have significantly improved HSS monitoring, and COs are continuing to request support for this work. Three COs reported that it was easier to monitor HSS when they had a specific output labelled as HSS (Myanmar, Nepal and Sierra Leone). However, these COs also noted that output-level monitoring of HSS was problematic as HSS outputs continue to contain support as well as strengthening activities, and non-HSS outputs continue to contain HSS-related initiatives and activities. This situation has led COs to request improvements to overall HSS monitoring, and further HQ and RO support for developing appropriate HSS outcomes, outputs and indicators.

UNICEF global and regional key informants confirmed that HSS monitoring continued to be challenging, but said that some progress was being made by particular divisions and groups. A regional key informant noted that, while there were some indicators related to HSS in the RAM, these were poor and did not always make sense. A global key informant noted that, while UNICEF was managing large amounts of Gavi Alliance HSS funding, no one was consistently monitoring or documenting HSS results, and this was a missed opportunity and something that should be addressed. However, another global key informant noted that some progress was being made in relation to PSCM, and, as noted above, that more intensive work was being done with 14 countries to agree a common language and metrics for supply chain strengthening, and to pilot the use of a supply chain maturity score card with set indicators.

Overall, there is limited evidence of UNICEF improving its capacity to monitor HSS results and expenditure over the past year, and there is a need for further action in this area. As found in the Preliminary Findings Report, UNICEF should strengthen its resource tracking and results monitoring for HSS and improve its system to track resources and monitor results in HSS, including short- and medium-term progress towards longer-term goals. It further noted that this should include development of a robust HSS indicator list that is incorporated into UNICEF’s standard RAM output and outcome indicator lists. In order to make further progress in this area, it would be useful for UNICEF to revisit and take action on this recommendation building on on-going work, such as what is being done by the National Supply Chain Strengthening Centre and experience from COs and ROs, and to address outstanding needs such as Gavi HSS monitoring. Attention should also be given to supporting COs to improve their HSS monitoring, including developing appropriate HSS outcomes, outputs and indicators.

3.2.2.2. Contribution to the HSS evidence base

There is some evidence of UNICEF contributing to the HSS evidence base but this continues to be limited. Four of the COs reviewed in 2018 (e.g. Bolivia, Kazakhstan, Myanmar and Nepal) reported that they had made, or planned to make, additional contributions to the HSS evidence base. For example, the Kazakhstan CO reported that it had prepared HSS-related articles and conference presentations and had incorporated HSS considerations into recent evaluations; the Myanmar CO intends to incorporate HSS considerations into three of its upcoming evaluations; and the Nepal CO is planning to incorporate HSS-related issues into the randomized control trial that it will use to evaluate progress in its 18 convergence municipalities (or palikas). However, COs also noted that resources for this type of work were limited, and the Nepal CO specifically noted that learning processes needed to be better resourced.

Final Report


In half of countries reviewed, development partners felt that UNICEF should do more in this area. This was true in Kazakhstan, Myanmar and Nepal, and is similar to what was found in 2017 and elaborated in the Preliminary Findings Report. Development partners also continued to note that UNICEF’s presence at different levels of the health system meant it was well positioned to document and disseminate lessons learned from every level in order to inform policy development.

Overall, there is limited evidence of UNICEF increasing its contributions to the HSS evidence base; there is no evidence of HQ or ROs taking any follow-up action over the past year, and there is a need for further action in this area. As an organization, UNICEF needs to further demonstrate that it has something meaningful to contribute to the HSS evidence base, and it needs to incentive staff to generate and disseminate HSS knowledge products. There are a number of ways that this can potentially be done, including adding these functions to job descriptions and annual performance assessment frameworks.

3.2.2.3. Use of data for decision-making

There is some evidence of UNICEF learning from its work on HSS but this continues to be limited. Four of the countries reviewed in 2018 could provide evidence of historical or planned learning on HSS. Bolivia reported that it had documented and learned from its QoC Continuous Improvement Cycles; Kazakhstan reported evaluating and learning from its piloting of the PNS (although the extent to which learning has been incorporated in the programme is less clear at this stage); Myanmar intends to use the results of its upcoming HSS-related evaluations to improve programming; and Nepal intends to undertake a randomized control trial to inform programming in its 18 convergent palikas. Global and regional key informants also noted that the Supply Division was supporting learning from national supply chain-strengthening efforts through the use of a standardized maturity score card in 14 countries.

There is organizational need and demand for additional work in this area. Three of the four TCS (DLG, QoC and PSCM) highlighted the need for and importance of further documenting, sharing and learning from experience to date. Multiple COs (e.g. Myanmar and Nepal) also expressed a need to better capture and learn from their own HSS work, and the Bolivia CO stated that it wanted the RO to share experiences and lessons learned in order to guide and help improve its work on HSS.

Overall, there is limited evidence of UNICEF learning from its work on HSS; limited evidence of HQ and RO support for these efforts over the past year; and a need for further action in this area. As found in the Preliminary Findings Report, UNICEF should strengthen its capacity to build and use evidence for HSS. It noted that the organization should move quickly to develop a joint learning agenda and consider developing (with other partners) a resourced learning hub to systematically share knowledge; develop and share tools to deal with constraints; and share lessons learned and success stories in relation to HSS programming. In order to make further progress in this area, it would be useful for UNICEF to revisit and take action on this recommendation either through developing a joint HSS learning hub or through developing an internal HSS community of practice. The latter is potentially easier and faster and would go a long way towards ensuring UNICEF is learning from what it is doing in the area of HSS, and systematically sharing knowledge, tools, lessons learned, constraints and success stories in relation to HSS programming. It would also be useful for HQ/ROs to review and discuss evaluation plans with COs at the outset of the CP to ensure they include evaluations of HSS-related interventions/outputs.

3.2.3. Strategies and factors that have enabled or hindered progress

Key findings

UNICEF has a specific role within the HSS ecosystem based on its position, expertise, mandate and vision. UNICEF has at its disposal a range of tools and approaches that can be applied to strengthen its work on HSS, to engage partners in a clear vision on HSS.

However, there are a range of factors at play that make it difficult for UNICEF to act on the basis of this comparative advantage for HSS.

Final Report


In view of these tensions, the shift towards HSS is not an easy one to make and requires careful consideration in terms of pace and scale of ambition.

We have identified above a range of factors that enable and constrain UNICEF’s ability to engage on, and its effectiveness in supporting, HSS. This section brings these together to discuss UNICEF’s comparative advantage – aspects of its positioning, capacities and purpose that make it well positioned to play a specific role on HSS – along with the levers UNICEF COs can use to transition towards this comparative advantage. We also discuss barriers or obstacles to their ability to use these levers.

UNICEF has a specific role within the HSS ecosystem based on its position, expertise, mandate and vision. As discussed above, summarized in Figure 3, UNICEF is well placed to engage on HSS because it is valued and trusted by government partners and has a seat in national-level policy-making forums. It complements this with the capability to work from ‘nose to tail’ and to reflect experience of working at all levels of the health system, and indeed in other sectors as well. As discussed in Section 3.2, UNICEF has a clear mandate – albeit underutilized at present – to generate and disseminate evidence and learning products that can support governments and other partners to ensure HSS policy and planning responds to evidence-based needs and changes in circumstances. UNICEF’s purpose and intention in terms of HSS has been set out in the HSS Approach, which is a positive move in terms of clarifying UNICEF’s role. We have identified a number of levers that UNICEF COs have used effectively to enable them to transition towards a stronger HSS focus (Figure 4). These are discussed in more detail below.

Figure 4: What will facilitate a shift to HSS? What are the levers to change? What are the barriers?

Figure 3 UNICEF’s comparative advantage in HSS

Final Report


3.2.3.1. Levers to support change

UNICEF has at its disposal a range of tools and approaches that can be applied to strengthen its work on HSS, to engage partners in a clear vision on HSS. There is strong evidence on convergence between the 2017 and 2018 findings on UNICEF’s use of a limited number of strategies and interventions for HSS. The most common strategies identified across TCS and country reports were capacity-building (see Section 3.1), use of data and evidence (see Section 3.2) – including contextual analysis of HSS deficits – and having capacity to work both at different geographic levels and in different sectors (see Section 3.2). However, these are approaches that UNICEF uses generally, and as such are not necessarily capacities specific to HSS work. Because they are not tailored to HSS interventions, we cannot say they are necessarily effective for HSS. However, there is scope for each of these core strategies to offer opportunities to contribute to HSS, but UNICEF needs to further reflect and provide guidance on how to do this in practice.

And UNICEF can better articulate its value added in its communication with partners, and in its strategy and staffing. The implications of the comparative advantage, described above, are that UNICEF needs to work in partnership with other stakeholders to effectively support health systems strengthening. To enhance the potential of partnership working, UNICEF can better describe its comparative advantage and communicate this to partners. As discussed above (Section 3.1), the HSS Approach does not address HF and HRH, which creates some confusion about UNICEF’s role in HSS (not least because UNICEF does intervene in these areas on specific issues). Making revisions to the HSS Approach, and indeed better use of transition planning (see Box 2), can provide a mechanism for communicating with partners about UNICEF’s role, and could be used to start to change the perception of UNICEF’s role as well as to contribute to generating additional funding for HSS from external sources. There is evidence (Section 3.1) that CPDs and other strategy documents – as periodic moments when UNICEF resets its strategic direction and resourcing – represent important levers in transitioning towards HSS. ROs have supported COs to incorporate HSS in recent CPDs (e.g. Ethiopia, Sierra Leone, Nepal). It is important that these opportunities be identified and seized, which may be more challenging in those ROs that have less health and HSS capacity. And UNICEF may want to review how this is being done in emerging CPDs and provide further guidance, as there is evidence of different approaches being used – that is, whether to separate out or mainstream HSS.

Box 2: Importance of transition planning

A review of published literature, on what works for sustainability in aid programmes, has highlighted the importance of an explicit sustainability or exit plan. Much of the evidence points towards processes that are important in planning for sustainability, including establishing joint goals and clarity on what will be sustained (e.g. activities, outputs, outcomes); ensuring effective communication; planning to ensure interventions are incorporated into the routines and institutions of the organizations that will continue with implementation; phasing out control; and monitoring progress in implementing a sustainability plan.

There is evidence that UNICEF works to achieve some of these things – e.g. joint annual work-plans with government ministries, which are standard practice as a means to operationalize CPD implementation – but these efforts are limited in scope in terms of driving towards sustainability. In some cases, lack of transition planning has resulted in UNICEF’s technical expertise being under-valued as part of transition. For instance, in Sierra Leone, donors funded another agency to provide technical assistance to strengthen procurement and supply, where previously UNICEF had been the leading procurement agency. Donors did not view UNICEF as the appropriate partner to transition from UNICEF support to government-led over-sight of supply chains. Government partners echoed this to some extent, as they wanted to see more explicit joint planning for transition plans, with clear timelines and more focus on embedded technical assistance models to strengthen capacity for policy-making and management of the health system. They perceived UNICEF as having too much control over the national supply chain system.

Final Report


3.2.3.2. Barriers to change

There are a range of factors at play that make it difficult for UNICEF to act on the basis of its comparative advantage for HSS, including the tension between HSS and implementation/emergency response and lack of operational clarity around HSS. As discussed in Section 3.2, UNICEF continues to be known for its technical skills and its capacity to implement, rather than its HSS competencies. This leads governments and donors to demand and commission UNICEF to work on more supportive interventions, including in responding to emergencies (see Section 3.1 for more on this). UNICEF’s staffing decisions contribute to the perception of UNICEF as a technical and implementation rather than an HSS partner. While governments value UNICEF technical expertise on thematic issues such as MNCH and immunization, UNICEF does not routinely have cross-cutting HSS expertise (although this is improving). And the implementation of UNICEF interventions, as described in Section 3.1, does not routinely emphasize HSS outcomes – examples of this include lack of emphasis on skills transfer in capacity-building and under-developed use of learning/data for decision-making. These factors contribute to a lack of clarity among COs on how to operationalize HSS; as highlighted above (Section 3.2), the HSS Approach requires some modification if it is to provide further clarity for both internal and external partners (not least in terms of HRH and HF).

We also highlight a number of UNICEF tools and strategies that require further guidance and thinking in order to optimize HSS effectiveness. Two further barriers to change are identified: the focus of interventions and accountabilities for HSS. Each is discussed below:

▪ Focus of programmes. Our work across four TCS has highlighted that UNICEF interventions are not routinely set up to achieve HSS results, even though they may be effective for achieving health outcomes. Even within interventions that fall under DLG, where it is reasonable to expect that projects will seek to build sustainable capacity at decentralized level, the focus is more often on outputs or short-term outcomes than on longer-term HSS goals. This is consistent with the discussion in Section 3.1 above, which highlights that UNICEF is more focused on service delivery, in line with its mandate to meet the needs of women and children. It also appears to be a function of the level of detail required in UNICEF documentation, in terms of setting out programme logic of interventions. There is some attempt to address this within UNICEF’s Kazakhstan Country Office (KCO), which is using a sine qua non model to support internal quality assurance of pilot projects and specifically to develop a ToC for each output within the CP. Developing this level of detail for each intervention would go some way to addressing concerns reported in some countries about lack of analysis on upstream issues, which is at the heart of the required shift in mindset towards HSS thinking.

▪ Unclear articulation of accountabilities for HSS. As explored above, the results framework does not adequately capture HSS results, and this in turn means staff are not supported to shift towards more HSS-focused programming. There is no clear basis for targeting and holding staff accountable for their focus on HSS, and staff may thus have a tendency to revert to more supportive types of programming within UNICEF’s current type. Given UNICEF’s decentralized model, without this kind of holding to account for HSS performance by management in COs and ROs, there is a risk that HSS programming will continue to be a secondary concern after core business, or will be viewed as a supplementary ‘add-on’ rather than integrated into core programming.

In view of these tensions, the shift towards HSS is not an easy one to make and requires careful consideration in terms of pace and scale of ambition. As highlighted above (Section 3.2), there is evidence of an incremental shift towards HSS in those COs studied. This is being supported by the training course in Melbourne, HSS trained team leaders and a wider contextual push for HSS by development partners. But it will take substantial time to effect organization-wide change if this is the main strategy to support the transition to HSS. As highlighted above (Section 3.2), UNICEF needs to think about the optimal staffing structure in COs and ROs to support transition to HSS in the context of a clearer articulation of where UNICEF sits within the HSS ecosystem. Evidence from middle-income Bolivia and Kazakhstan suggests resources and structures in these smaller COs have been geared towards HSS to a lesser extent

Final Report


than in larger COs in low-income countries, and UNICEF would benefit from giving further consideration to how to work in these contexts where it does not appear feasible to have dedicated HSS staff.

4. Lessons and implications for UNICEF

This section reflects on the findings presented above, to consider the implications for UNICEF and its transition to a focus on HSS. It picks up on key themes and challenges that are most strategic and relevant in terms of managing the organizational and programmatic change required to achieve a more effective transition. This is intended to maximize the utility of the evaluation report for UNICEF stakeholders.

UNICEF’s potential contribution to HSS is clear, but the scale and pace of ambition of organizational change for HSS appears to be constrained by tensions between HSS and UNICEF’s mandate and perceived capabilities. As described above (Section 3.2), UNICEF has a specific position, capacities and vision that make it well positioned to play a key role on HSS. However, it is not currently clear what ambition UNICEF has in terms of scale and pace of change in order to maximize its added value on HSS. A significant tension exists between investing in long-term, unattributable HSS results and UNICEF’s mandate to deliver results for women and children; put simply, UNICEF will always intervene to save the lives of women and children if required to do so. This was highlighted by key informants at all levels. One global key informant noted that, in purely pragmatic terms, UNICEF’s transition to HSS would be determined by its core mandated focus on MNCH, equity and gender, as the risk44 of making wholesale organizational changes to strengthen UNICEF’s HSS focus was too high. The challenge is therefore to identify opportunities to strengthen health systems within UNICEF’s existing mandate. This represents a different level of ambition and challenge to, for example, adapting to the extent that UNICEF becomes first and foremost an HSS provider, which would require wholesale change. It would be useful to clarify the level of ambition and the implications for organizational change. It is worth reflecting that the organizational structures and systems required to carry out emergency responses and HSS are markedly different.

The HSS Approach does not go far enough to articulate how HSS sits in practice within the existing comparative advantage. There is currently a gap between the thinking set out in the HSS Approach and experience in UNICEF country offices, and it is not always clear to COs how to operationalize HSS. It is critical for UNICEF to reflect on how having prominence of position at the development table can be translated into a more effective HSS Approach. For instance, leveraging position to influence policy-making is relatively easier to achieve for UNICEF compared with other development partners. However, in many instances, more could be done to make a more concerted shift to an HSS Approach. There was some evidence that this shift was underway in some countries, with a critical rethink on approaches. To provide greater clarity on how to work in thematic areas, more work should be prioritized between the HSS team and thematic sections; and it is important to ensure this work is contextualized in order to be useful to COs. A good example of what can be achieved is evident in the work between the HSS team and the supply chain team in Copenhagen.

UNICEF also needs to update the HSS Approach to recognize where it is making a contribution to HSS, including in HRH and HF. UNICEF currently often focuses on developing health workers’ skills and capacities for instance, but could use this on-going work to increase its focus on HSS. Similarly, for HF, while UNICEF is not the overall lead for this (the World Health Organization (WHO) and the World Bank take a prominent role), it has a role in related agendas, and articulating these would strengthen the shift to HSS. We do not conclude that UNICEF should seek to lead substantively on either HRH or HF, but that it should be explicit about the not insignificant work it already does in these areas and how it can leverage this work for HSS. Adapting the HSS Approach in this way, possibly using the WHO Building Blocks, will

44 The risk being that wholesale organizational changes to enable stronger working on HSS could undermine UNICEF’s capacity to deliver on its core mandate for women and children.

Final Report


help clearly situate UNICEF within the HSS ecosystem and address confusion among partners about how UNICEF plans to contribute on HSS.

Lack of clarity in terms of how to operationalize HSS can undermine all other aspects of the ToC (Volume 2, Annex B), starting with weak monitoring and evaluation for HSS. Lack of clarity on operationalizing HSS both drives and is driven by inadequate systems for measuring HSS: if it is not clear what is to be achieved, there is no clear basis for defining appropriate measures; and if there is no measurement, there is no firm basis on which UNICEF can develop a narrative to support its efforts to transition to a new role and how it can be an effective HSS partner. As a matter of urgency, UNICEF needs to develop an appropriate set of measures to track its HSS activities, expenditure and effectiveness. This should include improving routine documentation and sharing of HSS learning and achievements through developing a clear strategy and robust indicator list for HSS monitoring; documenting key HSS lessons learned and actively contributing to the HSS evidence base; and developing a HSS community of practice at the appropriate level(s). Together with a revised HSS Approach as an advocacy document and operational guide, this can form a platform for a change management process within UNICEF.

It is also likely that lack of measurement contributes to limited progress in terms of establishing the right organizational systems and structures, which has implications for implementation of HSS programmes. Currently, both lack of operational clarity and poor systems for measuring UNICEF’s work on HSS contribute to lack of communication internally and externally on how UNICEF plans to contribute to the HSS agenda.

▪ For internal stakeholders, while some UNICEF managers see the value of investing in HSS and have bought into HSS when it comes to allocating financial resources to fund HSS interventions, or human resources to work on HSS, this is not always the case. Global interviews highlighted that proposals to increasing funding or staffing for HSS, or to make changes to organizational systems and structures, were competing for finite resources with other priorities, and required commitment to HSS at the highest levels from senior managers. A clearer argument needs to be made on how, when and why HSS is a good investment to achieve goals relating to UNICEF’s core mandate (recognizing that HSS is not appropriate or feasible in some contexts). Without this, there is a risk that some UNICEF staff will continue to think results can be more easily achieved through vertical interventions – the sustainability of which will be threatened when donor funding flows change as countries graduate to middle-income status. As highlighted above, this would clearly be assisted through more effective monitoring and evaluation of HSS interventions.

▪ For external stakeholders, it is clear that donors continue to fund UNICEF to carry out implementation or to provide technical assistance to support effectiveness of their programmes, instead of ‘to do’ HSS. It is likely that there are a number of factors at play in this – not least that UNICEF has capacity, expertise and a mandate to carry out implementation, and that there is lack of clarity on UNICEF’s vision for its role on HSS. But a lack of proactive dialogue with external stakeholders is certainly a barrier to effecting change in this regard, which is in turn affected by lack of measurement. One UNICEF respondent reported that, ‘I don’t believe in engaging in discussions when not showing results. Where UNICEF is weak, sometimes, is when UNICEF advocates for things without showing results.’

Lack of resourcing for HSS within UNICEF is a barrier to making required changes and leads to a slow-moving roll-out of the HSS approach. This affects how HSS can be consistently rolled out, as ROs have a key role to play in supporting COs to contextualize the HSS Approach and to develop appropriate HSS strategies. Lack of HSS staff is contributing to lack of progress, even within the current, imperfect system. The HSS course in Melbourne is going some way, but the pace of this change is slow and there are opportunity costs to moving slowly (in terms of communications and organizational change, described above) – and the benefits of the training course take time to percolate out across the organization and it is left to individuals to take forward the HSS agenda through a piecemeal roll-out. We stop short of making recommendations on HSS staffing at CO level – the required outcome is increased HSS capacity in COs – but we do emphasize the importance of having adequate, dedicated HSS capacity at RO and HQ level.

Final Report


Taking steps in these areas will enable UNICEF to take strategic decisions on how to respond to requests from partners, and to progressively build demonstrable expertise and credibility as an HSS partner, within its MNCH mandate. Historically, UNICEF has been responsive to governments, most likely in response to its mandate but also as a means of demonstrating added value. There is, though, a major challenge with continuing to be responsive to requests that are based on perceptions of historical expertise and capacity, as this draws UNICEF away from moving in a new direction (albeit one where there is less demonstrated capacity). Having a clear vision and strategy for UNICEF’s contribution to HSS can provide a rationale or filter when responding (positively and negatively) to requests for support. This is essential if UNICEF is to position itself to continue to deliver on its mandate as countries transition to middle-income status.

The transition to a stronger focus on HSS is not an easy prospect, but UNICEF does have levers that it can more routinely use to accelerate momentum. The transition will require clear thinking and effective leadership to manage a process of change. There are opportunities for UNICEF to accelerate the momentum that has been started since the HSS Approach was published, through using the levers that we have identified above. These include:

▪ Incorporating measures of skills transfer into capacity-building interventions;

▪ Strengthening operational learning – documenting and sharing practice on HSS within the organization, and, in doing so, repositioning UNICEF as an HSS partner;

▪ Using political economic contextual analysis to develop a clear view of HSS needs and deficits in all countries, and at sub-national level where relevant and appropriate;

▪ Articulating a clearer position on UNICEF role in HF and HRH in terms of its work on HSS;

▪ Building transition plans into new interventions, and into existing interventions where opportunities exist;

▪ Ensuring that future CPDs, mid-term reviews and planned evaluations explicitly address HSS.

Priority conclusions

We summarize below the conclusions that the evaluation believes are of highest priority for UNICEF to consider in order to strengthen its organizational set-up for and performance on HSS.

Overall, we conclude that there is further scope for UNICEF to clarify its niche in the area of HSS and to translate this into being a core part of UNICEF programming. The HSS Approach Paper is a start, and there are pockets of progress, but the HSS Approach Paper is neither sufficient nor has been sufficiently rolled out to catalyse an organization-wide transition towards HSS.

Making the shift to HSS is complex and difficult to achieve. In many contexts, funding streams, government demands and UNICEF’s lead position as an implementing partner, including in emergency responses, can pull UNICEF into a more supportive versus strengthening approach.

UNICEF has a comparative advantage for HSS in specific areas, including in strengthening sub-national level governance in the era of decentralization, and in improving data for decision-making.

UNICEF has made limited progress in terms of making changes to structures and systems to support working on HSS, with implications in terms of HSS implementation. Focusing on organizational change to support the transition to HSS is a key priority, including on cross-sector working, guidance and support, staff capacity, and systems for monitoring and learning.

Limited focus on communication and dissemination around the UNICEF HSS Approach has impacted on UNICEF’s progress in transitioning to HSS, not least because of implications in terms of securing additional resources for HSS. However, lack of data on HSS effectiveness and insufficient clarity on UNICEF’s role are limiting factors in more proactive communication.

Final Report


4.1. Recommendations

Based on the findings and conclusions highlighted above, we make a series of recommendations that we believe will help UNICEF to accelerate its transition to HSS. These were discussed with UNICEF during a series of meetings, including with the Evaluation Reference Group45 and Programme Division staff, in New York on 16 January 2019; following which feedback was incorporated and is reflected in the following 7 recommendations. We group grouped these under four headings: 1) Clarifying vision and strategy; 2) support staff to work on HSS; 3) Building the evidence base; and 4) Making the case. They are listed in order of priority, and build progressively towards enabling UNICEF to achieve two overarching objectives: UNICEF and other partners understand and value UNICEF contribution on HSS; UNICEF current and future programmes maximise HSS potential within existing mandate. For each recommendation we provide a summary box, and then unpack this in following text – along with suggestions on who should take forward each recommendation.

1. Clarify vision and strategy

Recommendation 1.1: Clarify vision for UNICEF role in HSS and develop cross-organization strategy to deliver on this vision. This will provide clear direction to UNICEF staff and partners on what UNICEF intends when it talks about transitioning to HSS. It should include:

• Revisions to HSS Approach.

• Development of a cross-organization strategy to deliver the HSSA. Clarification on what success looks like and how HSS can be incorporated into thematic issues, starting with the five issue-specific areas of existing UNICEF capacity and perceived priority46.

• Development of operational guidance on HSS.

45 Evaluation Office, HSS team, Supply Division, MENA Regional Office. 46 See Fig. 2 above. These include: Improving data information systems; Procurement and supply chain management; Social protection and

welfare; Engagement and regulation of the private sector; and Quality of care at community and facility levels

Final Report


As the foundation of UNICEF work on HSS, the evaluation findings point to the need to strengthen UNICEF’s vision and strategy to achieve HSS, starting with an updated articulation of UNICEF’s comparative advantage and HSS service offer. This is needed for both internal and external audiences and is a critical component of making UNICEF transition to HSS. Below we describe in more detail some key components of this visioning and strategy work, with suggestions on who should take forward this work:

• Revisions to HSSA. The HSS team should revise the HSSA to further describe UNICEF comparative advantage, articulate how UNICEF work on HRH and HF will strengthen health systems, and to support communication of how UNICEF role fits into a bigger vision for how to strengthen health systems. This could engage with the fundamental tension between HSS and implementation/emergency response and give guidance to staff about how to manage this, including through a more explicit ToC for how UNICEF expects to contribute to HSS, based on its comparative advantage, and to include a better hierarchical visual of the relationship between the HSS approach and other HQ strategies. This could be based on an articulation of UNICEF’s ToC, using the WHO Building Blocks for HSS as a framework – i.e. specifying how UNICEF sees its role in relation to each. Any revision (or part thereof) should also be tailored for an external audience, which will help UNICEF be able to say what it is and is not doing on HSS (rec #4.2).

• Clarification on what success looks like and how HSS can be incorporated into thematic issues, starting with the pillars and beams. The HSS team and thematic section leads (immunisation, MCH, HIV, gender, ECD, WASH, nutrition, etc) should articulate what HSS means in their thematic area, including desired results and how to track these, examples of entry points, UNICEF role/comparative advantage compared to other stakeholders, how their strategies (Nutrition Strategy, Routine Immunization Roadmap, etc.) are contributing to the HSS roadmap, to help highlight organizationally where there are gaps and opportunities. This could use work done with Supply Division as a potential model. This is primarily a thought process, which would likely show up in a range of places, possibly including the HSSA, any strategy to operationalise this, specific guidance for each thematic area.

• Development of a cross-organization strategy to deliver the HSS Approach. Programme Division should lead on development of a strategy and plan to deliver the HSS Approach, clarifying roles and accountabilities across the organisation e.g. Supply Division, Programme Division etc. This should set out the process, mechanisms and pace at which UNICEF seeks to strengthen its focus on HSS.

• Development of operational guidance on HSS. The HSS team, thematic section leads, PD and ROs should develop guidance to help COs understand how to implement HSS in their contexts, including on thematic issues in the pillars and beams but also covering gender and equity and key areas such as developing a firm understanding of health systems deficits, piloting, sustainability/ transition and capacity skills transfer. Guidance on transition planning should take into consideration the context – i.e. how far and fast it is feasible to achieve sustainability, as well as set out 1) what is to be sustained and 2) steps to ensure conditions for sustainability are in place: capacity, finance, motivation. This should include providing a greater range of clear examples of where UNICEF has done HSS well, and development of tools to support (e.g. political economy analysis, financial assessments, transition planning, revising SitAns to cover health system deficits, checklists like the sine qua non tool in KCO).

2. Support staff to work on HSS

Recommendation 2.1: Ensure UNICEF staff have the capacity ‘to do’ HSS. This concerns ensuring that skills and knowledge to incorporate systems-thinking into their day-to-day work. We recommend three main elements to this focus on staff capacity:

• Ensure staff have relevant skills and knowledge through training and on-the-job mentoring.

• Ensure staff are focused on HSS through recruitment and performance management.

• Ensure all staff have appropriate responsibility and accountability for delivering on HSS objectives.

Final Report


As the key ‘change agents’ within the transition to HSS, UNICEF staff need to have the right skills and competences to incorporate systems thinking into their day-to-day work. This reflects that HSS is as much about how you do things, as about what you do. Below we describe in more detail some key components of work required to ensure staff have the right skills and competences, with suggestions on who should take forward this work:

• Ensure staff have relevant skills and knowledge through training and on the job mentoring. The HSS

team should continue to promote HSS training for all UNICEF staff (it is important that it continues to

be for all staff, not just in health sections). This should be seen not just as a professional development

tool but also as a programmatic intervention to change the culture and thinking within the

organization. The HSS team should continue to offer the Melbourne HSS course, or a version of it, and

continue with planned modifications to maximise its accessibility – e.g. to provide regional-level

courses, and lower-cost options such as MOOCs. ROs and COs should encourage staff to participate in

HSS training, and ensure that budgets are allocated and workflows managed to facilitate this.

Furthermore, where there are strong HSS leads or where the Chief of Health has relevant HSS

expertise, COs should ensure that the potential value of on-the-job mentoring is maximised; this

should the harness vital role that the Chief of Health or other staff with relevant HSS expertise can

play in this regard.

• Ensure staff are focused on HSS through recruitment and performance management. The HSS team,

ROs, Chief of Health and HR manager should explicitly incorporate HSS in staff recruitment and job

profiles:

- UNICEF ROs support COs to incorporate HSS competencies into health, nutrition, WASH, ECD, HIV staff profiles when they develop their new staff organograms at the start of a new CP;

- UNICEF HQ/ROs support COs to incorporate HSS into health, nutrition, WASH, ECD, HIV staff recruitment (e.g. through developing model HSS profiles, interview questions and testing tools).

- UNICEF HQ/ROs provide examples and support COs to incorporate HSS competencies into annual staff performance assessments;

• Ensure HSS staff have appropriate responsibility and accountability. HQ, ROs and COs should consider where HSS capacity should sit within the organization – to ensure that system strengthening is a cross-organization effort and not solely confined to health. This is likely to include reflection on the seniority of HSS leads, the accountabilities of these positions (i.e. to whom do they report) as well as where do these positions sit within the office. The objective is to ensure that there is sufficient authority and accountability to affect required organization-wide changes.

Recommendation 2.2: Support COs to strengthen their focus on HSS. This builds on recommendations 1 and 2, and on evidence that UNICEF staff have not found it easy to work on HSS and have requested proactive support from ROs and HQ to make this transition. Support could be provided in the following areas:

• Review and revise existing programmes.

• Design new HSS strategies and interventions.

• Incorporate HSS into new CPDs and MTRs.

• Support exchange of practical experience between UNICEF staff.

• Formalising support to COs by ROs and HQ.

Once the HSS vision and strategy have been refined, specific guidance developed, and relevant skills and capacity established at all levels across the organization, ROs need to provide proactive, systematic support to COs to help units and programmes within health explicitly acknowledge their influence on

Final Report


systems and frame their plans and goals/ways of working in terms of systems impact as well as health outcomes. This could be done through the following five approaches:

• Review and revise existing programmes. ROs should support CO teams to review existing interventions in the health and other sectors to identify opportunities to increase the focus on systems strengthening and move away from systems support. These reviews can draw on guidance/tools developed in recommendation 1. For instance, UNICEF should review investments in capacity building with the aim of better integrating sustainability considerations and measuring effective skills transfer. There is also potential to use the Chee cube (or a derivative of it) to map interventions on a continuum of support vs strengthening and to develop a strategy to explicitly move towards more systems-focused work.

• Design new HSS strategies and interventions. HQ and ROs should support COs to design funding proposals / programmes that are explicitly about HSS, but also that incorporate HSS thinking into thematic programmes. Also building on tools developed under recommendation 1 (e.g. checklists for how to incorporate HSS into programme design), new programmes can be more explicit about assessing health systems deficits, addressing policy and organizational barriers, ensuring transition from UNICEF to government, and ensuring conditions required for sustainability are in place.

• Incorporate HSS into new CPDs and MTRs. ROs proactively identify timing of key CO processes (CPDs, MTRs, strategic moment of reflection, retreats, evaluations) and offer support to ensure relevant information and thinking on HSS is available at the right time. The HSS team should review how HSS is being incorporated in emerging CPDs and provide further guidance, as there is evidence of different approaches being used – that is, whether to separate out or mainstream HSS.

• Support exchange of practical experience between UNICEF staff. ROs should create space for COs to share and learn lessons about experience in transitioning to HSS; this could be done through ROs facilitating regional-level Communities of Practice on HSS, and fostering interchange among HSS training graduates to support application of HSS approach at regional and country level, potentially drawing on emerging work in RoSA.

Formalise support to COs by ROs and HQ. Programme Division and the health section should look at the feasibility and practicability of establishing compacts between HQ, ROs and COs to formalise support, division of labour and agreed contributions on strengthening HSS at all levels. This could help create buy-in and operational space, as well as accountability. These might take the form of an HQ, RO and CO joint initiative in which select COs are supported as they implement the HSS operational guidance (Rec. 1.1). Depending on the CO programming cycle, resources would be made available to e.g. conduct political economy analysis in preparation for country programme development; preparation of transition plan, and support on monitoring and evaluation, documentation and learning.

3. Build the evidence base

Recommendation 3.1: Develop and implement a clear strategy for monitoring and evaluation of UNICEF work on HSS. In order to strengthen and institutionalize HSS monitoring, the HSS team, working with relevant parts of the organization, needs to ensure data and evidence is available to track performance on HSS, including through:

• Develop strategy to ensure data and analysis needs of organisation are met.

• Strengthen availability of evidence on effectiveness and efficiency of UNICEF HSS interventions.

In the same way that establishing a clear vision and strategy for HSS is foundational, so is UNICEF’s ability to track its work on HSS and demonstrate effectiveness. This recommendation seeks to establish a solid base from which staff will be able to confidently communicate to stakeholders within UNICEF on the added value of an HSS approach and of UNICEF’s role within this. This may not be straightforward– as

Final Report


evidenced by ongoing global discussions about how best to measure HSS – and so the initial focus could be placed on monitoring UNICEF interventions using internal (rather than global) measures47. Taking this forward should include:

• Develop strategy to ensure data and analysis needs of organisation are met. This could be part of

revisions to the HSS approach (rec. #1), providing that there is space for sufficient detail on how HSS

will be monitored and evaluated.

- Review requirements and purpose of M&E systems, e.g. to capture the needs of Gavi HSS monitoring, understand relative effectiveness of HSS interventions, and systematically describe what UNICEF is doing on HSS. As part of this work, it is particularly important to define and monitor the relevant indicators of progress and success on HSS.

- Learn lessons from previous UNICEF efforts, including: Supply Division’s work to look at progress in terms of maturity of health systems; and experience of COs and ROs in tracking HSS performance.

- Collect relevant quantitative and qualitative data and clarify responsibilities and timing of data

collection and analysis processes.

• Strengthen availability of evidence on effectiveness and efficiency of UNICEF HSS interventions:

- Support COs to develop HSS objectives and targets, and to measure progress towards these.

- Develop a robust HSS indicator list and incorporate into UNICEF’s RAM indicator lists

- Strengthen mechanisms for tagging activities/interventions as HSS to improve accuracy of reporting

- Support COs to incorporate HSS in scope of planned evaluations

- Complement evidence on effectiveness with data on UNICEF expenditure on HSS interventions,

including to demonstrate HSS effects of GAVI and GF HSS funds, and impact of UNICEF work to

support policy development at country-level.

Recommendation 3.2: Ensure UNICEF is learning from its work on HSS as an input to continuous improvement planning and to global public goods. UNICEF needs to develop and implement strategies to ensure it is reflecting on and learning from what it is doing on HSS, as a means to improve HSS programming. There are two key aspects to this learning agenda:

• Encouraging replication and adaptation within UNICEF.

• Informing policy development and the wider HSS community.

It is important that UNICEF has systems in place that enable it to document how it has sought to strengthen health systems and then to learn from this experience48, both for internal and external stakeholders. This includes two components that relate to adaptive management49, and to capturing UNICEF lessons for use by country- and global-level stakeholders:

• Encouraging replication and adaptation within UNICEF. The emphasis here is on ensuring adaptive

management or continuous improvement planning - to enable course correction, based on lessons

learned and in response to changes in context. This is particularly relevant where UNICEF is a key

country-level partner supporting the effective use of substantial HSS funding by GAVI and the Global

Fund. Building on the documentation of UNICEF interventions described in recommendation #3.1,

UNICEF at all levels needs to create or make use of existing systematic processes and spaces to ensure

review and decision-making based on evidence of what works and of changes in context.

47 It would be preferable to align internal and global measures on HSS, although important that this does not undermine internal progress on

operational measures of progress 48 This is not to overlook the potential contribution that is made through UNICEF’s implementation research (IR) in this area, but the evaluation

did not cover this in any detail, so we are unable to comment on how far IR goes in addressing this recommendation 49 We use USAID’s definition of adaptive management: “an intentional approach to making decisions and adjustments in response to new

information and changes in context” https://usaidlearninglab.org/qrg/adaptive-management [accessed 28/01/2019].

https://usaidlearninglab.org/qrg/adaptive-management

Final Report


• Informing policy development and the wider HSS community. As described above, a key aspect of

UNICEF’s comparative advantage is being able to bring capacity and insights from all levels of the

health system to policy- and evidence-based dialogue in other forums. UNICEF at all levels needs to

package and communicate findings from its work on HSS that can benefit and help external other

stakeholders improve their work on HSS. This could be done, for example, through convening

national/regional forums to highlight evidence on relevant HSS approaches – as has been done in

Kazakhstan, or publishing papers on UNICEF work on HSS.

Recommendations 1, 2 and 3.1 should facilitate UNICEF ability to deliver on recommendation 3.2. For example, UNICEF needs to incentivize staff to generate and disseminate HSS knowledge products, which can potentially be done through adding these functions to job descriptions and annual performance assessment frameworks (as per recommendation 2.1); to document how HSS is being implemented to promote operational learning both within and external to UNICEF (as per recommendation 3.1); and to develop mechanisms to share lessons, such as regional-level communities of practice (also as per recommendation 3.1).

4. Make the case

Recommendation 4.1: Advocate across UNICEF for an organization-wide change management process to increase UNICEF engagement on HSS, and ensure systems in place to strengthen effectiveness on HSS. Whilst there may appear to be overlap with recommendations 1-3 above, the emphasis here is on internal advocacy to ensure that recommendations 1-3 are taken forward. This should include cross-organization communication to encourage staff to participate in HSS training (rec. 2.1), to press for changes to systems to ensuring monitoring & evaluation of HSS (rec. 3.1 and 3.2), and to encourage staff to communicate with external partners (rec. 4.2). Additional objectives should include:

• adequate staff resources at all levels to work on HSS

• systems are in place to better articulate goals for HSS interventions, and support cross-sector engagement

To take forward recommendations 1-3 above, UNICEF needs to proactively and systematically make the case for HSS across the organization. Whilst acknowledging the work that has been done to disseminate the HSS Approach during 2017-18, it is clear that some staff within UNICEF remain to be convinced that HSS is a relevant strategy for achieving UNICEF’s core mandate. At the same time, required changes to organizational systems and structures have not been made that would facilitate a stronger, more systematic focus on HSS. Securing these changes will require leadership and internal advocacy with relevant parts of UNICEF. We therefore recommend that the HSS team and Programme Division jointly develop and implement a clear, agreed communication strategy to advocate for an organization-wide change management process to increase UNICEF engagement on HSS. As with monitoring and evaluation (rec 3.1), this could be part of revisions to the HSS approach (recommendation 1), providing that there is sufficient space to articulate objectives, messages differentiated for explicit target audiences, responsibilities and timeframes. It is also important that this dissemination reflects on UNICEF’s decentralised structure and avoids an overly ‘top-down’ dissemination which may limit uptake by COs. Objectives for this strategy should include cross-organization communication to encourage staff to participate in HSS training (rec. 2.1), to press for changes to systems to ensuring monitoring & evaluation of HSS (rec. 3.1 and 3.2), and to encourage staff to communicate with external partners and make them aware of resources to do so (rec. 4.2). Additional objectives should include: - Leadership

- Adequate staff resources at all levels to work on HSS. The HSS team should work with relevant

internal stakeholders to advocate for increased allocation of financial and human resources for HSS.

Consider ensuring that all ROs have at least one dedicated HSS post, or that HQ or ROs make available

resource pools to COs on specific themes or topics related to HSS.

Final Report


- Systems are in place to better articulate goals for HSS interventions, and support cross-sector

engagement. Programme Division should also review and revise requirements for setting out

programme-level interventions (i.e. to insist on articulation of HSS goals, explicit ToCs and integration

into Strategy Notes), learning from what it has been doing to increase internal coordination and cross-

sectoral engagement and reduce outstanding challenges and barriers.

Recommendation 4.2: Develop partnerships with external stakeholders to maximize UNICEF’s comparative advantage in HSS. There is a significant imperative for UNICEF staff to work with external partners to:

• Promote understanding of UNICEF contribution and added value as an HSS partner, and of UNICEF mandate.

• Leverage UNICEF comparative advantage to maximise effectiveness of HSS investments.

Evidence from this evaluation has shown that UNICEF is not perceived as a go-to partner on HSS for donors or governments. It is important to recognise that UNICEF is attempting to enter a thriving marketplace with many HSS providers, with two key implications: 1) UNICEF needs to be clear about and demonstrate its comparative advantage (section 3.1); 2) This comparative advantage is a niche role within what’s required overall to sustainably strengthen national health systems, and HSS objectives will only be achieved through partnership. There is therefore a significant imperative for UNICEF staff to work with external partners to:

• Promote understanding of UNICEF contribution and added value as an HSS partner, and of UNICEF mandate. Building on the vision and strategy set out in recommendation #1, which includes the development of a clearer theory of change to contextualise UNICEF’s contribution alongside that of other partners, UNICEF COs should articulate country-level partnership strategies that clarify and communicate UNICEF’s contribution and added-value on HSS. These should be developed in consultation with partners at country level, and with support from ROs and HQ (e.g. potentially through explicit guidance and tools developed under recommendation #1). It may, in the first instance, be more appropriate for partnership dynamics to be articulated at the thematic- or intervention-level and based on stakeholder analysis as part of the design/review of interventions. The key purpose is to recognise that UNICEF work on HSS is contributing to a wider-effort and its effectiveness is as much contingent on work done by others as on its own efforts.

• Leverage UNICEF comparative advantage to maximise effectiveness of HSS investments. Recognising that external partners (such as GAVI, the Global Fund, the Bill & Melinda Gates Foundation) have invested substantial funds in HSS, UNICEF should work with these organizations to ensure that its activities are complement and maximise the effectiveness of other partners investments. With other development partners, UNICEF should work to develop plans and funding modalities that articulate long-term vision and goals, and then make clear how short-term interventions contribute to these broader goals. Defining interventions in a longer-term context has the potential to manage expectations about what is achievable in the short-term and facilitate phased investment over multiple biennial cycles.

• Raise resources to ensure UNICEF’s status as an effective implementation partner is adequately funded. To support the actions identified throughout these recommendations, UNICEF should invest in raising resources which will allow it to engage across partners.

Final Report


Annexes (see Volume 2)

Annex A: Evaluation Terms of Reference

Annex B: Overarching Theory of Change for UNICEF’s approach to health systems strengthening

Annex C: Evaluation matrix

Annex D: Summary of organizational domain report findings (Year 1 reports)

Annex E: Summary of implementation domain (TCS) report findings