unicef and world bank findings from …...sources: unicef annual wash reports 2006–2016 and vision...

UNICEF and WORLD BANK

FINDINGS FROM GLOBAL EVALUATIONS

Global evaluation of UNICEF drinking water supply programming in rural areas and small towns 2006-2016

Selected findings on equity and sustainability

© U

NIC

EF/s

imo

nlis

ter

Jérémie Toubkiss

Evaluation SpecialistEvaluation Office

October 2018

RATIONALE & PURPOSE

WHY CONDUCTING THIS EVALUATION?

• Drinking water supply remains an important area of work for UNICEF,

included in the new Strategic Plan for 2018-21 and the UNICEF Global

WASH Strategy for 2016-2030

• The SDG emphasizes the need for Member States and their development

partners to scale up efforts in this area, establish efficient and sustainable

water supply management systems, ensure the quality of drinking water,

and adopt a stronger equity-lens in order to leave no one behind. Similar

commitment in the current UNICEF Global WASH Strategy

• Opportunity for UNICEF to reflect on the relevance and quality of its

drinking water supply portfolio, which, until now, had not been evaluated at

the global level

• Will feed into the development of a water supply strategic document

EVALUATION PURPOSE

• Contribute to evidence-based learning in the

water sector more broadly

• Assure internal and external stakeholders

regarding the level of performance and the quality

of the programmes that they have supported

• Guide UNICEF’s policies & programming in the

period of the new global WASH strategy and

SDGs

• Fill specific knowledge gaps and draw lessons

from past experience

Contribution to sector

Accountability

Decision making

Internal learning

CRITERIA & AREAS OF INQUIRY

Criteria Areas of inquiry

Effectiveness Contribution to the water MDG

Contribution to the sector EE

Contribution to building knowledge & capacities

Efficiency Achievement of commitments in terms of results & expenditure, and unit costs

Integration of cost-efficient approaches

HR: quantity, quality, management

Equity

Engagement and advocacy at global and country levels

Geographical targeting

Equity lens at each stage of the programme cycle

Results in reducing inequities

Sustainability Engagement & advocacy at global level

Measures supporting sustainable services

Evidence on sustainability of past interventions

Innovation /

upscaling

UNICEF’s capacities for innovation

Level of engagement in and scaling up of: real-time monitoring, innovative financing, PSP; accountability & regulation mechanisms

Process followed for innovating and scaling up

Relevance Alignment with the international and national agenda, and coordination with other sector stakeholders

Positioning & credibility to partners and other major players

Use by UNICEF of its comparative advantages, and added value in the sector

FINDINGS:EQUITY

EQUITY IN THE PROGRAMME CYCLE

To what extent has UNICEF targeted the most disadvantaged populations and contributed to reduce inequalities as per its Global Strategy?

Strong focus on the least advanced, low coverage regions

47% of UNICEF RWS investment spent in low income countries

(2012-2016), more than twice the share for all ODA (22%)

72% of UNICEF RWS investment spent in sub-Saharan

Africa, 3.5 times higher than total sector ODA (21%)

% of country population using improved water sources in 2006

2012-2016

2012-2016

Sources: UNICEF annual WASH reports 2006–2016 and VISION as of September 2017

Strong focus on rural areas, which lag behind

50-75% of UNICEF water supply investment in 2012-2016 went to rural areas compared to

31% for total water supply ODA – indicating a likely significant contribution of UNICEF to the

reduction of the gap between urban and rural areas

World – total

World – rural and urban

Sub-Saharan Africa

– total

Sub-Saharan Africa – rural

and urban

World – total


Sub-Saharan Africa

– total


and urban

World – total


Sub-Saharan Africa

– total


and urban

Water supply coverageUNICEF expenditure (million USD)

Annex E6: UNICEF RWS development expenditure 2012-2016, by country (all

resources, in USD)

Source: VISION data 2012–2016

Annex E7: UNICEF RWS development expenditure 2012-2016, by country

(un-earmarked funding only, in USD)


Countries with the greatest water needs did not benefit from proportionally more fundingIn 2006

Countries with lower water coverage With larger unserved population With slower progress towards MDGs

in 2006 in 2006 in 2006

Dependency on earmarked

funding (thus on donor

strategies and priorities)

constrained UNICEF’s ability

to direct its support to the

countries where it was most

needed.

All UNICEF RWS development expenditure 2012-2016 UNICEF RWS development expenditure 2012-2016 – un-earmarked funding only

But allocation of UNICEF’s

un-earmarked funding was

also suboptimal.

Annex E6: UNICEF RWS development expenditure 2012-2016, by country (all

resources, in USD)


Annex E7: UNICEF RWS development expenditure 2012-2016, by country

(un-earmarked funding only, in USD)


FINDINGS:SUSTAINABILITY

SUSTAINABILITY MEASURES AND RESULTS

To what extent has UNICEF effectively supported the financial viability of the service?

Figure 53: Proportion of communities in which tariff revenues allowed to recover recurrent costs

Source: Country case studies (visit of 61 communities in eight countries);

Note: the proportion of communities without a tariff in place was

significantly lower than indicated in sustainability checks; however, these

data are based on self-reports from water committees or service providers

Country

(report year)

% communities

with revenues

higher than

expenditure

Madagascar

(2014) 33%

Mozambique

(2015) 68%

Rwanda (2015) 82%

Kenya (2015) 9%

Myanmar

(2016) 43%

Ethiopia (2015) 35%

Pakistan (2016) 50%

Average 46%

Source: Evaluations and sustainability checks

Lack of analysis of life-cycle costs and users’

willingness to pay

Alignment of tariffs with government policy or

decision left for communities to decide

>50% of communities with tariffs non-existent or

not applied, and revenues insufficient to cover

O&M costs and ensure medium-term viability of

the service (country case study field visits, evaluations & sustainab checks)

Only 3 examples of UNICEF-supported financing

scheme supporting sustainability and/or pro-poor

objectives

Study or policy advocacy for increased public

budget allocations in 15-20 countries, with varying

levels of success

Cost recovery in communities visited in the 8 case study countries (% of 61 visited communities, based on water user committee’s self-report)

Financial viability of UNICEF supported RWS services: Low prioritization and weak results

Mapping of UNICEF-supported pro-sustainability or pro-poor financing schemes/tools, 2006-2016

To what extent has UNICEF effectively strengthened the institutional factors supporting

the sustainability of the service?

UNICEF significantly and increasingly engaged in strengthening

the sector enabling environment and institutional capacities at all

levels: local, national and global

UNICEF

supported all 5

‘building blocks’

of the sector

enabling

environment

UNICEF significantly and increasingly engaged in strengthening

the sector enabling environment and institutional capacities at all

levels: local, national and global

: Positive opinions on UNICEF’s investment in strengthening RWS capacities and institutional

sustainability in case study countries over the 2006–2016 period

UNICEF’s support was strongly recognized

and appreciated in all case study countries,

by the government, implementing partners

and other sector stakeholders

Positive opinions of sector stakeholders on UNICEF’s investment in

strengthening RWS institutional capacities and sustainability in countries

over the 2006-2016 period

Choice of management arrangements not based on

evaluation evidence, feasibility study or

comparative analysis, except in 2 countries

Mapping of UNICEF’s engagement with private sector participation

in RWS service management, 2006-2016

UNICEF rarely questioned or investigated the prevailing

community-based management model despite growing evidence

of shortcomings

Of the 33 RWS country programmes evaluated between 2006 and 2016

Little study and documentation of experiences and

lessons on private sector participation in UNICEF’s own

operations and in the broader water sector

Greater emphasis on PSP in UNICEF’s WASH Strategy

for 2016–2030 and interesting action-research in the field

Low-intensity and spread of UNICEF’s engagement in regulation

and accountability mechanisms – but has recently increased

Towards the end of the evaluation period, UNICEF

started supporting 7 regulation and accountability

mechanism initiatives in 5 countries

Anecdotal evidence of positive, sector-wide

outcomes for the sector (quality & sustainability of

the service), if effectively institutionalized

4 of these initiatives have focused on monitoring service

provider activities using real-time monitoring technologies

that UNICEF contributed to take to scale in some

countries and across countries

FINDINGS:EQUITY & SUSTAINABILITY IN CORPORATE AND COUNTRY

PROGRAM M&E

Program engagement and performance in equity and sustainability: How would UNICEF know?

Notable improvements over time, but corporate and program

monitoring & reporting systems not yet set up to track

engagement and results in equity in a periodic, accurate and

disaggregated manner

Number of RWS-related indicators in UNICEF’s monitoring and reporting

systems, 2006–2016

Special WASH reports

Inclusion of 5 RWS indicators in RAM

Inclusion of 4 RWS indicators in SMQs

Inclusion of 7 generic RWS indicators in new version of RAM Some progress towards UNICEF’s 2006

commitment to improve monitoring and

reporting, with more and better indicators

Results data increasingly sex-disaggregated

Insufficient progress in regard to measuring

results for other demographic groups –

‘equity’ too often conflated with ‘gender’

Number of RWS-related indicators in UNICEF’s monitoring & reporting systems

Similar finding on UNICEF’s ability to monitor and report on its

engagement and results in sustainability

Number of sustainability checks undertaken by UNICEF between 2006 and 2016

Most evidence comes from ad-hoc evaluations

and ‘sustainability checks’, conducted in about &

third of UNICEF RWS programme countries

This body of evidence is growing

…but still scattered and of uneven quality

Weaknesses in the inclusion of equity &

sustainability in UNICEF’s M&E and reporting

systems constrained its ability to continuously learn

and improve, despite high prioritization in global

Strategy and advocacy

Number of UNICEF sustainability checks, 2006-2016

Data sources on UNICEF RWS sustainability, 2006-2016

“A THIRST FOR CHANGE”

An Evaluation of The World Bank Group’s Support for

Water Supply and Sanitation, 2007-16

Ramachandra Jammi

Senior Evaluation Officer, Sustainable Development Unit, IEG

Joint Presentation with UNICEF focusing on Rural Water Supply findings

Water and Health Conference , 2018, University of North Carolina at Chapel HillOctober 31, 2018, 2:30-4:00 p.m., “Azalea”

Context – SDG 6

Independent Evaluation Group 33

SDGs are the core business of the World Bank Group, and thus SDG 6 now frames the World Bank Group’s strategy for Water Supply & Sanitation

SDG 6 has raised the bar from simple access to water supply and sanitation to sustainable and universal access to adequate, reliable, safe and affordable service delivery by 2030

Evaluation Questions

1. How effective has the WBG been in supporting Clients to improve WSS services?

2. How well is it equipped to support Client countries in keeping with SDG 6?


• Literature review: World Bank and External

• Portfolio Review of the lending and advisory/technical/knowledge assistance

• Country Case Studies:

Field-based: Egypt, India, Indonesia, Nigeria, Sri Lanka—jointly with ADB and

JICA—, Tunisia, and Zambia);

Desk-based: Brazil, Colombia, Haiti, Peru, the Philippines, and Uganda).

• Stakeholder Interviews with government and implementing agency officials;

WBG task team leaders, managers, and staff; civil society; beneficiaries.

Methodology


Urban and Rural Water Supply and Sanitation:

Investment Lending; Development Policy Finance; Advisory Services and

Analytics (ASA); Technical Assistance; Guarantees: FY2007-2016

Overall Evaluation Coverage

WBG assistance to WSS during evaluation period

Institution Projects (#)Commitment

(US$, billions)

World Bank 458 28.4

Water Global Practice 163 20.2

Other global practices 295 8.2

IFC investments 49 1.5

MIGA guarantees 9 0.4

WB Group Total 516 30.3

World Bank Support for WSS – Urban vs. Rural focus

Rural areasreceive lessattention in Sub-Saharan Africa

Urban areasreceive lessattention in South Asia

Small Towns receive lowemphasis in all regions


Category % projects with a subsector objective; total projects = 163; FY2007-16

SSA EAP ECA LAC MNA SAR

Urban water

Urban sanitation

RURAL WATER

Rural sanitation

SMALL TOWNS

SSA: Sub Saharan Africa; EAP: E. Asia and Pacific; ECA: E. Europe and C. Asia; LAC: Latin America and Caribbean;

MNA: Middle East and North Africa; SAR: South Asia


• focused more on access than on service adequacy, reliability, quality, affordability• more effective in improving access than adequacy, reliability, quality, and affordability

• Quality of rural water supply showed the lowest performance• Very few projects measured affordability

WB RWS projects: Priorities, Focus, and Performance

Access and Service Delivery Performance of World Bank Projects, Completed FY2007–16

Subsector ProjectsAccess

Service Delivery Attribute (% moderately satisfactory or better)

Adequacy Quality Reliability Affordability

U PU R U PU R U PU R U PU R U PU R

Water supply

Projects rated (#

67 15 45 45 * 22 35 * 12 27 * 7 * * *

MS+ (%) 81 73 78 73 * 77 69 * 58 74 * 86 * * *


WB Experience with the Community-based Model for Rural Water Supply

Performance of WB community-based RWS projects was favorable overall:

35 out of 45 projects (78 percent) rated moderately satisfactory or better

Good experiences• Indonesia’s Third Water Supply and Sanitation for Low Income Communities Project supporting

PAMSIMAS (the national program for rural water supply and rural sanitation in Indonesia), • India: several provincial projects in (Uttarakhand, Maharashtra, Punjab, and Kerala)• Sri Lanka’s Second Community Water Supply and Sanitation Project

Not so good experiences• Peru’s National Rural Water Supply and Sanitation Project• Uzbekistan’s Water Supply, Sanitation, and Health Project


WB Experience with the Community-based Model for Rural Water Supply

Factors driving performance

Attention to community capacity building (India, Indonesia, Sri Lanka); in a society accustomed to state institutions making decisions it requires extra effort in building community self-help (Uzbekistan)

Ownership from cash/in-kind contribution (India, Indonesia, Sri Lanka); undermining ownership e.g. through government grant-based programs that do not require any in-kind or cash contribution can adversely affect implementation (Peru)

Knowledge transfers from the WB’s global experience.

However, long-term sustainability depends on

ongoing financial and technical support from local government a transition strategy as villages grow into small towns and peri-urban communities


Focus• Small towns receive low attention overall, though this category is the fastest

growing segment in many countries

Challenges • High unit costs; low economies of scale; low scope for cross‐subsidies• Low water use and resulting revenues• Need for professional management capacity as opposed to community-based

Risk Factors• Lack of tariff reform; • Lack of finance for rehabilitation and asset expansion• Lack of capital and skills – despite legislative mandate to operate water utility• Above factors prevalent to greater degree in small towns than other situations

Anticipating the transition from Rural Areas to Small Towns

Tracking Outcomes – Large Gaps in measuring Service Delivery


Large gaps in measuring service delivery attributes for rural water supply

This is especially so for sanitation and for rural areas, and for affordability

Rural Water SupplyNo. of rated

projects

Number of projects measuring

ADEQUACY QUALITY RELIABILITY AFFORDABILITY

45 22 12 7 -

Focus on the Poor - Tracking Outcomes


Few projects track results for the poor: Only 15 out of 45 projects

covering 13 countries had explicit Key Performance Indicators (KPIs)

directed to outputs or outcomes for people classified as poor

Lack of baseline data: Overall, parameters relating to the poor were

inadequately defined; and poor implementation and feedback to operations

Focus on the Poor – WB’s WASH Poverty Diagnostic


• The WASH Poverty Diagnostic (World Bank 2016) is a flagship

initiative led by the WB Water Practice with the Poverty Practice

• It analyzes the linkages between poverty and WSS through an in-

depth analysis of 18 countries—across six regions.

• The WASH Poverty Diagnostic is informing the WBG’s Systematic

Country Diagnostics, and generating ideas for current and

pipeline projects in dialogue with governments.


PAMSIMAS project in Indonesia: Customized M&E system that using cell phones and smart technology to facilitate planning and program management for rural WSS

Rural Water and Sanitation Information System (SIASAR): Platform to monitor rural WSS use in Bolivia, Colombia, Costa Rica, Dominican Republic, Honduras, Nicaragua, Panama, Paraguay, Peru, the Mexican State of Oaxaca, and the Brazilian State of Ceara. Uses apps for collecting data for policy formulation, planning, and resource allocation, aiming to enhance the sustainability and quality of rural WSS services.

Program for Results in Vietnam: Introduced mobile data collection and access to data in real time through an online database system complete with mapping functionality.

MajiVoice in Nairobi, Kenya: MajiVoice, a platform for improving communication between citizens and utilities, was tested successfully, enabling an efficient means of registering and resolving complaints.

Innovations for Monitoring & Evaluation for Rural Water Supply


Addressing Gender Issues in Water Supply and Sanitation

Favorable gender-related interventions:

Lusaka and other townships in Zambia: women largely manage the water kiosks and sanitation facilities, and participants in focus groups expressed satisfaction with the services overall.

Sri Lanka: Systematic efforts in rural water supply programs for raising awareness and encouraging women’s participation in planning, implementing and managing water supply facilities.

India: Uttarakhand Rural Water and Sanitation Project - women may take an active role in planning and managing the facilities constructed

OVERALL, CROSS-CUTTING CONCLUSIONS

AND IMPLICATIONS FOR THE FUTURE

1. Significant contribution to the international development agenda and to a

strengthened sector enabling environment – increased over time

2. Focus more on increasing access to than ensuring sustainability of services and

on reducing inequities

3. The MDG target that did not prioritize and incentivize these objectives. A shift is

now needed in line with SDG 6.

4. Lack of a conceptual framework for and holistic and systematic approach to

equity and sustainability in RWS programming and M&E.

5. Need and opportunities for increased alignment and learning across the whole

sector prioritizing challenges related to SDG6. Efforts should be

coordinated/harmonized or in partnerships rather than isolated.

6. UNICEF & WB’s have a particular role to play: convening power and position as

knowledge brokers and policy influencers in the sector

World Bank IEG recommendations relevant to RWS (1/3)

RECOMMENDATION 1: Increase the World Bank Group’s diagnostic efforts for enhanced engagement on reducing disparities in WSS access between and within regions, countries, and urban and rural areas. This is especially relevant for LICs and LMICs of Sub-Saharan Africa, Asia, and Latin America, with a large concentration of the poor in several sub-regions, and peri-urban and rural areas.

In particular, rural WSS schemes need increased and dedicated technical and management support

World Bank IEG recommendations relevant to RWS 2/3

RECOMMENDATION 2: Align the results frameworks and key performance indicators of World Bank projects with SDG 6 needs and increase support to client countries to build their evidence base for WSS access and service delivery. Results frameworks and KPIs of World Bank projects should track service delivery outcomes (i.e. adequacy, reliability, quality, and affordability), and the degree of access and services to the poor. The World Bank should support client countries to set up systems to track WSS access and service delivery, drawing upon experience with harnessing information and communications technology for the purpose.

The lack of data on WSS service delivery is pervasive across World Bank client countries, hindering actionable policy assessments of service provision, performance targets, incentives for performance, and accountability for results

World Bank IEG recommendations relevant to RWS 3/3

RECOMMENDATION 3: Enhance knowledge and learning in the WSS sector in client countries through effective partnerships and capacity-building. Maintain and enhance the World Bank’s distinctive role in generating and sharing knowledge through analytical work – notably by the Water and Sanitation Program (WSP) and Water Partnership Program (WPP) – and technical assistance and capacity building through investment projects with a clear link to project outcomes in their results frameworks.

CONCLUDING REMARKS

Role of evaluation in guiding policyand programming efforts towardsthe SDGs – and how to maximize

this role

1. Evaluating equity and sustainability more systematically and better in order to fill current

knowledge gaps in the sector, overcome these critical challenges, and better inform

policy and programming efforts towards the SDGs

2. Design evaluations in a way that can help identify or assess specific innovations, or test

and compare new approaches for improved (sustainable, equitable) service delivery

models

3. More strategic, global/multicountry evaluations and evaluation syntheses such as these

ones, and more joint and government-led evaluations to inform national policies and the

sector as a whole rather than individual agencies and projects

unicef and world bank findings from …...sources: unicef annual wash reports 2006–2016 and vision...

Documents