1

Development Effectiveness Development Effectiveness at the World Bankat the World Bank

ODI 04/00

““...we will not measure our performance by dollars lent ...we will not measure our performance by dollars lent or projects approved, but by our development impact -- or projects approved, but by our development impact -- results on the ground. I cannot overstate the results on the ground. I cannot overstate the importance of this change.” importance of this change.”

James D. WolfensohnJames D. WolfensohnAnnual Meeting SpeechAnnual Meeting Speech

October 1996October 1996

2

The Development Environment The Development Environment Has EvolvedHas Evolved

Ascent of GlobalCapital Market

Greater Pluralism

End of Ideological Divide

Advent of Civil Society

Before Now

Money Bank

Planned Development

Conditionality

Resource Transfer

Full Service Bank

ParticipatoryDevelopment

Knowledge Management

and Partnerships

DevelopmentEffectiveness

3

The Very Concept of Development The Very Concept of Development Has ChangedHas Changed

FiftiesFifties SixtiesSixties SeventiesSeventies EightiesEighties NinetiesNineties

ReconstructionReconstruction GrowthGrowth Basic NeedsBasic Needs AdjustmentAdjustment Capacity BuildingCapacity Building

Technical AssistanceTechnical Assistance

ProjectsProjects

Sector InvestmentSector Investment

Adjustment LoansAdjustment Loans

Country Assistance Country Assistance StrategyStrategy

EngineeringEngineering

FinanceFinance

PlanningPlanning

NeoclassicalNeoclassical

Multi-disciplinaryMulti-disciplinary

DevelopmentDecades

Major Objective

Main Instruments

Dominant Discipline

Physical

NaturalHuman

Social

4

World Bank Performance Criteria Reflect World Bank Performance Criteria Reflect the New Development Agenda:the New Development Agenda:

andand

Governance: social development, economic management, public sector reform, etc.

Poverty/genderpopulation,

health,nutrition

Education

Human economy

Environment

Rural development

Infrastructure

Physical economy

Private sectordevelopment

Finance

Market economy

Energy/miningSocialprotection

Socially SustainableSustainableEnvironmentally Growth

5

The Meaning of Development The Meaning of Development Effectiveness Has Changed TooEffectiveness Has Changed Too

Development Effectiveness connotes a Development Effectiveness connotes a demonstrable and efficient contribution to demonstrable and efficient contribution to economically sound, socially responsible economically sound, socially responsible and environmentally sustainable growthand environmentally sustainable growth

6

OED methods are best practiceOED methods are best practice

According to two major reviews of the MDB’s According to two major reviews of the MDB’s evaluation methods and practices:evaluation methods and practices:

Bank implementation of evaluation criteria are Bank implementation of evaluation criteria are closest to the agreed criteria of all MDBsclosest to the agreed criteria of all MDBs

(according to Wyss report (according to Wyss report Harmonization of Evaluation Criteria Harmonization of Evaluation Criteria Among the MDBsAmong the MDBs, 1999), 1999)

The Bank is a leader in completion reporting and The Bank is a leader in completion reporting and performance review among all MDBsperformance review among all MDBs

(according to Eriksson report (according to Eriksson report Comparison of MDB Completion Comparison of MDB Completion Reporting and Performance ReviewReporting and Performance Review , 1996), 1996)

7

How is Development Effectiveness How is Development Effectiveness Assessed? Assessed?

At the project level, OED assesses five main criteria of development effectiveness:

Results

1.Outcome2. Sustainability

3. Institutional development impact

Inputs

4. Bank performance5. Borrower performance

Country conditions, exogenous factors

8

The outcome assessment The outcome assessment

““DOING THE RIGHT THINGS”DOING THE RIGHT THINGS”Relevance: Are objectives responsive to country needs andBank priorities?

““DOING THINGS RIGHT”DOING THINGS RIGHT”

Efficacy: Are we achieving objectives shared by the Bank and its partners?

Efficiency: Are we using Bank and partner resources economically?

Definition:the extent to which a project has achieved or is expected to achieve its major relevant objectives efficiently; rate of return threshold of 10% used when possible

6-point scale:•highly satisfactory•satisfactory•marginally satisfactory•marginally unsatisfactory•unsatisfactory•highly unsatisfactory

9

The sustainability assessment The sustainability assessment

Evaluation of 9 criteria on the risks to project net benefits over time:

technical environmental financial government ownership economic stakeholder ownership social conditions institutional support resilience to exogenous factors

Definition:The resilience to risk of net benefits flows over the long term

3-point scale:•likely•uncertain•unlikely

10

The institutional development The institutional development impact assessment impact assessment

Evaluation of 2 basic criteria:

capacity development of an organization

improving institutional arrangements (i.e., “rules of the game” such as the regulatory framework)

Definition:the improvement in a country’s ability to make effective use of its human, financial, and natural resources

3-point scale:•substantial•modest•negligible

11

What is the unit of account?What is the unit of account?

Investment projectsInvestment projects Sector investment programsSector investment programs Adjustment loansAdjustment loans Non-lending servicesNon-lending services Country assistance strategies (CAS)Country assistance strategies (CAS) Sector strategiesSector strategies

12

SupervisionReports

Mid-termReviews

Imple.Completion

Reports

PortfolioImprovement

ProgramReviews

PerformanceAudit Reports

and ImpactEvaluations

Regions/OPR

Regions/OPR

Regions/OPR

QAG OED

ARDE

RegionsAFREAPSASECAMNALAC

NetworksHDDESSDFPSI

PREM

Country AssistanceStrategy and Compacts

Sector Assistance Strategiesand Thematic Progress Reports

Lending ServicesNonlending

Services Lending ServicesNonlending

Services

ActivityCompletionReports onNonlending

Services

SectorImpactEval.

Networks/OED

AidCoordination

andCofinancing

Reviews

Regions/QAG/OED

ResearchActivity

CompletionReports

Trust FundsCompletion

Memos

Regions/OED

DEC/Networks

Regions/NetworksOED

13

Evaluation in The World BankEvaluation in The World Bank

OED is independent: it reports to the Board of OED is independent: it reports to the Board of Executive DirectorsExecutive Directors

5,000 projects have been evaluated since OED’s 5,000 projects have been evaluated since OED’s creation in 1970 creation in 1970 (644 operations, worth $52 billion in real terms, (644 operations, worth $52 billion in real terms, in FY98 -FY99 alone)in FY98 -FY99 alone)

The function spans a variety of sectors The function spans a variety of sectors (agriculture, energy, finance, transport, (agriculture, energy, finance, transport, water supply, urban development)water supply, urban development)

It evaluates projects, programs, processes, and policiesIt evaluates projects, programs, processes, and policies

14

Project Performance Project Performance Measurement SystemMeasurement System

• consistentconsistent• objectiveobjective

• continuouscontinuous• professionalprofessional

• credibilecredibile• coherentcoherent

(but need further (but need further harmonization)harmonization)

Quality at entryQuality at entry

Quality at exitQuality at exit

Monitoring duringMonitoring duringimplementationimplementation

OED

QAG QAG

15

Real Time Evaluation Means Real Time Evaluation Means Quality AssuranceQuality Assurance

Two Complementary ComponentsTwo Complementary Components Portfolio Improvement ProgramPortfolio Improvement Program Quality Assessment ProgramQuality Assessment Program

– Quality at EntryQuality at Entry– SupervisionSupervision– Economic and Sector WorkEconomic and Sector Work

16

It also Means Quality ManagementIt also Means Quality Management

Target managerial attention on “high risk” Target managerial attention on “high risk” clustersclusters

Regular monitoring and feedback to staff and Regular monitoring and feedback to staff and managersmanagers

Partnering to disseminate lessons of experiencePartnering to disseminate lessons of experience Linking evaluation to staff/managerial Linking evaluation to staff/managerial

evaluations and strategy (a long term goal)evaluations and strategy (a long term goal)

17

Where are we?Where are we?Where are we?Where are we?

Quality at exitQuality at exit have been improving over the 90s have been improving over the 90s Data on the active portfolio promises future improvementData on the active portfolio promises future improvement However, borrower implementation performance stagnantHowever, borrower implementation performance stagnant

1994 1995 1996 1997 1998-99*

Exit fiscal year

50

60

70

80

90

Per

cent

sat

isfa

ctor

y

OED - Outcome performance (exiting)

1997 1998 1999

Active fiscal year

50

60

70

80

90

Per

cent

non

-ris

ky

QAG - Non-risky portfolio (active)

18

(% satisfactory)(% satisfactory)

Exit fiscal yearExit fiscal year

RegionRegion 1990-93 1994-97 1998-99*1990-93 1994-97 1998-99*

AFRAFR 54 54 55 55 60 60 (+)(+)

EAPEAP 80 80 84 84 81 81 (-)(-)

ECAECA 73 73 72 72 83 83 (+)(+)

LACLAC 64 64 78 78 82 82 (+)(+)

MNAMNA 70 70 66 66 70 70 (+)(+)

SASSAS 73 73 67 67 68 68

BankwideBankwide 66 66 68 68 72 72 (+)(+)

Project Outcome by RegionProject Outcome by RegionProject Outcome by RegionProject Outcome by Region

19

Sustainability has been Sustainability has been improving...improving...Sustainability has been Sustainability has been improving...improving...

Sustainability Assessments

0

10

20

30

40

50

60

% Likely % Uncertain % Unlikely

Exit FY94-97Exit FY98-99*

47 49

3337

2014

20

… … as well as institutional as well as institutional development impact.development impact.… … as well as institutional as well as institutional development impact.development impact.

Institutional Development Impact Assessments

0

10

20

30

40

50

60

% Substantial % Modest % Negligible

Exit FY94-97Exit FY98-99*

31

3946 48

24

13

21

Regional ComparisonsRegional Comparisons

Exit FY90–93 Exit FY94–97 Exit FY98–99*

* Results are preliminary for FY99 exits (43 percent coverage).

Sustainability (percent likely)

0 10 20 30 40 50 60 70 80 90

AFR

EAP

ECA

LCR

MNA

SAR

ID Impact (percent substantial)

0 10 20 30 40 50 60 70 80 90

AFR

EAP

ECA

LCR

MNA

SAR

22

(% satisfactory)(% satisfactory)

Exit fiscal yearExit fiscal year

NetworkNetwork 1990-93 1994-97 1998-99*1990-93 1994-97 1998-99*

ESSDESSD 59 59 68 68 63 63 (-) (-)

FPSIFPSI 68 68 65 65 72 72 (+) (+)

HDNHDN 78 78 72 72 75 75 (+) (+)

PREMPREM 66 66 73 73 78 78 (+) (+)

BankwideBankwide 66 66 68 68 72 72 (+) (+)

Project Outcome by NetworkProject Outcome by NetworkProject Outcome by NetworkProject Outcome by Network

23

Network ComparisonsNetwork Comparisons

* Results are preliminary for FY99 exits (43 percent coverage).

Exit FY90–93 Exit FY94–97 Exit FY98–99*

Sustainability (percent likely)

0 10 20 30 40 50 60 70 80 90

ESSD

FPSI

HDN

PREM

ID Impact (percent substantial)

ESSD

FPSI

HDN

PREM

0 10 20 30 40 50 60 70 80 90