1

Monitoring and Evaluation: A Logical Frame

Evaluating the Impact of Projects and Programs

Beijing, China, April 10-14, 2006Shahid Khandker, WBI

2

What is M&E?

Monitoring and evaluation are tools that make it possible to identify and measure the results of projects, programs or policies

3

Why the Concern?To evaluate and adjust strategies and activitiesTo report on progress to interested parties, clients, taxpayers and the general publicTo identify and share with others best practices and lessons learnedTo improve the programming of new interventions and strategies

4

MonitoringProvides regular information on how things are workingDefn: A continuing function that uses: A) systematic data collection and analysis of specific indicators of progress.B) Provides management with indication of extent of progress towards goals

- achievement of deliverables-use of resources.

C) Contributes to performance improvement.D) Conducted by business unit.

5

Evaluation

Evaluation can only be done after a certain time and requires more thorough investigationsConducted by independent evaluators.Defn: A systematic and objective measurement of the results achieved by a project/program/policy in order to assess its relevance, the efficiency of implementation, its effectiveness, impact and sustainability.

6

Monitoring vs Evaluation

Monitoring assesses progress in implementation of ongoing programsEvaluation provides a snapshot against some benchmarks at a point in time of programs that may or may not be continuingMonitoring looks at progress relative to targets and assumes there is causalityEvaluation seeks to prove causality

7

Monitoring and Evaluation -RationaleMonitoring

Holds implementers accountable for delivery of inputsProvides basis for corrective actionProvides assessment of continued relevance

EvaluationAccountability - was money well spent?Learning - what could we do better next time?

8

… and an impact on development

Monitoring

Evaluation

Resources are mobilized…

… to undertake activities...

… the direct results of which...

… must have effects...

This chain is based on a series of logical relationships

(if… then) called the logframe

Monitoring and Evaluation, and the Chain of Results

9

Resources are mobilised

… to undertake activities...

… the direct results of which...

… must have effects…

… and an impact on development

building, training, delivery

budget funds

amount of services

provided, processes completed

Health, literacyconsumption,

life expectancy,

poverty

Allocation Inputs Outputs Outcomes Impact

Logframe of a Project or Program

10

What Type of Indicators Are Needed?-MDGs-

1. Eradicate extreme poverty and hunger2. Achieve universal primary education3. Promote gender equality and empower

women4. Reduce child mortality5. Improve maternal health6. Combat HIV/AIDS, malaria and other

diseases7. Ensure environmental sustainability8. Develop a Global Partnership for

Development

11

What Type of Indicators Are Needed? GOAL:Improve the health status of the population

through improved water supply

Impact Impact on living standards –Lower poverty; prevalence of malnutrition5-10 yrs

OutcomesAccess, usage, and satisfaction of beneficiaries - % hh with access to safe water supply; % hh with continuous water supply during the day1-3 yrs

Outputs Goods and services generated -Newly installed or rehabilitated water points;% wastewater treated

Annual

InputsResources (financial, material and human) provided - Real investment in water supply and sanitationAnnual

12

Monitoring the outcome and impact of development across sectors

Inputs

Impact

Outcomes

OutputsFocus on demand-side issues (data for decisions): logframe provides the foundation; largely project/program level: operationally-focused; tend to emphasize M&E at lower end of logframe [inputs/outputs); link to project design/implementation

Reflects a greater emphasis on: country-ownership, long-term capacity-building -also provides greater opportunities for civil society involvement and participatory M&E approaches. Link to PRSP

From project to program –The new development strategy

and its implications for M&E

13

Why Monitor?To ensure best use of scarce resourcesTo make user feedback integral to poverty reduction effortsTo mitigate the standard pitfalls in service provision to the poorTo enforce a concrete link between policy goals and policy inputsTo incorporate the many dimensions of poverty reduction into our decisions

14

What to Monitor

Identify few indicators, measure them well, and use the results for policy makersA prioritized list of input, output, outcome and impact indicators for monitoringDevelop a data collection system to institutionalize a monitoring system: budget and administrative data, facility and other survey data.

15

How to MonitorManage a monitoring system that integrates a variety of different types of information: MIS, surveys and censuses, and participatory exercisesDraw up a monitoring matrix which identifies:

-Data sources for each indicator-Frequency of measurements-The organization responsible for collecting

information.Collect information in a sequence that maximizes the complementaries between different types of data.Outputs:

-Annual progress report-Database for continuous monitoring-Desk studies

16

Features of Good IndicatorsRelevant; measure factors that reflect objectives

e.g. Number of eggs produced vs. Poverty incidenceDirect and unambiguous measure of progress

e.g. % Hh expenditures on health care vs. Immunization coverageVaries across areas, groups, over time; sensitive to changes in policies, programs and institutions

e.g. Life expectancy vs. Infant mortality ratesCan be tracked frequently; not too costly

e.g. Iron deficiency in pregnant women vs. Pre-natal care visits

Consistent with the decision making/budgetarycycle

17

How Many Indicators Should Be Monitored?

There is not a general rule but...It is more useful to identify a small number of indicators, measure them well and use the results for policy-making than to enumerate a long list of indicators, measure many of them badly and not use the results for policy decisions at all

18

At What Level of Disaggregation?Aggregate indicators are useful for giving an overall picture of country situation or cross-country comparisonsHowever, they tend to mask differences across areas, gender, or social groupAppropriate level of disaggregation depends on country conditions and the indicator itself

19

Types of DisaggregationGeographical

Rural/urbanAdministrative units (states, districts, municipalities, etc)Geoclimatic zones

GenderIncome or consumption levelSocially defined groups

20

Experience to DateLack of consistency between goals and indicators for monitoringMultitude of indicators without clear prioritizationUnbalance between intermediate (input/output) and final (outcome/impact) indicatorsInadequate/unclear indicators

21

Know,understand

and learn lessons

Manage, monitor

DecidePeople in charge

Manager,Operational

staff

Researcher

Know, be informed

Public, beneficiaries, contributors

4 mainpurposes

Purposes of an M&E

22

Key M&E Products

Before During At the end Afterwards

To check the design

To improve theimplementation

For accountabilitypurposes

To assessthe impact

Program

StartStart

23

Components of a M&E Strategy

Outcome-based monitoring systemComplemented with systematic impact evaluationCreating a feedback processBuilding capacity for monitoring and evaluationPromoting participation

24

Outcome-based Monitoring System

Setting goals and targets (including establishing the baseline) – where do we want to go?Identifying indicators that can be used to measure progress towards goalsCollecting data - what progress is being made? Providing feedback for decision making –what needs to be changed along the way

25

Monitoring System: Where to Start?

Stock taking:-Who are the actors? What are

their activities? What is their capacity? What are their roles and needs?Review the actions currently undertaken to build or re-enforce capacity.Review of needs in information

26

Monitoring System: What Limitations?

A series of issues are of concern:- Roles badly defined- Lack of coordination- Lack of reliability and relevance of

information- Difficulty in accessing information- Long delays in production of information- Lack of use of the data by the users

27

Example: Monitoring Poverty Reduction

28

Monitoring poverty reduction requires

Measuring povertyUnderstanding poverty reduction strategiesMonitoring outcomes and impacts—goes beyond project-based monitoringEvaluating m&e system and its capacity

29

Measuring povertyPoverty is a multi-dimensional concept—income, consumption, health, education, water, housing, sanitation, capability and empowermentPoverty is also dynamic—vulnerability and coping mechanismPoverty is also gender, caste, ethnic, and geographical based

30

Poverty reduction strategies

Market matters; state matters too – they can’t be substitutes; a right balance is required.Involve the poor in development process—poverty reduction requires both growth and targeted measuresDevelopment strategy needs (a) policies to improve the investment climate to raise income and employment growth, and (b) policies to equip and empower people to participate in growth.

31

Better investment climate and better participation

Macroeconomic stability and opennessGood governance and strong institutionsQuality infrastructureGrowth reduces poverty– but not all growth is equally pro-poorEmpowerment through social, human, and infrastructural investment to enable poor to participate in growth

32

Setting up an M&E system for Poverty Monitoring

1. Defining information needs. Deciding on indicators

2. What tools do we need? What tools do we have?

3. Defining institutional functions (including feedback processes)

4. Defining a 5-10 year work program5. Defining resource requirements

33

Step 2 – Measuring outcomes What tools do we need? What tools do we have?

34

Measuring outcomes - What tools do we need? What tools do we have?

Tools for measuring the extent and depth of poverty (poverty analysis)Tools for monitoring changes over time (poverty dynamics)Tools for comparing differences over space (poverty mapping)Participatory poverty monitoring (listening to the poor)

35

Measuring poverty and data quality

Tools: - A good household survey (there are various options - HBS, LSMS etc.)Measuring poverty is a data hungry exercise (means and distributions)Household Consumption (If you’re going to measure it - do it properly)Sampling and non-sampling errorMonitoring over time compounds the error problem

36

Monitoring changes over timeTools: Administrative data/MIS; Institution-based surveys; household surveysMonitoring leading indicators (Service Delivery)Key attributes of an ‘indicators survey

Simple to executeRapid reportingDisaggregatable to low levels

Core Welfare Indicators Questionnaire (CWIQ)

37

Monitoring Spatial/regional differences

Main source of information on distributional outcomes - household surveys - permit only limited disaggregation.Very large data sources (e.g. census) typically collect very limited information on welfare outcomes. Impute a preferred measure of welfare (e.g. comprehensive real consumption) from household survey into census, using statistical prediction methods

38

What are Poverty Maps?

39

Participatory Poverty monitoring (listening to the poor)

Limitations of statistical informationQualitative and participative approachesComplementary not alternative approaches

40

Data to monitor investment climate and empowerment

Firm and institutional surveysGood governance indicatorsEmpowerment indicatorsCommunity participation indicatorsData on quality of infrastructure Both quantitative and quality information required

41

Step 3 - Setting up a 5-year work program

National household budget survey with social indicators (LSMS), firm and institutional surveysAnnual monitoring survey (CWIQ)Participatory poverty assessments (PPA)Population and other censusConsumer and producer price collectionHousehold panel studyTimely national accountsCurrent public expenditure statistics

42

Illustrative S tructu re o f P overty and W elfare M onitoring S ystem

National statisticsoffice

Surveys, prices,natl.accounts

Line Ministries

Program & input monitoringAdmin. financial data

P.P.A Group

Qualitative studies

University

Panel studies

Welfare Monitoring Unit Welfare MonitoringAdvisory Group

National Poverty reduction committeeSr. Govt. Officials

Step 4-Defining Institutional functions

43

Step 5 - Defining Resource requirements

National budgetsCoordinated donor support through national budgetsStatistical capacity-building trust fund

44

Key messagesMonitoring poverty reduction goals goes beyond selection of indicators and measuring inputs and outputsA range of different monitoring tools are required Poverty reduction monitoring goals should include statistical capacity-building as a core elementMust be a national commitment to funding monitoring and statistical systemsMust ensure existence of appropriate incentives for good M&E?Feedback into budgeting and policy process