Social Development DepartmentThe World Bank

Participatory Approaches in

Impact Evaluation

Asli GurkanSocial Development DepartmentWorld BankDubai – Impact Evaluation workshopMay 31-June 4

Demand for Good Governance

Setting the Context Heightened attention to governance issues at

the World Bank since adoption on GAC strategy Increasing emphasis on outcomes/impact to

enhance development effectiveness More attention to transparency, access to

information, citizen-participation at all levels in World Bank operations (mandatory with Investment Lending reform)

Key focus of Social Development department-strengthening demand-side of governance (including PM&E methods)

Demand for Good Governance

What is demand for good governance (DFGG)?

The ability of citizens, civil society organizations and other non state actors to hold the state accountable and to make it responsive to their needs

In turn, DFGG enhances the capacity of the state to become transparent, accountable and participatory in order to respond to these demands

DFGG mechanisms can be initiated and supported by the state, citizens or both

but very often they are demand-driven and operate from the bottom-up.

The people have a right to know, a right to question, a collective Constitutional right to receive an answer.” Aruna Roy, MKSS Rajasthan, India

Demand for Good Governance

Key Demand-side mechanisms to address governance challenges

Transparency/Access to InformationConsultation/ParticipationGrievance Redress MechanismsThird party Monitoring/ Independent

verification of outcomes Participatory Impact Assessments

Demand for Good Governance

6

Possible methods/tools under PM&E:

Visual techniques Activity monitoring chart Participatory Rural Appraisal Citizens report cards Community scorecards SARAR (participatory problem solving tool) Participatory Impact Assessments

Demand for Good Governance

7

What’s Participatory Impact Assessment?

“Involves the adaptation of participatory tools combined with more conventional

statistical approaches specifically to measure the impact of humanitarian

assistance and development projects on people’s lives.”

Source: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners

Demand for Good Governance

8

Participatory versus Conventional IE

Conventional IE Participatory IEWho? External experts, IE

specialistsStakeholders, including communities and project staff; service providers and users

What? Predetermined indicators, to measure impact

Indicators identified by stakeholders, to measure impact

How? Questionnaire surveys, by outside “neutral” evaluators, distanced

Simple, qualitative and quantitativemethods, by stakeholders themselves guided by facilitator

Why? To make project and staffaccountable to funding agency

To empower stakeholders to takecorrective action

Method?

Pre-determined AdaptableSource: Adapted from Deepa Narayan, World Bank.

Demand for Good Governance

At the project level-3 key questions…

1. What changes have there been in the community since the start of the project?

2. Which of these changes are attributable to the project?

3. What difference have these changes made to people’s lives?

Demand for Good GovernanceDesigning and

Implementing PIA: StepsStage 1: Define QuestionsStage 2: Define the geographical and time-limits of

the projectStep 3: Identify locally defined impact indicatorsStep 4: Decide on ranking/scoring methods

on and testing sampling methodsStage 5: Choose Sampling MethodsStage 6: Assess project attributionStage 7: TriangulationStage 8: Feedback and verify results with

communitySource: Feinstein International Center: Participatory Impact Assessment” Guide for Practitioners

Demand for Good Governance

Step 2, Tool #1: Defining the project boundary: participatory mapping

A map of Zipwa Site, Zimbabwe Community members drawing a map in the sand

Demand for Good Governance

Step 2- Tool # 2: Define the project period by timelines established by the communities

Creating a timeline--- Identify a Knowledgeable person (or persons) in a community- Ask them to describe the

history of the community.- - In many rural

communities, such descriptions usually refer to key events such as drought, periods of conflict or disease epidemics

- The project start and end time should be related to these key events.

Demand for Good Governance

Step 3: Indicators-identified by communities themselves…

Ex: Drought projects in Zimbabwe and Niger

Impact indicator by project M&E specialist

Impact indicators by beneficiaries

increased crop production The ability to pay for school fees using project derived income (education benefits)

dietary diversity The ability to make home improvements Improved skills and knowledge from the projects training activities

Improved social cohesion

Tips for practitioners: Make sure to capture the views of different groups ofpeople within the community. (Women will often have different priorities and expectations of project impact than men.)

Demand for Good Governance

project participants identify all the food sources that contribute to thehousehold food basket.

Practitioner Tips- Where informants are literate you may choose to simply write the name ofThe indicator on a card.

Step 4: Methods for Scoring/EvaluationTool #1: Scoring of Food sources using counters- Evaluating the impact of a community garden

Demand for Good Governance

Step 4-Tool # 2: Impact calendars-post-harvest food balance

Monthly household utilization of the harvested maize until depletion (using 25 counters)

exercise -done with project participants for the agricultural year before and after the project and again for the agricultural year. The exercise then repeated with community members who had not participated in the project

Demand for Good Governance

Challenges with community-level PM&E approaches (from Bolivia

and Nepal) Clash of incentives: pressure from donors to ‘prove’ impacts

vs. adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations

Resistance from the project teams to changing their existing M&E practices

Project deadlines prevent project-staff from establishing a consistent PM&E practice

Availability of PME expertise within the field staff to facilitate the PM&E exercises, Insufficient transfer of community-level PME skills to interest

groups and grassroots organizations Lack of sufficient training/capacity-building programs

Human resource problems and lack of capacity in analyzing qualitative data and reporting results.

Demand for Good Governance

Possible recommendations to improve PM&E activities

Develop ready-to-use templates, a detailed Community Researcher manual to improve the research data and reports.

Conduct regular follow up visits to each case study sites to review the work of the Community researchers and provide feedback, advice and support

Identify mentors, encourage the community researchers to phone their mentors on a regular basis to share their progress and any problems they had.

Encourage the community researchers to contact each other regularly to share their experiences and reports. Promote “peer-to-peer learning’

Demand for Good Governance

Key messages Keep participatory evaluation methodologies

simple and practical

Develop your methods, standardize and repeat.

the more repetitions, or the larger the sample size, the more statistically reliable the results will be.

Remember to field test your methods with community members before the assessment–most methods look easy on paper but require fine tuning once you start to use them in the field.

Demand for Good Governance

Participatory Impact Monitoring Booklet I- V GTZ

World Bank Participation and Civic Engagement Website:

Feinstein International Center-Tufts University Participatory Impact Assessment: A Guide for Practitioners

Useful links/Resources

World Bank Social Accountability Sourcebook

Demand for Good Governance

Thank you