Assessment of Outcome Mapping as a tool for evaluating and monitoring support to civil society organisations

EASY-ECO Vienna 2008Stephen Powell, Joakim Molander & Ivona Čelebičić

proMENTE social research, Sarajevo

www.promente.org/om

Who are we

Steve Powell & Ivona ČelebičićproMENTE social research, Sarajevo

Joakim Molander: at the time of the evaluation, First Secretary at the Swedish Embassy, Sarajevo

Report: www.promente.org/sida2eng

www.promente.org/om

Background

Govt. in B&H barely functional Civil society fills the gap? Support people and networks How to evaluate? OM

www.promente.org/om

Evaluation TOR

1. Explorative evaluation of six civil society projects2. Sida-funded civil society programming: lessons

on sustainability3. Exploration of OM as a tool

www.promente.org/om

Messages

1. OM worked well for civil society evaluation2. OM plays well with other approaches3. Sustainability ↔ focus on “key players”4. OM can be part of a patchwork, rather than monolithic, approach to M&E

www.promente.org/om

Our project in B&H: 1 donor, 3 framework partners, 6 implementing partners, 6 projects: 2007Framework organisation

Partner organisation Evaluated activity

Kvinna til Kvinna

Zenski Centar Womens political lobby

Most Village activities (including round tables in towns)

Olof Palme International Center

Helsinki Citizens Assembly (HCA) – Academy for political leaders)

Academy for political leaders

Civil Society Promotion Center – GROZD, "Citizen in action"

"Citizen in action"– Project of community-based advocacy campaigns for solving priority citizens problems from "Civic Platform for 2006 Elections". "Local Government leadership building activities"

Swedish Helsinki Committee for Human Rights

BiH Press Council (funded by SHC)

Work with judges and journalists on Press Code

Vasa Prava "Improvement of access to justice – Raising public awareness on access to rights, legislative changes and legal procedures in BiH

www.promente.org/om

Intentional design Outcome & performance monitoring

Evaluation

1. Vision

2. Mission

3. ...Boundary partners

4. ...Outcome challenges

5. ...Progress markers 9. Outcome journals

6. …Strategy maps 10. Strategy journal

7. Organisational practices 11. Performance journal

8. Monitoring priorities 12. Evaluation plan

OM: 12 steps

Helping the implementing partner to learn

Internal M &E

Considering all the dimensions of strategy

Focus on outcomes

www.promente.org/om

Method: timeframeevaluation time-frame captures only a small slice of project implementation

Framework partner support: several years

Implementing partner project: several years

Duration of project activities being evaluated: Jan-Dec 2007

Research time-frame: May-Nov 2007

Bas

elin

e

Fina

l ass

essm

ent

www.promente.org/om

Method: OM Special use of OM for external evaluation

1. Mission, vision, progress markers, outcome challenge2. Assessment of planning strategies using strategy maps3. OM questionnaires4. OM interview with boundary partners (+ relevance interviews)5. OM interview with implementing partners (+ relevance interviews)6. Writing evaluation-start Outcome Journals 7. Confirming evaluation-start Outcome Journals8. OM questionnaires (same as evaluation-start)9. OM interview with boundary partners (on the basis of evaluation-start interview) (+ relevance

interviews)10. OM interview with implementing partners (on the basis of evaluation-start interview) (+

relevance interviews)11. Assessment of management progress using existing strategy maps12. Mission and vision: still relevant? Changed focus?13. Gathering contextual information and intervention timeline

Baseline

interimfinal assessm

ent

www.promente.org/om

Message 1: OM worked well with civil society evaluation

Most organisations succeeded in redefining changes in a small group of boundary partners as the main component of their vision

NGOs enthusiastic!? Focus on contribution is a big relief Gives richer feedback Strategy maps inspire and organise thinking about different dimensions of

planning OM standard method and questionnaires, (and interviews?) showed projects

making progress towards their vision Consider problem-based rather than vision-based programming. Vision is

not always about boundary partner change

www.promente.org/om

Message 2: OM plays well with other approaches

www.promente.org/om

2a: Qualitative methods

Gained a lot of additional information with systematic content analysis of independent “relevance interviews”– “A good part of the population is used to some sort of humanitarian aid,

some sort of social help, mercy, call it what you will. This means that nobody has to work and, at the same time, they get something. We can set our sights lower but we don’t need to invest anything.”

www.promente.org/om

2b: quantitative analysis

For four projects, a customised questionnaire was made on the basis of progress markers.

Questionnaire filled in at baseline and again at final assessment Results from around 100 baseline and 100 final assessment

questionnaires compared on a per-project basis. Validation of OM evaluation methods?

– Less clear progress with the least educated

www.promente.org/om

2b: quantitative analysis

Results for one project: average scores on progress marker questionnaires at baseline at evaluation end

Small but significant overall improvement

Corresponds to information from OM interviews & journals

Women start lower than men and improve more than men

In spite of a possible tendency to “raise the bar” during the project “soft” interpretation

0

0.5

1

1.5

2

2.5

3

m ž

pre

post

male female

www.promente.org/om

2c: implementing partners and boundary partners do not agree on “difficulty” of progress markers

Love to see / like to see / expect to see

Average baseline score for one project

involve boundary partners in planning!! – especially the poor

www.promente.org/om

2d: problems with OM + RBM

Avoiding a double workload: OM for monitoring OM for planning and reporting Better planning: OM could/should be adopted by donors

from the project application stage (or as a hybrid) Donors have to want to help organisations to learn, at

the cost of demonstrating effectiveness

www.promente.org/om

Message 3: Sustainability ↔ focus on real people!

www.promente.org/om

3a: differentiate your boundary partners

Especially in civil society, boundary partner groups can be usefully divided into "less powerful", "potentially powerful" and "powerful".

Even within one group, distinguish clearly between partners who are at different levels of development towards, or agreement with, the outcome challenge.

www.promente.org/om

3b: focus on “key players” = boundary partners or implementing partners?? They believe that change is possible through the

efforts of individuals. Civil society not a mass movement! Handful of

key (“career”) activists and organisations They have “first name and family name” and are

not interchangeable or easily replaceable; yet invisible on paper

www.promente.org/om

3c: A taboo: motivation Why would these people want to play along?

Money a taboo? If we aren’t using sticks, do we have juicy carrots?

Careers advice for activists? Business support for NGOs?

www.promente.org/om

3d: Another taboo: personality

Effective activists are not necessarily effective bureaucrats (Easterly). Should they be?

Personality is hard to change Differential skills are needed (talking to parliament, then villagers)

www.promente.org/om

Message 4: OM can be part of a patchwork, rather than monolithic, approach to M&E

www.promente.org/om

4a: the monolithic approach

RBM: killing the birds of project and program planning, control, monitoring and evaluation with one stone?

Impacts …

outcomes …

outputs …

Program

Project BProject A

Causation

Data aggregation

– Control: gathering data via a chain in which every link has a vested interest in lying

– Monitoring: ticking boxes at the expense of strategic thinking and organisational learning

– Evaluation: Can you calculate impacts by aggregating outputs? Are NGOs competent to measure outcome and impact directly? Is it their job?

www.promente.org/om

4b: OM is only part of a (patchwork) solution

Better control: release NGOs from exhausting box-ticking and implement fair random checking of outputs.

Better planning and monitoring: use OM where appropriate Better management: in exchange, require elements of strategic

planning and organisational learning. Better evaluation:

– Demonstrate donor-relevant impact: commission independent, external investigations of society change and why it happens/happened

– Sharpen the focus: help NGOs to do empirical explorations of a selection of specific issues that really interest them

OM isn’t everything