vMEASURING IMPACTAnne Kazimirski, Head of Measurement & Evaluation, NPC

Connect@Evolve, Monday 15th June 2015

NPC: TRANSFORMING THE CHARITY SECTOR

2

NPC works at the nexus between charities and

funders

Charity

SectorFunder

Increasing the impact of charities

eg, impact-focused theories of change

Strengthening the partnership

Eg, collaboration towards shared

goals

Increasing the impact of funders

eg, effective commissioning

Consultancy & Think tank

THE FOUR PILLARS APPROACH TO

MEASURING IMPACT

3

Map your

theory of

change

Prioritise

what you

measure

Choose

your level

of evidence

Select your

sources and

tools

Effective measurement

framework developed

Strategic vision

Leadership

Case for impact measurement

DEVELOPING AND USING

A THEORY OF CHANGE

WHAT IS A THEORY OF CHANGE?

Links activities intermediate outcomes final outcomes

A description of how activities lead to outcomes

5

- Clarifies what the activities aim to achieve and how

- Provides the case for why achieving intermediate outcomes is important

- Provides a structure for identifying what can be measured

HOW TO REPRESENT A THEORY OF CHANGE

6

Planning Triangle Logic Model Outcomes Chain

However you represent your theory of change, it should be supported by a written description.

7

Children’s emotional

resourcefulness

improves

Counselling

Clients’ ability to support

their children's healthy

development improves

Clients’ emotional or

psychological

difficulties decrease

Mother / Child

interaction

improves

Clients’ capacity

for self care

increases

EXAMPLE (SIMPLIFIED) THEORY OF CHANGE

MOTHERS’ COUNSELLING SERVICE

Activities Intermediate Outcomes Final outcome

PRIORITISING OUTCOMES

PRIORITISE OUTCOMES THAT:

• Are directly influenced (rather than indirectly supported)

• are important / material to the mission

• are not too costly to measure

• will produce credible data

9

CHOOSING THE RIGHT

LEVEL OF EVIDENCE

COUNTERFACTUAL

• Comparing the world with your

organisation in it with what the

world would be like without it.

• Control group

• Attribution

11

LEVELS OF EVIDENCE

12

Randomised

control trial

Anecdotes /

quotes

Before and

after survey

Self-reported

change

Case

studies

Control

groups

Credibility

Basic Advanced

SELECTING DATA

SOURCES AND TOOLS

• Quantitative data (numbers)

• Statistical estimates

• Prevalence of views, attitudes and experiences

• Admin data or questionnaires (paper, web, etc.)

• Qualitative data (words)

• Detailed understanding

• In-depth interviewing (telephone or face-to-face)

• Observation

• Stakeholders’ views

Proportion of beneficiaries

whose outcomes have

improved, and by how

much.

What did beneficiaries

think, did it make a

difference to them? How?

How could it have been

better?

DIFFERENT TYPES OF EVIDENCE

EVALUATION TRAPS

15

Collect data that matters, and work together

Don’t force

squares into circles

& don’t collect

arbitrary data

INSPIRING IMPACT: MEASURING UP! AND

THE IMPACT HUBMeasuring Up!

• online, step-by-step self-assessment tool

• looks at the way you plan, evidence, communicate and

learn from the difference your work makes

The Inspiring Impact Hub

• pulls together resources relevant to improving impact

practice

• enables users to search and filter results according to their

needs

inspiringimpact.org

16

ACTIVITY – BENEFITS AND CHALLENGES

17

Two questions to discuss:

• What are the benefits of measuring impact?

• What are the challenges of measuring impact?