v measuring impact kristy muir stephen bennett enactus november 2013

13
MEASURING IMPACT Kristy Muir Stephen Bennett ENACTUS November 2013

Upload: walter-armstrong

Post on 17-Dec-2015

220 views

Category:

Documents


1 download

TRANSCRIPT

MEASURING

IMPACT

Kristy MuirStephen Bennett

ENACTUS November 2013

WHY MEASURE?Does your or other social enterprises really make a difference?

How do you know? How can you prove it?

Why bother?

Purpose of social enterprise: balancing commercial strategy with social environmental or other public benefits

Other benefits from measuring impact• To understand your impact: determine the ‘merit, worth or

significance’ of something (Scriven 1967)• To determine the difference between model/enterprise/idea and

implementation failure/success • To inform and improve services/ideas/enterprises

Measuring Impact

2

There are a range of ways to measure programs/policies/interventions and there are hierarchies of quality /

3

MEASUREMENT THEORIES

Measuring Impact

Definition

"the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of ... Programs” (Rossi and Freeman, 1993)

The systematic collection of information about the activities, characteristics, and outcomes of programs that specific people use to reduce uncertainties, improve effectiveness, and make decisions regarding what those programs are doing and affecting’ (York 2000) [our emphasis]

Mullen et al. (2005)

4

MEASUREMENT THEORIES

Measuring Impact

Mullen et al. (2005)

Outcome evaluations study the effects of the program on participants

Process evaluations assess how the program works: establishment and implementation; facilitators and barriers to effective ways of working

Formative and summative evaluation

• Formative: help businesses, policy makers and practitioners refine and develop the process through which the program is implemented

• Summative: measure the impact of the program

Economic evaluation examples

• Social Return on investment

• Cost-benefit analysis

• Cost-effective analysis

• Is your approach to measuring impact ethical?

• How have you incorporated research ethics – consent and data collection

• Impartial evaluator

• Operational challenges

• Data availability

• Attribution

5

ETHICS AND MORALS OF EVALUATION

Measuring Impact

• Social enterprise models

• Geographic location

• Diverse client demographics

• What data is available?

• Policy/legislative framework

• Social, cultural, political, religious

• Comparative programs

• Existing evidence of intervention

6

CONTEXT

Measuring Impact

• What is best approach for collecting evidence based on the contextual factors for your organisation?

• What contexts do you need to consider in measuring your impact?

(Coatsworth, 2002, reproduced in Lippman 2004)

Evaluators of complex social programs should:

• Develop a theory

o E.g. marshmallow test & impulsive behaviour / impulse control

• Develop a hypothesis based on that theory (what might work for whom under what circumstances)

o If we intervene early to teach children impulse control we can decrease the likelihood they will end up in gaol. Intervention will involve…

• Conduct research to test the hypothesis

• Make conclusions based on the hypothesis and the findings

• Feed back into the development of theory

7

THEORY TO PRACTICE

Measuring Impact

What? A ‘program logic’ is a systematic, visual representation of the underlying assumptions of a planned program. A results logic illustrates why and how a program is presumed to work.

These models present a sequential, interactive account of how inputs will lead to outputs, which will result in the desired outcomes.

Designing a log frame/theory/hypothesis

1. What is the major outcome anticipated? Describe the problem your social enterprise is attempting to solve. What is your social mission?

2. What other goals or sub-outcomes anticipated or desired (which will help work towards the major outcome)? What are the near and long term desired results?

3. What’s the theory behind why the program will work? What broad systems and structures [or programs] are in place to achieve these outcomes? What are the successful strategies that have helped other organisations achieve the outcomes you desire? What other evidence is available?

4. What outputs from these systems and structures might contribute to the outcomes?

8

LOGIC FRAME APPROACH

Measuring Impact

9

A LOGIC MODEL

Measuring Impact

Inputs/Resources Activities Outputs Outcomes Impact

What goes in What happens Immediate results Short and long term results

Effects on root causes – sustained change

• Resources• Equipment• Knowledge/

expertise

• What does your organisation/ enterprise do?

• How much activity occurs?

• For each activity what are the results?

For example:• Improved living

conditions• Increased income

For example:• Change in poverty• Change in social

norm & attitudes

W.K. Kellog Foundation (2004), Ebrahim & Rangan (2010)

Logic model evaluation method

• Map appropriate methods against the program logic

• What information could or needs to be collected

• Client segmentation

• What population level information is available

Lead to [MAJOR OUTCOME]

Lead to [OUTCOMES]1 2 3 4 5

Which in turn will result in [OUTPUTS]1 2 3

And through ... [WHAT OTHER ASSUMPTIONS ARE IMPORTANT RE PROGRAM STRUCTURE]

By providing ... [WHAT AND HOW]

Supports ... [WHO is being supported]

Model [WHAT]

For example…

What is the business model for your social enterprise?

- Purpose

- Objectives and strategy & goals

- Theory/ research

- Finance model

How will you measure the social impact of your social enterprise?

12

WHAT DOES THIS MEAN FOR ME?

Measuring Impact

13

Q&A

Measuring Impact