A PRESENTATION FOR THE

OVCN INNOVATION & ACTION!

CONFERENCE

If Demonstrating Impact Seems Boring, You’re Doing it Wrong 

Pre - Test

!  When someone says “lets talk about demonstrating impact,” I: !  Get excited and energized. !  Get moderately enthusiastic. !  Picture my long to do list getting even longer, sigh, and say

“Ok, sure. I guess that’d probably be a good thing for me to think about.”

!  Make a mental note to bring my phone to the meeting so I can get caught up on some emails.

!  Think “just kill me now.”

A N D R E W @ T A Y L O R N E W B E R R Y . C A 5 1 9 5 4 6 4 7 8 9

@ T A Y L O R N E W B E R R Y C

Andrew Taylor Contact info

E v i d e n c e I n s i g h t A c t i o n

A Basic Overview of Outcomes Measurement

Acting

Listening Visioning and

Reflecting

Connecting

Program Planning; Strategic Planning;

Program Design and

Implementation

Consultation Partnership Building

Achieving Buy-in

Needs Assessment Outcome Measurement

Program Evaluation

Setting the stage for good program monitoring

1.  Get buy-in first. 2.  Make sure you and your partners

understand your program and its outcomes clearly and consistently.

3.  Remember that the key is using the data, not gathering the data. You don’t necessarily need to gather more data.

E N G A G I N G W I T H S T A K E H O L D E R S

1. Get Buy-in First

Thinking About Use before You Start

! Who do you want to act on the basis of your outcome measurement findings?

! How do you want them to act?

Why are people not interested in outcome measurement?

We tend to assume:

But the real reasons may be:

They don’t have the training to understand

They know that the big decisions aren’t really based on outcome measurement findings (despite the rhetoric).

They aren’t interested in learning

They see measurement as complicated, dry and boring. They don’t see how it will help them to learn.

They’ve got something to hide

They worry that people won’t understand the context. They are afraid of being unfairly judged.

They are too busy

They see no tangible “payoff” for the significant time investment.

Exercise: What’s in it for them? Stakeholder How might evaluation

help them? How might evaluation make their lives difficult?

Board Member

Manager

Front-line Staff Member Participant in a Program Funder

Partner Organization Volunteer

C L A R I F Y I N G T H E O R I E S O F C H A N G E

Make Sure You Understand the Program Clearly and Consistently

Theories of Change

! The beliefs or assumptions that inform the design of a program or an intervention

! In particular, those beliefs and assumptions about how change happens, and how the intervention will lead to change.

! Sometimes theories of change are very explicit and structured, but sometimes they are implicit and emergent.

The importance of short-term outcomes

Train volunteer

coordinators Program Activities

Long-Term Outcome or Goal

Publicize volunteer positions

Create a more engaged, connected community

The importance of short-term outcomes

Program Activities

Long-Term Outcome or Goal Create a more engaged, connected community

More diversity in volunteers

Volunteers have more meaningful experiences

More people

volunteer

Members use other

best practices

Members manage

volunteers better

Members are Stronger Short-Term Outcomes

Train volunteer

coordinators

Publicize volunteer positions

Volunteers are more engaged

Breaking Down the Complexity

Did our program have an impact???

What were we trying to change? (Long-Term Outcome Objectives)

How were we planning to make that contribution? (Activities)

What sorts of things would we see if the expected change was happening? (Indicators)

What particular contribution were we going to make to that change?

(Short-Term Outcome Objectives)

How can we document those observations in a systematic way?

(Methods)

Problems with Logic Models

!  They are dense !  They are abstract !  They talk about change in a very linear, mechanical

way !  They are good for getting your own thinking

organized, but not so good for communicating with outsiders, especially if those outsiders ALSO have a complex program logic model that they want to communicate with YOU.

Hello. My life goal is to get married and have kids. I am currently pursuing several

short term outcomes related to this goal, including meeting a nice person, getting in better physical shape, moving out of

my parents’ house, and choosing names for the kids.

Hello. I am new to town. I don’t know many people, or many places to go, and I’m

feeling a bit lonely. My outcome objectives are to

make some friends and get out more. My goal is to feel more connected to my new

community, although I recognize that this goal may

need to be made more measurable.

Yes! There’s a great place downtown. Want to go after

work?

Know anywhere good to go for coffee

around here?

Sounds great! I’ve been wanting to get

to know that neighbourhood. Me too, actually!

I’d love to live downtown someday.

What are the most important shared short-term outcomes for volunteer centres?

What are the short-term outcomes we are well positioned to achieve that “plug in” to the shared

“collective theory of change?”

What are the shared outcomes that the community has identified as important? What is the shape of the

“collective theory of change?”

The Really Useful Idea

Identify short-term outcome objectives that are

within your control, measurable and achievable,

and use a theory of change to explain how these

short-term achievements contribute to the

achievements of collective outcomes over time.

What’s the big, shared change that matters to

the community?

What do we do? What do we do?

What do we do? What do we do?

M A K I N G S M A R T U S E O F S I M P L E D A T A

3. Remember that the key is using the data, not gathering the

data.

What are Indicators?

! Bits of information that provide part of the answer to one of your questions

! Things you can see or touch or hear – things that are observable in the world and don’t involve ‘interpretation’

! Are often numbers but can also be (e.g.) stories, quotations, examples, pictures

Indicators are a Useful Idea Because...

! They help to break down the complex task of “outcome measurement” into manageable chunks

! They help others to understand what you mean in practical terms when you talk about a particular outcome

! They help you build up a strong meaurement plan by combining different kinds of information from several sources

The Notion of Triangulation

A Quick Way to See if You’ve got the Right Kind of Indicators (from Friedman, 2005)

Quantity Quality

How much work did we do? # of people who attended the workshop # of handouts created # of workshops run #

How well did we do it? % of invited people who attended % % %

# # # #

% of participants who are better prepared to complete outcome measurement plans % of measurement plans that are useful % of plans that are manageable and practical

Is anybody better off?

Different kinds of research on social programs

Measuring outcomes through ongoing program monitoring

! Carried out by program staff and volunteers on a routine basis

! Simple, easy to use methods ! Focused on short-term outcomes ! Usually measures a few basic things

well, rather than many different things

Measuring outcomes through ongoing program monitoring

! Ideally, becomes a seamless element of program delivery

! Periodic analysis of data ! Focuses on ongoing improvement of

programs

Outcome measurement planning sheet

What are the questions you need answered?

What indicators can you use from information you’ve already got?

What additional information do you need?

How can you get it?

These may be questions about outcomes, but they may also be questions about process, lessons learned, etc.

Be as specific as you can!

Think stories and examples and also quantitative data

What is the simplest, least intrusive method?

S H A R I N G F I N D I N G S

Making Your Evaluation“Worth It:”

Its all about “messaging”

!  The key to powerful communication about evaluation findings is to know: !  Who your audience is !  What that audience is already passionate about, and how that

passion connects to your work !  How you want them to act with the knowledge you give them !  How you can present your information in the way that

maximizes the chances of them acting

Acting

Listening Visioning and

Reflecting

Connecting

Program Planning; Strategic Planning;

Program Design and

Implementation

Consultation Partnership Building

Achieving Buy-in

Needs Assessment Outcome Measurement

Program Evaluation

Outcomes Measurement: Why do it?

The challenges

!  Steep learning curve !  Time consuming !  Different expectations from different funders

!  Lack of control over what you measure and how you measure it !  Funders that demand certain indicators but don’t explain what

they think those indicators mean !  Differing jargon across funders !  Knowing how much measurement is “enough” and what type is

expected !  Poor quality of data !  Lack of obvious “payoff”

The (potential) benefits

!  More mindful practice !  more engaged, satisfied staff and volunteers !  A better sense of “where you fit” in the system of services and

supports

!  Improved programming; more benefits for the people you serve

!  Improved capacity to explain the value of your work to donors, volunteers, board members, and the general public

Q U E S T I O N S ? C O M M E N T S ?

Yeah, but ….

Where do I start?

1.  Get buy-in first. 2.  Make sure you and your partners

understand your program and its outcomes clearly and consistently.

3.  Remember that the key is using the data, not gathering the data. You don’t necessarily need to gather more data.

Post - Test

!  When someone says “lets talk about demonstrating impact,” I: !  Get excited and energized. !  Get moderately enthusiastic. !  Picture my long to do list getting even longer, sigh, and say

“Ok, sure. I guess that’d probably be a good thing for me to think about.”

!  Make a mental note to bring my phone to the meeting so I can get caught up on some emails.

!  Think “just kill me now.”