how evaluation helps you design and implement programs ... · program design & evaluation –...

Program Design & Evaluation – Part 2 SAFE KIDS Conference

June 20, 2013

Carolyn Cumpsty Fowler, PhD, MPH

[email protected] 1

How Evaluation Helps You

Design and Implement

Programs That Work for Your Audience

Carolyn J. Cumpsty Fowler, Ph.D., M.P.H.


June 20, 2013


[email protected] 2

do you do what you do?


June 20, 2013


[email protected] 3

The Public Health Approach … again

1. Surveillance & systematic data collection

Assess the problem.

2. Identify risk and protective factors

Identify causes.

3. Develop & evaluate interventions

Develop programs and policies that work.

4. Implementation. Scale up effective policies & programs.

Implement & Disseminate

Original graphic from http://www.who.int/violenceprevention/approach/public_health/en/index.html - adapted by Carolyn Cumpsty Fowler 2012

Why Do So Many Programs Fail?


June 20, 2013


[email protected] 4

The Public Health Approach … again


Assess the problem.


Identify causes.


Develop programs and policies that work.


Implement & Disseminate


Modifiable barriers to implementation

of effective programs:

Overly broad problem definition

Incomplete diagnosis

Unrealistic goals

Poorly defined objectives

Inadequate implementation planning

Working in a vacuum

Turf wars

Planning gaps

Cruise control and tunnel vision

Absent or inadequate evaluation


June 20, 2013


[email protected] 5

Which is the most critical modifiable

barrier to implementation of effective

programs?



Unrealistic goals



Working in a vacuum

Turf wars

Planning gaps



Which is the most critical modifiable

barrier to implementation of effective

programs?



Unrealistic goals



Working in a vacuum

Turf wars

Planning gaps




June 20, 2013


[email protected] 6

When is evaluation inadequate?

What is

expected to

change

Outcome

This graphic is adapted from one originally developed with Ronda Zakocs, PhD, MPH

Idea You invest a lot of time, energy and resources


June 20, 2013


[email protected] 7

What is

expected to

change

Outcome

Outcome

Evaluation


Idea You invest a lot of time, energy and resources

What is

expected to

change

Outcome

Outcome

Evaluation


Inadequate Evaluation: Too Little & Too Late

At this point, we

have very little

control over the

outcome

You invest a lot of time, energy and resources


June 20, 2013


[email protected] 8

What is

expected to

change

Program Planning Implementation Outcome

Outcome

Evaluation



At this point, we

have very little

control over the

outcome

?...?...?

Assess

Plan

Develop or modify

activities

Implement

activities

What is

expected to

change


Outcome

Evaluation



At this point, we

have very little

control over the

outcome

Problems Always Happen

Evaluation used during planning and implementation increases your

ability to influence the outcome


June 20, 2013


[email protected] 9

Assess

Plan

Develop or modify

activities

Implement

activities

What is

expected to

change


Formative Evaluation Process Evaluation Outcome

Evaluation



At this point, we

have very little

control over the

outcome

Problems Always Happen

Evaluation used during planning and implementation increases your

ability to influence the outcome


June 20, 2013


[email protected] 10

Panel on the Understanding and Control of Violence

Understanding and Preventing Violence, 1993

“The only way to determine if

something works is to try it, in a

way that lends itself to reliable

evaluation”

Evaluative Thinking

Assess the

Problem Identify the

“Causes”

Design & Evaluate

Programs &

Policies

Implement &

Disseminate

Programs & Policies


June 20, 2013



Mission Critical – Informed Beginnings

If the first button of one’s coat is wrongly

buttoned, all the rest will be crooked.

Giordano Bruno (1548-1600)

Assess the

Problem Identify the

“Causes”

Design & Evaluate

Programs &

Policies

Implement &

Disseminate

Programs & Policies


June 20, 2013



A [person] should look for what is, and not for

what [s]he thinks should be.

Albert Einstein


June 20, 2013



The Golden Rule of Evaluative Thinking:

ALWAYS Question Assumptions

Cartoon courtesy of i-heart-god.com

The Evaluative Thinking Process as Coach

“Most of the value in a logic model is in

the process of creating, validating and

modifying the model … The clarity of

thinking that occurs from building the

model is critical to the overall success

of the program.”

W. K. Kellogg Foundation Evaluation Handbook 1998; p. 43.


June 20, 2013



The Public Health Approach …with evaluation questions


Have we asked the right questions? Have we defined the real problem? Have we checked?


Have we identified modifiable critical factors: causal, predisposing, enabling & reinforcing?


What works and for whom? Who should deliver? What are the costs, risks, benefits, & potential to sustain?


How do we take effective & promising interventions to scale, & demonstrate impact?



June 20, 2013



Are there other

important

questions that

evaluative

thinkers can

identify?

Always consider the intervention context

Factors that will support or inhibit our program

– resource issues

– physical environmental issues

– social environmental factors (social ecological)

Can you achieve this program; if yes, should it?

Where

we are

Where we

hope to be Change process


June 20, 2013



We can’t afford NOT to do evaluation!

(if we have limited resources)

Evaluation may help you discover

opportunities that help you implement

programs that work for your audiences

Program

Delivery

Injury

Prevention

Impacts

Problem

Analysis

Problem

Identification

Program

Design

Social and Political Contextual Influences

Capacity &

Infrastructure

Epidemiology & Assessment, Program Design, Implementation, Evaluation, Information-

Sharing, Human & Other Resource Issues, Funding, Partnerships, Socio-Political Influence

-Internal

- External

Graphic developed by Carolyn Cumpsty Fowler


June 20, 2013



Expanded Program Cycle Model developed by Carolyn Cumpsty Fowler PhD, MPH with input from Dr. Ronda Zakocs (2004)

The Expanded Program Cycle:

The set of activities involved in initiating,

planning, implementing, evaluating, and

improving programs.

Problem ID, Definition,

& Measurement

Monitoring of

Program Quality

Integrity

of Diagnosis

Reactive, Adaptive, Proactive

Program Responses

Intervention

Selection Strategy

Dissolution or

Institutionalization

Plan Quality

Resource

Mobilization


June 20, 2013



The Expanded Program Cycle can be viewed as a series of stages – each

with a desired outcome/product

Critical, evaluative thinking used throughout each step of the program

cycle, can serve as a program coach and build program capacity.

Resource

Mobilization

Problem ID, Definition,

& Measurement

Monitoring of

Program Quality

Integrity

of Diagnosis

Reactive, Adaptive, Proactive

Program Responses

Intervention

Selection Strategy

Dissolution or

Institutionalization

Plan Quality

Expanded Program Cycle Model developed by Carolyn Cumpsty Fowler PhD, MPH with input from Dr. Ronda Zakocs (2004)

E-valu-ation

Extremely Valuable Information

can provide


June 20, 2013



“I have not failed.

I have found ten thousand

ways that don’t work”

Thomas Edison

1847-1931

A multi-layered evaluation strategy is

needed to shed light on the change

process

Once we really believe this, we are more likely to prioritize and do evaluation


June 20, 2013



Many people undervalue evaluation

.. often because they don’t know what they’re missing

Where there’s a will, there’s a way

People who really want to do evaluation,

find ways to do it.

Evaluation may pay its own way

– $ invested in evaluation may save $$$ later

on in the program

– The earlier we start, the more we can save.

Evaluation can save your life – literally

and figuratively


June 20, 2013



Evaluation lessons from the fire department

“Size-Up”

The dangers of unfamiliar territory

The dangers of very familiar territory

The critical importance of sharing lessons learned Baltimore County Fire Department in action (2009) Photograph by Pete Hammond.

“A journey of a

thousand miles

begins with a single

step”

Lao-tzu (604 BC-531 BC)

Injury prevention

is an urgent

challenge but ..

don’t be

tempted to rush

into action


June 20, 2013



A journey of a thousand miles begins with a single step Lao-tzu 604 BC-531 BC

A new translation:

“A journey of a thousand miles begins at the spot under

one's feet, therefore deal with things before they happen;

create order before there is confusion”

As you embark on your injury

prevention program planning journey,

remember that evaluation is your

friend …and guide.


June 20, 2013



For Everything You Do

To Keep Kids Safe!


[email protected]

443-287-0541

Johns Hopkins Center for Injury Research and Policy: www.jhsph.edu/InjuryCenter

Valuable free web resources:

http://ctb.ku.edu

http://www.uwex.edu/ces/pdande/index.html


June 20, 2013



More excellent resources:

North Carolina Department of Health & Human Services (2011) Community Health Assessment Guide Book** http://publichealth.nc.gov/lhd/cha/docs/CHA-GuideBookUpdatedDecember15-2011.pdf

W. K. Kellogg Foundation www.wkkf.org

Evaluation handbook**

Logic Model Development Guide**

** These texts are free and available (as a pdf file) online

Coley, S.M., Scheinberg, C.A. (2013). Proposal writing:

Effective Grantsmanship (4th ed.) Los Angeles, CA: SAGE

Issel, L. M. (2013). Health Program Planning & Evaluation

(3rd ed.) Sudbury, MA: Jones and Bartlett

how evaluation helps you design and implement programs ... · program design & evaluation –...

Documents