heart of missouri united way · 1/6/2014 2 logic models definition, history & benefits...
TRANSCRIPT
1/6/2014
1
Heart of Missouri United Way Community Development Training Series
Module I: Logic Models for Program Evaluation Aaron M. Thompson
School of Social Work
University of Missouri
Overview of Training Modules
• Module I: The What of program evaluation
• Assist providers to clarify the description of their programs and how to link organizational activities to intended effects.
• Module II: The How of program evaluation
• Assist providers to develop measurement models tied to logic model outcomes to support process and outcome evaluation.
• Module III: The Why of program evaluation
• Assist providers to develop conceptual and statistical skills and strategies for analyzing program level data to effectively:
• Identify the malleable predictors your program targets
• Communicate evidence that your program makes a difference
• Develop competitive proposals when applying for funding
Agenda
Goal: To discuss logic models as a common strategic framework for planning, implementing, evaluating, and communicating the
purpose and effectiveness of your program.
1. Logic Models: Definition, History & Benefits
2. Program Evaluation & Logic Models: Stages & Phases
3. Social Problems & Logic Models: Mediators as Targets
4. Logic Model Elements: Common Language
a. Inputs & Budget
b. Activities & Targets
c. Outputs & Fidelity
d. Outcomes & Development
5. Application & Consultation: A logic model for your program
1/6/2014
2
LOGIC MODELS
Definition, History & Benefits
Definition
• A flat diagram of a program to represent proximal lower order program actions necessary to leverage higher order changes at the individual, group, or community level
• Facilitates transparency
• Communicates your theory of change
• Lists resources to inform budget development
• Organizes program activities to inform staff training needs
• Tracks key program elements for fidelity monitoring
• Lists a sequence of socially meaningful outcomes
• Connects activities to intended outcomes
• Serves as an organizing tool for process and outcome evaluation
Brief History
• Logic model history
• 1955—first termed a “conceptual model” for research purposes
• Klienberg (1955), James (1958), Herzog (1961)
• 1967—Evaluative Research: Principles and Practice in Public
Service and Social Action Programs (Suchman)
• From research to practice as a tool
• 1979—Evaluation Promise & Performance (Wholey)
• 1995—Measuring Program Outcomes (Hatry)
• United Way of America
• 2005—Program Logic Model Guide (Kellogg Foundation)
• 2006—Getting to Outcomes (Rand Corporation)
1/6/2014
3
Benefits
• Often required
• IES, NIH, NIMH, NIDA, OJJDP, State, local & Private
• Enhances clarity by creating shared meaning
• Improves stakeholder, funder, Policy maker, and public buy-in
• Visually organizes and structurally aligns complex programs
• Organizes complex multi-modal or multi-element programs
• Facilitates the development of a stepwise work plan
• Helps to unpack the structure and organization of a program
• Assists with fiscal orderliness—develop an investment schedule
• Permits the development of a quality assurance plan
• Permits causal claims surrounding the effects of a program
• Provides data to suggest mid-course corrections
So…what is the problem?
1. Transition is Hard
• Change is uncomfortable
• Even more difficult at the systems level
2. There are Language Barriers
• Too many voices increase confusion and ambiguity
• Disciplines talk differently about these concepts
3. Fear of Appraisal
• Focused on providing quality direct services
• Afraid to losing funding and support
4. Of HMUW funded agency proposals reviewed (N = 19)
• < 15% specified the problem properly
• < 26% proposed programs that are “scientifically-based”
• < 5% linked activities, outputs and outcomes sequentially
• < 5% specified an accurate measurement model
PROGRAM EVALUATION & LOGIC MODELS
Phases & Stages
1/6/2014
4
Stages & Phases (CDC, 2011)
• Stages of Program Evaluation
• Ascertain and Engage Stakeholders
• Identify and Map Scientifically-Supported Programs
• Develop Program Evaluation Design
• Collect Evidence Using Credible Indicators
• Justify Data-Based Conclusions
• Utilize Data in Program and Policy Decisions
• Phases of Program Evaluation
• Planning (Needs Assessment)
• Implementation (Process or Formative Evaluation)
• Maintenance (Outcome or Impact Evaluation)
INPUTS ACTIVITIES OUTPUTS OUTCOMES
Resources (i.e., capital, personnel,
facilities, & tools needed to do work)
Actions (i.e., tasks,
duties, events, and work that will be done)
Products (i.e., units offered,
persons served, quality and proof work was done)
Benefits (i.e., observable
changes accrued by stakeholders
effect of work done)
Planning Implementation
Process Evaluation
Outcome Evaluation
Program Evaluation & Logic Models
Focus on population needs before adding or
changing a program.
Summative Formative
Focus on agency: how well agency operates, what services are provided; how well services are allocated, usually occurs through fidelity—quantity
and quality—practitioner or worker and client satisfaction.
Focus is on effect, impact, goal attainment reflected by indicators.
Maintenance
Needs (i.e., required stakeholder
services/goals, work to be done)
PRIORITIES
Needs Assessment
Types of Evaluation Questions • Planning Questions:
• What are the major service gaps or what is the need?
• Is there public support/interest for the program?
• What resources are needed/exist to implement the program?
• Implementation Questions: • Is there compliance with the program (fidelity)?
• Is there increased support for the program?
• Are users/practitioners pleased with the program?
• Are some potential users not being accessed?
• Evaluation Questions: • What is the cognitive and behavioral impact of the program?
• What is the health impact of the program?
• What is the economic impact of the program?
• Are there dosage or moderation effects of the program?
• Are there iatrogenic effects from the program?
1/6/2014
5
LINKING SOCIAL PROBLEMS & LOGIC MODELS
Mediators as Targets
Defining Social Problems
• Step 1: Defining a problem
• Suggesting a small program will alter large scale issues is inaccurate • Poverty, Teen Pregnancy, Drug Abuse, Aggression, Dropout, etc….
• However, a problem statement includes: • Who: Population (sex, age, income, race or other defining characteristics)
• What: BIG Problem (i.e., poverty, AIDS, cancer, obesity, etc.)
• Why: List malleable mediators or predictors of the problem
• How: Broad strategy or approach to targeting the predictors
• Example:
We aim to reduce incidents of child abuse by offering Parent and Child interaction Therapy to low income families where parents are exposed to increased life
stressors without effective coping skills, adequate social supports, or positive parenting strategies.
Mediators & Social Problems • Mediators are the indicators or intervening mechanisms that
programs target to alter the social problem
• Mediators are highly associated with/predictive of the problem
• Mediators are immediate malleable outcomes
• Mediators are measured alongside medial and distal outcomes
• Mediators explain the variance in the outcome
EMPATHY
M
PROGRAM
X Y
RECIDIVISM
a b
c’
Proximal tx
outcome in
Logic Model
The following several slides are adapted from: MacKinnon, D. P. (20049 February). Methods to study treatment mechanisms of
action. NIDA Workshop on Mediation. Retrieved July 29, 2005 from http://www.public.asu.edu/~davidpm/ripl/NIDAM5.final.PPT
**
** **
1/6/2014
6
Examples of Mediators
• School Dropout • Engagement, bonding, student-teacher relations, formal social
support, social competency, decision making, problem solving, etc...
• School Readiness • Parent involvement, positive parenting strategies, parent-teacher
bonding, home-school communication, emergent literacy skills, social and emotional competence, authority acceptance, etc…
• Poverty • Financial literacy, secure housing, reduced use of “pay-day” loan
services, increased banking participation, etc…
• Teen Pregnancy • Knowledge of sex and STIs, attitudes toward condom use, peer group
normative attitudes about sex, propensity to identify risky scenarios, knowledge of escaping risky situations, etc…
• Violence and Aggression • Normative attitudes about violence, social information processing
skills, social competence, self-management, self-awareness, parent supervision, etc…
LOGIC MODEL ELEMENTS
Building Common Vision
Basic Components
• There is no “right” way to create a logic model
• Basic Ingredients:
A. INPUTS
B. ACTIVITIES
C. OUTPUTS
D. OUTCOMES
Resources (i.e., capital, personnel,
facilities, & tools needed to do work)
Actions (i.e., tasks,
duties, events, and work that will be done)
Products (i.e., units offered,
persons served, quality and proof work was done)
Benefits (i.e., observable
changes accrued by stakeholders
effect of work done)
1/6/2014
7
a. Inputs: Justifying Budgets • Resources or investments required to operate your program
• Financial, organizational, human or fiscal capital needed
• Staff AND time
• Contracted services
• Facilities
• Program materials, supplies, equipment
• Collaborative Relationships
• Items may be listed broadly on Logic Model figure, then…
• Use each item to inform budget development by:
• Estimating the number of units predicted/required
• Describe each unit in your budget justification parallel to your LM
• Why the unit is needed, the number of units needed, the cost per unit
• Build an itemized budget in an excel sheet
INPUTS ACTIVITIES OUTPUTS OUTCOMES
a. Inputs: Justifying Budgets (2) Not Specific Enough Just Right Too Specific
Home buying resources Clear financial records
W2 forms 1099s Tax returns Bank statements Pay stubs Utilities bills Credit report
Staff 3 full/1 part-time staff
1 project lead @ 40 hrs./wk. 2 project associates @ 40 hrs./wk. 1 part-time support person @ 20 hrs./wk.
Supplies Art Supplies
25 paintbrushes 50 bottles of paint 250 sheets of paper 25 coffee cans Dishwashing liquid
b. Activities: Altering Mediators
• Actions, tasks, processes, planned events, direct services, or what the program staff will do to fulfill its stated mission • Provide…
• Educate…
• Train…
• Create…
• Counsel…
• Use ACTION words to unpack what we will do
• Be sure activities TARGET known predictors or mediators associated with mitigating or improving the problem • These targets or mediators will logically bring about the desired
change
• There is research supporting the effect of the target on the social problem
INPUTS ACTIVITIES OUTPUTS OUTCOMES
1/6/2014
8
b. Activities: Altering Mediators
Activities
• Hire real-estate agent • Drive around the city
Not detailed enough.
• Google search •Interview friends • Read 3 books on neighborhoods • Hire a driver • Try restaurants • Set up meetings• Take friends on tour
Too detailed (more appropriate for a work plan).
• Research neighborhood amenities and prices • Hire a real-estate agent • Tour priority neighborhoods
Just right.
c. Outputs: Monitoring Fidelity
• Direct By-Products of Program Activities
• Number of classes provided
• Number of counseling sessions provided
• Number of materials created/used/distributed
• AND persons directly accessing services
• Number attending classes
• Number of persons counseled
• Number of persons using materials
• AND quality indicators of service provision/client access
• Client and practitioner tx acceptability
• Client engagement (i.e., behavioral, cognitive, emotional)
• Bonding between practitioner and client
INPUTS ACTIVITIES OUTPUTS OUTCOMES
Outputs (2) • Quantity (aka, surface exposure to tx elements)
• Exposure to the intervention activities • This is why we count outputs • Demonstration that people are indeed exposed to tx
• Quality (aka, subsurface response to exposure to tx elements) • Treatment acceptability
• Is tx cost effective and easy • What is practitioner tx acceptability • What is client tx acceptability • Levels of engagement (i.e., behavioral, cognitive, emotional) • Bonding between practitioner and client
• Field notes, focus groups, survey or questionnaires
• Don’t forget to measure what is happening in comparison • SSD—baseline • Group—all services exposed to comparison
• Estimating Fidelity (aka, total program dosage) • Ratio: # received/total elements = % or dosage • Compare compliers with noncompliers
• Investigate reasons for noncompliance
1/6/2014
9
d. Outcomes & Development • Measureable Benefits for participants after tx exposure
• If-then sequential developmental statements
• Links to outputs
INPUTS ACTIVITIES OUTPUTS OUTCOMES
Short-term Proximal
Immediate
Interim Medial
Intermediate
Long-term Distal
Impact
Initial changes most causally associated
with program targets
• Knowledge • Attitudes • Cognitions
Secondary changes reasonably expected
to stem from proximal effects
• Skills • Behaviors • Social Interactions
Terminal changes logically expected,
generally reflect goals of agency
• Health • Status • Condition
0-1 years 2-5 years 4-7 years
d. Outcomes & Development (2)
Immediate
Knowledge 1a. Reduced myths among targeted parents on immunization 2a. Increase knowledge in targeted parents of where have children immunized 3a….. …
Intermediate
Behavior 1b. Increase number of targeted parents taking children to immunization 2b. Increase number of targeted parents who report positive bonds towards health providers 3a….. …
Long-term
Condition 1c. Fewer children suffering from preventable diseases 2c. Better overall health for children 3a….. …
Developmental Impact of Immunization PSAs
Outputs vs. Outcomes
• Distinguishing characteristics:
• Outputs are direct and measurable products of a program’s activities and services
• Expressed in terms of volume or units delivered.
• Outcomes are the effects or impact of the activities and services.
• Each outcome usually corresponds to more than one output.
Outputs Outcomes
# of workshops Change in knowledge associated with workshop
# of new mom home visits Increased knowledge of child development
Action plan to clean and supervise playgrounds
Neighborhood volunteers sign up to clean and supervise playgrounds
# of funding proposals submitted # of potential individual donors
Increased and diversified resources for the program
1/6/2014
10
Logic Model Examples
• Policy: Bicycle Helmet Public Information Policy
• Single Program: STARS Integrated Conceptual Model
• Multimodal Program: SAMSHA Community Drug Abuse Prev.
Bicycle Helmet Public Information Policy
Inputs Staff
Space Equipment
Outcomes-Mid Increase bicycle helmets worn
Outcomes-Short Increased
awareness of bike TBI injuries
Activities -Gather data on
accidents -TBI, and injury
prevention -Develop press
kits, public service
messages -Attract key
spokesperson
Outputs Factsheets
TV and Radio PSAs
Outcomes-Long Fewer TBI bike related injuries
STARS
Autonomy Supports
+self-awareness
+self-monitoring
+self-management
Competency Supports
+social competence
+self-efficacy
+cue interpretation
+problem solving
Relational Supports
+social awareness
+communication
+staff & peer relations
Skills &
Opportunities
Increased Conventional Behaviors & Values
Reflective of the Social Unit
Improved
SEL &
Classroom
Behavior
Increased
Time On-
Task
Increased
Instructional
time
Improved
Academic
Performance
Involvement
& feedback
Social Rewards &
Increased Bonding
Figure 1.
STARS Integrated Theory of Change Impacting SEL, Behavior, and Academic Performance
Proximal Outcomes Distal Outcomes
1/6/2014
11
Multilevel or Multimodal Logic Model SAMSHA: Community Drug Abuse Prevention
Community- Risk Factors
Access Norms
Perceptions Parenting
Outcomes-Long
Reduced Delinquency Overdoses
Drug Arrests
Outcomes –Short
Increased Risk Awareness Storage options
Refusal strategies
Strategies
Drug Court Social Marketing
Curriculum Outreach
Drug Court
Inputs
Police Support Media Support
Education Court and LEA
Partners
Community Logic Model II: Prescription Drugs
Community- Specific Risk
Factors
Long-Term Outcomes
Reduction in Use
Delayed Onset
Reduction in Availability
Access Ease of
Availability
Short Term Outcomes
Increased awareness of risk of improper
storage
Increased use of safer storage and/or
disposal practices
Increased demand for take-back programs
Increased perception of harm or social
disapproval
Strategies Resources and Inputs
Community Norms
Perceived Risk
Drug Dispersal Take-Back Program
Social Marketing Lock Up Your
Medicine Cabinet
Collect and analyze data on the intervention Recruit local DEA or police department to staff take-back event.
Collect and analyze data on the intervention Develop a marketing plan Partner with media Secure earned and paid media
Community Logic Model I: Heroin
Community- Specific Risk
Factors
Long-Term Outcomes
Reduction in Lifetime Heroin Use
Reduction in Delinquency
Perception of Harm
Short-Term Outcomes
Increased Perception of Harm of Heroin Use
Increased Access to Mental Health and Treatment Services
Improved Access to
Parenting Skills Training via Drug Court Referrals
Strategies Resources and Inputs
Low Parental Care
Family Conflict
Curriculum
Drug Court
Collect and analyze data on the intervention
Curriculum, teacher training
Hire and train community outreach workers
Collect and analyze data on the intervention Court and enforcement partnerships
Community Outreach Worker
Education and Referral
1/6/2014
12
Caveats • We did not cover everything associated with Logic Models
• Logic Model Development Guide
• There is no right way to develop Logic Models • The grandmother test
• Cautionary word on the linearity of Logic Models • Most programs include more than one service
• Multi-Component Models (i.e., Master and Subcomponent models)
• Multi-modal Models (i.e., Master and Sublevel models) • Be sure the level targeted and the logic model elements align
• Cautionary word on outcome linkages & causality • Requirements for causality
a. Temporal Ordering
b. Association
c. Counterfactually ruling out other causes
• The ability to attribute causal inference diminishes the farther from program activities our outcome is
Considerations
• Keep it practical—time is money
• It should be to YOUR benefit, first and foremost!
• Keep it focused—quality assurance
• Do not dismiss the importance of process evaluation
• Be sure your outcomes reflect your organizational goal
• Consider how outcome evaluation results will be used
• As a reciprocal feedback tool
• To communicate impact
• To estimate a cost-effect size ratio
• Use explicit language—never assume your reader knows
• Use action words
• Use parallel construction between LM, narrative, and budget
Brief Activity and Peer Consultation
• Activity: Think-Pair-Share
• Think—(10 minutes)
• By yourself, scratch out a logic model for your program
• Use the Tip sheet and handouts
• Pair—(10 minutes)
• With a partner or partner, share your logic model
• Are the elements sequentially linked?
• Are there elements or activities that do not logically predict outcomes?
• Are the outcomes sequentially linked?
• Share—(20 minutes)
• 2 or 3 volunteers to discuss their logic model
1/6/2014
13
Questions and Comments
• Please Complete the Posttest Survey
• Feel Contact me for Consultation:
Aaron M. Thompson
School of Social Work
University of Missouri
718 Clark Hall, 65211
PH: 573.882.0124
THANK YOU!!!!