brittany rhoades, brian bumbarger, & melissa tibbits 2009... · anaheim, ca. today’s...
TRANSCRIPT
Brittany Rhoades, Brian Bumbarger, & Melissa TibbitsEvidence-based Prevention & Intervention Support Center (EPISCenter)
Pennsylvania State University
National Prevention Network 2009 ConferenceAnaheim, CA
Today’s Discussion: Sustainability What does the literature say?
What is happening in Pennsylvania?
What does our data say?
What do these results mean for YOU?
Prevention Science & Sustainability
That was then…• 30 years ago, there were NO empirically-validated
delinquency or drug prevention programs
• Efforts were guided primarily by “good intentions” and “gut instinct”
• Hundreds of millions of dollars were spent without any accountability
• Prevention was considered more “art” than “science”
This is now… We have learned more about what causes and what
works to prevent youth substance use, violence and delinquency in the last 30 years than we did in the previous 200 years
Today, there are many programs that have been proven effective in well-designed efficacy studies
There is clearly a “science” of prevention!
?
Science of Prevention Public Health Impact
Sustainability of Evidence-based
Programs
Sustainability
“the program components developed and implemented in earlier stages are maintained after the initial funding or other impetus is removed”
Scheirer (2005)
Implementation Readiness Motivation for change
Capacity to implement change
Support of influential leaders
Connection to Coalition Strength of Connection
Quality of Functioning
Types of Support
Program Support System Implementer
Characteristics & Skills
Training
Technical Assistance
Sustainability Planning Procurement of
additional funds
Discussion early in the implementation process
State of Sustainability Research Mostly theoretical and anecdotal
Few empirical studies
Program-specific results overgeneralization
Context rarely considered
Little information about what happens outside of controlled research trials – in the real world
A test-bed for sustainability research
Pennsylvania’s Evidence-based Program Initiative Nearly 200 EBPs funded in PA since 1998
Intentional focus on implementation quality & fidelity, impact assessment, and sustainability planning
Currently includes:
Olweus Bullying Prevention ProgramLifeSkills TrainingPATHSProject TNDBig Brothers/Sisters
Strengthening Families 10-14The Incredible YearsMultisystemic TherapyFunctional Family Therapy Multidimensional Treatment Foster Care
Resource Center for Evidence-based Prevention and Intervention Programs and Practices
NCJJLocal Innovative Programs
and Practices
The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University,and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare
as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.
A unique partnership between policymakers, researchers,
and communities to bring science to bear on issues of public
health and public safety
• Conduct Outreach and Advocacy• Provide Technical Assistance• Develop Training and Resources• Create and Facilitate Peer Networks• Conduct Translational Research
Resource Center for Evidence-based Prevention and Intervention Programs and Practices
NCJJLocal Innovative Programs
and Practices
The Goals: Preventing delinquency and youth violence to the
greatest degree possible (reducing delinquency rates)
Intervening effectively with youth for whom primary prevention is not sufficient (reducing further system penetration)
Allowing communities the flexibility to select strategies that meet local needs
Providing accountability and using scarce resources efficiently
Increasing local capacity
Preliminary results from one study in Pennsylvania
The Web-based Survey Annual survey completed by PCCD-funded grantees
Grantees are followed during the 4 years of their PCCD funding and after funding ends
Information collected on: Coalition connection, functioning, & support
Implementation readiness
Stakeholder buy-in & support
Training & TA
Implementer characteristics
Fidelity monitoring
Sustainability planning
Local evaluation
Post PCCD-funding
Sustainability
A first look….Implementation Barrier s(0 = not a barrier, 1 = somewhat, 2 = significant)
Predicted Sustainability
Implementing Agency -.31
Program Staff -.47*
Overall Community Support -.34
Support of Community Leaders -.05
Overall School Support -.88**
Support of School Administration -.71**
Communication with Coalition -.33
Alignment with Existing Goals -.33
Training & TA -.32
Actual Sustainability .70*
* p < .05, ** p < .01
Results from Tibbits, M., Bumbarger, B., Kyler, S., & Perkins, D. (under review) Sustaining evidence-based
interventions under real-world conditions: Results from a large-scale diffusion project . Prevention Science.
A first look….Hypothesized Predictors Actual
Sustainability
Community Leader Support .24
School Administration Support .12
Agency Administration Support .07
Community Readiness -.03
Financial Collaboration with Coalition .04
Instrumental Collaboration with Coalition .12
Quality of Training .10
Sustainability Financial Planning .27*
Sustainability Alignment Planning .37**
* p < .05, ** p < .01
Results from Tibbits, M., Bumbarger, B., Kyler, S., & Perkins, D. (under review) Sustaining evidence-based
interventions under real-world conditions: Results from a large-scale diffusion project . Prevention Science.
Current Study N = 43 respondents with 2005 data & sustainability info
Classroom-based: n = 12
Community/Mentoring: n = 14
Family Prevention: n = 8
Family Treatment: n = 9
Program was 2 or more years post PCCD-funding
Range: 2-6 years
Measures: ScalesConstruct No. of
ItemsSample Item Alpha
Coalition Connection 9 How often does a program rep. attend coalition meetings?
.89
Coalition Functioning 3 To what extent does the coalition encourage interagency collaboration?
.84
Coalition Support:Instrumental
5 To what extent does the coalition giveadvice/support in recruiting your target population?
.85
Coalition Support:Financial
3 To what extent does the coalition assist in seeking funding?
.83
Implementation Readiness 6 Because we were unprepared when funding began, we feel behind schedule.
.79
ImplementerCharacteristics
6 Staff were supportive & motivated to implement the program.
.92
Sustainability Planning: Financial
3 To what degree did you develop a fiscal plan outlining the funds needed to sustain the program?
.67
Sustainability Planning: Alignment
6 To what degree did you determine how the program aligns with the mission & goals of future stakeholders?
.77
Measures: Single Items Stakeholder Buy-In/Support
How supportive/resistant were the following stakeholders?
Training Formal training vs. not
Quality of the training
Communication with trainer after initial training
Fidelity Monitoring Are you formally monitoring attendance, number of sessions,
quality of interactions, quality of delivery?
Sustainability Planning
Year during implementation began sustainability planning
Local Evaluation Are you currently conducting a local evaluation?
Sustainability Rates by Program Type
6771
6367
3329
3733
0
10
20
30
40
50
60
70
80
Classroom-based Community/Mentoring Family Prevention Family Treatment
Percent Sustained Percent Not Sustained
Factors Related to Sustainability: Classroom-based Programs
3.8
3.9
4
4.1
4.2
4.3
4.4
Sustained Not Sustained
Implementer Characteristics: Support, Skills, Motivation & Attitude
N = 12, F = 3.99 (1, 10), p = .07
Factors Related to Sustainability: Classroom-based Programs
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No
Monitor Quality of Delivery Monitor Quality of Interactions
Percent Sustained Percent Not Sustained
N = 12, χ2 = 2.74 , p = .10
Factors Related to Sustainability: Classroom-based Programs
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No
Monitor Quality of Delivery Monitor Quality of Interactions
Percent Sustained Percent Not Sustained
N = 12, χ2 = 2.74 , p = .10
Factors Related to Sustainability: Community/Mentoring Programs
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Coalition Functioning
Coalition Instrm. Support
Trainer Communication
Financial Planning
Sustained
Not Sustained
N = 12, F = 13.58 (1, 10), p < .01
Factors Related to Sustainability: Community/Mentoring Programs
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Coalition Functioning
Coalition Instrm. Support
Trainer Communication
Financial Planning
Sustained
Not Sustained
N = 12, F = 3.76 (1, 10), p = .08
Factors Related to Sustainability: Community/Mentoring Programs
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Coalition Functioning
Coalition Instrm. Support
Trainer Communication
Financial Planning
Sustained
Not Sustained
N = 11, F = 4.64 (1, 9), p = .06
Factors Related to Sustainability: Community/Mentoring Programs
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Coalition Functioning
Coalition Instrm. Support
Trainer Communication
Financial Planning
Sustained
Not Sustained
N = 11, F = 6.10 (1, 9), p < .05
Factors Related to Sustainability: Community/Mentoring Programs
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No Yes No
Conducting Local Eval.
Formal Training Monitor Quality of Delivery
Percent Sustained Percent Not Sustained
N = 12, χ2 = 4.69, p < .01
Factors Related to Sustainability: Community/Mentoring Programs
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No Yes No
Conducting Local Eval.
Formal Training Monitor Quality of Delivery
Percent Sustained Percent Not Sustained
N = 11, χ2 = 2.93, p = .09
Factors Related to Sustainability: Community/Mentoring Programs
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No Yes No
Conducting Local Eval.
Formal Training Monitor Quality of Delivery
Percent Sustained Percent Not Sustained
N = 10, χ2 = 3.75, p = .05
Factors Related to Sustainability: Family Prevention Programs
0
1
2
3
4
5
6
Coalition Functioning Coalition Buy-in/Support
Sustained
Not Sustained
N = 5, F = 11.76 (1, 3), p < .05
Factors Related to Sustainability: Family Prevention Programs
0
1
2
3
4
5
6
Coalition Functioning Coalition Buy-in/Support
Sustained
Not Sustained
N = 7, F = 8.93 (1, 5), p < .05
Factors Related to Sustainability: Family Treatment Programs
0
1
2
3
4
5
6
Implementer Support & Skills
Yr. Begin Sustainability
Plan
Alignment Planning
Sustained
Not Sustained
N = 8, F = 10.55 (1, 6), p < .05
Characteristics
Factors Related to Sustainability: Family Treatment Programs
0
1
2
3
4
5
6
Implementer Support & Skills
Yr. Begin Sustainability
Plan
Alignment Planning
Sustained
Not Sustained
Characteristics
N = 8, F = 4.14 (1, 6), p = .09
Factors Related to Sustainability: Family Treatment Programs
0
1
2
3
4
5
6
Implementer Support & Skills
Yr. Begin Sustainability
Plan
Alignment Planning
Sustained
Not Sustained
Characteristics
N = 7, F = 8.65 (1, 6), p < .05
Factors Related to Sustainability: Family Treatment Programs
0%
20%
40%
60%
80%
100%
Yes No
Monitor Quality of Interactions
Percent Sustained Percent Not Sustained
N = 7, χ2 = 3.73, p = .05
Factors Related to Sustainability: Across all program types
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Trainer Communication
Implementer Support & Skills
Financial Planning
Alignment Planning
Sustained
Not Sustained
N = 38, F = 3.46 (1, 36), p = .07
Factors Related to Sustainability: Across all program types
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Trainer Communication
Implementer Support & Skills
Financial Planning
Alignment Planning
Sustained
Not Sustained
N = 39, F = 5.92 (1, 37), p < .01
Characteristics
Factors Related to Sustainability: Across all program types
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Trainer Communication
Implementer Support & Skills
Financial Planning
Alignment Planning
Sustained
Not Sustained
Characteristics
N = 35, F = 6.29 (1, 33), p < .01
Factors Related to Sustainability: Across all program types
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Trainer Communication
Implementer Support & Skills
Financial Planning
Alignment Planning
Sustained
Not Sustained
Characteristics
N = 35, F = 3.18 (1, 33), p = .08
Factors Related to Sustainability: Across all program types
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No
Monitor Quality of Delivery Monitor Quality of Interactions
Percent Sustained Percent Not Sustained
N = 36, χ2 = 6.89, p < .01
Factors Related to Sustainability: Across all program types
0%10%20%30%40%50%60%70%80%90%
100%
Yes No Yes No
Monitor Quality of Delivery Monitor Quality of Interactions
Percent Sustained Percent Not Sustained
N = 37, χ2 = 7.75, p < .01
Results SummaryPredictors Overall Classroom-
based
Comm. /
Mentoring
Fam.
Prevention
Fam.
Treatment
Coalition X X
Implementation
Readiness
Stakeholders Buy-
in/SupportX
Quality Training X X
Implementer
CharacteristicsX X X
Fidelity Monitoring X X X
Sustainability
PlanningX X X
Local Evaluation X
Study Limitations Small sample size
Limited ability to detect significant effects
Causality
Which direction does the arrow go?
Generalizability
Do these results generalize beyond PA?
Implications for research to practice
Points for Discussion How do these findings align or not with your
experiences?
Differences by program type
Additional factors not considered here?
Implications for Research We must consider program type/context
Larger sample size on diverse programs is needed
How do these predictors relate and interact with on another to predict sustainability?
Need more detailed view of sustainability
Implications for Practice Sustainability is more than funding
Sustainability planning should be integrated into the program implementation process – from the beginning!
Connection to a well-functioning coalition can be a great facilitator for sustainability, particularly for community and family prevention programs.
For more information on our research and services, contact:
Brittany Rhoades at [email protected]
Evidence-based Prevention and Intervention Support Center
(EPISCenter)
Prevention Research Center
Penn State University
206 Towers Bldg.
University Park, PA 16802
(814) 863-2568