brittany rhoades, brian bumbarger, & melissa tibbits 2009... · anaheim, ca. today’s...

Brittany Rhoades, Brian Bumbarger, & Melissa TibbitsEvidence-based Prevention & Intervention Support Center (EPISCenter)

Pennsylvania State University

National Prevention Network 2009 ConferenceAnaheim, CA

Today’s Discussion: Sustainability What does the literature say?

What is happening in Pennsylvania?

What does our data say?

What do these results mean for YOU?

Prevention Science & Sustainability

That was then…• 30 years ago, there were NO empirically-validated

delinquency or drug prevention programs

• Efforts were guided primarily by “good intentions” and “gut instinct”

• Hundreds of millions of dollars were spent without any accountability

• Prevention was considered more “art” than “science”

This is now… We have learned more about what causes and what

works to prevent youth substance use, violence and delinquency in the last 30 years than we did in the previous 200 years

Today, there are many programs that have been proven effective in well-designed efficacy studies

There is clearly a “science” of prevention!

?

Science of Prevention Public Health Impact

Sustainability of Evidence-based

Programs

Sustainability

“the program components developed and implemented in earlier stages are maintained after the initial funding or other impetus is removed”

Scheirer (2005)

Implementation Readiness Motivation for change

Capacity to implement change

Support of influential leaders

Connection to Coalition Strength of Connection

Quality of Functioning

Types of Support

Program Support System Implementer

Characteristics & Skills

Training

Technical Assistance

Sustainability Planning Procurement of

additional funds

Discussion early in the implementation process

State of Sustainability Research Mostly theoretical and anecdotal

Few empirical studies

Program-specific results overgeneralization

Context rarely considered

Little information about what happens outside of controlled research trials – in the real world

A test-bed for sustainability research

Pennsylvania’s Evidence-based Program Initiative Nearly 200 EBPs funded in PA since 1998

Intentional focus on implementation quality & fidelity, impact assessment, and sustainability planning

Currently includes:

Olweus Bullying Prevention ProgramLifeSkills TrainingPATHSProject TNDBig Brothers/Sisters

Strengthening Families 10-14The Incredible YearsMultisystemic TherapyFunctional Family Therapy Multidimensional Treatment Foster Care

Resource Center for Evidence-based Prevention and Intervention Programs and Practices

NCJJLocal Innovative Programs

and Practices

The EPISCenter is a project of the Prevention Research Center, College of Health and Human Development, Penn State University,and is funded by the Pennsylvania Commission on Crime and Delinquency and the Pennsylvania Department of Public Welfare

as a component of the Resource Center for Evidence-Based Prevention and Intervention Programs and Practices.

A unique partnership between policymakers, researchers,

and communities to bring science to bear on issues of public

health and public safety

• Conduct Outreach and Advocacy• Provide Technical Assistance• Develop Training and Resources• Create and Facilitate Peer Networks• Conduct Translational Research

Resource Center for Evidence-based Prevention and Intervention Programs and Practices

NCJJLocal Innovative Programs

and Practices

The Goals: Preventing delinquency and youth violence to the

greatest degree possible (reducing delinquency rates)

Intervening effectively with youth for whom primary prevention is not sufficient (reducing further system penetration)

Allowing communities the flexibility to select strategies that meet local needs

Providing accountability and using scarce resources efficiently

Increasing local capacity

Preliminary results from one study in Pennsylvania

The Web-based Survey Annual survey completed by PCCD-funded grantees

Grantees are followed during the 4 years of their PCCD funding and after funding ends

Information collected on: Coalition connection, functioning, & support

Implementation readiness

Stakeholder buy-in & support

Training & TA

Implementer characteristics

Fidelity monitoring

Sustainability planning

Local evaluation

Post PCCD-funding

Sustainability

A first look….Implementation Barrier s(0 = not a barrier, 1 = somewhat, 2 = significant)

Predicted Sustainability

Implementing Agency -.31

Program Staff -.47*

Overall Community Support -.34

Support of Community Leaders -.05

Overall School Support -.88**

Support of School Administration -.71**

Communication with Coalition -.33

Alignment with Existing Goals -.33

Training & TA -.32

Actual Sustainability .70*

* p < .05, ** p < .01

Results from Tibbits, M., Bumbarger, B., Kyler, S., & Perkins, D. (under review) Sustaining evidence-based

interventions under real-world conditions: Results from a large-scale diffusion project . Prevention Science.

A first look….Hypothesized Predictors Actual

Sustainability

Community Leader Support .24

School Administration Support .12

Agency Administration Support .07

Community Readiness -.03

Financial Collaboration with Coalition .04

Instrumental Collaboration with Coalition .12

Quality of Training .10

Sustainability Financial Planning .27*

Sustainability Alignment Planning .37**

* p < .05, ** p < .01

Results from Tibbits, M., Bumbarger, B., Kyler, S., & Perkins, D. (under review) Sustaining evidence-based

interventions under real-world conditions: Results from a large-scale diffusion project . Prevention Science.

Current Study N = 43 respondents with 2005 data & sustainability info

Classroom-based: n = 12

Community/Mentoring: n = 14

Family Prevention: n = 8

Family Treatment: n = 9

Program was 2 or more years post PCCD-funding

Range: 2-6 years

Measures: ScalesConstruct No. of

ItemsSample Item Alpha

Coalition Connection 9 How often does a program rep. attend coalition meetings?

.89

Coalition Functioning 3 To what extent does the coalition encourage interagency collaboration?

.84

Coalition Support:Instrumental

5 To what extent does the coalition giveadvice/support in recruiting your target population?

.85

Coalition Support:Financial

3 To what extent does the coalition assist in seeking funding?

.83

Implementation Readiness 6 Because we were unprepared when funding began, we feel behind schedule.

.79

ImplementerCharacteristics

6 Staff were supportive & motivated to implement the program.

.92

Sustainability Planning: Financial

3 To what degree did you develop a fiscal plan outlining the funds needed to sustain the program?

.67

Sustainability Planning: Alignment

6 To what degree did you determine how the program aligns with the mission & goals of future stakeholders?

.77

Measures: Single Items Stakeholder Buy-In/Support

How supportive/resistant were the following stakeholders?

Training Formal training vs. not

Quality of the training

Communication with trainer after initial training

Fidelity Monitoring Are you formally monitoring attendance, number of sessions,

quality of interactions, quality of delivery?

Sustainability Planning

Year during implementation began sustainability planning

Local Evaluation Are you currently conducting a local evaluation?

Sustainability Rates by Program Type

6771

6367

3329

3733

0

10

20

30

40

50

60

70

80

Classroom-based Community/Mentoring Family Prevention Family Treatment

Percent Sustained Percent Not Sustained

Factors Related to Sustainability: Classroom-based Programs

3.8

3.9

4

4.1

4.2

4.3

4.4

Sustained Not Sustained

Implementer Characteristics: Support, Skills, Motivation & Attitude

N = 12, F = 3.99 (1, 10), p = .07

Factors Related to Sustainability: Classroom-based Programs

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No

Monitor Quality of Delivery Monitor Quality of Interactions

Percent Sustained Percent Not Sustained

N = 12, χ2 = 2.74 , p = .10

Factors Related to Sustainability: Classroom-based Programs

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No

Monitor Quality of Delivery Monitor Quality of Interactions

Percent Sustained Percent Not Sustained

N = 12, χ2 = 2.74 , p = .10

Factors Related to Sustainability: Community/Mentoring Programs

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Coalition Functioning

Coalition Instrm. Support

Trainer Communication

Financial Planning

Sustained

Not Sustained

N = 12, F = 13.58 (1, 10), p < .01

Factors Related to Sustainability: Community/Mentoring Programs

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Coalition Functioning

Coalition Instrm. Support

Trainer Communication

Financial Planning

Sustained

Not Sustained

N = 12, F = 3.76 (1, 10), p = .08

Factors Related to Sustainability: Community/Mentoring Programs

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Coalition Functioning

Coalition Instrm. Support

Trainer Communication

Financial Planning

Sustained

Not Sustained

N = 11, F = 4.64 (1, 9), p = .06

Factors Related to Sustainability: Community/Mentoring Programs

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Coalition Functioning

Coalition Instrm. Support

Trainer Communication

Financial Planning

Sustained

Not Sustained

N = 11, F = 6.10 (1, 9), p < .05

Factors Related to Sustainability: Community/Mentoring Programs

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No Yes No

Conducting Local Eval.

Formal Training Monitor Quality of Delivery

Percent Sustained Percent Not Sustained

N = 12, χ2 = 4.69, p < .01

Factors Related to Sustainability: Community/Mentoring Programs

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No Yes No

Conducting Local Eval.

Formal Training Monitor Quality of Delivery

Percent Sustained Percent Not Sustained

N = 11, χ2 = 2.93, p = .09

Factors Related to Sustainability: Community/Mentoring Programs

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No Yes No

Conducting Local Eval.

Formal Training Monitor Quality of Delivery

Percent Sustained Percent Not Sustained

N = 10, χ2 = 3.75, p = .05

Factors Related to Sustainability: Family Prevention Programs

0

1

2

3

4

5

6

Coalition Functioning Coalition Buy-in/Support

Sustained

Not Sustained

N = 5, F = 11.76 (1, 3), p < .05

Factors Related to Sustainability: Family Prevention Programs

0

1

2

3

4

5

6

Coalition Functioning Coalition Buy-in/Support

Sustained

Not Sustained

N = 7, F = 8.93 (1, 5), p < .05

Factors Related to Sustainability: Family Treatment Programs

0

1

2

3

4

5

6

Implementer Support & Skills

Yr. Begin Sustainability

Plan

Alignment Planning

Sustained

Not Sustained

N = 8, F = 10.55 (1, 6), p < .05

Characteristics

Factors Related to Sustainability: Family Treatment Programs

0

1

2

3

4

5

6

Implementer Support & Skills

Yr. Begin Sustainability

Plan

Alignment Planning

Sustained

Not Sustained

Characteristics

N = 8, F = 4.14 (1, 6), p = .09

Factors Related to Sustainability: Family Treatment Programs

0

1

2

3

4

5

6

Implementer Support & Skills

Yr. Begin Sustainability

Plan

Alignment Planning

Sustained

Not Sustained

Characteristics

N = 7, F = 8.65 (1, 6), p < .05

Factors Related to Sustainability: Family Treatment Programs

0%

20%

40%

60%

80%

100%

Yes No

Monitor Quality of Interactions

Percent Sustained Percent Not Sustained

N = 7, χ2 = 3.73, p = .05

Factors Related to Sustainability: Across all program types

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Trainer Communication

Implementer Support & Skills

Financial Planning

Alignment Planning

Sustained

Not Sustained

N = 38, F = 3.46 (1, 36), p = .07

Factors Related to Sustainability: Across all program types

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Trainer Communication

Implementer Support & Skills

Financial Planning

Alignment Planning

Sustained

Not Sustained

N = 39, F = 5.92 (1, 37), p < .01

Characteristics

Factors Related to Sustainability: Across all program types

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Trainer Communication

Implementer Support & Skills

Financial Planning

Alignment Planning

Sustained

Not Sustained

Characteristics

N = 35, F = 6.29 (1, 33), p < .01

Factors Related to Sustainability: Across all program types

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Trainer Communication

Implementer Support & Skills

Financial Planning

Alignment Planning

Sustained

Not Sustained

Characteristics

N = 35, F = 3.18 (1, 33), p = .08

Factors Related to Sustainability: Across all program types

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No

Monitor Quality of Delivery Monitor Quality of Interactions

Percent Sustained Percent Not Sustained

N = 36, χ2 = 6.89, p < .01

Factors Related to Sustainability: Across all program types

0%10%20%30%40%50%60%70%80%90%

100%

Yes No Yes No

Monitor Quality of Delivery Monitor Quality of Interactions

Percent Sustained Percent Not Sustained

N = 37, χ2 = 7.75, p < .01

Results SummaryPredictors Overall Classroom-

based

Comm. /

Mentoring

Fam.

Prevention

Fam.

Treatment

Coalition X X

Implementation

Readiness

Stakeholders Buy-

in/SupportX

Quality Training X X

Implementer

CharacteristicsX X X

Fidelity Monitoring X X X

Sustainability

PlanningX X X

Local Evaluation X

Study Limitations Small sample size

Limited ability to detect significant effects

Causality

Which direction does the arrow go?

Generalizability

Do these results generalize beyond PA?

Implications for research to practice

Points for Discussion How do these findings align or not with your

experiences?

Differences by program type

Additional factors not considered here?

Implications for Research We must consider program type/context

Larger sample size on diverse programs is needed

How do these predictors relate and interact with on another to predict sustainability?

Need more detailed view of sustainability

Implications for Practice Sustainability is more than funding

Sustainability planning should be integrated into the program implementation process – from the beginning!

Connection to a well-functioning coalition can be a great facilitator for sustainability, particularly for community and family prevention programs.

For more information on our research and services, contact:

Brittany Rhoades at blr162@psu.edu

Evidence-based Prevention and Intervention Support Center

(EPISCenter)

Prevention Research Center

Penn State University

206 Towers Bldg.

University Park, PA 16802

EPISCenter@psu.edu

(814) 863-2568