Brian K. BumbargerNational Prevention Network

September, 2009

Community-level Impact of a Strategic Prevention Framework

The CTC Model in Pennsylvania

Investigators and Authors:

Mark Feinberg Mark Greenberg

Brian Bumbarger Wayne Osgood

Louis Brown Nate Riggs

Ty Ridenour Damon Jones

Jennifer Sartorious Daniel Bontempo

Brendan Gomez Richard Puddy

Michael Cleveland

Collaborative Policy Innovators:

Mike Pennington Clay Yeager

This research is supported by a grant from the Pennsylvania Commission on Crime and

Delinquency

Special thanks to the staff at PCCD

3

• Local people solve local problems best.

• People support what they help create.

• Science matters.

SAMHSA’s Prevention Basics

SAMHSA’s Strategic Prevention Framework

Steps

Sustainability & Cultural Competence

Profile population needs, resources, and readiness to address

needs and gaps

Monitor, evaluate, sustain, and improve or replace those that

fail

Implement evidence-based prevention

programs and activities

Develop a Comprehensive Strategic Plan

Mobilize and/or build capacity to address needs

Assessment

Capacity

PlanningImplementation

Evaluation

Why don’t communities see greater success in prevention?

• Chasing money rather than outcomes

• Inability to sustain programs

• No single guiding philosophy (many separate but disconnected efforts)

• Little accountability

• The lack of good data to drive decision-making and resource allocation

• Reliance on untested (or ineffective) programs

• Poor implementation quality

There are a number of proven-effective delinquency prevention programs

• Effectiveness demonstrated in rigorous scientific evaluations (randomized controlled trials)

• Large longitudinal studies or multiple replications (results that are generalizable)

• Significant effects on aggression, youth violence, delinquency, substance use, school failure

These evidence-based programs give us great confidence that if implemented well they will be effective at promoting better youth outcomes

If you build it, they will come…..or

Moving from prevention science to

community-level public health impact

“If you build it”….they may never know about it

…they may not understand what it is

…they won’t know how to get there

…they won’t think it fits

…they’ll think they already have it

…they’ll see it as competition

…they’ll decide they should build their own

…they’ll adapt it into something unrecognizable

…they’ll only use the pieces of it they like

OR

…they will come, and love it.

Then they’ll want you to make ten more just like it in surrounding communities. Now.

Emshoff, J. (2008). Researchers, practitioners, and funders: Using the framework to get us on the same page. American Journal of

Community Psychology, 41 (393-403).

…and miles to go before I sleep• EBPs still represent the minority of prevention programs

• Research has shown that many (most?) aren’t being implemented with sufficient quality or fidelity

• There is tension between advocates of strict fidelity and those who encourage local adaptation

• Very few programs measure or monitor implementation fidelity and quality

• Policy work is needed to capture & redistribute dollars saved through prevention

• Programs are often “pull-out” and operate singularly

• Sustainability remains a challenge – no permanent infrastructure

Greenberg, M. T. (2007). School-Based Prevention: Current Status and Future Challenges. Working paper for IOM/NRC Committee on Prevention of Mental Disorders.

Bumbarger, B. and Perkins, D. (2008). After Randomized Trials: Issues related to dissemination of evidence-based intervention. Journal of Children’s Services,3(2), 53-61.

The challenges

• Having community-based prevention work be more focused and strategic

• Increase (carefully planned) adoption of EBPs by more communities

• Ensure high quality implementation

• Sustain programs long-term

Bridging The Gap Between Science And Practice

• The goal: to ensure better outcomes for children, youth, families, and communities

• The strategy: creating sustained, community-wide public health impact through effective community coalitions using effective programs targeted at strategically identified risk and protective factors

Creating Fertile Ground for EBPsRisk-focused Prevention Planning

(the Communities That Care model)

Collect local data on

risk and protective

factors

Use data to

identify priorities

Select and implement

evidence-based program that

targets those factors

Re-assess risk

and protective

factors

Form local coalition

of key stakeholders

Leads to community

synergy and

focused resource allocation

What is CTC?

• An “operating system” to mobilize communities and agency resources

• Follows a public health model of preventing poor outcomes by reducing associated risk factors and promoting protective factors

• Coalition model that is data-driven and research-based

• Follows a specific sequence of steps

• Focuses on the use of targeted resources and evidence-based prevention programs

Outcomes Targeted by CTC

• Delinquency

• Substance Abuse

• Violence

• Academic Failure & Dropout

• Teen Pregnancy

How is CTC different?

• Uses local data to set priorities and focus resources

• Starts with quantifiable goals

• Engages the whole community

• Addresses youth problems by identifying their (actual) root causes, rather than dealing with them after they occur

• Involves a realistic view of adolescent development and the length of time necessary to change outcomes

• Focuses on the use of proven-effective programs

• Has a built-in process of assessment and accountability

The “Operating System” as a cycle

• Identify and involve key leaders and stakeholders in the community through a community coalition with a shared vision

• Collect data on the prevalence of known risk and protective factors, and problem behaviors, in the community

• Use data to prioritize risk and protective factors to be addressed collectively

• Select and implement evidence-based programs to address prioritized risk and protective factors

• Re-assess prevalence and continue the cycle

Key Leaders and Prevention Board

• Key leaders can impact policy, control resources, and influence public opinion

• Prevention Board members represent all the constituencies that have a stake in the healthy development of children and families

• Community Mobilizer to maintain focus (fidelity) and forward momentum

Risk Assessment

• Based on a public health model that acknowledges known risk and protective factors

• Empirical data on poor outcomes and on mediators that proximally relate to those outcomes

Prioritizing Risk and Protective Factors

• Which factor(s) does the community want to address?

• Which are most prevalent?

• Which can be addressed with available resources?

• Which are most amenable to change?

• Which are politically popular?

• Which will have the greatest impact on outcomes?

Creating Fertile Ground for EBPsRisk-focused Prevention Planning

(the Communities That Care model)

Collect local data on

risk and protective

factors

Use data to

identify priorities

Select and implement

evidence-based program that

targets those factors

Re-assess risk

and protective

factors

Form local coalition

of key stakeholders

Leads to community

synergy and

focused resource allocation

Resource Assessment

• What resources currently exist to address the prioritized risk and protective factors?

• Are these research- or evidence-based?

• Is there evidence that existing efforts are working?

CTC in Pennsylvania

• Adopted as a statewide initiative in 1994

• Over 120 communities trained over 16 cycles

• 88 currently functioning CTC communities

• Dedicated regional technical assistance for CTC to improve coalition functioning

• New technical assistance infrastructure for evidence-based programs (EPISCenter)

• Opportunity to study CTC in a long-term large-scale implementation under real-world conditions

Why PCCD chose CTC?

• Community readiness – prepares “fertile ground” to support a comprehensive community prevention effort before selecting specific programs.

• Programming had often been selected based on the availability of grant funds – we have money so let’s do a program!

• CTC reverses this process – identify a need before choosing a program.

• Make sure that chosen program(s) fits with the risks and strengths of each individual community.

• The CTC risk-focused prevention strategy was the foundation for Evidence-Based Program implementation in Pennsylvania.

The PA CTC initiative currently supports 10 EBPs

• Big Brothers/Sisters

• PATHS

• LifeSkills Training

• Project TND

• Strengthening Families 10-14

• MST

• FFT

• MTFC

• Incredible Years

• Olweus Bullying Prevention

Other Commonwealth agencies also support some of these and other EPBs

Evidence-based Programs Funded Throughout Pennsylvania*

*programs funded under the EBP initiative 1998-2007

Building Infrastructure for Policy,

TA and Evaluation

• Nearly all state agency RFA’s contain language

re: risk and protective factors and quantitative

risk assessment to define need

• Many are also restricted to funding research-

based or evidence-based programs

• Significant TA and Evaluation are also funded

Pennsylvania Youth Survey

PAYS

• A voluntary survey conducted in schools every other year for youth in 6th, 8th, 10th, and 12th grades.

• Adapted from the Communities That Care Youth Survey, with additional questions added to gather data on areas such as gambling, prescription drug abuse and other anti-social behaviors.

• Administered to 19,853 students in 2007; next cycle is Fall 2009.

• Summary of 2007 findings, as well as trend data, available on PCCD’s website.

On Community Coalitions…

“Collaboration is an unnatural act committed by (un)consenting adults.”

Anonymous

“Community mobilization is a lot like a football game… 11 people do all the work while thousands, who could really benefit from the exercise, just watch.”

Joe Paterno…(ok, not really)

“Never doubt that a small group of thoughtful, committed citizens can drive an ambulance off a cliff and throw starfish back into the ocean….indeed, it’s the only blah, blah, blah.”

Margaret What’s her face

32

CTC DomainsThe areas that were studied in CTC-Web 2008

include the following:

CTC Process

Board Leadership

Board Work Style

Board Relationships

Board Membership

Barriers Experienced

Technical Assistance

Programs Implemented

33

CTC Research Team, Prevention Research Center

Pennsylvania State University

Sponsored by

2003 2004 2005 2006 2007 2008

# of CTC Sites 68 79 75 73 72 67

# of participants 570 867 799 929 988 946

% individual

participation

- 46% 50% 60% 62% 62%

Communities That Care

Web-Based SurveyCTC

Web

34

6

7

8

9

Board Cohesion Staff Board Communication

2003 2004 2005

2006 2007 2008

Board Relationship

Board Cohesion

Unity and group spirit

felt within the board

Staff-Board Communication

Frequency and productiveness of

communication between CTC

staff and board members

35

CTC Process

CTC Model Fidelity

Board is able to follow

CTC model elements

and programs are

implemented well

6

7

8

9

Community Support CTC Model Fidelity Sustainability

2003 2004 2005

2006 2007 2008

Sustainability

Board has explored financing

and resource development

strategies, and believes the CTC

process will continue beyond

PCCD funding

Community Support

Understanding & supportive

relationships from

community leaders and

institutions for CTC

PRC’s research on CTC in PA

• Process evaluation of first 21 sites (3

cycles) in 2001

• Identified structural/organizational issues; made

recommendations for strengthening the statewide initiative (www.pccd.state.pa.us/pccd/lib/pccd/stats/final_report-psuctceval.pdf)

• Cross-sectional quasi-experimental study

of 98,000 students in 147 communities

• Longitudinal study of students/schools

over a five-year period (2001-2005)

PRC’s research on CTC in PA

• Process evaluation of first 21 sites (3

cycles) in 2001

• Cross-sectional quasi-experimental study

of 98,000 students in 147 communities

• Found youth in CTC communities reported

lower rates of risk factors, substance use, and

delinquency than youth in similar non-CTC

communities

• First evidence of the effectiveness of a large-

scale community coalition approach

Risk Factors 6 8 10 12

Family Supervision + + + +

Family Discipline + + + +

Family History of Antisocial Behavior + + + +

Parental Attitudes Favorable to ATOD Use + + - +

Academic Failure + - + +

Low School Commitment + + - -

Rebelliousness + + - -

Friends’ Delinquent Behavior + + + +

Friends’ ATOD Use + + + +

Peer Rewards for Antisocial Behavior + + + +

Favorable Attitudes Toward Antisocial Behavior + + - +

Favorable Attitudes Toward ATOD Use + + + +

Low Perceived Risk of ATOD Use + + + +

Early Initiation of Problem Behavior + + + +

Sensation Seeking + + - -

Favors CTC sites at p<.05

Favors CTC sites at p<.10

CTC Sites versus non-CTC sites – 2003 PAYS

Risk Factors 6 8 10 12

Family Supervision + + - +

Family Discipline + + + +

Family History of Antisocial Behavior + + + +

Parental Attitudes Favorable to ATOD Use + + - -

Academic Failure - - + +

Low School Commitment + + + +

Rebelliousness + + - +

Friends’ Delinquent Behavior + + + +

Friends’ ATOD Use + + + +

Peer Rewards for Antisocial Behavior + + + +

Favorable Attitudes Toward Antisocial Behavior + + - +

Favorable Attitudes Toward ATOD Use + + + +

Low Perceived Risk of ATOD Use + + + +

Early Initiation of Problem Behavior + + + +

Sensation Seeking + + - -

Favors CTC sites at p<.05

Favors CTC sites at p<.10

CTC Sites With Age Appropriate Evidence-based Programs versus

non-CTC sites

ATOD and Delinquency Outcomes 6 8 10 12

Past 30-day Alcohol Use + + + +

Past 30-day Cigarette Use + + + -

Past 2-week prevalence of Binge Drinking + + + +

Drunk or high at school in past year + - + +

Delinquency + + + +

Drug Use + + + +

Favors CTC sites at p<.05

Favors CTC sites at p<.10

CTC Sites versus non-CTC sites – 2003 PAYS

ATOD and Delinquency Outcomes 6 8 10 12

Past 30-day Alcohol Use + + + +

Past 30-day Cigarette Use + + + -

Past 2-week prevalence of Binge Drinking + + + +

Drunk or high at school in past year + + + +

Delinquency + + + +

Drug Use + + + +

Favors CTC sites at p<.05

Favors CTC sites at p<.10

CTC Sites With Age Appropriate Evidence-based Programs versus

non-CTC sites

PRC’s research on CTC in PA

• Process evaluation of first 21 sites (3 cycles) in 2001

• Cross-sectional quasi-experimental study of 98,000 students in 147 communities

• Longitudinal study of students/schools over a five-year period (2001-2005)

• Over 231,000 student self-reports

• 419 age/grade cohorts over a 5-year period

• Earlier waves of data act as a control, allowing examination of within-unit change over time

Does the CTC strategy reduce risk and problem behaviors over time?

Risk and problem behaviors typically increase across adolescence. Is there a lower rate of increase in risk and problem behavior for youth in CTC districts?

1. Examined change in risk and problem behaviors from 6th to 12th grade for youth responding to Pennsylvania Youth Survey.

2. Compared change over time for CTC vs. Comparison school districts.

3. CTC cohorts limited to “Expected Impact”

Longitudinal Study Sample

Grade 6th 8th 10th 12th Total

Non-CTC 18,116 25,766 25,094 13,726 82,702

CTC 39,988 43,230 38,466 26,996 148,680

Expected Impact (subsample of CTC)

15,917 18,158 15,525 9,721 59,321

TOTAL 58,104 68,996 63,560 40,722 231,382

Year (# schools) 2001 (91) 2003 (154) 2005 (174)

6th 4 5 6

8th 3 4 5

10th 2 3 4

12th 1 2 3

Condensed Risk/Protective Factors to 7 Indices

• Community Cohesion

• Drug-Firearms Availability

• School Prosocial Support

• Family Cohesion

• Family Risk Factors

• Antisocial Attitudes/Behavior

• Antisocial Peer Influence

Feinberg, M.E., Ridenour, T. A., and Greenberg, M.T. (2007). Aggregated indices of risk and

protective factors in the Communities That Care survey. Journal of Adolescent Health, 40, 506-513.

Average score

at 6th grade,

comparison

districts

Annual

change,

comparison

districts

% reduction in

change due to

CTC

Community Cohesion .36 -.14 4.6

Drug-Firearms Availability .56 -.20 3.0

School Prosocial Support .22 -.12 16.4

Family Cohesion .29 -.12 7.7

Family Risk Factors 1.67 .37 6.7

Antisocial Attitudes/Behavior -.36 .12 6.7

Antisocial Peer Influence 1.59 .40 10.8

Academic Grades 2.25 -.09 33.2

Delinquency 0.18 .21 10.8

Youth in CTC communities report less negative change over time

than youth in comparison districts on the following :(all p < .05)

LONGITUDINAL STUDY RESULTS

Summary of Findings

• Overall, CTC communities showed lower levels of risk factors, substance abuse, and delinquency (7x as many as chance)

• Even greater impact where evidence-based prevention programs were used (11x)

• CTC community youth exposed to universal prevention programs showed slower increase in risk and delinquency, and slower decrease in protection and academic grades.

CTC and Sustainability

• Does CTC aid in sustaining programs?

• Are CTC sites themselves sustainable?

• What fosters CTC sustainability?

PA Communities That Care

0.5

0.6

0.7

0.8

0.9

1.0

0.00 1.00 2.00 3.00 4.00 5.00 6.00

T ime from L a unc h (yea rs)

Cu

mu

lati

ve S

urv

ival

+ /- 1 S E

S urvival

CTC Sustainability: Survival Post State Funding – N=110 Sites

90% of CTC coalitions continued after the three-year initial funding

period, over 60% sustained 6 years post-funding

Is Coalition Connection Related to Program

Sustainability?

67

3643

33

6457

0

10

20

30

40

50

60

70

80

None CTC Other

Pe

rce

nt

Program Grantees off of funding 2 or more years, percentages

Operating

Not operating

PSU Model of CTC: Version 5.0

51

TrainingandT.A.

CTC Coalition

Functioning

-internal -external

CommunityOutcomes

Community readiness

SustainabilitySustainabilityPlanning

CTCModel

Fidelity

ImplementationSupport

Attitudes and

Knowledge

Barriers:infighting, turnover

Bridging The Gap Between Science And Practice:

Moving from Lists to Public Health Outcomes

Enhancing the effectiveness of coalitions: the web-based survey and technical assistance

Enhancing the effectiveness of evidence-based programs: implementation quality, sustainability, and impact assessment

Key Changes to the Process

• Must complete a Risk and Resource Assessment every 2-3 years.

• Matching funds requirement - 25% in Year Three and 50% in Year Four (cash or in-kind)

• Must obtain “Letter of Support” from Program Developer, the CTC Coalition, and other local stakeholders

• Implementation Quality Assurance by Program Developer

• Standardized Performance Measures for each program, plus universal measures of impact

• Focus on Sustainability throughout entire grant, from application through implementation.

Some Lessons Learned

• Focus on risk and protective factors rather than

problems

• Good community-specific data to drive decision

making and resource allocation

• Single state epidemiological survey

• Becomes community needs assessment tied to

multiple agency RFPs

• Community needs assessment and mobilization, as

well as EBP implementation, requires proactive

Technical Assistance

Some Lessons Learned (cont.)

• Fix community readiness and infrastructure issues

first

• Find a small number of things that work, and do

them well

• Multi-year funding is necessary to get to stable

effective services

• Tie funding to quality implementation and outcomes

(objective criteria de-politicizes the process)

• Build in evaluation for continuous quality

improvement

The take-home message…CTC works!

Community readiness+ Data-driven local decision making+ Evidence-based prevention programs+ High-quality implementation & sustainability

= Reduce rates of delinquency and ATOD use ANDConsiderable cost savings

* Feinberg, M.E., Greenberg, M.T., Osgood, W.O., Sartorius, J., Bontempo, D.E. (In Press). Can Community Coalitions Have a Population Level Impact on Adolescent Behavior Problems? CTC in Pennsylvania, Prevention Science.

** Jones, D., Bumbarger, B., Greenberg, M., Greenwood, P., and Kyler, S. (2008). The Economic Return on PCCD’s Investment in Research-based Programs: A cost-benefit assessment of delinquency prevention in Pennsylvania. Prevention Research Center, Penn State University.

CTC Research from the Penn State Prevention Research Center

• Cleveland, M.J., Feinberg, M.E., Bontempo, D.E., & Greenberg, M.T. (in press). The Role of Risk and Protective Factors in Substance Use across Adolescence. Journal of Adolescent Health.

• Feinberg, M., Greenberg, M.T., & Osgood, W.O. (2002). The effects of training in prevention science on community leaders and projects. Evaluation and Program Planning, 25: 245-259.

• Feinberg, M.E. & Greenberg, M.T., & Osgood, W.O. (2004). Readiness, functioning, and perceived effectiveness in community prevention coalitions: A study of Communities That Care. American Journal of Community Psychology, 33: 163-177.

• Feinberg, M.E. & Greenberg, M.T., & Osgood, W.O. (2004). Technical assistance in prevention programs: Correlates of perceived need in Communities That Care. Evaluation and Program Planning, 27: 263-274.

• Feinberg, M.E., Bontempo, D.E., & Greenberg, M.T. (in press). Predictors and Level of Sustainability of Community Prevention Coalitions. American Journal of Preventive Medicine.

• Feinberg, M.E., Gomez, B., Puddy, R., & Greenberg, M. (2008). Evaluation and community prevention: Validation of an integrated web-based/technical assistance model. Health, Education, and Behavior, 35:9-21.

PRC Articles on CTC (cont.)

• Feinberg, M.E., Greenberg, M.T., Osgood, W.O., Sartorius, J., Bontempo, D.E. (In Press). Can Community Coalitions Have a Population Level Impact on Adolescent Behavior Problems? CTC in Pennsylvania, Prevention Science.

• Feinberg, M.E., Ridenour, T.A., & Greenberg (In press). The Longitudinal Effect of Technical Assistance Dosage on the Functioning of Communities That Care Prevention Boards in Pennsylvania. Journal of Primary Prevention.

• Feinberg, M.E., Ridenour, T.A., & Greenberg, M.T. (in press). Aggregated indices of risk and protective factors in the Communities That Care survey. Journal of Adolescent Health.

• Feinberg, M.E., Riggs, N.R., & Greenberg, M.T. (2005). A network analysis of leaders in community prevention. Journal of Primary Prevention, 26: 279-298.

• Gomez, B.J., Greenberg, M.T., & Feinberg, M.E. (2005). Sustainability of prevention coalitions. Prevention Science, 6: 199-202.

• Greenberg, M.T., Feinberg, M.E., & Osgood, W.O. (2005). Testing a community prevention focused model of coalition functioning and sustainability. Editors: Stockwell, T., Gruenewald, P., Toumbourou, J., & Loxley, W. Preventing Harmful Substance Use: The Evidence Base for Policy and Practice.

PRC Articles on CTC (cont.)

• Riggs, N.R., Feinberg, M.E., & Greenberg, M.T. (2002). Positional and gender differences among community prevention leaders. Journal of Community Psychology, 30: 709-721.

• Wells, R., Feinberg, M.E., Alexander, J., & Ward, A. (In Press). Factors Affecting Member Perceptions of Coalition Impact. Nonprofit Management and Leadership.

• Wells, R., Ward, A.J., Feinberg, M., Alexander, A. (In Press). What motivates people to participate more in community-based coalitions? American Journal of Community Psychology.

Thank You!For more information on our research and services, contact:

Brian K. Bumbarger

Evidence-based Prevention and Intervention Support Center (EPISCenter)

Prevention Research Center

Penn State University

206 Towers Bldg.

University Park, PA 16802

EPISCenter@psu.edu

(814) 863-2568