of ptip revention and it tii ntervention rograms on and ...€¦ · inclusion criteria •...

Eff t f P ti d I t ti PEffects of Prevention and Intervention Programs on School Completion and Dropout

Results from a Systematic ReviewResults from a Systematic ReviewSandra Jo Wilson

b d hPeabody Research Institute

The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org

Acknowledgments

• Funding provided by the Campbell Collaboration and the Peabody Research Institute

• My co‐authors:– Mark W. LipseyMark W. Lipsey– Emily E. Tanner‐SmithChiungjung Huang– Chiungjung Huang

– Katarzyna Steinka‐Fry– Jan Morrison– Nianbo Dong

The Campbell Collaboration www.campbellcollaboration.org

Overview

• Discussion of different kinds of interventions that target school dropout.

• Systematic review methodology.• Preliminary results from the systematic review• Preliminary results from the systematic review.


Risk‐oriented Programming

• There are a great variety of strategies for preventing school dropout.

• The different strategies are generally developed out of different theories about the causes of dropout.p

• The biggest risk factors for school dropout are pregnancy low school attendance and poor academicpregnancy, low school attendance, and poor academic performance.


Risk‐oriented Programming

Risk Factor for Dropout Targeted Program or StrategyTeenage pregnancy Provide child care services or financial

assistance to young mothers.Poor attendance Monitor and reward attendance.P d i f P id l t l d i iPoor academic performance Provide supplemental academic services.

Lack of support for higher education Provide college‐oriented curricula and advising for students.g

Family and personal problems Connect students with appropriateservices through case management, provide mentors or counselorsprovide mentors or counselors.

Traditional school structure doesn’t work for some students

Restructure or reorganize the school day, curriculum, etc. to better fit the needs of different types of students.

Students lack purpose, goals, understanding of the purposes of

Create career/work oriented courses;involve students in community.


g p peducation

y

Objectives of the ReviewObjectives of the Review

• Summarize the available evidence on the effects of prevention and intervention programs aimed at increasing school completion or reducing dropout.c eas g sc oo co p et o o educ g d opout

• Primary emphasis on the comparative effectiveness of different types of programs in the context of differentdifferent types of programs, in the context of different methodologies, different levels of implementation quality and different configurations of studentquality, and different configurations of student characteristics.


Inclusion Criteria

• Interventions are school‐based, school‐affiliated, or community‐based programs.

• Outcomes must be dropout, graduation, or attendance.p g• Students are between the ages of 4‐18.

Recent dropouts (18 22) are eligible if school completion is– Recent dropouts (18‐22) are eligible if school completion is explicit goal.For students under the age of 12 dropout must be assessed– For students under the age of 12, dropout must be assessed.

• Experimental and quasi‐experimental designs are l d dincluded.

• 1985 and later, any language, any publication type.


Literature Search• Comprehensive literature search of major educational databases, including grey literature.databases, including grey literature.

• Search of international databases not yet completed.Id ifi d 16 962• Identified 16,962 reports– 2,627 reports were deemed potentially relevant and retrieved

– 878 reports were determined to be eligible878 reports were determined to be eligible• 416 studies (188 with dropout; 228 with attendance)

– 164 coded studies (so far)164 coded studies (so far)– 450 dropout effect sizes from 302 independent study samples


Study CodingStudy Coding

• Study coding included information about the programs• Study coding included information about the programs under evaluation, the students in those programs, and the methods used to evaluate the programs.methods used to evaluate the programs.

• Outcomes indexed using odds ratios.Ratio of the odds of completing school for treatment– Ratio of the odds of completing school for treatment participants to the odds of completing school for control participantsparticipants

– Odds ratio < 1 means control groups are favored; Odds ratio > 1 means treatment groups are favored; Oddsratio > 1 means treatment groups are favored; Odds ratio = 1 means odds are equal for the two groups.


Treatment Typesyp

• The 302 independent samples were divided into twoThe 302 independent samples were divided into two groups

269 general programs– 269 general programs• 299 effect sizes measuring dropout, graduation, GED, or enrollment for general population & at‐risk samples.e o e t o ge e a popu at o & at s sa p es

– 33 teen parent (mostly teen mothers) programs• 150 effect sizes measuring dropout, graduation, GED, or150 effect sizes measuring dropout, graduation, GED, or enrollment for teen parents.

– NOTE: enrollment defined as not dropped out, not yet pp , ygraduated, and still attending school.


How effective were dropout programs overall?

Mean ORLower CI

Upper CI nes Ksamples Q tau2

G l T 1 68 1 52 1 88 300 269 1712 3* 23General Tx 1.68 1.52 1.88 300 269 1712.3* .23Teen Parent Tx 1.75 1.41 2.17 150 33 191.3* .37Random effects analysis

Mean Odds Ratios Translated to PercentagesMean Odds Ratios Translated to Percentages

TX % CT %

General Tx Programs (Percent dropped out) 33% 45%Teen Parent Tx Programs (Percent enrolled) 41% 28%


How effective are dropout programs overall?• After treatment, the odds of completing school were significantly better for students in preventionsignificantly better for students in prevention programs than for students who received no special programmingprogramming.– Treated students were less likely to drop out, and more likely to graduate obtain a GED or stay enrolled inlikely to graduate, obtain a GED, or stay enrolled in school.

Heterogeneity was evident for both the general• Heterogeneity was evident for both the general programs and for the teen parent programs.

• What study characteristics are associated with that heterogeneity?


Characteristics of the Research

• Literature on dropout largely unpublished; 71% of the studies were technical reports (66%) or dissertations (5%).

• Evaluator role & routine practicep– Some evaluators were closely involved in the research; other programs had independent evaluators.other programs had independent evaluators.

– About 50% were routine practice programs.55% of the studies used random assignment• 55% of the studies used random assignment.


Characteristics of the Students

• 88% were high school aged samples; 12% were middle school or younger samples.

• Mixed ethnicity groups of students were common. y g p75% of the student samples were mostly minority students.

• With the exception of specialized programs for teenage mothers most programs were delivered toteenage mothers, most programs were delivered to mixed gender groups.N l ll t d t t i k f d i t d• Nearly all students were at risk for dropping out and were from low socioeconomic status families.


Characteristics of the Dropout ProgramsCharacteristics of the Dropout Programs

• Community‐based programs make up 15% of the sample; the remaining 85% are school‐based or school‐affiliated.

• Duration averaged 90 weeks (typical school year was coded as 38 weeks).)

• Frequency of treatment varied, but over half of the programs involved daily contactprograms involved daily contact.

• Implementation quality also varied: 32% experienced l bl 15% ll d d t ibl blclear problems, 15% alluded to possible problems, and 53% experienced no problems or mentioned no

blThe Campbell Collaboration www.campbellcollaboration.org

problems.

Treatment Types: General ProgramsTreatment Types: General ProgramsTreatment Type Description N

School or class restructuring Small learning communities block schedules 109School or class restructuring Small learning communities, block schedules, career academies, small class size.

109

Vocational training Coursework, internships, or employment oriented d k i

41toward work or career interests.

Supplemental academicservices

Remedial education, tutoring, homework assistance, etc.

25

Community service Programs involved planning and carrying out a community service project (commonly coupled with a weekly life skills curriculum).

24

y )Attendance monitoring & contingencies

Monitoring and services to increase attendance; some offer financial incentives.

17

C ll i t d C ll t i l ll i t d 12College‐oriented programming

College preparatory curriculum, college‐oriented academic advising.

12

Case management Programs revolved around connecting students & 11


families with appropriate services.

Treatment Types: General ProgramsT T D i iTreatment Type Description nMentoring, counseling Programs provided adult mentors or trained

counselors for students. Though mentors focused 11

more on career/work, both mentors and counselors dealt with students’ personal issues.

Skills training, including CBT Generally oriented toward improving self‐esteem 6g, g y p gor attitudes about school, or preventing drug use.

Multi‐service package Large, comprehensive programs; generally included academic vocational and case

6included academic, vocational, and case management services (e.g., Job Corps)

Family outreach School contact with parents and families 3

Other Recreational, residential services for homeless, etc.

4


Treatment Types for Teen ParentsT T D i iTreatment Type Description nAttendance monitoring & contingencies

Monitoring and services to increase attendance; most offer financial incentives and/or

22

contingencies tied to welfare.Case management Programs revolved around connecting teen

parents with appropriate services for parents and 6

p pp p pchildren.

Parenting skills training Parent skills training (and often child care services) for teen mothers

3services) for teen mothers.

School or class restructuring Teenage Pregnancy & Parenting Program 1

l k l hMulti‐service package Family Growth Center 1


Analytic PlanAnalytic Plan• Meta‐regression models with robust standard errors used to examine influence of study characteristics on treatment effects.

• Regression models were then used to calculate covariate adjusted posttest effect sizes for eachcovariate adjusted posttest effect sizes for each treatment type, adjusting for method and subject h t i ti th t f dcharacteristics that may confound group differences.

• Models run separately by program groups (general vs. teen parent)


(g p )

Meta‐regression Results for General ProgramsGeneral Programs b seMethod VariablesRandom assignment design ‐.159 .157Matched groups design ‐.051 .177

( )Attrition (%) ‐.041 .400Data in OR adjusted ‐.300 * .113Subject CharacteristicsSubject CharacteristicsGender mix .049 .045Ethnicity mostly white .059 .123Treatment CharacteristicsRole of evaluator (higher=more independent) ‐.144 † .083Implementation quality (higher better) 173 * 065Implementation quality (higher=better) .173 * .065Treatment duration (weeks) ‐.000 .001Frequency of service per week (hours) ‐.004 .004


00 00nes=299; k=268

Study Characteristics associated with Treatment yEffectiveness: General Programs

• Effect sizes calculated with covariate‐adjusted data tended to be smallertended to be smaller.

• Study design and attrition not significantly associated with treatment effects.

• Gender mix and ethnicity not associated with differential effects.


Study Characteristics associated with Treatment yEffectiveness: General Programs

• Studies in which evaluators were closely involved produced larger effectsproduced larger effects.

• Implementation quality resulted in better outcomes.• Treatment length and treatment frequency not associated with effects.


Treatment

Family Outreach

Type

3

n

3

k

0.88 (0.68, 1.14)

OR (95% CI)

0.88 (0.68, 1.14)

OR (95% CI)

Monitoring & Contingencies

Multi‐service Package

Case Management

21

8

18

17

6

11

1.09 (0.93, 1.27)

1.17 (0.67, 2.04)

1.21 (0.79, 1.87)

1.09 (0.93, 1.27)

1.17 (0.67, 2.04)

1.21 (0.79, 1.87)Case Management

Skills Training

Counseling, Mentoring

S l A d i T i i

18

5

13

27

11

5

11

25

( , )

1.34 (1.07, 1.70)

1.43 (0.98, 2.09)

1 52 (1 13 2 05)

( , )

1.34 (1.07, 1.70)

1.43 (0.98, 2.09)

1 52 (1 13 2 05)Suppl. Academic Training

School or Class Restructuring

Other Treatments

27

122

5

25

109

4

1.52 (1.13, 2.05)

1.78 (1.54, 2.06)

2.23 (1.31, 3.81)

1.52 (1.13, 2.05)

1.78 (1.54, 2.06)

2.23 (1.31, 3.81)

Vocational Training

Community Service

College‐oriented Program

41

24

12

41

24

12

2.32 (1.91, 2.82)

2.57 (1.48, 4.47)

2.85 (1.69, 4.81)

2.32 (1.91, 2.82)

2.57 (1.48, 4.47)

2.85 (1.69, 4.81)

Favors CT Favors TX

1.5 1 1.5 2 2.5 3 5.1


Favors CT Favors TX

Mean Adjusted Odds Ratios for General Programs

Summary for General Programs• All general programs except family outreach had positive effects on dropout, graduation, or continued enrollment.– Attendance monitoring, case management, multi‐service g, g ,packages, and counseling/mentoring programs not significant.

– College‐oriented programming, community service, vocational training, supplemental academic training, and g gschool restructuring programs were the most effective, net of study characteristics in the regression models.

– Overlapping confidence intervals for program types indicates that programs were not significantly different


from each other.

Translating Adjusted Means

• Recall that our overall average odds ratio = 1.68– Translation: 45% of control students drop out vs. 33% of treated students

• Adjusted OR for school restructuring programs = 1.78– Translation: 45% of control students drop out vs. 32% ofTranslation: 45% of control students drop out vs. 32% of treatment students

• Adjusted OR for vocational training = 2 32• Adjusted OR for vocational training = 2.32– Translation: 45% of control students drop out vs. 26% of treatment studentstreatment students


Meta‐regression Results for Teen Parent Programs

Teen Parent Programs b seM th d V i blMethod VariablesRandom assignment design ‐.596 .391Matched groups design ‐2.051 * .490g p g 2.051 .490Attrition (%) 1.188 * .520Data in OR adjusted .534 * .234Subject CharacteristicsEthnicity mostly white .043 .138Treatment CharacteristicsTreatment CharacteristicsRole of evaluator (higher=more independent) ‐.459 * .178Implementation quality (higher=better) .979 * .287Treatment duration (weeks) ‐.014 * .005Frequency of service per week (hours) ‐.005 .019

k


nes=150; k=33

Study Characteristics associated with Treatment Effectiveness: Teen Parent Programs

• Effect sizes calculated with covariate‐adjusted data tended to be smallertended to be smaller.

• Study design and attrition both significantly associated ith t t t ff twith treatment effects.

– Random assignment and matched designs produced ll dd h d h dsmaller odds ratios than non‐random, non‐matched

designs.• Ethnicity not associated with differential effects.


Study Characteristics associated with Treatment Effectiveness: Teen Parent Programs

• Studies in which evaluators were closely involved produced larger effectsproduced larger effects.

• Implementation quality resulted in better outcomes.• Shorter treatments were associated with larger treatment effects.


Treatment

Type n k OR (95% CI)OR (95% CI)

School or Class Restructuring

Parent Training & Child Care

4

4

3

3

1.59 (1.31, 1.93)

1.59 (1.31, 1.93)

1.59 (1.31, 1.93)

1.59 (1.31, 1.93)g

Case Management 15 6

( , )

1.62 (1.33, 1.98)

( , )

1.62 (1.33, 1.98)

Monitoring & Contingencies 125 22 1.74 (1.56, 1.94)1.74 (1.56, 1.94)

Favors CT Favors TX 1.5 1 1.5 2 2.5 3

d d dd f


Mean Adjusted Odds Ratios for Teen Parent Programs

Summary for Teen Parent Programs

• All four treatment strategies for teen parents were significantly better than controls for keeping students enrolled in school.

• The most common strategy (attendance monitoring gy ( gand financial incentives) had the largest adjusted odds ratio, though no strategy was significantly better thanratio, though no strategy was significantly better than another.


Final CommentsFinal Comments

Dropout programs are generally effective with• Dropout programs are generally effective, with negligible differences between different strategies h t d h t i ti t ll dwhen study characteristics are controlled.

• Study methods are influential.• Implementation quality is strongly associated with better treatment outcomes.

• Studies in which evaluators were closely involved with treatment delivery tended to produce bettertreatment delivery tended to produce better outcomes.


Sandra Jo Wilsonemail: [email protected]

The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org

of ptip revention and it tii ntervention rograms on and ...€¦ · inclusion criteria •...

Documents