ORI GI NALPAPERA Meta-Analysis of After-School Programs That Seek to PromotePersonalandSocialSkillsinChildrenandAdolescentsJosephA.DurlakRogerP.WeissbergMollyPachanPublishedonline:19March2010SocietyforCommunityResearchandAction2010Abstract Ameta-analysisof after-school programsthatseektoenhancethepersonal andsocial skillsofchildrenand adolescents indicated that, compared to controls,participants demonstrated signicant increases in theirself-perceptions and bonding to school, positive socialbehaviors, school gradesandlevelsofacademicachieve-ment, and signicant reductions in problem behaviors. Thepresence of four recommendedpractices associatedwithpreviously effective skill training (SAFE: sequenced,active, focused, andexplicit) moderatedseveral programoutcomes. Oneimportantimplicationofcurrentndingsisthat ASPs should contain components to foster the personaland social skills of youth because youth can benet inmultiple ways if these components are offered. The secondimplicationisthatfurtherresearchiswarrantedonidenti-fyingprogramcharacteristicsthat canhelpusunderstandwhysomeprogramsaremoresuccessfulthanothers.Keywords After-school Meta-analysis Socialcompetence Socialskills YouthdevelopmentIntroductionWhatisknownabouttheimpactofafter-schoolprograms(ASPs)? Considerable attention has focused on the academicbenets of ASPs. The results of two large-scale evaluationsof twenty-rst CenturyCommunityLearningCenters(21CCLCs) that is, centers that received federal fundingthroughNoChildLeft Behindlegislationhavegeneratedcontroversy. Neithertheevaluationofcentersservingele-mentary (James-Burdumy et al. 2005) or middle schoolstudents (Dynarski et al. 2004) found any signicant gains inachievement test scores, although there were some gains insecondary outcomes such as parental involvement in schoolandstudentcommitmenttohomework.ThesendingsledsometosuggestdrasticreductionsinthelevelsoffederalnancialsupportforASPs, whichhadreachedonebilliondollars a year by 2002 (Mahoney and Zigler 2006).However, researchershavediscussedseveral methodo-logical issues that limit the interpretation of the resultsof the national evaluations of 21 CCLCs (Kane 2004;Mahoney and Zigler 2006). Depending on the age group inquestion, these include the lack of initial group equivalence,high attrition among respondents, lowlevels of studentattendance, and the possible nonrepresentativeness ofevaluatedprograms. Thereisalsotheproblemoftreatingcenters as though they provided a uniformapproach toacademic assistance when they clearly did not. While some21CCLCs providedstudents withintensive small groupinstructionorindividualtutoring,othersmerelyaskedstu-dentstoworkindependentlyonhomework.Insteadof focusingontheresultsof onlytwoevalua-tions out of many, a well-done meta-analysis that evaluatesabroadsampleofrelevantstudiescarefullycanassessthemagnitudeof changeondifferent outcomes andidentifysomeoftheimportant characteristicsofprogramsassoci-atedwithmore positive results. For example, the meta-analysis of 35 outcome studies by Lauer et al. (2006) led totheconclusionthat ASPscanhavepositiveeffectsonthe achievement of academically at-risk students (p. 303).J. A.Durlak(&) M.PachanDepartmentofPsychology, LoyolaUniversityChicago,6525N.SheridanRoad, Chicago, Il60626, USAe-mail:jdurlak@luc.eduR. P.WeissbergDepartmentofPsychology, UniversityofIllinoisatChicago&CollaborativeforAcademic, Social, andEmotionalLearning(CASEL), Chicago,Il, USA1 3AmJCommunityPsychol(2010)45:294309DOI10.1007/s10464-010-9300-6Signicant gains inreadingor math achievement, or inboth areas, were observed for elementary, middle, and highschool students, and the latter group showed the mostimprovement in both areas. Although the results of a meta-analysis are never denitivelyconclusive, Lauer et al.s(2006)resultsbegintoclarifywhichprogramparticipantsmight be more likely to derive academic benets fromASPs.WhatAboutPersonalandSocialBenets?The recent focus on the academic benets of ASPs tends tooverlookthe fact that manyASPs were initiallycreatedbased on the idea that young peoples participation inorganizedactivities after school wouldbe benecial fortheirpersonalandsocialgrowth.Whileotherfactorshaveinuencedthe growth ofASPsinthe United States,oneofthe goals of manycurrent programs is tofoster youthspersonalandsocialdevelopmentthrougharangeofadult-supervisedactivities.Moreover,substantialdevelopmentalresearchsuggests that opportunities toconnect withsup-portive adults, and participate with peers in meaningful andchallengingactivities inorganizedASPs canhelpyouthdevelopandapplynewskillsandpersonal talents(EcclesandTempleton2002; Mahoneyet al. inpress; NationalResearch Council and Institute of Medicine 2002). In otherwords, ASPs can be a prime community setting forenhancingyoungpeoplesdevelopment.Nevertheless,studiesevaluatingthepersonalandsocialbenetsofASPshaveproducedinconsistent ndingsthatarefurther complicatedbyvariationsinthedesigns, par-ticipants, andtypes of outcomes assessedacross studies(HarvardFamilyResearchProject2003;Mahoneyetal. inpress; RiggsandGreenberg2004). Just asthemeta-anal-ysisbyLaueretal.(2006)soughttoclarifythenatureandextentofsomeoftheacademicbenetsofASPs, thecur-rentstudyappliedmeta-analytictechniquesinanefforttoexamine the personal and social benets of participation inASPs. No previous meta-analysis has systematicallyexaminedtheoutcomesof ASPsthat attempt toenhanceyouthspersonal andsocial skillsinordertodescribethenature and magnitude of the gains from such programs, andto identify the features that characterize more effectiveprograms. Thesearethetwoprimarygoalsofthecurrentreview.All the programs inthe current reviewwere selectedbecause they included within their overall mission thepromotion of youths personal and social development.AlthoughsomeASPsofferamixofactivitiesthatincludeacademic, social, cultural, and recreational pursuits, thecurrent reviewconcentratesonthoseaspectsofeachpro-gramthat aredevotedtodevelopingyouthspersonal andsocialskills.ImpactofSkillTrainingThereisextensiveevidencefromawiderangeofpromo-tion, prevention, and treatment interventions that youth canlearn personal and social skills (Collaborative for Aca-demic, Social, and Emotional Learning [CASEL] 2005;CommissiononPositiveYouthDevelopment 2005; Loseland Beelman 2003). Programs that enhance childrenssocial and emotional learning (SEL) skills cover such areasasself-awarenessandself-management (e.g., self-control,self-efcacy), social awareness and social relationships(e.g., problemsolving, conict resolution, andleadershipskills) and responsible decision-making (Durlak et al.2009). Our rst hypothesis was that ASPs attemptingtofosterparticipantsSELskillswouldbeeffectiveandthatyouthwouldbenet inmultipleways. Weexaminedout-comesinthreegeneral areas: feelingsandattitudes, indi-catorsof behavioral adjustment, andschool performance.Positiveoutcomeshavebeenobtainedinthesethreeareasfor school-based SEL interventions that target youthspersonal and social skills (Durlak et al. 2009), and wehypothesized that a similar pattern of ndings wouldemergeforsuccessfulASPs.RecommendedPracticesforEffectiveSkillTrainingSeveral authors haveofferedrecommendations regardingtheprocedurestobefollowedfor effectiveskill training.Forinstance,thereisbroadagreementthatstaffarelikelyto be effective if they use a sequenced step-by-step trainingapproach, emphasize active forms of learning so that youthcan practice new skills, focus specic time and attention onskill training, andclearlydenetheir goals(Arthur et al.1998; Bond and Hauf 2004; Durlak 1997, 2003; Dusenburyand Falco 1995; Gresham1995; Ladd and Mize 1983;SalasandCannon-Bowers2001).Moreover,thesefeaturesareviewedas important incombinationwitheachotherrather than as independent contributing factors. Forexample, sequenced training will not be as effective ifactive forms of learning are not used, and the latter will notbeas helpful unless theskills that aretobelearnedareclearlyspecied.Althoughtheaboverecommendations aredrawnfromskilltraininginterventionsthathaveprimarilyoccurredinschool andclinical settings, weexpectedthemtobesim-ilarly important in ASPs. Therefore, we coded for thepresence of the four above features using the acronymSAFE (Sequenced, Active, Focused and Explicit). Wehypothesizedthatstaffthatfollowedallfourofthesefea-tureswhentheytriedtopromotepersonalandsocialskillswouldbemoreeffectivethanstaffthatdidnotincorporateallfourduringskilldevelopment.AmJCommunityPsychol(2010)45:294309 2951 3For example, newskills cannot be acquired immedi-ately. It takes time and effort to develop new behaviors andmorecomplicatedskillsmustbebroken downinto smallerstepsandsequentiallymastered. Therefore, acoordinatedsequence of activities is requiredthat links the learningsteps and provides youth with opportunities to connectthesesteps. Usually, this occurs throughlessonplans orprogrammanuals, particularly if programs use or adaptestablishedcurricula. Gresham(1995)hasnotedthat it isimportant to help children learn how to combine, chainand sequence behaviors that make up various social skills(p. 1023).Youthdohavedifferent learningstyles, andsomecanlearnthroughavarietyof techniques, but evidencefrommanyeducationalandpsychosocialinterventionsindicatesthatthemosteffectiveandefcientteachingstrategiesformanyyouthemphasize active forms of learning. Youngpeople often learn best by doing. Salas and Cannon-Bowers(2001)stressthatItiswelldocumentedthatpracticeisanecessaryconditionforskillacquisition(p. 480).Active forms of learning require youthtoact onthematerial. That is, after youth receive some basic instructionthey should then have the opportunity to practice newbehaviors andreceivefeedbackon theirperformance.Thisis typicallyaccomplishedthroughroleplayingandothertypes of behavioral rehearsal strategies, andthecycleofpracticeandfeedbackcontinuesuntilmasteryisachieved.Thesehands-onformsoflearningaremuchpreferredoverexclusively didactic instruction, which rarely translates intobehavioralchange(Durlak1997).Sufcient time and attention must be devoted to any taskforlearningtooccur(Focus). Therefore, staffshoulddes-ignatetimethatisprimarilydirectedatskilldevelopment.Some sources discuss this feature in terms of training beingof sufcient dosage or duration. Exactly how many trainingsessions are neededis likelytodependonthetype andnatureof thetargetedskills, but implicit inthenotionofdosage or duration is that specic time, effort, and attentionshould be devoted to skills training. We coded programs onfocus because of its relevanceto the current meta-analysis.Although all reviewed programs indicated their intention todevelopyouths personal andsocial skills, somedidnotmention any specic program components or activities thatwere specically devoted to skill development. Weexaminedhowprogramdurationrelatedtooutcomesinaseparateanalysis.Finally, clear andspeciclearningobjectives arepre-ferredover general ones (Explicit). Youthneedtoknowwhat they are expected to learn. Therefore, staff should nottarget personal andsocial development ingeneral terms,but identifyexplicitlywhatskillsintheseareasyouthareexpected to learn (e.g., self-control, problem-solving skills,resistanceskills, andsoon).Insum, thecurrent meta-analysisofASPsthat attempttofosterthepersonalandsocial skillsofprogrampartici-pants was conductedwiththeexpectationthat suchpro-grams would yield signicant effects across a range ofoutcomes, andthat theapplicationof four recommendedpractices during the skill development components of ASPswouldmoderateprogramoutcomes.MethodAnASPinthismeta-analysiswasdenedasanorganizedprogramofferingoneormoreactivitiesthat: (a)occurredduring at least part of the school year; (b) happened outsideofnormalschoolhours;and(c)wassupervisedbyadults.Inadditiontomeetingthisdenition,theASPhadtomeetthe inclusioncriterionof havingas one of its goals thedevelopment of one or morepersonal or social skills inyoungpeoplebetweentheagesof5and18.Thepersonaland social skills could include any one or a combination ofskillsinareassuchasproblem-solving,conictresolution,self-control, leadership, responsible decision-making, orskills relatedtotheenhancement of self-efcacyor self-esteem.Includedreportsalsohadtohaveacontrolgroup,presentsufcientinformationsothateffectsizescouldbecalculated,andappearbyDecember31,2007.Althoughitwasnotaformalcriterion, alltheincludedreportsdescri-bedprogramsconductedintheUnitedStates.Evaluations that only focused on academicperformanceor school attendance and only reported academic outcomeswere excluded, as were reports on adventure education andOutwardBoundprograms, extra-curricular school activi-ties, andsummer camps. These types of programs havebeen reviewed elsewhere (Bodilly and Beckett 2005;CasonandGillis1994; HarvardFamilyResearchProject2003).LocatingRelevantStudiesThemajor goal of thesearchprocedureswastosecureanonbiased representative sample of studies by conducting asystematicsearchfor publishedandunpublishedreports.Four primaryprocedureswereusedtolocatereports: (a)computersearchesofmultipledatabases(ERIC,PsycInfo,Medline, andDissertationAbstracts)usingvariantsofthefollowing search terms, after-school, out-of-school-time,school, students, social skills, youth development, children,andadolescents(b)handsearchesofthecontentsofthreejournals publishingthe most outcome studies (AmericanJournalofCommunityPsychology,JournalofCommunityPsychology, and Journal of Counseling Psychology),(c) inspection of the reference lists of previous ASPreviews and each included report, and (d) inspection of the296 AmJCommunityPsychol(2010)45:2943091 3database on after-school research maintained by theHarvard Family Research Project (2009) from which manyunpublished reports were identied and obtained. The datesof the literature search rangedfromJanuary1, 1980toDecember 31, 2007. Although no review can be absolutelyexhaustive, we feel that the study sample is a representativegroupofcurrentprogramevaluations.StudySampleResultsfrom75reportsevaluating69different programswereevaluated.Severalreportspresenteddataonseparatecohorts involved in different ASPs, each with its owncontrol group, and these interventions were treated asseparate programs. In the 75 evaluations, 68 assessedoutcomesat post; 8alsocollectedsomefollow-upinfor-mation, and7onlycontainedfollow-updata. Post effectswerebasedontheendpoint of theyouths programpar-ticipation. That is, onthose occasions whentworeportswereavailableonthesameparticipantsandonecontainedresults after 1year of participation while the secondofferedinformationafter2yearsofparticipation,onlythelatter data were evaluated. The nal study sample con-tainedexamples of 21st CCLCs, programs conductedbyBoysandGirlsand4-HClubs, andavarietyoflocal ini-tiatives developed and supported by various communityandcivicorganizations.IndexofEffectThe indexof effect was a standardizedmeandifference(SMD) that was calculated whenever possible by sub-tracting the mean of the control group from the mean of theafter school group at post (and at follow-up if relevant) anddividing by the pooled standard deviation of the twogroups. If meansandstandarddeviationswerenot avail-able, theneffectswereestimatedusingproceduresdescri-bed by Lipsey and Wilson (2001). When results werereportedas nonsignicant andnoother informationwasavailable,theeffectsizeforthatoutcomemeasurewassetat zero. There were 38 imputed zero effects and thesevalues were not signicantlyassociated with anycodedvariables.Each effect was corrected for small sample bias andweighted by the inverse of its variance prior to any analysis(HedgesandOlkins1985). Largereffectsaredesiredandreecta strongerpositiveimpactontheafter-schoolgroupcomparedtocontrols. Wheneverpossible,weadjustedforanypre-interventiondifferences betweengroups oneachoutcomemeasurebyrstcalculatingapreSMDandthensubtractingthis pre SMDfromthe obtainedpost SMD.This strategy has been used in other meta-analyses (Derzon2006;Wilsonetal. 2001).Theconsistent strategyintreatingSMDswastocalcu-lateoneeffect sizeper studyfor eachanalysis. Inotherwords, fortherst analysisoftheoveralleffectsfromall68 programs at post, we averaged all the effect sizes withineachstudysothateachstudyyieldedonlyoneeffect.Forthe subsequent analyses by outcome category, if there weremultiplemeasuresfromaprogramfor thesameoutcomecategory, they were averaged so that each study contributedonly one effect size for that type of outcome. For example,if SMDs frommeasures of self-esteemandself-conceptwere available in the same study, the data were averaged toproduceasingleeffectreectingself-perceptions.Arandomeffects model was usedintheanalyses. Arandomeffects model assumes that variation in SMDs acrossstudiesistheresultofbothsamplingerroranduniquebutrandom features of each study, and the use of such a modelpermits a broader range of generalization of the ndings. Atwo-tailed.05probabilitylevel was usedthroughout theanalyses. Meaneffects for different studygroupings arereported along with .05 condence intervals (CI). Moreover,homogeneityanalyses were conductedtoassess whethermeanSMDsestimatethesamepopulationeffect. Homo-geneityanalyses were basedonthe Qstatistic whichisdistributedasachi-squarewith k-1degreesoffreedom,where k=the number of studies. For example, when studiesaredividedforanalysistoassesspossiblemoderatorvari-ables, Qstatisticsassessthestatistical signicanceof thevariabilityineffectsthat existswithinandbetweenstudygroups. Inaddition, wealsousedtheI2statistic(Higginsetal.2003)whichindicatesthedegreeratherthanthesta-tistical signicanceofthevariabilityofeffects(heteroge-neity) among a set of studies along a 0100% scale.CodingAcoding systemwas developed to capture basic studyfeatures, methodological aspects of the programevalua-tion, andcharacteristicsoftheASP, participants, andout-comes. The coding of most of the variables isstraightforward and only a fewvariables are describedbelow.MethodologicalFeaturesTwo primary methodological features were coded aspresent orabsent:useofarandomizeddesign, anduseofreliableoutcomemeasures. Thereliabilityof anoutcomemeasurewasconsideredacceptableifitsalphacoefcientwasC0.70, oranassessmentofinter-judgeagreementforcodedorratedvariableswasC.70(forkappa, C.60). Wecodedreliabilityinadichotomousfashionbecauseseveralreports offered no information on reliability. A thirdmethodvariable, attrition, wasmeasuredonacontinuousAmJCommunityPsychol(2010)45:294309 2971 3basis as the percentage of the initial sample that wasretainedinthenalanalyses(possiblerange0100%).OutcomeCategoriesOutcomedataweregroupedintoeightcategories. Twoofthese assessed feelings and attitudes (child self-perceptionsand bonding to school); three were indicators of behavioraladjustment (positivesocial behaviors, problembehaviors,anddruguse), andthreeassessedaspects of school per-formance (achievement test scores, grades, and schoolattendance).Self-perceptionsSelf-perceptions included measures of self-esteem, self-concept, self-efcacy and in a fewcases (four studies)racial/cultural identityor pride. School bondingassessedpositivefeelings andattitudes towardschool or teachers(e.g., likingschool, or reports that the school/classroomenvironment or teachers are supportive). Positive socialbehaviors measured positive interactions with others.These are behavioral outcomes assessing suchthings aseffectiveexpressionoffeelings, positiveinteractionswithothers, cooperation, leadership, assertiveness in socialcontexts or appropriate responses to peer pressure orinterpersonal conict. Problembehaviors assessed dif-culties that youth demonstrated in controlling theirbehavioradequatelyinsocialsituations, andincludeddif-ferent types of acting-out behaviors suchas noncompli-ance, aggression, delinquent acts, disciplinary referrals,rebelliousness,andothertypesofconductproblems.Druguse primarily consisted of youth self-reports of their use ofalcohol, marijuana, or tobacco. Achievement test scoresreectedperformanceonstandardizedschoolachievementtests typically assessing reading or mathematics. Schoolgradeswereeitherdrawnfromschoolrecordsorreportedby youth and reected performance in specic subjectssuchasreading, mathematicsorsocial studies, oroverallgradepoint average. School attendanceassessedthefre-quencywithwhichstudentsattendedschool.SAFEFeaturesThepresenceofthefourrecommendedpracticesforskilltraining was coded dichotomously on a yes/no basis.Sequenced: Doestheprogramuseaconnectedandcoor-dinatedsetofactivitiestoachievetheirobjectivesrelativetoskilldevelopment?Active:Doestheprogramuseactiveformsoflearningtohelpyouthlearnnewskills? Focused:Doestheprogramhaveatleastonecomponentdevotedtodeveloping personal or social skills? Explicit: Does theprogramtargetspecicpersonalorsocialskills?Programsthat met all four criteria were designatedas SAFEpro-gramswhilethosenotmeetingallfourcriteria were calledOtherprograms.ReliabilityofCodingReliability was estimated by randomly selecting 25% of thestudies that were then coded independently by the rstauthorandtrainedgraduatestudentassistantswhoworkedat different timeperiods. Kappacoefcientscorrectedforchangeagreementwereacceptableacrossallcodes(0.700.95, average=0.85) anddisagreements incodingwereresolvedthroughdiscussion. Theproduct moment corre-lationsforcodingcontinuousitemsincludingthecalcula-tionofeffectswereallabove0.95.ResultsTable1 summarizes several features ofthe 68 studies withpost data. Sixty-seven per cent of the studies appeared after2000,andthemajoritywereunpublishedtechnicalreportsordissertations(k=51, or68%). Nearlyhalfofthepro-grams served elementary students (46%), over a thirdservedstudentsinjuniorhigh(37%), andafewinvolvedhigh school students (9%; six evaluations did not report theageof participants). Intermsof methodological features,35%employedarandomizeddesign, meanattritionwas10%, andreliabilitywasreportedandwasacceptablefor73%oftheoutcomemeasures.Twenty-vestudiesdidnot specifytheethnicityoftheparticipants at post, and the remaining 43 reported thisinformation in various ways. Among the latter studies,participating youth were predominantly ([90%) AfricanAmericanintenstudies; Latinoinsixstudies, AsianorPacicIslander inthreestudies, andAmericanIndianinone study. There was no information on the socioeconomicstatus of the participants families innearlyhalf of thereports (k=31, or 46%). Basedonthewayinformationwasreported inthe remainingstudies,17 studiesprimarilyservedalow-incomegroup(25%) and13studies (19%)servedyouthfrombothlow-andmiddle-incomelevels.OverallImpactatPostFirst, weinspectedthedistributionofeffectsandWinsor-ized three values that were C3 standard deviations from themean(i.e., reset thesevaluestothreestandarddeviationsfrom the mean). The Winsorized study level effects, whichranged in value from -0.16 to ?0.85, had an overall meanof ?0.22(CI=0.160.29), whichwas signicantlydif-ferent fromzero. Thesedataindicatethat ASPs haveanoverall positive and statistically signicant impact on298 AmJCommunityPsychol(2010)45:2943091 3participatingyouth. However, there was statisticallysig-nicant variabilityinthedistributionof effects basedontheQstatistic(Q=306.42, p \.001), andahighdegreeof variabilityaccordingtotheI2value(78%) suggestingthe need to search for moderator variables that mightexplainthisvariabilityinprogramimpact.InWhatWaysDoYouthChange?Table2presents themeaneffects obtainedfor theeightoutcome categories, their condence intervals, and thenumber of studies contributing data for each category.Signicant meaneffects rangedinmagnitude from0.12(forschool grades)to0.34forchildself-perceptions(i.e.,increased self-condence and self-esteem). The meaneffects for school attendance (0.10) anddruguse (0.10)were the only outcomes that failed to reach statisticalsignicance. Inother words, ASPs were associatedwithsignicantlyincreasedparticipants positive feelings andattitudes about themselves and their school (child self-perceptionsandschool bonding), andtheirpositivesocialbehaviors. In addition, problembehaviors were signi-cantlyreduced.Finally,therewassignicantimprovementinstudentsperformanceonachievementtestsandintheirschool grades. These data support our rst hypothesis.ParticipationinASPsisassociatedwithmultiplebenetsthatpertaintoyouthspersonal, social,andacademiclife.ModeratorAnalysisTherewere41SAFEprogramsevaluatedat post that fol-lowed all four recommendedskill trainingpractices; 27Other programs did not use all four practices. Table3contains themeanSMDs for SAFEandOther Programsoverall andwithineachof theeight outcome categoriesalongwithQandI2values. Theuseof I2aids ininter-pretationbecausetheQstatistichaslowpowerwhenthenumber of studiesissmall andconverselymaybestatis-tically signicant when there are a large number of studies,even though the amount of heterogeneity might be low(Higginsetal.2003).WhenstudiesaregroupedaccordingTable1 Descriptivecharacteristicsofreviewedstudiesatpostk %PublicationfeaturesDateofreport19791990 3 4.419912000 19 27.920012008 46 67.6SourceofreportPublishedarticle 24 35.3Unpublishedreport 44 64.7MethodologicalfeaturesExperimentaldesignRandomized 24 35.3Quasi-experimentaldesign 44 64.7ReliabilityofoutcomemeasuresAcceptablereliability 270 73.2Unknown/unacceptable 99 26.8Meanpercentofattrition 10CharacteristicsofparticipantsMeaneducationallevelElementaryschool(K-5) 31 45.6Middleschool(68) 25 36.8Highschool(912) 6 8.8Didnotreport 6 8.8PresentingproblemsNone(universalintervention) 61 89.7Somepresentingproblems 7 10.3Predominantethnicityofparticipants[90%African-American 10 14.5[90%Latino 6 8.8[90%Asian/PacicIslander 3 4.4[90%AmericanIndian 1 1.5Didnotreportethnicity 25 36.8SocioeconomicstatusPredominatelylowincome 17 25.0Mixedincome 13 19.1DidnotreportSES 31 45.6ProgramfeaturesDurationLessthan1year 45 66.212years 12 17.6Morethan2years 11 16.2Thepercentagesdonotalwaysaddto100%duetomissingdataTable2 Meaneffectsfor68studiesatpostineachoutcomeareaOutcomes SMD k 95%CondenceintervalFeelingsandattitudesChildself-perceptions 0.34* 23 {0.23, 0.46}Schoolbonding 0.14* 28 {0.03, 0.25}IndicatorsofbehavioraladjustmentPositivesocialbehaviors 0.19* 36 {0.10, 0.29}Problembehaviors 0.19* 43 {0.10, 0.27}Druguse 0.10 28 {0.00, 0.20}SchoolperformanceAchievementtestscores 0.17* 20 {0.06, 0.29}Schoolgrades 0.12* 25 {0.01, 0.23}Schoolattendance 0.10 21 {-0.01, 0.20}*Denotesmeaneffect issignicantlydifferent fromzeroat the.05levelAmJCommunityPsychol(2010)45:294309 2991 3tohypothesizedmoderators, thereshouldbelowhetero-geneitywithingroups(reectedinlowI2valuesandnon-signicant Q statistics) but high and statistically signicantlevels of heterogeneity between groups (reected bycorresponding highI2values andstatistically signicantQ-betweenvalues). BenchmarksforI2suggestthatvaluesunder 15%indicatenegligibleheterogeneity, from15to24% reect a mild degree of heterogeneity, between 25 and50% a moderate degree, and values C75% a high degree ofheterogeneity(Higginsetal. 2003).ThedatainTable3indicatethat whereas SAFEpro-gramsareassociatedwithsignicant meaneffectsfor alloutcomes (mean SMDs between 0.14 and 0.37), Otherprograms do not yield signicant mean effects for anyoutcome. There is empirical support for moderation forfouroutcomesintermsofsignicantQ-betweenstatisticsand correspondingly high (7493%) I2values (positivesocial behaviors, problem behaviors, achievement testscores, and grades). However, the Q-between statisticswerenot signicant andtheI2valuesweregenerallylowfor the other four outcomes (self-perceptions, schoolbonding, drug use and school attendance). Furthermore,there is a moderate degree of within group variabilityamong SAFE programs (I2values between 34 and 76%) forfour outcomes(problemsbehaviors, druguse, test scoresandgrades) suggestingthepossibilityof additional mod-eratorsthatmightimprovethemodelt.Althoughwerequiredthat programstaffhadtofollowall four SAFE practices, there was some relationshipbetweentheabsolutenumber of practices usedandout-comes. The mean study-level ESs for staff using none, one,two, or four of the SAFE practices (three practices were notpresent in any report) were 0.02 (k=4), 0.07 (k=7), 0.10(k=16), and0.31(k=0.31), respectively.RulingoutRivalExplanationsToexamineotherpotentialexplanationsfortheresultswerst comparedthe effects ineachoutcome categoryforstudies groupedaccordingtoeachof thefollowingvari-ables: randomization (yes or no), use of a reliable outcomemeasure (yes or no), presence of an academic component inthe ASP (yes or no), and the educational level (elementary,middle,orhighschool)andgenderoftheparticipants.Wealso computed product moment correlations between SMDsand sample size, program duration, and per cent of attrition.There were too fewdata on participants ethnicity andsocioeconomic status to examine these variables ade-quately.Settingwasstronglyassociatedwiththepresenceofanacademiccomponentsoweonlyexaminedthelattervariable(i.e., school-basedprogramsweremorelikelytooffersomeformofacademicassistance).Theseproceduresresultedin64analyses(eightvariablescrossedwitheightoutcomecategories).Fortheseanalyses,signicanteffectsemergedinonlytwocases, whichwouldbeexpectedbychance. The use of randomized designs was associated withhigher levels of positive social behaviors (Q-between=4.80, p \.05), and there was a signicant positiveTable3 Outcomesfortheuseofrecommendedskilltrainingpracticesasamoderator(SAFECriteria)SAFEprogramsatpost Otherprogramsatpost BetweengroupsSMD k 95%CI Q-within I2values SMD k 95%CI Q-within I2values Q-between I2valuesAllprograms 0.31* 41 {0.24, 0.38} 48.07 17 0.07 27 {-0.01, 0.16} 11.94 0 17.69** 94FeelingsandattitudesChildSelf-perceptions0.37* 21 {0.24, 0.50} 21.22 6 0.13 2 {-0.33, 0.59} 0.69 0 0.69 0Schoolbonding 0.25* 13 {0.08, 0.41} 14.65 18 0.03 15 {-0.12, 0.19} 6.86 0 3.33 70IndicatorsofbehavioraladjustmentPositivesocialbehaviors0.29* 19 {0.21, 0.37} 15.27 0 0.06 17 {-0.03, 0.15} 23.75 33 13.97** 93Problembehaviors0.30* 22 {0.17, 0.42} 31.74 34 0.08 21 {-0.05, 0.20} 17.01 0 5.85* 82Druguse 0.16* 12 {0.05, 0.27} 17.86 38 0.03 16 {-0.08, 0.13} 16.36 8 2.94 66SchoolperformanceAchievementtestscores0.20* 10 {0.13, 0.27} 36.93** 76 0.02 10 {-0.04, 0.07} 1.75 0 15.14** 93Grades 0.22* 9 {0.07, 0.36} 19.71 60 0.05 16 {-0.04, 0.13} 10.71 0 3.91* 74Schoolattendance0.14** 9 {0.05, 0.24} 10.69 25 0.07 12 {0.01,0.13} 10.52 0 1.65 39*Denotesmeaneffectissignicantlydifferentfromzeroatthe.05level**Denotesmeaneffectissignicantlydifferentfromzeroatthe.01level300 AmJCommunityPsychol(2010)45:2943091 3correlationbetweenfemalegender andhigher test scores(r=0.69, p \.01). Overall, these analyses suggest that theabovevariablesdonotserveasanalternativeexplanationfor the positive ndings obtained by SAFE programs.Additional comparisons indicated that SAFEand Otherprogramsdidnot differ signicantlyonanyof theabovevariables.We also examinedpublication source andfound thatpublished reports (k=24) yielded signicantly higherstudy-level ESs than unpublished (k=44) reports(respective meanSMDs=0.34and0.10). Uponfurtherexamination, thiseffect wasrestrictedtoOtherprograms.Whereas 25unpublishedOther studies yieldedanonsig-nicant meanSMDof 0.05(CI=-0.03, 0.13), thetwopublishedstudiesofOtherprogramshadameanSMDof0.69 (CI=0.21, 1.17). In contrast, there was no SMDmeandifferencebetweenthe19unpublishedand22pub-lished reports of SAFE programs (mean SMDs=0.31 and0.30, respectively).SensitivityAnalysesWeconductedseveral sensitivityanalyses of study-leveleffects inconsiderationof different features of programevaluations.Amongthe68evaluationsatpost,eightwereconductedbyresearchers whohadapparentlydevelopedthe skills content of the ASPand might have a vestedinterest initspositiveoutcomes; therewere17studiesinwhichinvestigatorsfailedtoconrmthepre-interventionequivalence of program and control groups; there were fourcases of differential attritionoccurringbetweenprogramand control groups; and in eight cases, some criterionrelatedtoattendancewasusedwhencomposingtheinter-ventionsample(e.g.,onlychildrenwhoattendedacertainpercentageofavailabletimeswereassessed).Ontheotherhand, in 22 studies an intent-to-treat analysis was con-ducted in which all youth assigned to the ASP wereassessedregardless of whether or not theyattendedfre-quently or not at all. In two cases, the same ASPwasevaluated in two separate reports. Separate analysesremovingthestudieswitheachof abovefeaturesdidnotchangethemainoutcomendings.Finally, although published and unpublished SAFEstudies yieldedsimilar results, wealsoconductedatrimandllanalysis(DuvalandTweedie2000)toestimatethepossibilityof publicationbias onstudy-level effect sizes(i.e., todetermine if additional but missingunpublishedstudies wouldchange the mainnding). This proceduresuggested the trimming and lling of four studies andresultedinanadjustedmeanestimateforSAFEprogramsthat remained statistically signicant fromzero (meanES=0.22,p \.05).TheImpactofPreSMDsThe impact of computing pre SMDs is reected in themeancomparisons betweenthe81outcomes inwhichitwaspossibletocalculatesuchSMDs(group1)versustheremaining334outcomesinwhichthesedatacouldnotbecalculatedduetolackofinformation(group2).Whilethepost meanSMDs are similar for bothgroups (0.20and0.18, respectively), themeanpreSMDfor group1was-0.10. SubtractingthepreSMDfromthepost SMDtocreateanadjustedpost SMDforgroup1producedasig-nicant mean difference at post favoring group 1 (0.29versus 0.18, respectively, p \.01). Thevalues of thepreandpostmeanSMDsforgroup1indicatethaton20%ofthe outcomes the after-school group started at a docu-menteddisadvantagecomparedtocontrols, but overcomethisdisadvantageovertimeandweresuperiortothecon-trol group at post. Including pre SMDs increased theoverall meaneffect by61%ontheseoutcomes. (0.29vs.0.18). PreSMDswerenot morelikelyforsomeoutcomecategoriesthanothers,norweretheyassociatedwithothercoded variables except for SAFE and Otherprograms. TheformerweremorelikelytohavepreSMDwhichmightbeonefactorcontributingtotheirlargereffects.PuttingCurrentFindingsintoContextIt may seem customary to view the effects achievedin thisreview (i.e., mean SMDs in the 0.20 and 0.30s) as smallin magnitude. However, methodologists nowstress thatinsteadofsimplyresortingtoCohens(1988)conventionsregardingthesizeof obtainedeffects, ndingsshouldbeinterpretedinthecontext ofpriorresearchand, wheneverpossible, in terms of their practical value (Vacha-Haase andThompson 2004). If one does so, the impact for ASPprogramsachievesmoreprominence.For example, Table4 compares the mean SMDsachievedbythe41effectiveSAFEprogramstotheresultsreported in meta-analyses of other interventions for school-aged youth. The SMDs of SAFE programs are similar to orbetter thanthose producedbyseveral other community-and school-based interventions for youth assessing out-comessuchasself-perceptions, positivesocial behaviors,problembehaviors, drug use, and school performance(DuBoiset al. 2002; DurlakandWells1997; HaneyandDurlak 1998; Losel andBeelman2003; Tobler et al. 2000;Wilsonetal.2001,2003).Forthesecomparisons,weusedthendingsfromothermeta-analysesregardinguniversalinterventionswhereverpossiblebecausethevast majorityof effective ASPs in our review did not involve youth withidentiedproblems.Of particular note, themeanSMDobtainedbySAFEprograms on achievement test scores (0.31) is not onlyAmJCommunityPsychol(2010)45:294309 3011 3larger than the effects obtained in reviews of primarilyacademically-orientedASPsandsummerschoolprograms(Cooperetal.2000;Laueretal. 2006), butiscomparabletotheresultsof87meta-analysesofschool-basededuca-tionalinterventions(Hilletal. 2008).It ispossibletoconvert ameanSMDintoapercentileusingCohensU3indextoreect theaveragedifferencebetween the percentile rank of intervention and controlgroups(Institutefor EducationSciences 2008a). Ameaneffect of 0.31 translates into a percentile difference of 12%.Putanotherway,theaveragememberofthecontrolgroupwould demonstrate a 12 percentile increase in achievementiftheyhadparticipatedinaSAFEafter-schoolprogram.ResultsatFollow-upThe15reportscontainingfollow-updatacollectedinfor-mationondifferent outcomecategories. Thecell sizesatfollow-uprangedfromzeroforschool attendancetoninefor self-perceptions(meanES=0.19; p \.05). Unfortu-nately, thereistoolittleinformationat follow-uptoofferanyconclusionsabout thedurabilityofchangesproducedbyASPs.DiscussionThis is the rst meta-analysis to evaluate the outcomesachievedbyASPsthat seektopromoteyouths personalandsocial skills. ThisreviewincludedalargenumberofASPs(k=75),andisthersttimemanyofthesereportshavebeenscrutinized.Two-thirdsoftheevaluatedreportsappearedafter2000. Asaresult,thisreviewyieldsanup-to-date perspective on a rapidly growing research literature.CurrentdataindicatethatASPshadanoverallpositiveandstatisticallysignicant impact onparticipatingyouth.Desirable changes occurred in three areas: feelings andattitudes, indicatorsof behavioral adjustment, andschoolperformance. More specically, there were signicantincreases inyouths self-perceptions, bondingtoschool,positivesocial behaviors, school grades, andachievementtest scores. Signicant reductionsalsoappearedforprob-lembehaviors. Finally, SAFEprograms were associatedwithpractical gainsinparticipantstest scoressuggestinganaveragedifferenceof12percentilepointsbetweentheafter-schoolandcontrolgroup,andachievedresultsinthisandseveral otherareasthat weresimilartoorbetterthanthose obtained by many other evidence-based psychosocialinterventionsforschool-agedpopulations.Theimplicationof current ndingsisthat ASPsmerit support andrecog-nitionas animportant communitysettingfor promotingyouthspersonalandsocialwell-beingandadjustment.An important qualication is that not all ASPs wereeffective. OnlythegroupofSAFEprogramsyieldedsig-nicant effects on any outcomes. Commenting on theresults of our reviewas well as several others, Granger(2008) notedthat although some ASPs achieve positiveresults, manyothersdonot, indicatingthat thereismuchroomfor improvement amongcurrent programs. As wediscuss, below, thishasimportant implicationsfor futureresearchandpractice.Several steps weretakentoincreasethecredibilityofthendings. Wesearchedcarefullyandsystematicallyforrelevant reports to obtain a representative sample ofTable4 ComparingthemeaneffectsofSAFEprogramstotheresultsofotheruniversalinterventionsforchildrenandadolescentsOutcomes Currentreview OtherreviewsMeaneffectsFeelingsandattitudesSelf-perceptions 0.37 0.19aSchoolbonding 0.25 IndicatorsofbehavioraladjustmentPositivesocialbehaviors 0.29 0.15b,0.39cProblembehaviors 0.30 0.21b,0.27c,0.09d,0.17e0.30fDruguse 0.16 0.11b,0.05e,0.15gSchoolperformanceAchievementtestscores 0.20 0.11b,0.30f,0.24hGrades 0.22 Schoolattendance 0.14 Resultsfromothermeta-analysesarefromoutcomecategoriesmost comparabletothoseinthecurrent reviewandresultingfromweightedrandomeffectsanalyseswheneverpossibleaHaneyandDurlak(1998),bDuBoisetal. (2002),cLoselandBeelman(2003),dWilsonetal. (2003),eWilsonetal. (2001),fDurlakandWells(1997),gTobleretal. (2000),hHilletal. (2008)302 AmJCommunityPsychol(2010)45:2943091 3publishedandunpublishedevaluations, andarecondentthatoursampleofstudiesisanunbiasedrepresentationofevaluations of ASPs meeting our inclusion criteria thathaveappearedbytheendof2007.Wealsoexaminedandwereabletoruleoutsomeplausiblerivalexplanationsforourmainndings. Furthermore, thecurrentreviewunder-estimatesthetrueimpactofASPsforatleasttworeasons.One has to do with the nature of the control groups used incurrent evaluations; the second has to do with the dosage oftheinterventionreceivedbymanyprogramyouth.ControlGroupsThe intent of this reviewwas tocompare outcomes foryouth attending a particular ASP to those not attendingtheprogram, but this does not meanthat comparisonyouthconstituted a true no intervention control group. Forexample, itiswellknownthatinanyonetimeperiodnotonly do many youth spend their out-of-school time indifferent pursuits (e.g., in ASPs, extra-curricular schoolactivitiesandchurchgroups, aswell ashangingout withfriends, andbeingalonesomeofthetime), but alsotheymay change their level of participation across activitiesover time (Mahoney et al. 2006, in press). In ve reviewedreports,authorsnotedthatyouthintheircontrolconditionwere participatinginalternative ASPs or other types ofpotentiallybenecialout-of-schooltimeactivities(Brookset al. 1995; Philliber et al. 2001; Rusche et al. 1999; Tebesetal.2007;Weismanetal. 2003). Itisrecommendedthatevaluators monitor the types of alternative services that arereceived by control groups, so a truer estimate of theimpactofinterventioncanbemade.ProgramDosageIt is axiomatic that recipients must receive a sufcientdosagefor aninterventiontohaveaneffect. However, itappears this did not happen in several of the reviewedprograms, which may be an explanation for the poor resultsobtainedforsomeprograms.Althougheachreportdidnotcontainspecicdataonprogramattendance, whensomeinformationwaspresenteditwasapparentthatattendancewasaproblemforseveralprograms.Forexample,youthsattendance ranged from 15 to 26% in 11 evaluations (Bakerand Witt 1996; Dynarski et al. 2004; James-Burdumy et al.2005; LaFranceet al. 2001; Lauver2002; Maxeldet al.2003, twocohorts; Philliber et al. 2001; Prenovost 2001,threecohorts).Moreover, analyses conducted in some reports indicatedthat attendancewaspositivelyrelatedtoyouthoutcomes.This occurred in six of the seven studies that examined thisissue, although signicant differences did not alwaysemergeoneveryoutcomemeasure(BakerandWitt1996;Fabianoet al. 2005; Lauver 2002; Morrisonet al. 2000;Prenovost2001;Vandell,etal.2005;Zief2005).Reviewsof other ASPs have also reported a signicant positiverelationship between attendance and positive outcomes(Simpkinsetal. 2004, butalsoseeRothetal. 2010).Furthermore, attendanceis onlyoneaspect of partici-pation.Informationisalsoneededonthebreadthofyouthactivities within any program and their level of engagementineachactivity.Forexample,studiessuggestthatyouthslevelofengagementpredictspositivesocialandacademicoutcomes(Mahoneyet al. 2007; Shernoff 2010). Insum,thereceipt ofalternativeafter-school activitiesbycontrolgroupsandthelowattendanceachievedinsomeprogramsworkedagainst ndingpositiveoutcomes. Thenext sec-tionsdiscussseveralotherissuessuggestedbythecurrentndings.ElementsofEffectiveASPsAs hypothesized, the use of four recommendedtrainingpractices (i.e., SAFE) moderated several outcomes anddistinguishedbetweenASPs that wereor werenot asso-ciatedwithmultiplepositiveoutcomes.Moreover,thereisconvergent evidencefromnumerousothersourcesontheimportance of SAFEfeatures. Althoughthe terminologymay differ, others have mentioned the importance of one ormoreSAFEfeaturesinASPprograms(Gerstenblithet al.2005; Granger and Kane 2004; Mahoney et al. 2001, 2002;Miller 2003, National ResearchCouncil andInstitute ofMedicine2002). For example, Granger (2008) notedthatour data were consistent with a developing consensus in theafter-school eld that being explicitabout programgoals,implementing activities focused on these goals, and gettingyouth actively involved are practices of effective pro-grams (p. 11). We recommend that future research shouldcontinuetoexaminethevalueof thesefeaturesinASPs.Fortunately, SAFE practices can be applied to a widevarietyofinterventionapproaches.GainsinAchievementTestScoresSAFEASPs yielded signicant improvement in partici-pants standardizedtest scores andat a magnitude (i.e.,SMDof 0.31), whichis over twotimes larger thanthatfoundintheprevious meta-analysis of academically-ori-entedASPs (Lauer et al. 2006). Whywere current pro-gramssoeffectiveintheacademicrealm?Thereareseveral possibleexplanations. First, itshouldcomeasnosurprisethat programspromotingskill devel-opment can also improve school performance. There isnowagrowingbodyof researchindicatingthat interven-tionsthatpromoteSELskillsalsoresultinimprovedaca-demic performance (Collaborative for Academic, Social,AmJCommunityPsychol(2010)45:294309 3031 3andEmotional Learning[CASEL] 2005; WeissbergandGreenberg1998; Zins et al. 2004). We have obtainedameanSMDof similar magnitude (i.e., 0.27) for school-based interventions promoting students personal andsocialskills(Durlaketal. 2009).Second, current results are based on a set of recentevaluationsofASPs, onlyafewofwhichhaveeverbeenpartofanypreviousreview.Althoughwedidnotcodetheacademiccomponents of ASPs, it is possiblethat devel-opersofnewerASPsmayhaveusedstrategiesthatwouldstrengthen their impact. For example, others have sug-gestedthatgainsinacademicachievementaremorelikelytooccur if staff arewell-trainedandsupervised, useevi-dence-based instructional strategies, are supportive andreinforcing to youth during learning activities, conduct pre-assessmentstoascertainlearners strengthsandacademicneeds, and coordinate their teaching or tutoring with schoolcurricula(e.g., Birminghamet al. 2005; Southwest Edu-cational Development Laboratory2006). Arecent multi-siteevaluationindicatedthatASPparticipantsdomanifestacademic progress if evidence-basedinstructional strate-giesareusedandarewell-implemented(Sheldonetal. inpress). Moreresearchneedstoanalyzehowdifferent fea-tures of the academic components of future ASPs con-tributetooutcomes. Third, it must beacknowledgedthatonly 20 programs collected outcome data on academicachievement, socurrent results needreplicationinmoreprogramstoconrmtheirgenerality.LimitationsandDirectionsforFutureResearchThere are four important limitations in our reviewthatsuggestdirectionsforfutureresearch.1. Currentconclusionsrest uponoutcomeresearchthatshouldbeimprovedinseveral ways. Manyreportslackeddata on the racial and ethnic composition or the socio-economic status of participants, so we could not relateoutcomestotheseparticipant characteristics. Missingsta-tistical data at pre or post limited the number of effects thatcould be directly calculated. At a minimum, future programevaluations should provide complete information on thedemographiccharacteristics of participants, their preandpost scores onall outcomes, and, if pertinent, their prioracademic achievement, and any presenting problems youthmight have. Thegoals, procedures andcontents of eachprogram component should be specied and described, anddataonlevelsof participationandbreadthanddegreeofengagement in different activities should be included.Reliableandvalidoutcomemeasuresshouldbeusedand,wheneverpossible, data should be collected using multiplemethodologies (e.g., fromschool records, questionnaires,and behavioral observations) and from multiple informants(e.g., youth,parents, teachers,andASPstaff).Futureevaluationsshouldalsobeawareoftheanalyticproceduresthatshouldbeusedfornesteddesigns.Thatis,whenaninterventionis conductedinagroupcontext orsettingsuchas inanASP, participant dataarenot inde-pendent andanalysestreatingindividual dataasindepen-dent can greatly increase TypeI error rates. Unfortunately,virtuallyall thereviewedreportsemployedoneinterven-tionandonecontrolgroupsothat appropriatecorrectionsfornesteddatacouldnot bemade(Baldwinet al. 2005).Guidelines are available for the appropriate analyses ofnested data (Institute for Education Sciences 2008b;RaudenbushandBryk2002).Care is also needed in designating program participants.Eight studies only analyzed data from participants who hadattended a certain number of programactivities usingunique criteria in each circumstance. This method con-founds the impact of intervention with dosage. A preferredstrategyusedinsomestudies(e.g., Philliber et al. 2001;Weisman et al. 2001; Zief 2005) is an intent-to-treatanalysis in which all participants data are evaluatedregardless of their programdosage. Additional analysescan then be conducted to examine the relationship betweenprogramattendanceandoutcomes.Current ndingsillustratehowtheimpact of interven-tioncanbemorecompletelyportrayedbyincludingpreSMDs inthenal calculationof effects. On19%of theoutcomes,theafter-schoolgroupstartedatadisadvantage(meanpre SMD=-0.10) but overcamethis disadvantageover time (mean post SMD=0.20). Incorporating preSMDs increased the nal SMD for these outcomes by 69%(0.29 versus 0.20). More journals are now requiring authorstoreport SMDsfor individual studies(Durlak2009) andfuture researchers should consider calculating adjustedSMDsthat takeintoconsiderationanyinitial differencesbetweengroups.2. Although the four SAFEfeatures we assessed diddistinguish between moreandless effectiveprograms,itisimportant toput these ndings incontext. First, authorshave noted additional aspects of skill training that areimportant, suchasthetrainersinterpersonal skills, sensi-tivitytothelearnersdevelopmental abilitiesandculturalbackground, andtheimportanceof helpingyouthgener-alize their newly-developed skills to everyday settings(Dusenbury and Falco 1995; Gresham1995). Unfortu-nately, informationontheseadditional recommendedele-mentswasnotavailable.Second, although previous authors have stressed that thefourfeaturesweassessedworkincombinationwitheachother, their relativeinuencemight neverthelessvarynotonly in relation to youths developmental level and culturalbackground,butalsoonthenatureandnumberoftargetedskills. For example, younger children will likely need morepracticethanolderyouthwhenattemptingtomastermore304 AmJCommunityPsychol(2010)45:2943091 3complexskills. Therelativeinuenceofdifferenttrainingprocedures on eventual skill development also deservesattentioninfutureresearch.Third, it would be preferable to evaluate SAFE practicesascontinuousrather thandichotomousvariables. That is,programstaff can be compared in termsof how muchtheyfocus on skill development and use of active learningtechniques instead of viewing these practices as all-or-nonephenomena. Observational systemshavenowbeendevel-oped to record the use of SAFEpractices in ASPs ascontinuousvariables(Pechmanetal. 2008).Fourth, basedonQandI2values there was strongerempirical support for SAFEpractices as moderators forsome outcomes over others (e.g., for positive socialbehaviors,problembehaviors,testscores,andgrades)andit waspossibleto calculatepre SMDsfor moreSAFEthanOther programs. Therefore, current dataarejust abegin-ning in exploring the black box of ASPs, that is, inunderstandingall thestructuresandprocessesthat consti-tuteaneffectiveprogram.CurrentdataarecorrelationalinnatureandwecannotconcludethatSAFEfeaturescausedthepositivechangesinprogramparticipants. Becausethecurrent meta-analysis focused only on the skill-buildingcomponents of ASPs, it is possible that additional programvariables play a role in the effectiveness of ASPs. Forexample, programquality is one feature that comes tomind, andhasbeenemphasizedintheoperationofASPs(Birmingham et al. 2005; Granger 2008; High/ScopeEducational Research Foundation 2005; Miller 2003;Vandelletal. 2004).Several independent groups have focusedonsixcorefeaturesthat contributetothequalityofayouthdevelop-ment program(Yohalemet al. 2007). Inadditiontoskill-buildingopportunities, theseincludethecharacteristicsofinterpersonal relationships (betweenstaff andyouthandamong youth), the programs psychosocial environment,thelevel andnatureof youths engagement inactivities,social norms, andprogramroutineandstructure. Inturn,thesefeaturesarerelatedtosuchvariablesasstaffbehav-ior, program policies, youth initiative, and issues related tocommunitypartnershipsandsupport.Informationonthesevariables were not available in reviewed reports, and futureresearchersshouldexploretheirinuence.Two additional, important foci for future researchinvolve the creation and evaluation of effective staffdevelopment andtrainingprogramsanddataonprogramimplementation.Whatarethemostefcientwaysforstafftolearnnewtechniquesandimplementthemeffectively?Some authors stress the importance of less structuredactivities that might stimulateyouthinitiativeandfosterheightened leadership skills and autonomy. The SouthBaltimoreYouthCenter(Bakeretal.1995)whichdidnotfollowSAFEpractices usedanempoweringstrategybyhaving participants assume responsibility for all majorYouthCenteractivities. Thisstrategywasassociatedwithimpressive signicant improvement in adolescents self-reporteddelinquentbehavioranddruguse(SMDsof1.10and0.82, respectively). Datafromotherstudiesalsocon-rmthevalueof empoweringstrategiesinASPs(Hirschand Wong 2005; Hansen and Larson 2007), but morecontrolledoutcomedataareneeded.Nevertheless, the ndings on the value of structured skilldevelopment practices do not necessarily contradict thevalueof lessstructuredactivitiesfor threereasons. First,alternativestrategiescanleadtosimilar outcomes. Thereare many possible ways to get from Point A to Point B andsomecompetenciesmaybebetterpromotedviaonestrat-egythananother. Studies directlycomparingtherelativebenets of different strategies on different skills andadjustment outcomes would be helpful. Second, most ASPscontainmultiple components somorestructuredapproa-ches can be used some of the time and less structured onesatothertimes.Third,empowermentstrategiescanbeusedwithin structured components, for example, by asking moreskilled youth to be role models, trainers, or co-groupleaders for others. Assuming such roles could promoteyouthsleadershipskillsandsenseofself-efcacy.Futureresearchthatcanclarifyhow differentaspects ofprogram quality inuence different youth outcomes will beextremelyhelpful inimprovingASPs. Because programqualityisamulti-dimensional construct, assessingqualityacross its dimensions and relating these to a range of youthoutcomes can provide an empirical basis for understandingtheprocesses withinASPsthatleadtodifferentresults.Asresearchonthistopicaccumulates, it will bepossibletodevelopaclearerunderstandingofwhatconstitutesahighquality program and in what respects current programs canbeimproved.3. Unfortunately, few reports have collected any follow-up data, so we cannot offer any conclusions about the long-termeffects of ASPs. Hopefully, future evaluations willcontainfollow-upinformationtodeterminethedurabilityofanygainsemanatingfromprogramparticipation.4. Althoughtheinitial studysampleseemssufcientlylarge (68 studies with post data), dividing studies rstaccording to outcome categories, and then according tootherpotentiallyimportant variablesreducedthestatisticalpower of the analyses. Therefore, the failure to obtainstatisticallysignicant ndingsfor someof thevariablesexaminedhereshouldbeviewedcautiously.AsmoreASPevaluationsappear,researcherswillhavemore power to detect the inuence of potentially importantvariables. Attheindividuallevel, weneedinformationonhowgender, race/ethnicity, age, income status, and thepresence of academic or behavioral problems are related toparticipantsparticipation, engagementanddifferenttypesAmJCommunityPsychol(2010)45:294309 3051 3of outcomes. At the ecological level we need to understandhowfamily, school, andneighborhoodcharacteristicsandresourcesareassociatedwithconsistent andactivepartic-ipation in ASPs, and interact with various programprocesses and structures to inuence youth outcomes(Mahoney et al. 2007; Weiss et al. 2005). Such data wouldhelp us maximize the t between programfeatures andlocalneedstoincreasethereachandbenetsofASPs.Notwithstanding the above limitations, the currentreviewoffers empirical support for thenotionthat ASPscanbesuccessful inachievingtheir historical missionoffostering the personal andsocial development of youngpeople. Althoughnot conclusive, current ndings shouldstimulatemoreinterestininvestigatingandunderstandinghow ASPs programs affectyouth, and what can be done toenhancetheireffectiveness.Acknowledgments This article is based on a grant from theWilliamT. Grant Foundation(grant#2212)awardedtotherstandsecond authors. We wish to express our appreciation to DavidDuBois, Mark Lipsey, Robert Granger, and Nicole Yohalemwhoprovided helpful comments on an earlier draft of this manuscript. Weoffer additional thanks toMarkLipseyandDavidWilsonfor pro-vidingthe macros used for calculatingeffects fromeachrelevantoutcomeandconductingthestatistical analyses. Finally, wewishtothank Heather Weiss and Chris Wimer fromthe Harvard FamilyResearch Project who supplied copies of relevant reports that we wereunabletoobtain.ReferencesReferencesmarkedwithanasteriskindicatestudiesincludedinthemeta-analysisArthur, W., Jr., Bennett, W., Jr., Stanush, P. L., &McNelly, T. L.(1998). Factors that inuence skill decay and retention: Aquantitativereviewandanalysis. HumanPerformance, 11, 57101.*Astroth, K. A., &Haynes, G. W. (2002). More than cows andcooking: Newest researchshowstheimpact of4-H. JournalofExtension,40,110.*Baker, K., Pollack, M., &Kohn, I. (1995). Violence preventionthrough informal socialization: An evaluation of the SouthBaltimoreYouthCenter. Studies onCrimeandPrevention, 4,6185.*Baker, D., &Witt, P. A. (1996). Evaluationof theimpact of twoafter-school programs. Journal of ParkandRecreationAdmin-istration,14,6081.Baldwin, S. A., Murray, D. M., & Shadish, W. R. (2005). Empiricallysupported treatments or Type I errors? Problems with theanalysisofdatafromgroupadministeredtreatments.JournalofConsultingandClinicalPsychology,73,924935.*Belgrave, F. Z., Chase-Vaughn, G., Gray, F., Addison, J. D., &Cherry, V. R. (2000). The effectiveness of a culture- and gender-specic intervention for increasing resiliency among AfricanAmericanpreadolescent females. Journal of BlackPsychology,26, 133147.*Bergin,D.A.,Hudson,L.M.,Chryst,C.F.,&Resetar,M.(1992).An afterschool intervention programfor educationally disad-vantagedyoungchildren.TheUrbanReview,24,203217.Birmingham, J., Pechman, E. M., Russell, C. A., &Mielke, M.(2005). Shared features of high-performing after-school pro-grams: Afollow-uptotheTASCevaluation. Washington, DC:Policy Studies Associates. Retrieved May 29, 2007 fromwww.sedl.org/pubs/catalog/items/fam107.html.*Bissell, J., Dugan, C., Ford-Johnson, A., Jones, P., &Ashurst, J.(2002). Evaluationof theYS-CAREafter school programforCalifornia work opportunity and responsibility to kids (Cal-WORKS). Department of Education, University of California,IrvineandResearchSupportServices.Bodilly, S., &Beckett, M. K. (2005). Making out-of-school timematter: Evidence for an action agenda. Santa Monica, CA: RandCorporation. Retrieved September 10, 2005, fromwww.rand.og//pubs/monographs/MG242/index.html.Bond, L. A., & Hauf, A. M. C. (2004). Taking stock and putting stockin primary prevention: Characteristics of effective programs.JournalofPrimaryPrevention,24,199221.*Brooks, P. E., Mojica, C. M., & Land, R. E. (1995). Final evaluationreport:LongitudinalstudyofLAsBESTafterschooleducationandenrichment program, 19921994. LosAngeles: Universityof California, Graduate School of Education &InformationStudies, CenterfortheStudyofEvaluation.Cason, D., &Gillis, H. L. L. (1994). Ameta-analysis of outdooradventureprogrammingwithadolescents. Journal of Experien-tialEducation,17,4047.*Chase, R. A. (2000). Hmong American partnership: 2HTNnalreport. St.Paul, MN:WilderResearchCenter.Cohen, J. (1988). Statistical power analysis for the behavioralsciences(2nded.). Hilllsdale, NJ:Erlbaum.Collaborative for Academic, Social, and Emotional Learning[CASEL].(2005).Safeandsound:Aneducationalleadersguideto evidence-based social and emotional learning programsIllinois edition. Retrieved January 10, 2007, fromhttp://www.casel.org.CommissiononPositiveYouthDevelopment. (2005). Thepositiveperspective on youth development. In D. W. Evans, E. B. Foa, R.E. Gur, H. Hendin, C. P. Obrien, M. E. P. Seligman, &B. T.Walsh(Eds.), Treating and preventing adolescent mental healthdisorders: What weknowandwhat wedont know(pp. 497527). NY:OxfordUniversityPress.Cooper, H., Charlton, K., Valentine, J. C., & Muhlenbruck, L. (2000).Making the most of summer school: A meta-analytic andnarrative review. Monographs of the Society for ResearchinChildDevelopment,65(1),260.Derzon, J. (2006). Howeffectiveareschool-basedviolencepreven-tionprograms inpreventingandreducingviolence andotherantisocialbehaviors?Ameta-analysis.InS.R.Jimerson&J.J.Furlong (Eds.), The handbook of school violence and schoolsafety:Fromresearchtopractice(pp. 429441). Mahwah, NJ:LawrenceEarlbaum.DuBois, D. L., Holloway, B. E., Valentine, J. C., &Cooper, H.(2002). Effectiveness of mentoring programs for youth: Ameta-analyticreview. AmericanJournal of CommunityPsychology,30, 157198.Durlak, J. A. (1997). Successful prevention programs for children andadolescents.NewYork:Plenum.Durlak, J. A. (2009). Howtoselect, calculate, andinterpret effectsizes.JournalofPediatricPsychology,34,917928.Durlak, J. A.,Weissberg, R. P., Dymnicki, A. B., Taylor, R, D., &Schellinger, K. B. (2009). The impact of enhancingstudentssocial andemotional development: Ameta-analysisof school-based universal interventions. Manuscript submitted forpublication.306 AmJCommunityPsychol(2010)45:2943091 3Durlak, J. A., &Wells, A. M. (1997). Primary prevention mental healthprogramsforchildrenandadolescents:Ameta-analyticreview.American Journal of Community Psychology, 25, 115152.Dusenbury, L., &Falco, M. (1995). Elevencomponentsofeffectivedrugpreventioncurricula. Journal of School Health, 65, 420425.Duval, S., & Tweedie, R. (2000). Trim and ll: A simple funnel-plot-basedmethodof testingandadjustingfor publicationbias inmeta-analysis.Biometrics,56, 455463.*Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke,J., &Manseld, W. (2004). Whenschoolsstayopenlate: Thenational evaluationof the 21st Century Community LearningCentersProgram:Newndings. USDepartment of Education,National Center for EducationEvaluationandRegional Assis-tance. Washington,D.C.:USGovernmentPrintingOfce.Eccles, J. S., & Templeton, J. (2002). Extracurricular and other after-school activities for youth. Review of Research in Education, 26,113180.*Fabiano, L., Pearson, L. M., &Williams, I. J. (2005). Puttingstudents on a pathway to academic and social success: Phase IIIndings of the Citizen Schools evaluation. Washington, DC:PolicyStudiesAssociates, Inc.*Foley, E. M., & Eddins, G. (2001). Preliminary analysis of Virtual Yafter-school program participants patterns of school attendanceandacademic performance: Final evaluationreport programyear 19992000. NY: National Center for Schools andCom-munities, FordhamUniversity.*Fuentes, E. G. (1983). Aprimarypreventionprogramfor psycho-logical and cultural identity enhancement: Puerto Rican childrenin semi-rural northeast United States. Dissertation AbstractsInternational,44(05), 1578B.Gerstenblith, S. A., Soule, D. A., Gottfredson, D. C., Lu, S.,Kellstrom, M. A., & Womer, S. C. (2005). After-schoolprograms, antisocial behavior, andpositiveyouthdevelopment:Anexplorationoftherelationshipbetweenprogramimplemen-tationandchanges inyouthbehavior. InJ. L. Mahoney, J. S.Eccles, R. W. Larson, et al. (Eds.), Organized activities ascontextsofdevelopment:Extracurricularactivities,after-schooland community programs (pp. 457478). Mahwah, NJ: Erlbaum.*Gottfredson, D. C., Soule, D. A., &Cross, A. (2004). Astatewideevaluation of the Maryland after school opportunity fundprogram. Department of Criminology and Criminal Justice,UniversityofMaryland.Granger, R. C. (2008). After-school programs and academics:Implications for policy practice, and research. Social PolicyReport,22(311), 1419.Granger, R. C., &Kane, T. (2004). Improvingthequalityof after-schoolprograms.EducationWeek,23,7677.*Grenawalt,A.,Halback,T.,Miller,M.,Mitchell,A.,ORoarke,B.,Schmitz, T., et al. (2005). 4-H animal science programevaluation: Spring 2004What is the value of the Wisconsin4-H Animal Science Projects? Madison WI: University ofWisconsinCooperativeExtension.Gresham, F. M. (1995). Best practices insocial skills training. InA. Thomas & J. Grimes (Eds.), Best practices in schoolpsychology-III (pp. 10211030). Washington, DC: NationalAssociationofSchoolPsychologists.*Hahn, A., Leavitte, T., &Aaron, P. (1994). Evaluation of theQuantum Opportunities Program (QOP): Did the programwork? Waltham, MA: Brandeis University, Heller GraduateSchool, CenterforHumanResources.Haney, P.,&Durlak,J. A.(1998). Changingself-esteeminchildrenand adolescents: Ameta-analytic review. Journal of ClinicalChildPsychology,27,423433.Hansen, D. M., & Larson, R. W. (2007). Ampliers of developmentaland negative experiences in organized activities: Dosage,motivation, lead roles, and adult-youth ratios. Journal of AppliedDevelopmentalPsychology,28,360374.HarvardFamilyResearchProject. (2003). Areviewof out-of-schooltimeprogramquasi-experimental andexperimental evaluationresults. Cambirdge, MA:HarvardFamilyResearchProject.Harvard Family Research Project (2009). Out-of-school-time programresearchandevaluationdatabaseandbibliography. RetrievedJune 5, 2009 fromhttp://www.hfrp.org/out-of-school-time/ost-database-bibliography.Hedges, L. V., &Olkin, I. (1985). Statistical methods for meta-analysis.NY:AcademicPress.Higgins, J. P., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003).Measuring inconsistency in meta-analyses. British MedicalJournal,327,557560.High/Scope Educational Research Foundation. (2005). Youth pro-gramqualityassessment validationstudy: Findingsforinstru-ment validation. Retrieved April 6, 2006 fromhttp://www.highschope.org/EducationalPrograms/Adolescent/YouthPQA/YouthPQASummary.pdj.Hill, J. C., Bloom, H. S., Black, A. R., &Lipsey, M. W. (2008).Empirical benchmarksfor interpretingeffect sizesinresearch.ChildDevelopmentPerspectives,2,172177.Hirsch, B., &Wong, V. (2005). Aplacetocall home:After-schoolprogramsforurbanyouth.Washington,DC:AmericanPsycho-logicalAssociationandNewYork:TeachersCollegePress.*Huang, D. (2004). Exploringthelong-termimpactofLAsBestonstudentssocial andacademicdevelopment. LosAngeles, CA:National Center for Research on Evaluation, Standards, andStudentTesting(CRESST).*Huang, D., Sung Kim, K., Marshall, A., & Perez, P. (2005). Keepingkids in school: AnLAs BESTexample National Center forResearch on Evaluation, Standards, and Student Testing(CRESST) Center, Center for theStudyof Evaluation(CSE),GraduateSchoolofEducationandInformationStudies,Univer-sityofCalifornia, LosAngeles.*Hudley, C. (Ed.). (1999). Problembehaviorsinmiddlechildhood:Understanding risk status and protective factors. Montreal,Quebec, Canada: CaliforniaWellnessFoundation. (ERICDoc-umentReproductionServiceNo.ED430066).InstituteforEducationSciences.(2008a).Whatworksclearinghouseprocedures andStandards Workbook, Version2.0, December,2008. RetrievedJune18, 2009fromhttp/ies.ed.gov/ncee/wwc/references/idocViewer/Doc.aspx?docid=198tocid=1.Institute forEducationSciences.(2008b). Technical details of WWC-conducted computations. Retrieved June 6, 2008 fromhttp//ies.ed.gov.ncee.wwc/pdf/conducted_computations.pdf.*James-Burdumy, S., Dynarski, M., Moore, M., Deke, J., &Manseld,W. (2005). Whenschool stayopenlate: Thenationalevaluation of the 21st Century Community Learning CentersProgram Washington, DC: US Department of Education,Instituteof EducationSciences, National Center for EducationEvaluationandRegionalAssistance.Kane, T. J. (2004). The impact of after-school programs: Interpretingthe results of four recent evaluations. Retrieved January 17,2006 from www.wtgrantfondation.org/usr_doc/After-school_paper.pdf.Ladd, G. W., & Mize, J. (1983). A cognitive social learning model ofsocialskilltraining.PsychologicalReview,90,127157.*LaFrance,S.,Twersky,F.,Latham,N.,Foley,E.,Bott,C.,Lee,L.,et al. (2001). Asafe place for healthy youthdevelopment: Acomprehensive evaluation of the Bayview Safe Haven. SanFrancisco, CA:BTWConsultantsandLaFranceAssociates.Lauer,P.A.,Akiba,M.,Wilkerson,S.B.,Apthorp,H.S.,Snow,D.,&Martin-Green, M. (2006). Out-of school time programs: Ameta-analysisof effectsfor at-riskstudents. Reviewof Educa-tionalResearch,76,275313.AmJCommunityPsychol(2010)45:294309 3071 3*Lauver, S. C. (2002). Assessing the benets of an after-schoolprogramfor urbanyouth: Animpact andprocess evaluation.DissertationAbstractsInternational,63(02), 533A.*LeCroy, W. W. (2004). Experimental evaluation of GoGrrrlspreventive intervention for early adolescent girls. Journal ofPrimaryPrevention,25, 457473.Lipsey, M. W., &Wilson, D. B. (2001). Practical meta-analysis.ThousandOaks,CA:Sage.*LoSciuto, L., Hilbert, S. M., Fox, M. M., Porcellini, L., & Lanphear,A. (1999). A two-year evaluation of the Woodrock YouthDevelopment Project. Journal of Early Adolescence, 19, 488507.Losel, F., &Beelman, A. (2003). Effectsof childskillstraininginpreventingantisocialbehavior:Asystematicreviewofrandom-izedevaluations. Annalsof theAmericanAcademyof PoliticalandSocialScience,587,84109.Mahoney,J.P.,Parente,M.E.,&Zigler,E. F.Afterschoolprogramparticipation and childrens development. In J. Meece &J. Eccles (Eds.), Handbookof researchonschools, schooling,andhumandevelopment. NewYork:Wiley(inpress).*Mahoney,J. L., Lord,H.,& Carryl,E. (2005).Afterschoolprogramparticipation and the development of child obesity and peeracceptance.AppliedDevelopmentalScience,9,202215.Mahoney, J. L., Parente, M. E., &Lord, H. (2007). Program-leveldifferencesinafterschool programengagement: Linkstochildcompetence, program quality and content. The ElementarySchoolJournal,107,385404.Mahoney, J. L., Schweder, A. E., &Stattin, H. (2002). Structuredafter-school activities as a moderator of depressed mood foradolescentswithdetachedrelationstotheirparents. Journal ofCommunityPsychology,30,6986.Mahoney, J. L., Stattin, H., & Magnusson, D. (2001). Youthrecreation centre participation and criminal offending: A20-yearlongitudinal studyofSwedishboys. International JournalofBehavioralDevelopment,25, 509520.Mahoney,J.P.,&Zigler,E.F.(2006).TranslatingsciencetopolicyundertheNoChildLeft Behindact of2001: Lessonsfromthenational evaluation of the 21st-Century Community LearningCenters. Journal of Applied Developmental Psychology, 27,282294.*Mason, M. J., & Chuang, S. (2001). Culturally-based after-school artprogramming for low-income urban children: Adaptive andpreventiveeffects.JournalofPrimaryPrevention,22, 4554.*Maxeld, M., Schirm, A., &Rodriguez-Planas, N. (2003). TheQuantumOpportunityProgramdemonstration:Implementationandshort-termimpacts. Washington, DC: MathematicaPolicyResearch.*McClanahan, W. S., Sipe, C. L., &Smith, T. J. (2004). Enrichingsummerwork:Anevaluationofthesummercareerexplorationprogram.Philadelphia,PA:Public/PrivateVentures.Miller, B. M. (2003). Critical hours: After-school programs andeducationalsuccess.NewYork:NellieMaeEducationFounda-tion. Retrieved December 13, 2005 fromwww.nmefdn.org/uploads/Critical_hours_Full.pdf.*Monsaas, J. (1994). Evaluation reportFinal validation: ProjectEMERGE,CriptCountyAtlanta,GA:EmoryUniversity.*Morrison, G.M., Storino,M.H., Robertson,L. M.,Weissglass, T.,&Dondero, A. (2000). Theprotectivefunctionof after-schoolprogrammingandparent educationandsupport for studentsatrisk forsubstanceabuse. Evaluation and Program Planning, 23,365371.National Research Council and Institute of Medicine. (2002).Communityprogramstopromoteyouthdevelopment. Washing-ton, DC:NationalAcademyPress.*Neufeld, J., Smith, M. G., Estes, H., & Hill, G. C. (1995). Rural afterschool childcare: Ademonstrationproject inaremoteminingcommunity.RuralSpecialEducationQuarterly,14,1216.*Oyserman, D., Terry, K., &Bybee, D. (2002). Apossibleselvesinterventiontoenhanceschoolinvolvement.JournalofAdoles-cence,25,313326.Pechman, E. M., Russell, C. A., &Birmingham, J. (2008). Out-of-school time (OST) observation instrument: Report of thevalidationstudy. Washington, DC: PolicyStudies Associates,Inc. RetrievedJuly30,2009fromwww.policystudies.com.*Philliber, S., Kaye, J., & Herrling, S. (2001). The national evaluationof the Childrens Aid Society Carrera model program to preventteenpregnancy.Accord, NY:PhilliberResearchAssociates.*Phillips, RubyS. C. (1999). Interventionwithsiblingsof childrenwith developmental disabilities fromeconomically disadvan-tagedfamilies.FamiliesinSociety,80, 569577.*Pierce, L. H., & Shields, N. (1998). The Be A Star community-basedafter-school program: Developingresiliencyinhigh-riskpread-olescent youth. Journal of Community Psychology, 26, 175183.*Prenovost, J. K. E. (2001). Arst-year evaluationof after schoollearning programs in four urban middle schools in the Santa AnaUniedSchoolDistrict. Dissertation Abstracts International,62(03), 884A.Raudenbush, S. W., &Bryk, A. S. (2002). Hierarchical linearmodels: Applications and data analysis methods. ThousandOaks, CA:Sage.Riggs, N. R., & Greenberg, M. T. (2004). After-school youthdevelopment programs: Adevelopmental-ecological model ofcurrent research. Clinical Child and Family Review, 7, 177190.*Ross, J. G., Saavadra, P. J., Shur, G. H., Winters, F., & Felner, R. D.(1992). The effectiveness of an after-school program for primarygrade latchkey students on precursors of substance abuse.JournalofCommunityPsychology,OSAPSpecialIssue,2238.Roth, J. L., Malone, L. M., &Brooks-Gunn, J. (2010). Does the amountofparticipationinafterschoolprogramsrelatetodevelopmentaloutcomes: Areviewof the literature. American Journal ofCommunity Psychology. doi:10.1007/s10464-010-9303-3.*Rusche,S.,Kemp,P.,Krizmanich,J.,Bowles,E.,Moore,B.,CraigJr., H. E., et al. (1999). Helping everyone reach out: Club Hero,nal report. Atlanta, GA: NationalFamiliesin Action & EmstarResearch.Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: Adecadeofprogress. Annual Review of Psychology, 52,471499.*Schinke, S. P., Orlandi, M. A., Botvin, G. J., Gilchrist, L. D.,Trimble, J. E., &Locklear, V. S. (1988). Preventingsubstanceabuse amongAmerican-Indianadolescents: Abicultural com-petenceskillsapproach. JournalofCounselingPsychology, 35,8790.*Schinke,S.P.,Orlandi,M.A.,&Cole,K.C.(1992).Boys&GirlsClubsinpublichousingdevelopments: Preventionservicesforyouthatrisk.JournalofCommunityPsychology,OSAPSpecialIssue,118128.Sheldon, J. Arbreton, A., Hopkins, L., &Grossman, J. B. (inpress).Investing in success: Key strategiesfor building quality in after-school programs. AmericanJournalof CommunityPsychology.doi:10.1007/s10464-010-9296-y.Shernoff, D. J. (2010). Engagement inafter-school programs as apredictor of social competence and academic performance.American Journal of Community Psychology. doi:10.1007/s10464-010-9314-0.Simpkins, S., Little, P., &Weiss, H. (2004). Understanding andmeasuring attendance in out-of-school time programs. Cam-bridge, MA: Harvard Family Research Project. Available atwww.gse.harvard.edu/hfrp/projects/afterschool/resources/issuebrief7.html.*Smith,R.E.,Smoll,F.L.,&Curtis,B.(1979).Coacheffectivenesstraining: A cognitive-behavioral approach to enhancing relation-shipskillsinyouthsportcoaches.JournalofSportPsychology,1,5975.308 AmJCommunityPsychol(2010)45:2943091 3*Smoll, F. L., Smith, R. E., Barnett, N. P., &Everett, J. J. (1993).Enhancement of childrens self-esteemthroughsocial supporttraining for youth sport coaches. Journal of Applied Psychology,78, 602610.Southwest Educational Development Laboratory (2006). Time forAchievement:Afterschoolandout-of-schooltime. SEDL Letter,18, Number 1. Retrieved January 10, 2006 from http:/www.sedl.org/pubs/sedl-letter/v18n01/SEDLLetter-v18n01.pdf.*St. Pierre, T. L., &Kaltreider, D. L. (1992). Drugpreventioninacommunitysetting: Alongitudinal studyof therelativeeffec-tivenessofathree-yearprimarypreventionprograminBoys&GirlsClubacrossthenation. AmericanJournal of CommunityPsychology,20, 673706.*St. Pierre, T. L., Mark, M. M., Kaltreider, D. L., &Aikin, K. J.(1997).Involvingparentsofhigh-riskyouthindrugprevention:Athree-yearlongitudinalstudyinBoys&GirlsClubs.JournalofEarlyAdolescence, 17,2150.*St. Pierre, T. L., Mark, M. M., Kaltreider, D. L., &Campbell, B.(2001). Boys and Girls Clubs and school collaborations: Alongitudinal study of a multicomponent substance abuse pre-vention program for high-risk elementary school children.JournalofCommunityPsychology,29,87106.*Tebes,J.K.,Feinn,R.,Vanderploeg,J.J.Chinman,M. J.,Shepard,J. Brabham, T., et al. (2007). Impact of a positive youthdevelopment programin urban after-school settings on thepreventionof adolescent substanceuse. Journal of AdolescentHealth,41,239247.Tobler, N. S., Roona, M. R., Ochshorn, P., Marshall, D. G., Streke,A. V., & Stackpole, K. M. (2000). School-based adolescent drugprevention programs: 1998 meta-analysis. The Journal ofPrimaryPrevention,20,275336.*Tucker, C. M., &Herman, K. C. (2002). Usingculturallysensitivetheories and research to meet the academic needs of low-incomeAfrican American children. American Psychologist, 57, 762-773.Vacha-Haase, T., &Thompson, B. (2004). Howto estimate andinterpret effect sizes. Journal of Counseling Psychology, 51,473481.*Vandell,D.L.,Reisner,E.R.,Brown,B.B.,Dadisman,K.,Pierce,K. M., &Lee, D. et al. (2005). Thestudyof promisingafter-school programs: Examination of intermediate outcomes in year2. Retrieved June 16, 2006, from http://www.wcer.wisc.edu/childcare/statements.html.*Vandell,D.L.,Reisner,E.R.,Brown,B.B.,Dadisman,K.,Pierce,K. M., &Lee, D. (2004). Thestudyof promisingafter-schoolprograms: Descriptive report of the promising programs.University of Wisconsin, Madison: Wisconsin Center forEducation Research. Retrieved June 16, 2006, fromhttp://www.wcer.wisc.edu/childcare/statements.html.*Vincent, V., &Guinn, R. (2001). Effectiveness of a Coloniaeducational intervention. Hispanic Journal of Behavioral Sci-ences,23,229238.*Weisman, S. A., Soule, D. A., &Womer, S. C. (2001). Marylandafter school community grant program report on the 19992000school year evaluationof thephase1after-school programs.UniversityofMaryland, CollegePark.*Weisman, S. A., Womer, S. C., Kellstrom, M., Bryner, S., Kahler,A., Slocum, L. A., et al. (2003). Marylandafterschool grantprogrampart 1: Report onthe20012002school yearevalu-ation of the phase 3 after school programs. University ofMaryland, CollegePark.Weissberg, R. P., & Greenberg, M. T. (1998). School and communitycompetence-enhancement and prevention programs. In I. E.Siegel & K. A. Renninger (Eds.), Handbook of child psychology.Vol 4: Child psychology in practice (5th ed., pp. 877954). NewYork:Wiley.Wilson, D. B., Gottfredson, D. C., &Najaka, S. S. (2001). School-based prevention of problem behaviors: A meta-analysis.JournalofQuantitativeCriminology,17,247272.Wilson,S.J.,Lipsey,M.W.,&Derzon,J.H.(2003).Theeffectsofschool-basedinterventionprograms onaggressivebehavior: Ameta-analysis. Journal of ConsultingandClinical Psychology,71, 136149.Yohalem, N., Wilson-Ahlstrom, A. Fischer, S., &Shinn, M. (2007).Measuringyouthprogramquality: Aguidetoassessment tools.Retrieved April 20, 2007 fromwww.forumfyi.org/Files/Measuring_youth_ program_quality.pdf.*Zief, S. G. (2005). Amixed methods study of the impacts andprocesses of an after-school programfor urban elementaryyouth.Unpublisheddoctoraldissertation,UniversityofPennsyl-vania, Philadelphia.Zins,J.E.,Weissberg,R.P.,Wang,M.C.,&Walberg,H.J.(Eds.).(2004). Building academic success on social and emotionallearning: What does the researchsay?. NewYork: TeachersCollegePress.AmJCommunityPsychol(2010)45:294309 3091 3