finding what works helping young adults transition into adulthood
DESCRIPTION
TRANSCRIPT
The Need for Rigorous Evaluation of Interventions to Improve the Transition to Adulthood for Youth in State Care Mark E. Courtney School of Social Service Administration and Chapin Hall University of Chicago
My Purpose Today
Present recent research on foster youths’
transitions to adulthood Describe the focus of social policy on this
population Summarize the weak evidence regarding the
effectiveness of interventions for this population
How do foster youth fare during the transition?
Midwest Study Design and Sample
Largest prospective study of foster youth making the transition to adulthood since the Foster Care Independence Act of 1999
Collaboration between state child welfare agencies and the research team
Foster youth in Iowa, Wisconsin and Illinois who: Were still in care at age 17 Had entered care before their 16th birthday Had been placed in care because they were abused, neglected or
dependent Not originally placed because of delinquency
Data from in-person interviews (structured and in-depth qualitative) and government program administrative data
Study Design and Sample (continued)
Wave Year Number Interviewed
Response Rate
Age at interview
1 ’02 – ’03 732 96% 17 – 18
2 ‘04 603 82% 19
3 ‘06 591 81% 21
4 ‘08 602 82% 23-24
5 ’10 – ’11 596 83% 26
Young Women’s Educational Attainment
Young Men’s Educational Attainment
Young Women’s Educational Enrollment
Young Men’s Educational Enrollment
Young Men’s and Young Women’s Employment
72% employed during year; mean earnings among employed = $13,989
Family Formation Among Young Women
19% of women with children have a nonresident child
Family Formation Among Young Men
66% of men with children have a nonresident child
Young Women’s Criminal Justice System Involvement
Young Men’s Criminal Justice System Involvement
Summary of What We Know About Early Adult Outcomes Post Chafee
Outcomes are relatively poor across a variety of domains Trends are generally problematic:
Declining engagement in education, though some are still in school Gradually increasing but poor engagement in the workforce Many non-resident children Troubling levels of justice system involvement continuing through mid
20s Functioning in other domains (e.g., mental and behavioral health, risk
behaviors, victimization) is also poor
Outcomes vary by gender; males fare worse Despite a sobering picture overall, many young people leaving
the care of the state do well
16
US Social Policy and the Transition to Adulthood for Foster Youth
U.S. Demographic, Developmental, and Policy Context
The transition to adulthood in the U.S. is taking longer Markers of the transition are happening later; half of
young people between 18-24 live with a parent $38k in direct support between 18-34
Developmental psychologists describe a new period of “emerging adulthood”
Yet, U.S. policy provides little support for young adults
U.S. Policy on Foster Youth in Transition
Research from 1990s continues to show poor outcomes
1999 Foster Care Independence Act $140 million per year allocated to states Funds a broad range of services Up to 30% of funds can be used for room and board Allows states to extend Medicaid to foster youth through age 21 Amendment to law allows appropriation up to $60 million per
year to fund education/training vouchers for up to $5000 per year through age 23
Creates outcome reporting requirements and devotes 1.5% of funds to rigorous evaluation of promising programs
A Brave New World: The Fostering Connections to Success Act of 2008
Among its provisions, the law: Extends Title IV-E funding (including guardianship and
adoption subsidies), at state option, to age 21 Youth must be 1) completing high school or an
equivalency program; 2) enrolled in post-secondary or vocational school; 3) participating in a program or activity designed to promote, or remove barriers to, employment; 4) employed for at least 80 hours per month; or 5) incapable of doing any of these activities due to a medical condition
Existing IV-E protections remain, including ongoing court oversight of state foster care provision
But…Evidence of What Works is Lacking Cochrane collaboration review of evaluation research on IL programs
(Montgomery et al, 2006) found no rigorous studies: “Further research incorporating randomized designs is both feasible and necessary”
Recent ACF-funded randomized evaluations: No impact of life skills training, tutoring/mentoring, and employment support Massachusetts Adolescent Outreach had some positive effects, but those
appear to be mediated by the program’s impact on youth remaining in care past age 18.
The bottom line: Too many programs are poorly targeted, have poorly
developed logic models, and are not intensive enough to influence outcomes for youth making the transition to adulthood from foster care.
WE NEED RIGOROUS EVALUATION RESEARCH!!!
Evaluation of Youth Villages’ Transitional Living Program
John Martinez
Deputy Director, Health and Barriers to Employment Policy Area
Overview of Session
Introduction to MDRC and evaluation Who is in the study Evaluation Status
2
Who is conducting the evaluation?
MDRC, a non-profit, non-partisan education
and social policy research organization and intermediary
Based in New York City with a regional office in Oakland, CA
Dedicated to learning what works best to improve the lives of low-income families
Nearly 40 years of experience evaluating social policy programs
3
Who is funding the evaluation?
The Edna McConnell Clark Foundation The Annie E Casey Foundation The Bill and Melinda Gates Foundation
4
Components of the Evaluation (I)
Impacts: To what extent does TL improve outcomes for
youth? Which approaches are most effective for whom? On what outcomes (e.g. housing stability, educational
outcomes, labor market success, reduced crime) Over what period of time (e.g. length of follow-up) To what magnitude? For what subgroups?
Use Random Assignment Research Design Data sources: public records data, baseline forms, and surveys
5
Components of the Evaluation (II)
Program implementation: What services are provided? How are they delivered? What challenges are encountered? Are the challenges related to serving youth aging out of state care? Measuring enrollment and participation rates Measuring implementation fidelity Measuring the contrast Data sources: surveys, field research, YV MIS, in-depth
interviews with youth
Costs and Benefits: What are the costs? Do benefits outweigh the costs?
6
Impact Analysis Design
Evaluation uses random assignment, the most reliable way to assess what difference a program makes
Youth eligible for TL were assigned, by chance, to one of two groups: TL Program Group: receives TL services Community Services(CS) Group (Control Group): not eligible
to receive TL services for up to five years
MDRC will follow both groups for at least three years (and possibly longer, depending on funding)
7
Random Assignment
Participants meet program criteria
Participants give consent
Baseline data collected
Random Assignment Control group
Receive other services in the community
Program group Enroll in program
8
Why Random Assignment?
Gold standard of research as it is the most reliable way to measure impacts: Ensures motivation levels and personal characteristics of youth in
program and control groups are same on average at beginning of program
Youth in the control group illustrate what would have happened if not for the program
Any subsequent difference in outcomes can be attributed to the program with the highest confidence
Widely used in public service settings
Endorsed by OMB, DOE and other federal agencies
Fair and equitable way to determine who receives the program
9
Who is in the study?
What is the baseline information form?
Youth completed a baseline survey prior to random assignment
Baseline data provides a “picture” of these youth at study entry
Multiple uses Allows researchers to assess whether random assignment
worked Could be used in impact analysis models Can also be used to determine whether TL works differently
for different types of youth
Presenting data today on 1,225 study participants
Did RA work?
Goal: two research groups in which the only difference is that one group was eligible to receive TL and one was not
Baseline data can help support that the two groups were equivalent
Data suggests this was the case: of 22 variables, only 2 had a SS difference between the 2 groups (and we would expect that to happen by chance)
Demographics
Characteristic Full Sample (%)
Gender
Male 52
Female 48
Race
Hispanic 5
White/non-Hispanic 51
Black/non-Hispanic 38
Other/non-Hispanic 6
Age at random assignment
71%
20%
9%
Age at RA
181920-24
Age at 1st custody placement
1%
6% 6%
23%
64%
LT 11-56-1011-1415-18
Contact with biological mother
Contact with biological father
43%
16% 8%
8%
25%
Every day
At least1X/wkAt least1X/moLT 1X/mo
Never
16%
12%
8% 11%
53%
Every day
At least1X/wkAt least1X/moLT 1X/mo
Never
Contact with biological parent
Other characteristics
Characteristic Full Sample (%)
Contact with any other relative at least 1X/mo 88
Pregnant at baseline 4
Has any children 17
Enrolled in school 54
Ever repeated a grade or held back 43
Ever suspended from school 81
Ever arrested 64
Current Status
Random assignment complete 1322 youth randomly assigned within two years
Most field work associated with the implementation study is complete
Survey fielding is ongoing Very high response rates (about 85 percent)
What’s Next
Survey fielding will continue (survey firm should wrap up winter/spring 2014)
Implementation report slated for publication in January 2014
Impact report slated for publication in spring/summer 2015
All contents ©2011 by Youth Villages, Inc. with all rights reserved
Presentation to First Focus/SPARC March 19, 2013
The promises and perils of random assignment evaluations – a provider’s perspective
Discussion Topics
• Embarking on a random assignment evaluation
• Confronting the ethical issues
• Recruiting study subjects
• Monitoring program fidelity
• Controlling study costs
• Awaiting results
Why do a random assignment evaluation?
• Frightening
• Time consuming
• Referral sources/ service purchasers don’t require it
• Youth Villages already has an on-going outcome evaluation process
• Provides a benchmark
• Value in having independent evaluation
• Exciting
• Opportunity for program improvement
• Push toward evidence-based practices from govt. and foundation funders
• Meets organizational goal of increasing use of evidence-based services
PRO CON
Confronting the Ethical Issues of Random Assignment
Forty percent of youth in the study are denied entry into the
TL Program.
How do we justify that?
Recruiting Study Subjects The expectation:
The reality:
0
100
200
300
400
500
600
700
800
900
1,000
1,100
1,200
1,300
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
Month
Goal Actual
Original goal – 1,600 Adjusted goal – 1,300
It wasn’t as easy as we thought!
Monitoring Program Fidelity
In some ways, this has been the easy one!
• Program Model Adherence Reviews
• Balanced Scorecard
Is that enough?
Controlling Study Costs
The evaluation is funded by the Edna McConnell Clark Foundation and the Bill and Melinda Gates Foundation
What are the provider’s costs?
• Study coordinator (full time) • Leadership staff time • Assessors (to determine eligibility and
appropriateness for program) • Training and travel
Awaiting Study Results
Planning Begins – July 2008
Study Recruitment Begins – October 2010
One Year Follow-up Begins – November 2011
Study Recruitment Ends – October 2012
One Year Follow-up Completed – January 2014
Preliminary Report on One Year Outcomes – July 2014
Six Years!
Lessons learned?
• Go in with eyes wide open
• Take time to prepare
• Be ready for the unexpected