Improving Education through Accountability and Evaluation: Lessons from Around the WorldRome, Italy3 October, 2012

Building Evidence that Counts: Evaluating Career

Pathways Interventions for Disconnected Youth and

Adults in the U.S.

David FeinAbt Associates Inc

david_fein@abtassoc.com

Acknowledgments

Work on Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation sponsored by Administration for Children and Families, U.S. Department of Health and Human Services with major support from the Open Society Foundations, additional support from Joyce and Kresge foundations

Presentation represents the author’s views only Paper available at:

http://www.projectisis.org/wp-content/uploads/2012/05/ISIS-Career-Pathways-Framework-OPRE-2012-30_5-16-12.pdf

2

3

Introduction

Large population of adults and youth in the U.S. with low skills, little/no post-secondary education, and poor prospects for self-sufficiency

In the 1990s-2000s, national policies emphasized labor force attachment (versus education and training)

Evidence from social experiments influenced this emphasis– Consistent modest positive findings for employment-focused interventions;

disappointing findings for education and training– Some argued human capital investments should be limited to younger ages

Findings characterize traditional educational approaches not well-suited to second-chance populations

The Ground Is Shifting

Increasing evidence of limits to strictly employment-focused strategies, especially in a weak economy

Recession hits low-skilled jobs hardest and accelerates technological shifts widening skills premiums

Improved understanding of challenges facing individuals seeking post-2ndary, and deficiencies in second- (and first-) chance systems

Outpouring of promising innovations addressing these issues But these policies and systems remain small scale and highly fragmented… Enter Career Pathways – a body of ideas and practices for integrating

promising innovations promoted by a growing movement in the U.S. And, with work, a conceptual framework for building systematic, high-quality evidence on these ideas and practices

– It’s not about work vs. education, it’s about joining the two– I.e., “We’ve got the… stuff, we’ve just got to bring it together!” (Obama, on the

economy in general, 9-28-12)

4

Building Evidence on Career Pathways that Counts

Abt Associates, a global social policy research and technical assistance firm, is leading several national random assignment studies of promising career pathways interventions

The work has involved mapping out a conceptual framework and identifying priorities and opportunities for useful experiments

I will introduce this framework, show how it can help to organize a wide range of studies, and discuss some ways we are designing these projects so that their findings will count

5

Key Career Pathways Ideas

Address the wide range of skills and needs of second-chance populations

Create manageable, well-articulated training steps Provide credentials valued in high demand occupations/sectors Build effective partnerships

– E.g., public agencies (education, labor, human services); community colleges; community-based organizations; employers

– Vs. K-12, more diverse actors, complexity

Basic ideas also being applied in strengthening career pathways in secondary education

6

V. BA+ Programs Upper-Skilled Jobs V. BA+ Programs Upper-Skilled Jobs

IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs

IV. 1-2-Year Certificate to AA Programs Mid-Level Skilled Jobs

III. Short-Term Certificate Programs Entry-Level Skilled Jobs

III. Short-Term Certificate Programs Entry-Level Skilled Jobs

II. Sectoral Bridge Programs Semi-Skilled Jobs

II. Sectoral Bridge Programs Semi-Skilled Jobs

I. Basic Bridge ProgramsI. Basic Bridge Programs

Pro

spec

ts f

or

go

od

-pay

ing

, st

ab

le e

mp

loym

en

t

Occupational, academic, and life skills

The Basic Career Pathways Model

8

Program Inputs: Signature CP Service Strategies Comprehensive assessment

– Academic and non-academic skills

Basic and technical skills instruction– Modularization, contextualization, acceleration, flexible delivery,

active learning

Supports – Pro-active advising and guidance, supplemental instruction,

social supports, supportive services, financial assistance

Employment connections– During and after training

Foundational Academic SkillsFoundational Academic Skills

Intermediate Outcomes

Increase Performance & Persistence in Training

• Certificate/Diploma• 2- year, 4- year Degree

Improve Performance & Advancement in Jobs

• Earnings• Benefits• Job security

Improve Other Outcomes

• Income & assets• Child & adult well being• Local economic growth

Participant Characteristics

•Demographic•Educational•Economic

Program Inputs Primary Outcomes

TO

NE

XT

ST

EP

Occupational SkillsOccupational Skills

Psycho-social Factors Psycho-social Factors

Career Orientation and Knowledge Career Orientation and Knowledge

Resource Constraints Resource Constraints

Other Personal and Family ChallengesOther Personal and Family Challenges

Contextual Factors: Institutional, Economic, SocialContextual Factors: Institutional, Economic, Social

Initial Targeting & Placement Decisions

Initial Targeting & Placement Decisions

Theory of Change for Career Pathways

Illustration #1: Pima Community College Pathways to Healthcare Program (Arizona)• Basic elements

– In partnership with county One-Stop, college provides training and credentials in 16 health care occupations, organized into 5 broad pathways (+ 3 broad levels), programs varying from 1-24 months. Targets low-income adults, substantially Latino population. Funded by federal HPOG program.

Signature strategies– Instruction: Upfront assessment and training planning, w/10-week

contextualized basic skills development program if below required entry skill levels. Many programs offer streamlined/compressed formats, translate “clock-hours” at lower levels to prerequisites to higher levels.

– Supports: More intensive and coordinated case management (via One-Stop) and academic advising and services (via college), financial support covers training and related expenses.

– Employment: Outreach to engage local employers in training, internships, employment; use One-Stop services; MOUs linking to various county and state agencies.

10

Illustration #2: Year Up (National)• Basic elements

– National non-profit operating in 8 urban areas. Partners with community colleges and major employers in financial services and information technology. Targets disadvantaged youth, 18-24, with high school diploma/equivalent.

Signature strategies– Instruction: Six-month “learning development” phase provides customized

training in technical and contextualized basic and professional skills (e.g., communication, behavior). Earn 14+ college credits through agreement w/local colleges.

– Supports: “High support, high feedback” environment—learning communities, pro-active counseling, structured group sessions, supportive services, and weekly performance-based stipend of up to $260 throughout the year. Post-program follow-up and engagement.

– Employment: Second six-month internship phase places students in entry-level financial services/IT positions with major employers, who contribute about half of total program costs per participant.

11

Illustrative Evidence on More Explicit Career Pathways Approaches: Comprehensive, well-targeted “first step” training programs can have substantial impactsSectoral training (Maguire et al. 2010)•3 experienced CBOs provide customized short-term training in varied high-demand fields•Disadvantaged adults, HS+, careful screening (n=1,014)

Year Up (Roder & Elliott 2011)•National org provides 6 months customized training + 6 month paid internship in IT and finance sectors•Disadvantaged urban youth 18-24, HS+•Program articulates w/local colleges, completers earn 14+ credits•Results from small initial experiment (n=164)

12

Examples of Recent Experimental Findings on Signature CP Strategies

Learning communities– As bridge program (Fein et al. 2006)– By linking courses (Sommo et al. 2012; Visher et al. 2012)

More intensive, specialized personal guidance and support– Coaching (Bettinger & Baker 2011)– Help with financial aid applications (Bettinger et al. 2012)– Student success courses + support (Weiss et al. 2011)

Increased financial support– Performance-based scholarships (Patel & Richburg-Hayes 2012)– Unconditional needs-based grants (Goldrick-Rab et al. 2011)

Psycho-social interventions– E.g., implicit theories of intelligence, normalizing worries about fitting in and

success (Yeager & Walton 2011)

13

Designing Evaluations that Matter

Evaluations should be designed in ways that invite practitioners to make use of the results and adopt solid practices based on evidence. We need to recognize the role of motivated program leaders at the center of evaluation efforts, to ensure that these efforts advance program theory and practice, rather than merely fulfilling a funder request. Research tells us that organizations and their leaders need to own and trust information in order to use it. Partnership on the ground between practitioners and evaluators, along with the long-term support of committed public and private funders, is indispensable if the goal is to deliver evaluations that actually improve program quality and effectiveness.

– Public/Private Ventures, Priorities for a New Decade: Making (More) Social Programs Work (Better), 2011.

PrinciplesElucidate and address theories of program effectivenessDevelop strong common metrics that will be easily understood and widely acceptedDesign to anticipate interest in scaling and replicating successful approachesFoster appreciation for technical rigor (e.g., experimental designs)Produce evaluation products useful to policymakers and practitioners

– Emphasis on active interaction: evaluators-practitioner, practitioner-practitioner

14

Example: The ISIS Evaluation

First national random assignment study of U.S. career pathways programs

Separate tests of nine relatively comprehensive career pathways programs, with implementation and cost-benefit studies

Central goals– Assess overall effectiveness of each program– Understand “why and under what conditions?”

• Work within well-specified theory of change• Strong focus on impacts on exposure to program inputs, intervening outcomes,

heterogeneity• Non-experimental analysis of mediators of impacts on primary outcomes

Strong emphasis on collaboration helps ensure findings will count…

15

Collaboration Increases Chances that Findings Will Matter Extensive stakeholder outreach + literature review identify career pathways

focus in ISIS… Early and ongoing exchanges foster interest and create positive climate for

experiments among potential sites– Working w/sites as partners w/complementary expertise and interests– Building an ongoing “learning community”

• Learn from experts on the team, peer-to-peer exchange, participation in external forums

• Help plan and execute dissemination strategies

Maintain and extend relationships with wider stakeholder network– Including private and public funding agencies, policy makers, professional associates,

advocates, practitioners, – Through meetings, website, public webinars, varied publications– On diverse interests: related projects, substantive topics/directions in ISIS, evaluation

findings, evaluation products,

So that findings aren’t limited to a small number of reports, but shared in dynamic, diversified ways with engaged audiences

16

Summary

Career pathways framework useful in program and evaluation design, organizing emerging findings

Genuine partnerships with practitioners and other stakeholders key to designing evaluations that matter

Thank you!

17