Changing Perspectives on Workforce System

Performance- Adjustment ModelsWorkforce Innovations Conference

July 21, 2004

2

Presenters

• USDOL Grant• Project Overview• Proposed Model

• Panel Response• Q & A

• Amanda Ahlstrand, USDOL-ETA

• Marcia Black-Watson, MI• Randall Eberts, WE Upjohn

Institute

• Neil Ridley, Heldrich Center• Dan O’Shea, Univ. of Texas• Craig Schrader, Michigan

Works!• Janet Howard, MI

3

Background: US DOL-ETA GrantPerformance Environment-- Focus on results for customers-- Better utilize available information to inform program decisions

Grant awarded to Michigan: Develop strategies and guidance for state and local workforce investment system goal setting and performance adjustment on behalf of the states

– Review information needed to establish a process for goal setting

and managing programs under WIA– Confer with other states on suggested approaches– Discuss ways to frame and analyze issues, data, and mechanisms

for system performance management– Develop proposals for comment and review– Implement pilot projects for testing

4

Project Overview

Two-tier Project With a Focus on Performance

• Develop a framework to assist states and local workforce investment areas in performance goal setting

• Provide management tools to help local workforce investment areas performance management

5

Project Team

• Department of Labor & Economic Growth

• Corporation for a Skilled Workforce

• Public Policy Associates

• W.E. Upjohn Institute for Employment Research

6

Informed Decisions

• Meeting the needs of our customers

• Evaluating agency performance fairly and equitably

• Determining the effectiveness of the delivery of services

• Evaluating and improving programs using “real time” measures

7

Evidence-based Decision Tools

Administrators are able to:

• Develop management tools

• Understand what factors contribute to success

• Adjust their course of action during the program year

8

Project Components: Tools to Identify the Factors That Contribute to Participant Success

• Distinguish between:– Factors outside the control of the local administrators – Factors that are within their control – “Value-added

Performance”

• Customize the tools for local areas using individual administrative data and wage records

9

Project Components:Framework for Negotiating Performance Goals

• Adjusts for factors outside the local administrator’s control

• Tracks the progress of local areas in meeting their performance goals

• Offers evidence-based decision making in referring participants to services and in improving the quality of services

10

Performance Adjustment Model: Purposes

• To develop “fair” measures of local workforce program performance using the new “common measures”

• To develop “value added” measures of the program

• To develop “timely” predictors for local program managers on how well their local area will do on common measures

11

Performance Adjustment Model: Methodology

• Statistically adjust for how outcomes are influenced by customer characteristics and local economic conditions

• Use statistical model, data on local customers, and forecasts of local economic conditions to provide “real time” forecast of how a local area’s performance will be adjusted for performance standards

• Use data on intermediate outcomes to predict the local area’s performance on common measures

• Combine predictions of outcomes and adjustment to predict whether the MWA will exceed each performance standard

12

Adjustment Procedure

• Quantify the impact client and economic conditions have on individual outcomes

• Aggregate these impacts to determine statewide or agency-based “expected” outcomes

• Compare “expected” outcomes to actual outcomes when data become available

13

Requisites for a “Good” Model

• High predictive power

• Understandable by administrators

• Objective

• Useful as a management tool

• Generate performance measures in a timely fashion

14

Requisites for Explanatory Variables

• Make intuitive sense

• Significant explanatory power

• Available on real-time basis at the one-stop when person registers for program

• Parsimonious– For example, pared 84 variables to 41 in common

measure 1 regression

15

DOL’s Common Measures

Adult Measures• Entered employment• Employment

retention• Earnings increase 1• Earnings increase 2• Program efficiency

Youth Measures

• Placement in employment and/or education

• Attain a degree or certificate

• Literacy and numeracy gains

• Program efficiency

16

Common Measure 1: Employment Rate

• Example is from WIA Adult Program

• Whether adult WIA participant who was not employed at registration is employed one quarter after exiting the program

• Employment in a quarter is measured by earnings greater than zero in wage record

17

Common Measure 1: Employment Rate

Registration Exit Q1 afterexit

Q2 afterexit

Q3 afterexit

Not Employed at registration AND Employed in Q1 after exit

E=1E=0

18

Estimate Impacts on Q1 Employment

• Yij = BxXij + Wj + eij

• Estimation based on individual observations of participants from 7/00-9/02

• Wage records and administrative data

• Adult WIA Programs (10,000+ observations)

19

Predicting Common Measures• Yij = BxXij + Wj + eij

• Variation in common performance measures (Y) explained by three sets of factors:

1) Personal characteristics (X)2) Local labor market conditions (X)3) The effect of program intervention (W)—value added measure

• Bx : effects of customer characteristics on post-exit employment

• Wj: Value added impact of program intervention

20

Explanatory Variables

Variable Empl. Rate

Retention Earnings Chg 1 Earnings Chg 2

Age X X

Gender X X X X

Education X X X X

Race X

Wages X X X

Barriers X X X X

Industry X X X

WIA areas X X X X

Unemp rate X X X X

21

Selected Results: Adult CM1

• Individuals, age 29 or less, are 8% more likely to become employed compared with those 50 or older

• Local areas that experience unemployment increases of 1% from one quarter before registration to one quarter after exit are expected to have 1% lower employment rate one quarter after exit

• Individuals with disability are 7% less likely to become employed one quarter after exit

• High school dropouts are 4% less likely to become employed• Several others . . . . . .

22

Adjustment Factors

• Adjustment factor– The difference between the weighted average characteristics

of the individual MWA and the weighted average characteristics of the state

• Value Added Performance – Difference between Local Area Performance and Statewide

Average Performance

– Adjusted for the difference between Local Client & Economic Conditions and Statewide Client & Economic Conditions

23

Example of Adjusting Common Measure 1 for WIA Adults

MWA

Actual employment rate for adult WIA Participants, 1 quarter after exit

Difference from state mean

Portion of difference from state mean due client or local area adjustment factors

Portion of difference from state mean not due to adjustment factors—e.g. Value Added

A 0.728 -0.026 -0.004 -0.022

B 0.716 -0.037 -0.002 -0.035

C 0.867 0.114 0.016 0.097

D 0.809 0.055 0.032 0.024

E 0.786 0.032 -0.003 0.034

F 0.747 -0.006 -0.009 0.002

G 0.652 -0.101 0.006 -0.108

H 0.849 0.096 0.038 0.057

I 0.864 0.110 0.015 0.095

J 0.746 -0.008 -0.000 -0.008

24

Example of Adjusting Common Measure 1 for WIA Adults

-0.15

-0.1

-0.05

0

0.05

0.1

0.15

A B C D E F G H I J

% D

iffe

ren

ce

Difference from Statewide Mean Adjustment Factor Value-Added Performance

25

Real-time Forecasts• Use the model to predict performance of each MWA using all

estimated effects except for MWA area effects and effects of changes in unemployment– Use this to adjust the MWAs actual performance relative to state standard or to

adjust state standard to a local standard

• Adjustment factor can be computed for each individual for individual characteristics at time of registration

• Local economic trends can be assumed or forecasted• Allows local administrators to glean the direction of

adjustments to performance standards

26

Final vs. Predicted Overall Adjustment (Correlation = 0.934)

-0.06

-0.05

-0.04

-0.03

-0.02

-0.01

0

0.01

0.02

A B C D E F G H I J

Ad

justm

en

t

Final Performance Adjustment Predicted Performance Adjustment

27

Forecasting Actual Performance based on Intermediate Outcomes

• This model with real-time adjustment can be used to forecast actual performance for each MWA

• Common measure outcomes for each local area may be forecasted at any point in time by using a forecasting equation that includes intermediate outcomes in addition to the other explanatory variables

• Intermediate outcomes are often highly correlated with common measures; e.g. for CM1—someone employed at exit is 35% more likely to be employed at +Q1

• These predictions have a 0.79 correlation with actual performance

28

Predicting Common Measure 1 for WIA Adults with Exit Variables

MWAs Actual employment rate 1 quarter after exit

Predicted employment rate using data on employment at exit

A 0.728 0.742

B 0.717 0.752

C 0.867 0.778

D 0.809 0.826

E 0.786 0.731

F 0.747 0.748

G 0.652 0.708

H 0.849 0.805

I 0.864 0.791

J 0.746 0.761

K 0.749 0.722

L 0.657 0.717

29

Correlation of Actual and Predicted Performance = .789

0.6

0.65

0.7

0.75

0.8

0.85

0.9

0.6 0.65 0.7 0.75 0.8 0.85

Predicted

Actu

al

30

Predicting whether MWA Meets Performance Standards

• Combine predictions using intermediate outcomes to forecast actual outcomes, and forecast adjustment using registration data to predict how each MWA will do on performance standard

• Correlation = .693

31

Other Common Measures, Programs and Groups

• Similar adjustment and forecasts using intermediate outcomes are developed for all common measures based on employment, earnings, or educational attainment: – WIA youth and displaced worker programs; – ES; TAA; WorkFirst

• Models so far have been estimated for 4 common measures for WIA adult programs and 2 common measures for WIA youth

32

Selected Results

• Many good individual predictors for:– Adult CM2 (job retention)– Adult CM3 (earnings gain - Q1 to + Q1)– Youth CM1 (job entry)– Youth CM2 (attainment of degree or certificate)

• Adult CM4 is problematic; relatively few good predictors for CM4 (earnings gain – Q1 to + Q3)

• Possible to construct good real time predictors for all common measures but adult CM4

• Possible to construct good predictors using intermediate outcomes for adult CM1 and CM3

33

Timeline

_______________________ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Registration Exit 1 Quarter after Exit

Unemployment data 1 quarter after exit available

Wage record data on one quarter after exit available

Estimate model Using Historical data, integrate into local data management system

Collect participant data, including prior earnings from wage records. Combine with model βs to predict adjustment

Collect exit data. Use to predict actual common measure and to predict success or failure on performance standards

With unemployment and wage record data, calculate final performance adjustments and final success or failure on performance standards

34

What’s the Payoff?• Timely data

– enhances accountability – improves program management and planning– greater satisfaction for clients and business

• Accurate and fair performance evaluation– prevents “creaming”– gives programs “bonus points” for working with the hard-to-place

• Better and more targeted use of resources– identifies what works best– tracks improvements over time

35

What is needed to succeed?

• Administrative data at the local program level– Including Social Security Numbers

• Ability to match wage-record and administrative data using SSN as the link

• Opportunity to track wage-record data for individuals who work in or move to other states

• Assistance from DLEG’s Bureau of Labor Market Information and Strategic Initiatives

• Information system enhancement to handle increased data reporting and analysis

36

Panel

• Neil Ridley, Heldrich Center, Rutgers University

• Dan O’Shea, University of Texas

• Janet Howard, State of Michigan: Moderator

37

38

For Further Information

• Marcia Black-Watson, State of Michigan,

Black-WatsonM@michigan.gov

• Martha Reesman, Corporation for a Skilled Workforce, mreesman@skilledwork.org