Outcome-Based Scrutiny:Outcome-Based Scrutiny:Putting a Price on PerformancePutting a Price on Performance

Government Accountability Project

Mercatus Center

George Mason University

Arlington, VA

http://www.mercatus.org

What is Outcome-Based What is Outcome-Based Scrutiny?Scrutiny?

Assessment of federal programs to determine:

1. What specific benefits were produced (or harms averted) for citizens?

2. How could resources be reallocated to make government more effective?

The 4The 4thth Step in the GPRA Step in the GPRA ProcessProcess

Planning (Strategic Plans)

Implementation

Disclosure (Performance Reports)

Scrutiny

Why Outcome-Based Scrutiny?Why Outcome-Based Scrutiny?

• Improve quality of agency reporting

• Improve congressional resource allocation decisions

• Drive fundamental management change within federal agencies

A 7-Step ProcessA 7-Step Process

Mercatus demonstration study: Mercatus demonstration study: Vocational Training ProgramsVocational Training Programs

Note: This is a demonstration study only, not a full analysis that provides sufficient information for congressional decisions.

1. Outcome: “What is the issue 1. Outcome: “What is the issue and the intended outcome?”and the intended outcome?”

Focus on programs whose primary purpose is to enhance employability

Outcome measure is the number of people who moved into work or became more employable as a result of the program

2. Agencies: “Who is 2. Agencies: “Who is responsible?”responsible?”

• Labor• Education• Health & Human Services• Interior• Housing & Urban Development• Justice

[Narrowed down from 12 agencies]

3. Programs: “How do they do 3. Programs: “How do they do it?”it?”

Department # of Programs

Labor 17

Education 11

Health & Human Svcs. 7

Interior 4

Housing & Urban Dev. 2

Justice 3

Narrowing the list…Narrowing the list…

105 programs mention vocational training

44 (grouped into 25 areas in APRs) have it

as a primary goal

34 (in 14 areas) have outcome data

How comprehensive is this?How comprehensive is this?

Our analysis covers

• 77 percent of vocational programs

• 56 percent of vocational program areas defined in performance reports

• 97 percent of spending we identified

Where’d we get this stuff?Where’d we get this stuff?

• Programs: Agency Strategic Plans, Catalog of Federal Domestic Assistance (CFDA), agency web sites

• Outcomes: GPRA Annual Performance Reports, agency analysts

• Cost figures: CFDA, U.S. Budget, agency budget justifications

4. Results: “Are they effective?”4. Results: “Are they effective?”

Program Area # of PlacementsVocEd - State Grants and Tech-Prep (Education) 1,600,000Veterans in Need (Labor) 288,404School-to-Work (Education) 257,600Adult Disadvantaged Job Training (Labor) 207,000Dislocated Workers (Labor) 163,438Welfare-to-Work (Labor) 79,771Youth Transition (Labor) 71,397Job Corps (Labor) 58,010Economic Independence - Refugees (HHS) 52,298BIA Community Development (Interior) 11,158Indian and Native Americans (Labor) 7,980Migrant and Seasonal Farmworkers (Labor) 7,300Trade-Affected Workers (Labor) 4420Tribal Postsecondary Vocational Institutions (Education) 187

5: Cost/Benefit: “What is the best 5: Cost/Benefit: “What is the best value?”value?”

Program Area Appropriation Placements/$million Cost/placement

School-to-Work (Education) $149,000,000 1729 $578

Veterans in Need (Labor) $167,000,000 1727 $579

VocEd - State Grants and Tech-Prep (Education) $1,056,000,000 1515 $660BIA Community Development (Interior) $12,928,000 863 $1,159

Adult Disadvantaged Job Training (Labor) $955,000,000 217 $4,608

Indian and Native Americans (Labor) $53,700,000 149 $6,711Economic Independence - Refugees (HHS) $423,000,000 124 $8,065

Dislocated Workers (Labor) $1,347,000,000 121 $8,264

Migrant and Seasonal Farmworkers (Labor) $73,000,000 100 $10,000Youth Transition (Labor) $1,201,000,000 59 $16,949

Welfare-to-Work (Labor) $1,476,000,000 54 $18,519Job Corps (Labor) $1,188,000,000 49 $20,408Tribal Postsecondary Vocational Institutions (Education) $4,100,000 46 $21,739Trade-Affected Workers (Labor) $131,100,000 34 $29,412

6: Reallocation: “What changes 6: Reallocation: “What changes give the greatest benefits?”give the greatest benefits?”

• Total expenditure: $8.4 billion• Total benefit: 2.8 million placements• Average cost: $3000/person

• Hypothetical total benefit if most cost-effective program received all resources: 14.5 million placements

Reallocation within 4 youth Reallocation within 4 youth programs with outcome info.programs with outcome info.

• Total cost: $3.6 billion• Total placements: 2 million

• Hypothetical placements if 2 most cost-effective programs received all resources: 5.5 million

7: Forgone Opportunity: “What do 7: Forgone Opportunity: “What do we lose under the status quo?”we lose under the status quo?”

• All programs example:14.5 million hypothetical placements

- 2.8 million actual placements11.7 million people not helped!

• Youth example:5.5 million hypothetical placements

- 2 million actual placements3 million young people not helped!

Implementation TipsImplementation Tips

• Alternative values define public benefit

• Correlation is not causation

• It can be OK to mix apples and oranges

• Unit costs may not be constant

• Match results with expenditures

Alternative values define public Alternative values define public benefitbenefit

• Forgone opportunity may be 11.7 million people…

• …or 281,178 additional trade-affected workers who might have been helped if all the resources went to that program.

• If it only costs $578 per placement, do those students really need government help?

Correlation is not causationCorrelation is not causation

Observed placements

- Placements that would have occurred in the absence of the program

_____________________________________

Outcome actually attributable to the program

Apples and OrangesApples and Oranges

• Complaint: Comparisons “unfair” due to differences among target populations.

• Judgment required: Compare all vocational programs or all youth vocational programs?

• The forgone opportunities are still real.

Unit costs can changeUnit costs can change

• Easy results may already have been achieved.

• Economies of scale may not be fully realized.

• Forgone opportunity may be larger or smaller than our calculations suggest.

Marginal and average costs can Marginal and average costs can vary with outputvary with output

Match results with expendituresMatch results with expenditures

• Cost is a single year’s appropriation, but placement figures are for a single year.

• Ideal data would be time series, not snapshot.

• Match this year’s outcomes with past expenditures contributing to the outcomes?

Outcome-Based Scrutiny Outcome-Based Scrutiny improves improves KnowledgeKnowledge and and

IncentivesIncentives

• Reveals what kind of information is needed from agencies

• Reallocation improves govt. performance• Comparison promotes interagency learning• Prospect of reallocation motivates

improvement