district management council academic return on investment (a-roi)

D I S T R I C T M A N AG E M E N T C O U N C I L

ACADEMIC RETURN ON INVESTMENT (A-ROI)

MINDSET

THE WHY OF A-ROI

White Paper: Results-Driven Decision-Making and Academic Return on Investment

• “K-12 public education performance has been stagnant for years despite increased fiscal investments, both at the state and local level”.• “…school districts have ramped up their focus on

measuring academic outcomes, most measures do not promote looking at sustained improvement or recognize the need to differentiate programming..”• “.. It’s hard to know what works, what works best for

different students, how long a program or investment should be in place prior to producing intended outcomes.”

STRATEGIC FRAMEWORK

How does A-ROI align with our score card and strategic

framework?

THREE CRITICAL COMPONENTS

We need:

1. A shift in mindset2. An analytical tool3. Align staff to develop and implement a new

approach

EVOLUTION OF SCHOOL DISTRICT’S MINDSET

Focus on Outcomes Focus on Cost Reduction

Focus on Value

No Child Left Behind forces states to set achievement standards and implement standardized tests

The fiscal crisis strained local tax revenue

Rising pressure on school districts to improve outcomes with reduced funding

Governments created annual progress benchmarks and report cards.

School districts faced a reduction in funds

Simultaneous consideration of cost and outcomes to drive sustained improvement.

School districts instituted multiple strategies to improve outcomes.

Increasing competition for public funds

2000 2008 2014

FOCUS ON OUTCOMES

• The focus on outcomes has resulted in a vast number of metrics being calculated and tracked across school districts in the country.

• Many metrics are not actionable and may mostly be compliance or accountability- related.

TYPES OF METRICS REQUESTED BY KEY STAKEHOLDERS

State

District

School Community

School Leaders

Teachers

CHALLENGES WITH OUTCOME FOCUS

• The data is used in aggregate form and lacks granularity to make real improvements because it assumes most students have similar needs• The data is not tied to instructional approaches

and thus can not be replicated or improved• The data does not show which teachers are

effective with which students• The data does not take into account other

relevant attributes

FOCUS ON COSTS

• Different districts spend very different amounts to serve similar students populations.

• Within the same district, the cost of educating a student may vary based on teaching strategy or program (special ed. vs. Title vs. ADSIS vs. AVID or resource vs. co-teaching vs. inclusion)

Per Pupil Amount

$0 $5,000 $10,000

District ADistrict BDistrict C

CHALLENGES WITH COST APPROACHES

• Costs estimates typically only include teacher costs and leave out cost of administration, support services, etc.

Incomplete

• Costs are calculated by time or by program and do not reflect whether or not students realize the intended outcome.

Not tied to specific

outcomes

• Costs are captured by department or by school, which are not granular enough to show changes in instructional process.

Not captured at specific student

level

DMC’S VALUE-BASED APPROACH

DMC Value-Based Analysis Framework

Student Segmentation

• Educational need• Other relevant

attributes

Student Outcomes

• Mastery

Costs

• Direct and indirect• Duration for attainment

KEY BENEFITS OF A VALUE-BASED APPROACH

Key Benefit1. Focus on value and

effectiveness not absolute results

2. Promote innovation and experimentation

3. Compare results across schools, district, and even regions

RationaleAllows identification of cost-effective programs or interventions that have the greatest impact on student performance.

Frees up resources to invest in other areas and allows the district to test newer strategies.

Allows normalization of outcomes and costs across districts and may allow easy identification and replication.

STUDENT SEGMENTATION

In most districts, segmentation of students takes place based on NCLB categories or funding streams and only classroom teachers have a student by student look.Issues• This does not provide sufficient information for

tailored interventions- instead, all students receive the same treatment.• All thStuden

tMath Score

Current Segmentation

A 59 ESL

B 60 F/R

C 61 Spec. Ed.

All three students receive math intervention

STUDENT OUTCOMES

Potential outcome measures

• Improvement from level F to G in Fountas & Pinnell in 6 months

• 80% score on Algebra 1 final

• Score of 4 on 6th grade writing rubric-persuasive essay

Key Practices

• Monitors progress toward academic goals

• Measures outcomes associated with specific student needs

• Includes both absolute and growth metrics

• Assesses performance frequently.

Good outcome measures are actionable

STUDENT OUTCOMES

StudentsFormative Assessmen

ts

Desired Results

• Student specific

• Individualized need and key attributes

• Allows for timely and targeted interventions

COSTS

Identify Major Steps

and Map Process

Calculate Direct Costs

Calculate Indirect Costs

Compute Total Cost

per student

Complete Cost Analysis

Choose an achievement Benchmark

Create a map that outlines the major steps for student to reach benchmark

Estimate the direct cost of each service

Allocate services that indirectly support a student’s learning

Add direct and indirect costs to obtain annual cost

KEY TAKE-AWAYS

• Schools have many measures of outcomes, but few actionable.• In order to capture actionable outcome measures,

schools need performance data segmented by student need and key attributes.• Capture all costs connected with serving student,

not just the easy ones.• Calculate the cost to achieve an outcome, not just

by the budget year.

USING A VALUE-BASED APPROACH

1. Find or create natural experiments to evaluate programs• Existing programs: Use student-level data when

analyzing programs that already exist and look for natural experiments to compare value.

• Proposed programs: Ensure that the rollout of new programs have measurable outcomes and meaningful comparisons.

2. Give value-based decision-making a seat at the table

3. Formalize the program approval, modify, or abandonment process• Base on data and separate from the budget cycle

A-ROI TIMELINE

Aug Sept Oct Nov Dec Jan Feb Mar

Select target programs

Conduct value-based analyses

Find meaning

Budget Cycle

OUTCOME MEASURES

Monitors progress toward academic goals

Does not measure process or satisfaction metrics

Includes both absolute and growth metrics

Assess performance frequency

Measures outcomes associated with specific student needs

Key Attributes

of Effective Outcome Measures

POSSIBLE PROGRAMS FOR A-ROI

• AVID• ADSIS• LLI