academic return on investment: doing the most good with ...€¦ · essay review • personalized...

© 2016 THE DISTRICT MANAGEMENT COUNCIL

Academic Return on Investment:

Doing the most good with limited funds

www.dmcouncil.org 2

As funding tightens, the traditional approach of adding resources to fund

new initiatives may not be an option in many districts.

Traditional Approach to Funding New Initiatives INTRO

Time

Money

People

Traditionally districts have relied on

increasing money, expanding time, or

adding people to fund new initiatives

www.dmcouncil.org 3

Funding in real dollars is decreasing, and may not return to past levels

for the foreseeable future in many districts.

Change in Per Student Spending 2005 – 2013

Inflation Adjusted to 2004

1.4% 1.7%

3.0%

2.1%

2.8%

-0.6%

-3.0%

-2.1%

-0.6%

-4%

-2%

0%

2%

4%

2005 2006 2007 2008 2009 2010 2011 2012 2013

Source: U.S. Census Bureau, U.S. Bureau of Labor Statistics.

INTRO

www.dmcouncil.org 4

Investments Programs and Initiatives Better Outcomes?

It is difficult to know which investments are actually leading to positive

outcomes for students.

Common Challenges with District Programming

Time

People

Money

Core

instruction Tutors

Speech and

language Special

education

Peer

tutoring

ELL

Read 180

Reading

Recovery

After school

Paraprofessionals

Extra Small

Group Reading

Block • Which programs drive

student achievement?

• Which programs should

be expanded to serve

more students?

• Which programs are an

ineffective use of funds?

• Do some programs work

for some students and

not for others?

? ? ? ? ?

www.dmcouncil.org 5

Objective studies of a program’s Academic Return on Investment (A-ROI)

can be powerful ways to build shared understanding and reduce pushback.

Typical Disagreements (Absent A-ROI Studies)

Increased anxiety

and friction

It works! It doesn’t work!

www.dmcouncil.org 6

In one district, a practice of co-teaching had been a source of

long-running debate, but no data clearly indicated its effectiveness.

Common Opinions of Co-teaching

Supporters

Doubters

“Inclusion can’t work without co-teaching”

“Many students are really benefiting!”

“Co-teaching is the best of both worlds”

“In my old district we didn’t need co-teaching”

“Too many of our students still struggle”

“Co-teaching is simply not worth the cost”

Differing opinions make it difficult to

move beyond the status quo

www.dmcouncil.org 7

A-ROI analysis involves the interplay of three components: student

segments, student outcomes and fully-loaded costs.

DMC Academic Return on Investment (A-ROI) Framework

Fully-Loaded Costs

A-ROI

www.dmcouncil.org

• Focused students

• Very positive staff feedback

Districts use a wide variety of programs and strategies to help struggling

students, but often do not know which are effective for which students.

Components of Middle School Remediation Programs

• Blended learning

• Targeting skill remediation

• Large group

• Essay review

• Personalized instruction

• Small group

Math Intervention English Intervention

• Loud, rambunctious room

• Stressed staff Observations:

Delivery Model:

www.dmcouncil.org

Using hard data can help illuminate the effectiveness of a program and

make decisions about that program’s future much simpler.

Comparing Program Results

18 months

50%

6 months

Median student

growth per year

95%

Typical student

attendance

$4,000

$1,000

Cost per student

The math program produced

greater student

outcomes…

…and the math program also

operated at lower costs

.

Ma

th

En

gli

sh

Without assessing effectiveness there would have been intense pushback to shifting

resources to expand the math program and curtail the English program

www.dmcouncil.org 10

DMC has developed a detailed process, with associated worksheets and

protocols, for conducting an A-ROI analysis.

Ten Step A-ROI Process

Plan Collect

data

Evaluate Act

1. Select target

2. Define success

3. Design analysis

4. Collect segment

and background data

5. Collect outcomes

data

6. Collect costs data

7. Evaluate program

effectiveness

8. Analyze cost-

effectiveness

9. Draw insight

10. Take action


A-ROI can be utilized to evaluate existing programs implemented in the

past or new/ongoing programs going forward.

Evaluation

approaches

Backwards-looking approach:

• e.g. Did a new curriculum lead to

better outcomes?

Forwards-looking approach:

• e.g. Will these new iPads lead to

better outcomes?

Existing Programs Proposed Programs 1 2

Why use this approach?

• A key initiative implemented in the past has

unclear effectiveness and

cost-effectiveness for students

• Existing data in can provide sufficient

information to evaluate the program

• Quick to get started

Why use this approach?

• An untested, new program will need to be

evaluated to guide implementation,

expansion, or continued funding

• Evaluation can be based on data that is

specifically aligned with the program’s

goals

• Few doubts about the data available for

analysis


With a proper A-ROI analysis, decisions about a program’s future can and

should go beyond simply “keep” or “eliminate.”

Options for Future of Program

AROI findings

Expand

Keep

Fix

Streamline

Eliminate

• If found particularly cost effective, the district may

want to expand the program to all students in the

segment

• If found cost effective, but all students in the

segment already receive the program, it may be

kept as is

• If only some pieces of the program are found to be

effective, the ineffective portions may be fixed

• The district may seek to cut out pieces of the

program found ineffective or costly

• If the program does not work for particular

segments, the district may want to stop providing

the program to that segment


A-ROI seeks to measure both effectiveness and cost effectiveness.

Sample Data from A-ROI Analysis

Using a small group intervention block achieves the same

outcomes as Reading Recovery, but costs one-quarter as

much.

Program Used

Years Growth/Year of

Students in Program

Cost Per

Student

Struggling Readers

Before Intervention 0.6 $0

Reading Recovery 1.3 $4,500

Read 180 1.0 $1,500

Additional Small Group

Reading Block 1.3 $1,125


Formalize the program approval and abandonment process, and separate

it from the budget review cycle.

Program Approval Cycle

Prior School Year Summer & Fall Nov Dec Jan Feb Mar

Select target programs

Conduct AROI analyses

Find meaning

Budget cycle


Provide “A-ROI champions” a formal voice in budget discussions by

locating them high enough in the organizational structure.

Example Org. Structure

Superintendent

Grants Chief Academic

Officer

Data &

Account-ability Budget

A-ROI

technical

skills

Instruction, budget and data must work as team.


Selecting a target program and potential student segments.

Districts can analyze just a few programs a year


Technology

based Math

program ? ? 6

Teacher’s

professional

development

program ? ? ? ? 4

Expansion of

arts courses

for high

schools

? ? ? ? ? ? 2

Selecting a target program and potential student segments.

What programs, efforts or strategies would you want to study?


DMC assessed 8 districts to determine what skills were present, and what

capacities would need to be built for A-ROI.

A-ROI Capacity Rubric

Ready

Willing

Able

• Does the district have the

necessary structures to support

A-ROI?

• Will the student data, budget data,

and budget process systems

support A-ROI?

• How invested are the leadership,

team, and school staff in A-ROI?

• How well do the key players

understand the work they need

to do?

• How prepared is the A-ROI team

to undertake each step of the

analysis process?

• What skills do the team members

need to build in order to be

successful?

Most Common

Recommendations

• Communicate early

and often

• Allocate adequate time

for data collection

• Clarify team roles and

structure

• Teach team throughout

the process


Additional resources are available.

A Better Way To Budget by Nathan Levenson Focuses on how to

overcome political and social pushback that frequently gets in the way of

shifting limited resources to student-centered initiatives

• Introduces eight strategies for shifting funds and winning support

• Available at Amazon.com

Spending Money Wisely by Nathan Levenson, Karla Baehr, James Smith, and

Claire Sullivan

• Highlights 10 strategies for improving the reach of scarce district resources

• Available at dmcouncil.org/spending-money-wisely-ebook

The District Management Journal

• Provides a mix of practical, actionable case studies, interviews, and thought

pieces that highlight best practices around the nation and current issues in education

• Available at dmcouncil.org/the-district-management-journal


Q&A

© 2013 THE DISTRICT MANAGEMENT COUNCIL www.dmcouncil.org

If you have any comments or questions about

the contents of this document, please contact

Nathan Levenson, President,

The District Management Council:

• Tel: (877) DMC-3500

• Email: [email protected]

• Fax: (617) 249-0910

• Web: dmcouncil.org

• Mail: 133 Federal Street, Boston, MA 02110

academic return on investment: doing the most good with ...€¦ · essay review • personalized...

Documents