tatiana melguizo & bo kim (usc) hans bos (american institutes for research)

21
Using a Regression Discontinuity Design to Estimate the Impact of Placement Decisions in the Los Angeles Community College District Tatiana Melguizo & Bo Kim (USC) Hans Bos (American Institutes for Research) George Prather (LACCD, Retired) Presented at the Research and Planning Group conference, April 2, 2013 This research is funded by a grant from the U.S. Department of Education’s Institute of Education Sciences (IES).

Upload: powa

Post on 09-Feb-2016

29 views

Category:

Documents


0 download

DESCRIPTION

Using a Regression Discontinuity Design to Estimate the Impact of Placement Decisions in the Los Angeles Community College District. Tatiana Melguizo & Bo Kim (USC) Hans Bos (American Institutes for Research) George Prather (LACCD, Retired) - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Using a Regression Discontinuity Design to Estimate the Impact of

Placement Decisions in the Los Angeles Community College

District

Tatiana Melguizo & Bo Kim (USC)Hans Bos (American Institutes for Research)

George Prather (LACCD, Retired)

Presented at the Research and Planning Group conference, April 2, 2013

This research is funded by a grant from the U.S. Department of Education’s Institute of Education Sciences (IES).

Page 2: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

The Problem

• Community college students have widely varying initial skill levels and the majority arrive with substantial remedial needs in math

• Colleges have to offer classes to meet the needs of their diverse students and have to keep heterogeneity in the classrooms manageable

• Placing students incorrectly can reduce the likelihood that students succeed

Page 3: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Math Placement in the Los Angeles Community College

District

About sixty percent of the entering

students need at least one remedial

class before taking a math class that

counts towards a degree

Page 4: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Some Controversy about Impact of Remedial Education

• Some research finds that remedial education provides the preparation necessary for students to succeed in college (Boylan, Bliss, & Bonham, 1994; 1997; Lazarik, 1997)

• Critics contend that remedial education may hold students back and be ineffective (Calcagno & Long, 2008; Martorell & McFarland, 2011)

• We argue that it depends on how students are placed and how cut points between levels are set

Page 5: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Focus on Cut Points

• Math faculty set the cut points between the different levels based on who applies and how their course offerings are distributed

• If the cut points are too high, too many students languish in remedial courses

• If the cut points are too low, too many students fail higher-level courses and present a challenge to the instructors

• Getting the cut points just right is important

Page 6: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Changing Cut Scores

Page 7: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Different Pathways to Success(Arithmetic vs. Pre Algebra)

Test

Placed in Pre-

Algebra

Placed in Arithmeti

c

Enroll in Pre-

Algebra

Enroll in Arithmetic

No enrollme

nt

Success

Failure

Success

Failure

Next course

Page 8: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Success Above Cut Point

Test

Placed in Pre-

Algebra

Placed in Arithmeti

c

Enroll in Pre-

Algebra

Enroll in Arithmetic

No enrollme

nt

Success

Failure

Success

Failure

Next course

Direct but academically challenging

Page 9: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Success Below Cut Point

Test

Placed in Pre-

Algebra

Placed in Arithmeti

c

Enroll in Pre-

Algebra

Enroll in Arithmetic

No enrollme

nt

Success

Failure

Success

Failure

Next course

A less academically challenging but more time-consuming trajectory

Page 10: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

The Impact Question

• For a student at the margin, does the longer road to success produce better results?

• Is greater likelihood of success worth the extra time and effort?

The Policy Question• Are cut points between math placement

options set correctly?

Page 11: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Using Regression Discontinuity to Evaluate Placements

• Regression discontinuity analysis is the strongest non-experimental method to estimate causal effects

• It depends on a continuous forcing variable and an exogenously established cut point

• Those two conditions are present in this situation

Page 12: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Out

com

e

Placement Test Score

Regression Line

Arithmetic

Pre-Algebra

Impa

ct

Page 13: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Analytical Details

• Not everyone follows the placement test’s recommendation

• Some students enroll below or above their placement level

• Compliance was high

• Impacts are estimated using discrete time survival analysis implemented within a regression discontinuity framework

• The density of the forcing variable around the cut point was normal (confirmed with McCrary test)

Page 14: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

The Impact of Placement Decisions Needs to be Observed

Over Time

Page 15: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

The Impact of Placement Decisions Needs to be Observed

by College

Page 16: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

The Impact of Placement Decisions Needs to be Observed

by Level

Page 17: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Students are not Being Placed Effectively in the Higher Levels

(Elementary versus Intermediate Algebra)

Page 18: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Students Placed in Lowest Levels Accumulate More Degree

Applicable Credits

Page 19: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Placement in Less Effective in Terms of Transfer Degree Credit

Accumulation

Page 20: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

Conclusions• Cut points between elementary algebra

and intermediary algebra may be set too high as marginal students do worse in elementary algebra

• At lower levels the cut points appear to be in the right place

• For students placed in lower levels of math sequence, there is no penalty in terms of accumulating 30 degree applicable credits

• For students placed in higher levels there is a penalty in terms of completing 30 transfer credits

Page 21: Tatiana Melguizo & Bo Kim (USC) Hans  Bos  (American Institutes for Research)

THANK YOU!Questions

Tatiana [email protected]

http://www.usc.edu/dept/education/rossier_faculty/tmelguizo/