developmental education assessment, placement, and progression thomas bailey based on research by...

Developmental EducationAssessment, Placement, and Progression

Thomas Bailey

Based on Research by

Katherine Hughes, Shanna Jaggars, Judith Scott-Clayton

National Context

• For many (most?) entering CC students, assessment center is one of first places they will visit

• For the majority of students sitting for these exams, the result is placement into developmental education

• Yet research has not consistently found that this process actually improves student outcomes

CCRC Literature Review(Hughes & Scott-Clayton)

• Examined three questions:1. Is there consensus regarding the proper

purpose and role of assessment in CCs?2. Are the most commonly used assessments

valid for their intended purpose?3. Are there alternative models of assessment

that may improve outcomes for underprepared students?

• CUNY study brings new data to bear on similar set of questions

No Consensus on Meaning of College Ready

• Many assessments

• Many cut off scores

• Many policies with respect to – Mandatory Testing– Mandatory Placement

Figure 3: Educational Outcome by Math CPT Score and Estimated Discontinuity

0.2

.4.6

.8

-50 -40 -30 -20 -10 0 10 20 30CPT Score Relative to Math Cutoff

Estimated Discontinuity = -0.014(0.012)

Passing First College-Level Course

0.2

.4



2 yr Degree Completion

20

25

30

35

40

45

50


Estimated Discontinuity = 3.590(0.657)

Total Credits Earned

0.2

.4.6

.8



Fall-to-Fall Retention

0.2

.4



Transfer to 4 yr

20

25

30

35

40

45

50



Total College-Level Credits Earned

Are Dev Ed Assessments Valid?

• CUNY uses COMPASS math & reading tests (published by ACT, Inc.; one of two most common assessments)

• There are lots of different ways to think about validity:– Construct validity: does the test measure what you think it does?– Predictive validity: does the test predict some measure of later success?– Argument-based approach to validity: “It is the interpretation of test

scores required by proposed uses that are evaluated, not the test itself” (Standards for Educational and Psychological Testing)

• Focus here is on predictive validity– This is a necessary, but not sufficient component of overall validity of

the test– “[U]ltimately, it is the responsibility of the users of a test to evaluate this

evidence to ensure the test is appropriate for the purpose(s) for which it is being used” (College Board, 2003, p. A-62)

– Broadest analysis of validity eventually requires a program evaluation: when students are assigned to some treatment on the basis of a score, do better outcomes result?

Predictive Validity Analysis

• Research questions:– How well do placement test scores predict “success”

in the relevant gatekeeper course?– How well do other measures (such as high school

performance) predict success, either instead of or in addition to placement test scores?

– How many students are “correctly placed” using current placement test cutoffs to divide students, versus assigning all students to the same level?

What is “Gatekeeper Success”?

• “Gatekeeper” course: first college-level course

• We look at three measures:– Completed course with B or higher– Completed course with C or higher– Passed course (D- or higher)

• These measures of success are all conditional upon actually enrolling in a gatekeeper course

Research Method Overview• Focus on first-time 2004-2007 entrants at two-year

colleges only, who have CAS and placement test data• First, estimate statistical relationships between

placement test scores (and/or other predictors) and gatekeeper success– Restrict sample to students who took gatekeeper without taking

developmental coursework (“estimation sample”)– Then, regress gatekeeper success on placement test scores

(and/or other predictors) to estimate relationships– Examine two summary measures: R-squareds and correlation

coefficients• Second, use logistic regression to predict which students

are likely to be “correctly placed” using different placement criteria

Methodological Concerns

• Restriction of range– R-squareds, correlations are measured only for those

who were placed directly into gatekeeper course– In general this tends to depress r-squareds and

correlations

• Extrapolation– For placement accuracy analysis, we must use

relationships estimated on about 25% of the data to predict likelihood of “success” for the other 75%

– So we must hope that the other 75% aren’t that different (not totally implausible)

Table 1 (R-squareds)

Proportion of Variation in Gatekeeper Outcomes That Can Be ExplainedBy Alternative Sets of Predictor Variables ("R-Squareds")

Test Scores, HSPlacement Test High School Subj. Test Scores and GPA/Units, LocalHS,

Scores Only GPA/Units Only HS GPA/Units YearsSinceHS

MathEarned B or higher in GK 0.121 0.094 0.160 0.184Earned C or higher in GK 0.069 0.070 0.104 0.122Passed GK (D- or higher) 0.040 0.054 0.070 0.080

EnglishEarned B or higher in GK 0.022 0.029 0.048 0.098Earned C or higher in GK 0.009 0.028 0.035 0.066Passed GK (D- or higher) 0.004 0.024 0.027 0.053

Table 1 (Correlations)

Correlation Coefficients Between Gatekeeper OutcomesAnd Alternative Sets of Predictor Variables

Test Scores, HSPlacement Test High School Subj. Test Scores and GPA/Units, LocalHS,

Scores Only GPA/Units Only HS GPA/Units YearsSinceHS

MathEarned B or higher in GK 0.349 0.306 0.400 0.429Earned C or higher in GK 0.263 0.265 0.322 0.349Passed GK (D- or higher) 0.199 0.233 0.265 0.282

EnglishEarned B or higher in GK 0.148 0.169 0.218 0.314Earned C or higher in GK 0.093 0.167 0.188 0.257Passed GK (D- or higher) 0.063 0.154 0.165 0.229

Placement Accuracy Rates

• We know who will be placed in dev ed or gatekeeper based on test scores

• We can estimate whether or not given individual is predicted to succeed based on test scores

• Can then assign each person to one of four cells• Placement accuracy rate is sum of bottom left/upper right cells• Can also compare this to accuracy rates without using test at all

Predicted to Succeed in GK

Not Predicted to Succeed in GK

Placed in Dev Ed “False negative”

(Type II error)

Accurately Placed

Placed in GK Accurately Placed “False positive”

(Type I error)

Figure 1Probability of Gatekeeper Success,

By Math Part 2 Scores

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99

Math Part 2 Score

Per

cen

t M

eeti

ng

Su

cces

s C

rite

rio

n

Earned B or Better

False positives

Accurately placed

Acc. placed

False neg.

Figure 1

Probability of Gatekeeper Success, By Math Part 2 Scores

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99

Math Part 2 Score

Per

cen

t M

eeti

ng

Su

cces

s C

rite

rio

n

Passed GK Earned C or Better Earned B or Better

Figure 2 (By Writing Score)

Probability of Gatekeeper Success, By Writing Placement Scores

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

2 3 4 5 6 7 8 9 10 11 12

Writing Placement Scores

Per

cen

t M

eeti

ng

Su

cces

s C

rite

rio

n


Table 2Predicted Placement Accuracy Rates Using Placement Test Scores,

Versus Placing All Students in College Level or Remedial

Accuracy Rate, Accuracy Rate, Accuracy Rate, Incremental IncrementalUsing Placement All Students In All Students In Validity vs. Validity vs.

Test Cutoffs Developmental College Level All Dev Ed All Coll. Lev

MathEarned B or higher in GK 0.701 0.691 0.309 0.010 0.391Earned C or higher in GK 0.585 0.496 0.504 0.089 0.081Passed GK (D- or higher) 0.487 0.348 0.652 0.140 -0.165

EnglishEarned B or higher in GK 0.650 0.650 0.350 0.000 0.301Earned C or higher in GK 0.463 0.379 0.621 0.085 -0.158Passed GK (D- or higher) 0.381 0.275 0.725 0.105 -0.344

Table 2

Predicted Placement Accuracy Rates Using Placement Test Scores,Versus Placing All Students in College Level or Remedial



MathEarned B or higher in GK 0.701 0.691 0.309 0.010 0.391Earned C or higher in GK 0.585 0.496 0.504 0.089 0.081Passed GK (D- or higher) 0.487 0.348 0.652 0.140 -0.165

EnglishEarned B or higher in GK 0.650 0.650 0.350 0.000 0.301Earned C or higher in GK 0.463 0.379 0.621 0.085 -0.158Passed GK (D- or higher) 0.381 0.275 0.725 0.105 -0.344

Predicted Placement Accuracy Rates Using Placement Test Scores, Predicted Placement Accuracy Rates Using Placement Test Scores,Versus Placing All Students in College Level or Remedial, Versus Placing All Students in College Level or Remedial

Sample Restricted to Narrow Range Around Current Cutoffs



MathEarned B or higher in GK 0.643 0.722 0.278 -0.079 0.365Earned C or higher in GK 0.519 0.512 0.488 0.007 0.031Passed GK (D- or higher) 0.423 0.352 0.648 0.071 -0.225

EnglishEarned B or higher in GK 0.600 0.628 0.372 -0.028 0.228Earned C or higher in GK 0.430 0.345 0.655 0.085 -0.226Passed GK (D- or higher) 0.368 0.253 0.747 0.115 -0.379

Table 1. Summary of the Evidence on Placement Accuracy Ratesfor COMPASS® and ACCUPLACER®

Success Criterion: B or Higher Success Criterion: C or Higher

Test Target CourseCorr. Coeff.

Average/ Median

Accuracy Rate

Median Increase in Accuracy

RateCorr. Coeff.

Average/ Median

Accuracy Rate

Median Increase in Accuracy

Rate

COMPASS (Source: ACT Inc., 2006, pp. 103-104)Writing Skills Composition n/a 66 19 n/a 67 2Reading Skills Composition n/a 60 10 n/a 67 2Reading Skills Psychology n/a 68 31 n/a 67 4Numerical Skills/Prealgebra Arithmetic n/a 70 16 n/a 72 4Numerical Skills/Prealgebra Elementary Algebra n/a 67 25 n/a 63 6Algebra Intermediate Algebra n/a 71 25 n/a 68 5Algebra College Algebra n/a 72 43 n/a 67 20

ACCUPLACER (Mattern & Packman, 2009, p. 4)Sentence Skills Composition, Reading 0.19 59 n/a 0.13 75 n/aReading Comprehension Composition, Reading 0.17 62 n/a 0.10 80 n/aArithmetic Basic Math to Precalc 0.29 66 n/a 0.23 84 n/aElementary Algebra Basic Math to Precalc 0.27 65 n/a 0.25 73 n/a

Caveats

• Maximizing placement accuracy rates may not be the goal

• Our computation treats false positives and false negatives equally, but may care more about one than the other

• Values about which type of error is worse can be inferred from where the cutoff is placed– Ex: Pr(passingGK) for math at cutoff is 67%– This means those that are just below cutoff are wrongly placed—

false negatives—67% of the time– Could increase placement accuracy by lowering cutoff– But if we think failing someone in GK is 2x worse than making

someone take developmental unnecessarily, then cutoff is in the right spot

Figure 1

Probability of Gatekeeper Success, By Math Part 2 Scores

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99

Math Part 2 Score

Per

cen

t M

eeti

ng

Su

cces

s C

rite

rio

n


Predictive Validity:Take-Away Messages

• Placement tests are much better at predicting who is likely to do well in gatekeeper than at predicting who is likely to fail

• Placement tests are more predictive of gatekeeper success in math than in english

• High school academic measures are almost as predictive as math test scores, and more predictive than english test scores

• Placement accuracy rates are only modestly higher in some cases, and substantially worse in others, than what would result if no tests were used– But weighting false positives and false negatives differently may

change this conclusion• Analysis of effectiveness of remediation still to come

For more information:

Please visit us on the web at http://ccrc.tc.columbia.edu,where you can download presentations, reports,

CCRC Briefs, and sign-up for news announcements.

CCRC is funded in part by: Alfred P. Sloan foundation, Bill & Melinda Gates Foundation, Lumina Foundation for Education, The Ford Foundation, National Science Foundation (NSF), Institute of Education Sciences of the U.S.

Department of Education

Community College Research CenterInstitute on Education and the Economy, Teachers College, Columbia University

525 West 120th Street, Box 174, New York, NY 10027 E-mail: [email protected]: 212.678.3091

http://ccrc.tc.columbia.edu/

mailto:[email protected]

developmental education assessment, placement, and progression thomas bailey based on research by...

Documents

placement test datafirst

test measure

placement test scores

interpretation of test

current placement test

regress gatekeeper success

underprepared students

majority of students