compass math test validity april 2011
DESCRIPTION
Compass Math Test Validity April 2011. Mike Orkin, Jo Ann Phillips, Hui Zhang Office of Institutional Research Peralta Community College District. Compass Math Test. Peralta students take assessment tests for placement into Math, ESL, and English courses. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/1.jpg)
Compass Math Test Validity April 2011
Mike Orkin, Jo Ann Phillips, Hui ZhangOffice of Institutional Research
Peralta Community College District
1
![Page 2: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/2.jpg)
Compass Math Test• Peralta students take assessment tests for
placement into Math, ESL, and English courses. • Course placement based on test scores is
recommended not mandatory. • A student may take courses without taking
placement tests.
2
![Page 3: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/3.jpg)
Compass Math Test • Peralta uses the ACT Compass test for math
placement.• When students take Compass, they advance
through “subtests” until they reach a “placement domain.”
• The higher the placement domain, the higher the difficulty level of the recommended courses.
3
![Page 4: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/4.jpg)
Data• Students taking Compass-designated math courses
in F08, S09, F09, S10. • Took Compass test immediately preceding
semester when they took math course, or• Never took Compass test prior to semester when
they took math course.
4
![Page 5: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/5.jpg)
Data Totals• 12,657 student enrollments. • 4,915 (39%) took Compass test.• 7,742 (61%) did not.
5
![Page 6: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/6.jpg)
Data by Placement Domain Courses
Placement Domain Courses Compass
No Compass Total
% Take Compass
MPRAL 3,749 2,644 6,393 59%
MALGE 755 5,268 6,023 13%
MTRIG 179 643 822 22%
MCOAL 232 3,590 3,822 6%
6
![Page 7: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/7.jpg)
Placement Domains and Key Courses• MPRAL – Prealgebra
Arithmetic – Math 250 (0-32)Pre-Algebra – Math 253 (33-39)Elementary Algebra – Math 201 (40-100)
7
![Page 8: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/8.jpg)
Placement Domains and Key Courses• MALGE – Algebra.
Elementary Algebra – Math 201 (0-34)Intermediate Algebra – Math 203 (35-49)Statistics – Math 13 (50-100)
8
![Page 9: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/9.jpg)
Placement Domains and Key Courses• MTRIG – Trigonometry.
Trigonometry – Math 50 (0-45)Pre Calculus – Math 1 (46-100)
9
![Page 10: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/10.jpg)
Placement Domains and Key Courses• MCOAL – College Algebra.
Statistics – Math 13 (0-34)Pre Calculus – Math 1 (35-49)Calculus – Math 3A (50-100)
10
![Page 11: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/11.jpg)
Compliance Within Placement Domain• Most complied with cutoff score assignment
within their placement domain.
11
![Page 12: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/12.jpg)
Compliance Within Placement Domain
Placement Domain Compliance Total %MPRAL COMPLY 3,264 87% NONCOMPLY 485 13%MPRAL Total 3,749MALGE COMPLY 682 90% NONCOMPLY 73 10%MALGE Total 755MTRIG COMPLY 125 70% NONCOMPLY 54 30%MTRIG Total 179MCOAL COMPLY 169 73% NONCOMPLY 63 27%MCOAL Total 232
Grand Total 4,915
12
![Page 13: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/13.jpg)
Validity• A test is “valid” if it measures what it’s supposed
to measure. • We looked at this in two ways:
I. Are grades and success positively related to Compass scores?
II. Do students who take Compass do “better” than students who don’t take Compass?
13
![Page 14: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/14.jpg)
Validity• Are grades positively related to Compass scores?
Analysis of Variance (ANOVA) determines if there are significant differences in average Compass scores across course grades.
On average, as grades increase, so do scores.
14
![Page 15: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/15.jpg)
Validity ANOVA - Assessment test averages by grade.
Placement Domain A - Test ave B - Test ave C - Test ave D - Test ave F/W- Test ave Sig.
MPRAL 38.49 35.77 34.41 34.97 32.58 0.000
MALGE 48.77 44.73 43.33 41.36 44.51 0.014
MTRIG 39.55 39.03 38.62 37.45 36.56 0.401
MCOAL 71.87 69.6 65.23 64.43 64.77 0.037
ALL 44.12 39.35 37.25 36.54 34.53 0.000
15
![Page 16: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/16.jpg)
Validity Grade Counts
Placement Domain A B C D F W Total
MPRAL 767 569 471 275 504 1,107 3,693
MALGE 239 139 113 47 70 144 752
MTRIG 69 39 26 11 9 25 179
MCOAL 127 43 22 7 10 21 230
ALL 1,202 790 632 340 593 1,297 4,854
16
![Page 17: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/17.jpg)
Validity• Do assessment test scores predict a student’s
chance of success (C or better) in a course?
Logistic regression estimates the probability of success for a given assessment test score.
For all subtests, as scores get higher, the predicted chance of success increases, indicating predictive validity.
17
![Page 18: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/18.jpg)
Validity
18
![Page 19: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/19.jpg)
Validity
19
![Page 20: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/20.jpg)
Validity
20
![Page 21: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/21.jpg)
Validity
21
![Page 22: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/22.jpg)
Validity
22
![Page 23: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/23.jpg)
Perfect Success• We have shown that the chance of success
increases as Compass scores increase.• However, if the assessment process works
perfectly, there will be an appropriate course for every student, even those who get low Compass scores, and so everyone (or almost everyone) will succeed, regardless of their Compass score.
23
![Page 24: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/24.jpg)
Perfect Success• If everyone succeeds, the chance of success will be
100% no matter what the Compass score, and so the regression analysis will not be meaningful.
• In the same way, if there is an appropriate course for everyone, and if everyone succeeds, there will be no low grades, and, more generally, there won’t be a relationship between Compass scores and grades.
24
![Page 25: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/25.jpg)
Perfect Success• In other words, looking for a relationship between
Compass scores and success can be clouded by too much success.
• It thus becomes useful to compare those who take the Compass test and those who don’t.
25
![Page 26: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/26.jpg)
Compass v. no Compass• Do students who take the Compass test do better
than students who don’t take the test?
Assume that those who don’t take the Compass test essentially select a class at random.
26
![Page 27: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/27.jpg)
Compass v. no Compass• For non-Compass students, just by chance, some
strong students will take lowest-level courses (too easy) and some weak students will take higher level courses (too difficult).
• If Compass is properly assessing students, only weaker students will take lowest level courses and stronger students will take higher level courses.
27
![Page 28: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/28.jpg)
Compass v. no Compass• Thus, students who take Compass should do worse
than those who don’t at lowest subtest and better than those who don’t at higher levels.
• This is exactly what we observe in the data.
28
![Page 29: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/29.jpg)
Compass v. no CompassChi Square - Success – Compass Test v. No Test
Placement Domain Courses
Take Test % Success
No Test % Success Sig.
MPRAL 48.2% 50.9% 0.020
MALGE 65.0% 57.5% 0.000
MTRIG 74.9% 56.9% 0.000
MCOAL 82.8% 60.8% 0.000
29
![Page 30: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/30.jpg)
Compass v. no Compass• Similar result when comparing grades of students
who take Compass with those who don’t.
30
![Page 31: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/31.jpg)
Compass v. no Compass
ANOVA - Test v. No Test Course GPA
Placement Domain Courses Test GPA No Test GPA Sig.
MPRAL 2.3 2.6 0.000
MALGE 2.7 2.6 0.245
MTRIG 3.0 2.8 0.083
MCOAL 3.3 2.8 0.000
31
![Page 32: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/32.jpg)
Summary• All four methods of analysis yield statistically
significant results supporting Compass validity.
32
![Page 33: Compass Math Test Validity April 2011](https://reader036.vdocuments.site/reader036/viewer/2022081517/568163ee550346895dd56259/html5/thumbnails/33.jpg)
Summary• On average, as grades increase, so do Compass scores.• As Compass scores increase, the predicted chance of
success increases. • Students who take Compass do better than those
who don’t in the higher subtests and worse in the lowest subtest, in both course success and grades.
33