SAT Suite of Assessments:Recommended Performance Levels and Cut

ScoresMarch 2020 Board Meeting

Joyce ZurkowskiChief Assessment Officer

1

Review the process for establishing recommended cut scores for SAT Math and Evidence-Based Reading and Writing.

Present the recommended cut scores and performance levels for PSAT/SAT Math and Evidence-Based Reading and Writing.

Action Requested:Board adoption of performance levels and cut scores for PSAT/SAT Math and Evidence-Based Reading and Writing based on the intended-to-be assessed Colorado Academic Standards.

Today’s Goals

2

SAT Background:Revised Version starting in 2016

• Revised SAT was designed to:

• Be a measure of student achievement of high school academic expectations

• Emphasize• Knowledge, skills and understandings identified as most important for

college and career readiness and success• Meaning of words in context (eliminated analogies)

• Have simplified scoring – one point for each correct answer

• Be more transparent

• Be accessible to all students and measure performance influenced by successful completion of challenging coursework

• Supported by free content-based practice and refresher materials• Allow English learners to test with accessibility features, including

extended time and word-to-word glossaries

3

SAT Background:Current College Board Reporting

Types of Scores:

1. Scale scores• Total: 400-1600 • Section: 200-800

• Evidence-based Reading and Writing (ERW)• Math

2. College and Career Readiness Benchmarks: 75% likelihood of getting a C or higher in a first semester, credit-bearing college course• ERW: 480• Math: 530

4

Student Score Report

5

Standard Setting:Establishing Performance Levels

• Standard setting buckets scale scores into performance levels

• Cut scores and their associated performance levels anchor scale scores to actual achievement of specific skills and concepts being measured

• Performance levels:• Provide additional achievement information• Support the interpretation of results

• Translate scale scores into a metric that may have a more practical meaning

• Can be used in accountability metrics: ex. growth to standard

6

CMAS Cut Score Setting Process

7

A Content Based Cut Score Setting Approach

Colorado Academic Standards

Assessments

Performance Standards(cut scores)

Purpose of standard setting: To make a series of recommendations about the locations of the cut scores on the Evidence-Based Reading and Writing (ERW) and Math sections of the SAT

8

Policy-level Performance Level Descriptors

9

Level 1Did Not Yet Meet

Expectations

Level 2ApproachedExpectations

Level 3Met

Expectations

Level 4Exceeded

ExpectationsStudents performing at this level may minimally approach the academic expectations for the knowledge, skills and practices known to be most relevant for success in college and careers contained in the assessed high school CAS.

With significant additional coursework, students may engage successfully in entry-level, credit-bearing courses.

Students performing at this level approach the academic expectations for the knowledge, skills and practices known to be most relevant for success in college and careers contained in the assessed high school CAS.

With continued coursework, students are likely to engage successfully in entry-level, credit-bearing courses.

Students performing at this level meet the academic expectations for the knowledge, skills and practices known to be most relevant for success in college and careers contained in the assessed high school CAS.

Students are likely to engage successfully in entry-level, credit-bearing courses.

Students performing at this level exceed academic expectations for the knowledge, skills and practices known to be most relevant for success in college and careers contained in the assessed high school CAS.

Students are very likely to engage successfully in entry-level, credit-bearing courses.

11th Grade Assessment: Which Standards Should Be Addressed?

• State Assessment Content• Concepts and skills students have had opportunity to learn• Concepts and skills known to be most relevant for success in college and

careers assessable content• Some content is measured validly within other components of the

assessment system • Securely held knowledge from prior grades

• High School Math Standards• Not grade specific• Developed with the end in mind: college and career readiness

• Local Control over Scope and Sequence• Two pathways• Multiple course sequences

10

Traditional Pathway Courses: Algebra I, Geometry, Algebra II, Advanced Math and StatisticsIntegrated Math Pathway Courses: Math I, Math II, Math III, Advanced Math and Statistics

Pre-HS

Securely Held Knowledge• Creating, interpreting, and solving multiple representations of linear equations,

both with and without context• Utilizing statistical data to draw conclusions• Utilizing the coordinate plane to show transformations of geometric objects• Using the Pythagorean Theorem with and without context

Sequence 1 Sequence 2 Sequence 3 Integrated

Year 1 Algebra I Algebra I Algebra IAMath I

(excl. Geom)

Year 2 Geometry Algebra II Algebra IBMath II

(excl. Geom)

Year 3 Algebra II Geometry Algebra II Math III

Year 4Statistics

Advanced MathStatistics

Advanced MathGeometry

StatisticsAdvanced Math

• 35 Educator Panelists:• Experts in the concepts and skills reflected in the Colorado

Academic Standards (CAS) for math and Reading, Writing and Communicating

• Experts in student development of the high school concepts and skills

• Knowledgeable about range of characteristics in the population tested

• Interested in the results of the standard setting process and consequences associated with the results

• Selected through open recruitment process by CDE.

Who participated in the SAT standard setting meetings?

12

Summary of Panelists-1

13

English Language Arts Math Grand Total

Content Expert 18 17 35

Teacher 13 16 29

District 7 7 14

Students with Disabilities/ English Learners 7 6 13

Higher Ed 4* 5* 7*

*Higher Ed panelists also work at the district and school levels

Summary of Panelists-2

14

Content AreaDenver-Metro

Urban-Suburban

RuralSmallRural

Total

English Language Arts 3 10 2 3 18

Math 3 9 3 2 17

Total 6 19 5 5 35

Standard Setting:Methodology

The Bookmark Method: one of the most widely used content-based procedures to set cut scores for large-scale state assessments

• Uses an online Ordered Item Booklet (OIB)• Focuses on threshold students: assumes students who have been instructed in

the relevant skills and concepts, using locally determined methods and curriculum

• Identifies the cut score associated with the “just barely” threshold level of a given performance level

• Level 1 - lowest• Level 2• Level 3• Level 4 - highest

Above the “bar”(on track toward college and career readiness)

15

Standard Setting:Bookmark Method

16

80

2221201918171615

1413121110

9876

54321

OrderedItemSet

. . . . . .

. .EasiestItem

Most Difficult Item

What students should know

and be able to do

Bookmark Placement

. . . . .

. . .

Focus on Threshold Students

Threshold Students

Level 1

Recommended Performance Standards

Level 4

(Exceeded?)

Level 2

(Approached?)

Low Achievement High Achievement

Level 3

(Met?)

17

Panels’ SAT Recommendations

Math

18

Recommendations: Math

Evaluation Committee

Recommendation

CDE Recommendation

CorrespondingScale Score Scale Score

Level 4 Cut Score 660 660

Level 3 Cut Score 530 530

Level 2 Cut Score 460 460

19

Panelist Support of Cut Score Recommendations:Math

No Response (Panelist did not answer question)

Do Not Support

Support with Some

Reservation

Moderately Support

Strongly Support

Level 4 Cut 0 0 0 2 15

Level 3 Cut 0 0 1 4 12

Level 2 Cut 0 0 0 3 14

20

Way Too Low A Bit Low Appropriate A Bit High Way Too High

Level 4 Cut 0 0 15 2 0

Level 3 Cut 0 0 11 6 0

Level 2 Cut 0 3 13 1 0

Math: Percent of Students in each Performance Level Based on CDE Cut Score Recommendations

36.3

24.3

30.0

9.4

0.010.020.030.040.050.060.070.080.090.0

100.0

Total

Percent of CO Students at Each Performance Level:Math 2018

Level 1 Level 2 Level 3 Level 4

21

SAT Suite of Assessments: MathRecommended Performance Levels and Cut Scores

22

Math

Did Not Yet Meet

Expectations (DNYM)

Approached Expectations

MetExpectations

ExceededExpectations

Met/Exceeded Expectations

SAT Scale Scores 200 – 450 460 – 520 530 – 650 660 – 800 530 – 800

SAT Percentage2018 36.3% 24.3% 30.0% 9.4% 39.4%

MathDNYM/

Partially Met Expectations

Approached Expectations

MetExpectations

ExceededExpectations

Met/Exceeded Expectations

SAT Percentage2019 37.2% 23.8% 29.8% 9.2% 39.0%

CMAS Grade 8Percentage 2019

39.9%23.3% 30.7% 6.2% 36.9%

20.2% 19.7%

SAT Suite of Assessments: MathRecommended Performance Levels and Cut Scores

23

Math

Level 1Did Not Yet

Meet Expectations

Level 2Approached Expectations

Level 3Met

Expectations

Level 4Exceeded

Expectations

SAT 200 – 450 460 – 520 530 – 650 660 – 800

PSAT 10 160 – 420 430 – 470 480 – 580 590 – 760

PSAT 9 120 – 400 410 – 440 450 – 550 560 – 720

Adopt the recommended Colorado math performance levels and cut scores for the SAT Suite of Assessments based on the intended-to-be assessed Colorado Academic Standards..

Requested Action

24

Panels’ SAT Recommendations

Evidence-Based Reading & Writing

25

Recommendations: Evidence-Based Reading & Writing

Evaluation Committee

Recommendation

CDE Recommendation

Scale Score Scale Score

Level 4 Cut Score 640 640

Level 3 Cut Score 490 480

Level 2 Cut Score 450 440

26

Panelist Support of Cut Score Recommendations:Evidence-Based Reading & Writing

No Response (Panelist did not answer question)

Do Not Support

Support with Some

Reservation

Moderately Support

Strongly Support

Level 4 Cut 1 1 2 8 6

Level 3 Cut 1 2 8 5 2

Level 2 Cut 1 3 4 5 5

27

Way Too Low A Bit Low Appropriate A Bit High Way Too High

Level 4 Cut 0 0 12 3 3

Level 3 Cut 0 0 5 12 1

Level 2 Cut 0 0 9 6 3

Evidence-Based Reading & Writing: Percent of Students in each Performance Level based on CDE Cut Score Recommendations

26.7

13.4

45.9

14.0

0.010.020.030.040.050.060.070.080.090.0

100.0

Total

Percent of CO Students at Each Performance Level: Evidence-Based Reading & Writing

Level 1 Level 2 Level 3 Level 4

28

SAT Suite of Assessments: Evidence-based Reading and WritingRecommended Performance Levels and Cut Scores

29

ERW

Did Not Yet Meet

Expectations (DNYM)

Approached Expectations

MetExpectations

ExceededExpectations

Met or Exceeded

Expectations

SATScale Scores 200 – 430 440 – 470 480 – 630 640 – 800 480 – 800

SAT Percentage2018 26.7% 13.4% 45.9% 14.0% 59.9%

SAT Suite of Assessments: English Language ArtsRecommended Performance Levels and Cut Scores

30

Level 1Did Not Yet

Meet Expectations

Level 2Approached Expectations

Level 3Met

Expectations

Level 4Exceeded

Expectations

SAT 200 – 430 440 – 470 480 – 630 640 – 800

PSAT 10 160 – 380 390 – 420 430 – 590 600 – 760

PSAT 9 120 – 360 370– 400 410 – 560 570 – 720

Adopt the recommended Colorado English language arts (evidence-based reading and writing) performance levels and cut scores for the SAT Suite of Assessments based on the intended-to-be assessed Colorado Academic Standards.

Requested Action

31