A New Generation of Assessments Three Things You Need to Know

October 30, 2013

Sean McGrew, Ph.D.

Poll 1: What is this?

A: A tool for chopping wood

B: A murder weapon

Three Things to Know

1. Everything depends on use of assessments

2. Assessment distinctions have fuzzy borders

3. Computer-based testing is (finally) changing the practice

Validity in the Context of Use

“Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests. Validity is, therefore, the most fundamental consideration in developing and evaluating tests. The process of validation involves accumulating evidence to provide a sound scientic basis for the proposed score interpretations. It is the interpretations of test scores required by proposed uses that are evaluated, not the test itself.” AERA/APA/NCME Standards for Educational and Psychological Testing, 1999.

Reliability

Reliability “refers to the consistency of . . . measurements when the testing procedure is repeated on a population of individuals or groups” (AERA, 1999)

Poll 2: What’s the most important attribute?

What’s the most important attribute of an educational assessment?

Utility

Validity

Reliability

It depends

Fuzzy Borders

Formative vs. Summative

Norm-Referenced vs. Criterion-Referenced

Interim/Benchmark/Formative

Test/Assessment

Poll 3: OK: Of a formative or diagnostic?

What’s the most important attribute of an diagnostic of formative assessment used to diagnose skill deficits and/or inform instruction?

Utility

Validity

Reliability

Poll 4: Of a high-stakes test?

What’s the most important attribute of a high-stakes test used to rank or rate students?

Utility

Validity

Reliability

Poll 5: Reporting Categories Important to You

Poll 6: Scores

Poll 7: Logistics

Computer-Based Coming of Age

PROS

Easy to edit/update

Rapid scoring and reporting

Possibility of multimedia content

Possibility of adaptive testing

CONS

Hardware requirements

Often need internet access

Assumes comfort level with computers

Harder to ensure same experience for all students

Less proven metrics for reliability (if get away from multi-choice)

Not all Computer-Based tests are Adaptive

Adaptive Tests

Basic idea goes back to Binet’s 1905 intelligence tests

First done by computers in 1970s and 1980s

Widespread by late 1990s

PROS

Shorter

More accurate for wider range of students

CONS

Harder to know ‘what’s on the test’

Harder to review item and standard results for a group of students

Common Core Test Consortia

Smarter Balanced: Computer-Based ADAPTIVE

PARCC: Computer-Based, NOT ADAPTIVE

Next-Generation Reporting

Focus on Longitudinal Analysis

Teachers expected to incorporate results provided online

GROWTH

INTERIM TEST REPORTING TOOL Catapult Learning’s Common Core

21

22

23

Grade 3 Assessment Blueprint

24

Standard Section

Number

of Items

2.OA.A.1: Use addition and subtraction within 100 to solve

one- and two-step word problems. Pre-req 4

2.OA.C.4: Use addition to find the total number of objects

arranged in rectangular arrays with up to 5 rows and 5

columns. Pre-req 4

2.OA.B.2: Fluently add and subtract within 20 using mental

strategies Pre-req 4

3.OA.A.2: Interpret whole-number quotients of whole

numbers Unit 1: A,B 4

3.OA.A.3: Use multiplication and division within 100 to solve

word problems in situations involving equal groups, arrays,

and measurement quantities. Unit 1: A,B 4

3.OA.C.7: Fluently multiply and divide within 100 Unit 1: A,B 8

Total 28 items

To teach, or re-teach?

To teach, or else re-teach, that is the question:

Whether ’tis Nobler in the mind to suffer

The Slings and Arrows of outraged Bosses,

Or to take Arms against student weaknesses,

And by opposing end them: Re-teach, press on

No more; and by Re-teach, to say we end

The Heart-ache, and the thousand Natural shocks

Common Core enforces. ’Tis a consummation

Devoutly to be wished. Re-teach, press on,

Press on, perchance to fail; Aye, there’s the rub.

Adapted from William Shakespeare, Hamlet III.i

Connect with Us!

Stay connected with our latest updates, blog posts, upcoming webinars, and more.

Catapult Learning on Twitter @catapultlearn

Dr. Sean McGrew on Twitter @SeanPMGrew

Share your successes! If you put our methods into action, let us know about it. Use the hashtag #catapulting

26