copyright © 2015 by educational testing service. computer-based testing: advances in k12 and...

Copyright © 2015 by Educational Testing Service.

Presented at the Association on Higher Education and Disability 2015 conference

Computer-Based Testing: Advances in K12 and Implications for Higher

Education

Elizabeth StoneCara Laitusis

Educational Testing Service


Overview

• Operational use of accommodations– Overview of accommodations typically offered in

K12– Newer or developing accommodations– Overview of two major K12 test consortia

• Research on accommodations– Resources on research to validate accommodations

in different contexts– Upcoming predictive validity research study


3

Standard accommodations (brief overview)

• Timing and scheduling (length of time)- extended time [not breaks]

• Presentation (changes to method or format)- read aloud, ASL, braille

• Setting- separate room: attention disorder, distracting accommodation [used with other accommodations]

• Response (alternative ways to respond to test items)- scribe, answer in test booklet, electronic braille notetaker, calculator


4

Variations on delivery example (read aloud or audio accommodation)

• Read aloud by human proctor• Recorded human delivery (on CBT, video,

MP3…)• Text to speech (screen readers, CBT)

Different modes of audio delivery require different conditions (e.g., separate room) and raise different questions about score comparability (e.g., standardization of delivery).


5

Newer accommodations

• Masking tools• Strike-out tool• Pop-up glossaries (English, translated)• Translation of test into multiple

languages (including ASL)• Refreshable braille• On-demand embossing


• ASL vs. Signed Exact English vs. Pidgin Sign

• Video vs. Avatar • Construct Definition– Directions only–Math only– Text in math but not numbers– ELA but not reading comprehension

Signed Presentation of Items

6


Source: https://www.youtube.com/watch?v=lpOqI7daiBI 7

Example of ASL ItemSmarter Balanced


Refreshable braille• Hardware that translates text to braille– braille text is displayed through matrix of pins

that can be individually raised to form letters and words

– Can include braille input keys, speech synthesizer for text to speech

• Most students currently use refreshable braille via braille notetakers

• Some students have refreshable braille devices for computers or tablets


9


Challenges for Delivery

• Math requires braille embosser for Nemeth code and tactile figure supplements

• Embossers are noisy and require separate room and additional staff to transport braille to student (plus possible security issues)

• Increased item development (unique twins) for items that can not be delivered with refreshable braille and embossers

• Increased challenges for new item types


11

Some future directions for accommodations

• Haptics: provide tactile feedback about visuals (e.g., through vibration)

• Sonification: can use different pitches to describe quantity, density, or proximity

• On-demand 3D printing of manipulatives


12

To give or not to give accommodations…

• Want test to be accessible (reduce barriers that prevent students from being able to demonstrate proficiency); eliminate construct-irrelevant variance

• Still want to measure the same construct so that scores are comparable and meaningful (validity)

• Accommodation policies reflect testing program approaches to balancing these goals, and policies differ


13

Some criteria for choosing accommodations

• Effectiveness: effective in making an assessment more accessible

• Validity: should not alter the focal construct, i.e., the outcomes of accommodated and non-accommodated assessments should be comparable.

• Differential Impact: should be sensitive to student’s background characteristics, and their academic standing

• Relevance: should be appropriate for the recipients • Feasibility: must be logistically feasible to implement in the

assessment

• Abedi, J. (2012). Validity issues in designing accommodations. In: Fulcher, G. and Davidson, F. The Routledge Handbook of Language Testing in a Nutshell. Florence, KY: Routledge, Taylor & Francis Group.


14

Partnership for Assessment of Readiness for College and Careers (PARCC)

• Linear assessment (multiple forms per grade)• Computer delivered (except for disability or

infrastructure challenges)• Increased use of technology enhanced items• Includes elements of both Universal Design

and Evidence Centered Design• Platform compliant with Accessible Portable

Item Profile (APIP) and Web Content Accessibility Guidelines (WCAG) 2.0


15

PARCC Accessibility

• Driven by Personal Needs Profile (prepared by IEP coordinator)

• Features for All Students: answer masking, color contrast, highlight, flag for review

• Accessibility Features (pre-identified): refreshable braille, extended time

• Accommodations: calculator on non-calculator math sections, read aloud or ASL on ELA


16

Smarter Balanced

• Computer adaptive• On grade level Off grade level after 2/3 of test

(if needed)• Increased use of technology enhanced items • Design elements refer to Universal Design,

Access by Design, and Evidence Centered Design

• Can be delivered on SmarterApp open-source platform or other vendor platforms following accessibility standards (e.g., WCAG 2.0)


17

Smarter Balanced Accessibility

• Universal Tools: highlighter, writing tools, zoom, mark for review, breaks, customized English glossaries

• Designated Supports (pre-identified): masking, scribe, color contrast, color overlay

• Accommodations (IEP/504): braille, ASL


18

Accessibility challenges

• New technology enhanced items– Highly visual, difficult to describe in

words– Require fine motor movements, –May increase demand on working

memory • Require production of alternative formats

on demand for adaptive tests


19


• Difficulty level of items is increasing– Removal of modified assessment (2%

test)– CCSS are more demanding than many

previous state standards• Increased language requirements for English

Language Arts– Complex text– Authentic text


20


• Are practice materials available and accessible?

• How well does assistive technology (AT) work with browsers displaying varying content types?

• If third-party AT is used, what are the security (and validity: standardization) implications?

• In making policy, do we know who actually uses accommodations and how?


21

Resources and Research


22

Resources for research on disability and accommodation

• Smarter Balanced report: Literature review of testing accommodations and accessibility tools for students with disabilities– Audio presentation for mathematics– Audio presentation for English language arts (ELA)– Refreshable braille for ELA, mathematics– American Sign Language for mathematics– Calculators for mathematics– Writing tools for ELA (spellcheck, grammar check, word

processor, speech to text, word prediction, text to speech for proofreading)

– http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/08/Smarter-Balanced-Students-with-Disabilities-Literature-Review.pdf

http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/08/Smarter-Balanced-Students-with-Disabilities-Literature-Review.pdf




23


• ETS Research Reports:– E. Stone, L. Cook, & C. Laitusis. (2013). Evaluation of a

condition-adaptive test of reading comprehension for students with reading-based learning disabilities.

– H. Buzick & E. Stone (2011). Recommendations for conducting differential item functioning (DIF) analyses for students with disabilities based on previous DIF studies.

– E. Stone & T. Davey (2011). Computer-adaptive testing for students with disabilities: A review of the literature.

– http://search.ets.org/researcher/

http://search.ets.org/researcher/

http://search.ets.org/researcher/


24


• ETS R&D Connections:– M. Hakkinen (2015). Assistive technologies

for computer-based assessments– E. Stone & L. Cook (2009). Leveling the field

on math and science tests for students with learning disabilities

• http://

www.ets.org/research/policy_research_reports/r_and_d_connections

http://www.ets.org/research/policy_research_reports/r_and_d_connections




25


• NCEO accommodation bibliography: Search by accommodations, grade level, content, disability, and specialized terminology– http://

www.cehd.umn.edu/NCEO/OnlinePubs/AccommBibliography/AccomStudies.htm

http://www.cehd.umn.edu/NCEO/OnlinePubs/AccommBibliography/AccomStudies.htm




26

Upcoming study

• Evaluating validity of text-to-speech (TTS) accommodation on the PARCC English Language Arts grade 11 assessment: Fall 2015

• Will recruit institutes of higher education (IHE) who will help us identify students with disabilities who are typically provided the TTS accommodation and a matching (#) sample of students without disabilities


27

Upcoming study

• Participant characteristics:– 18 yrs old +– Native English-speaking students or English

language learning students who have an advanced level of English who are U.S. Citizens or permanent residents

– Graduated from HS in 2015 from one of the following states: AR, CO, DC, IL, LA, MD, MA, NJ, NM, NY, OH, or RI (other students will be considered, but these states are a priority)


28

Upcoming study

• Students will be randomly assigned to take PARCC 11 End of Year and Performance Based Assessments with or without a TTS accommodation; students will be compensated ($150)

• IHE will provide grades, other academic information (upon student consent); IHE will be compensated ($100 per participant)


29

Upcoming study

• Identify, send study invitation to freshmen who are typically eligible to receive a text-to-speech (TTS) or audio accommodation and matching number of freshmen who do not receive TTS.

• Send ETS transcript information (courses enrolled/dropped, class grades) for all consenting study participants. Send course information so ETS can identify comparable courses between institutions.

• Work with ETS, Pearson to coordinate sessions at the school (e.g., find locations, assign study times, provide headphones to participants).


30

Upcoming study

Predictive validity goal:Evaluate the relationship between PARCC 11 ELA and grades in entry-level, credit-bearing courses focusing on reading


31

Upcoming study

Evaluate the appropriateness of the accommodation for this assessment for the target group from the perspective of the differential boost hypothesis

– Do students without disabilities have significantly higher scores with TTS?

– Do students with disabilities have significantly higher scores with TTS?

– Is the gain for students with disabilities significantly larger? (disability * accommodation interaction)


32

Implications for institutions of higher education

• Tests are designed to measure standards aligned with readiness for college and career

• Students take the test under varying conditions, and policies differ about which test changes affect score comparability

• IHE evaluating these scores need to understand how to interpret them for the purposes of admission or placement


33

Contact with questions, comments, or interest in validity study

• Any of the above: Elizabeth Stone ([email protected])

• Validity study participation information: Teresa King ([email protected])

Thank you!

mailto:[email protected]

mailto:[email protected]

copyright © 2015 by educational testing service. computer-based testing: advances in k12 and...

Documents