copyright © 2011 educational testing service. all rights reserved. accessibility issues of...

Copyright © 2011 Educational Testing Service. All rights reserved.

Accessibility Issues of Scenario-Based Computer-Delivered

Mathematics Items for Students with Disabilities and English Language

Learners

Teresa KingEducational Testing Service

National Conference on Student Assessment June 20, 2011

Copyright © 2011 Educational Testing Service. All rights reserved.Copyright © 2011 Educational Testing Service. All rights reserved.

• Maria Martiniello• Cara Laitusis• Lucy Amato • Mike Eckert

• Mike Wagner• Karen Harris• Peggy Redman• Malcolm Bauer• Joni Lakin

2

Acknowledgements


Overview

• What is CBAL?• Purpose of current research study• Interview findings by student group

and assessment features• Validity implications• Introduction of 2011 accessibility

research study

3


CBAL

• Cognitively Based Assessment of, for, and as Learning– Program of research intended to create a model for a balanced K-12

assessment system that:• Documents what students have achieved (“of learning”)• Helps identify how to plan instruction (“for learning”)• Is considered by students and teachers to be a worthwhile educational

experience in and of itself (“as learning”)

– Has conceptual, summative, formative, and professional support components

– Prototype development has focused primarily on middle-school reading, writing, and mathematics

4


Purpose

• Examine the accessibility of the linguistic complexity, the test layout, and the technology features of Grade 8 Cognitively Based Assessment of, for, and as Learning (CBAL) summative mathematics items

5

Copyright © 2011 Educational Testing Service. All rights reserved.Copyright © 2011 Educational Testing Service. All rights reserved.6

An item appeared here that assessed a student’s understanding of the linear relationship between time and distance traveled, as represented in a table. The item featured an online tool that the student could access to illustrate the time/distance relationship, as well as an online calculator, and a numeric-entry response format.


An item appeared here that assessed a student’s understanding of similarity of geometric shapes or figures. The item featured an online protractor and online calculator that could be accessed by the student, and a mixed response format that included both text entry and numeric entry.


Materials – CBAL test

• Sections of three CBAL (PAA) summative Grade 8 Mathematics item sets were selected

• Items were selected to have various linguistic features, simulations and tools

• Linguistic features– E.g., Directions embedded in 1st slide or items

with separate directions screen, lengthy descriptions or shorter sentences

8


Tools and Simulations

• Tools - Calculator, protractor, slider• Simulation - Moving sidewalk • Precise mouse manipulations – clicking on a graph

to draw a line, rotating tiles, dragging tiles, using a protractor

• Response format - enter a # or text into a text box, selecting radio button, select response in a drop down box

9


Mathematics Knowledge• Lines of symmetry• Rotation• Reflection• Concept of similar figures when one figure is an

enlargement of the other or when all the corresponding angles are the same

• Determining speed when the formula is given • Slope

– Find slope from graph– Interpretation of slope– Y-intercept

• Rate of change

10


Students

• Native Spanish-speaking ELL students• Students with a disability that affects

their motor skills• Comparison group of students– Native English speaking students– No learning or physical disabilities

11


Research Questions1. Are CBAL math assessments fair for students who are ELLs

and for students who have fine-motor disabilities?Specifically, which aspects of the test instructions and tasks may include construct-irrelevant factors affecting the performance of these groups.

2. What challenges do ELLs find in comprehending and communicating their knowledge in CBAL summative tasks? How are these challenges different for ELLs than for non-ELLs?

3. How can we increase the accessibility and validity of assessment for ELLs and for students who have fine-motor disabilities regarding test design and administration with accommodations?

12


Procedure

• Introduction/training of the CBAL layout, tools, and demonstration of thinking aloud

• Cognitive Interviews – One-on-one administration– ELL interviews conduced by a trained

bilingual Spanish/English interviewer

13


Cognitive Interview Details

• Students think aloud while reading instructions and questions

• Retrospective questions followed each screen of questions– ELL and SWD specific questions– Generic questions

14


Spanish Interviews• Directions and follow up questions could be given

in English or Spanish• The text of the test items, answer options, and

directly related test questions were only permitted in English

• Thoughts spoken aloud could be in English or Spanish

• Test responses = English only

15


ELL Findings

• All students were verbal, all but one student preferred to speak in Spanish and one student said it was difficult to think aloud and answer

16


ELL Findings

• No technology issues observed– No students tried to use keyboard

shortcuts– No students had issues with making

precise manipulations with the mouse–Mistakes or misconceptions may be

eliminated with more directions or by providing Spanish directions

17


Linguistic Findings

• One ELL did not know the word “tiles”

• Many ELLs did not know the concept of slope or what the word assumption meant

18


SWD Findings

• Extra time was imperative –Students demonstrated perseverance

• Some observed difficulty interacting with a graph with small lines–Students were able to click where

they intended but it required more effort

19


An item appeared here that assessed a student’s understanding of similarity of geometric shapes or figures. The item featured an online protractor and online calculator that could be accessed by the student, and a mixed response format that included both text entry and numeric entry.


Accessibility Findings

• No students tried to use keyboard shortcuts• No major issues with any of the tools or

simulations• Suggestion by some students to modify the slider

tool

• Most response formats were not found to be problematic; students knew how to interact and respond

• Enlarging the text may be helpful

21


Validity Implications

• Extent of the linguistic challenges posed by content assessment items impact ELLs’ interactions with and performance on them – Specific wording issues– ELP correlated with ability to interact

appropriately with the test– Further examine language support options,

simplified text, and changes to the test directions

22


Validity Implications• Extent of the technology features impact

students with motor disabilities impact their interactions and item performance– Students were able to demonstrate what they know;

however they need more time to take the test– Option to enlarge the test material would be helpful– Additional research is underway

23


2011 Current Research• Use the principles of universal design to

make existing items more accessible for students with disabilities and English language learners

• Selected revisions examined for 3 similar student groups– Native Spanish speaking ELLs– Students with learning disabilities– Comparison students

24


• Read aloud of all items and directions

• Spanish translations• Language

simplification• Scribe for extended

responses

• Warnings for incomplete items

• Increased white space• Add/remove graphics• Scratchpad for note-

taking• Access to tool

instructions throughout item

25

Language Modifications

User-InterfaceModifications


Preliminary Accessibility Thoughts• Consult with item developers to ensure the additions or

changes do not affect the construct being measured• Work early and closely with bilingual test developers to

allow proper consideration of linguistic features of test items– Avoid terms that are lengthy to translate or cannot

be translated– Awareness of cognates that may benefit or confuse

ELLs when translated

26


Thank you!

Email: [email protected]

CBAL information: http://www.ets.org/research/topics/cbal/initiative

27