Adding Value to Accommodations Decision-Making (AVAD)

2006 Enhanced Assessment Instruments Grant Project

Elizabeth Jones and Christopher Webster, South Carolina Department of Education

2006 Enhanced Assessment Instruments Grant Award

The South Carolina Department of Education received a 2006 Enhanced Assessment Instruments Grant award to undertake the AVAD project.

Overall goal of AVAD:

To ensure the valid, effective use of accommodations designed to increase access to large-scale assessments for English language learners and students with disabilities

AVAD Project Partners

South Carolina Department of Education, lead state and fiscal agent

LEP SCASS ASES SCASS Michigan State University: Rebecca Kopriva University of Oregon: Gerald Tindal University of Nebraska/Psychometric

Inquiries: Barbara Plake Phoebe Winter, project evaluator

AVAD Project Period

Current project period from October 1, 2006 to March 31, 2008

One-year no-cost extension will be requested SCASS grant activities will begin in

spring/summer 2007 SCASS grant activities will be scheduled

during or immediately before/after regularly scheduled SCASS meetings

AVAD’s Objectives:

1) to validate and enhance the Selection Taxonomy for English Language Learner Accommodations (STELLA) and publish the program on the SDE’s Web site

2) to create and validate a decision-making taxonomy for the Accommodation Station (AS) based on the ASES Accommodations Manual and publish the program on the SDE’s Web site

AVAD’s Objectives, cont.:

3) to develop 20 prototypes of computer-based access enhanced science items for grade four, and

4) to disseminate results through reports, the SDE’s Web site, User and Technical Manuals for STELLA and AS, the ASES Professional Development Guide video, and Web-Ex teleconferences

What role does the LEP SCASS play? LEP SCASS will serve as expert panel to provide input and

feedback on STELLA and computer-based access-enhanced prototype items

Opportunity to participate in a subgroup of states within the SCASS focusing on accommodations issues

Accommodations subgroup will commit to participation in AVAD meetings/conference calls

Updates on AVAD will be provided to the larger SCASS group

(ASES SCASS will play a similar role in Accommodation Station design and development)

What is STELLA?

STELLA=Selection Taxonomy for Testing English Language Learners an online system to assist educators in

assigning testing accommodations to English language learners (ELL) so that individual students receive appropriate accommodations

developed by a 2003 Enhanced Assessment Instruments grant

Data Collection Teacher Form Parent Form Records Form

Preloaded in System Student Information Variables Test Accommodations Conversion and Consolidation Rules Decision-Making Rules

Output Student Profile Accommodation Decisions for Each Student

Associated Materials Data Collection Manual Output Interpretation Manual Technical Manual

STELLA Overview

Current Accommodations loaded in STELLA system

Pre-Test Best Practice Accommodations Family Assessment Night Tailored Classroom Support

Forms Standard or some Universal Design Forms Access-based Form L1 or Side-by-Side forms as available

Tools Bilingual word list, general or test specific Picture-word dictionary Problem solving tools

Current Accommodations loaded in STELLA system

Administration Small group Individual Administration Oral English Oral L1 Language Liaison Extra time More frequent breaks

Response Written L1 or code switching Oral English Oral L1 or code switching Demonstrated or modeled response

STELLA Screenshots

What does STELLA look like?

Records Form

Language of instruction

English language proficiency test

L1 proficiency test

Records Form

Type of ELL program

L1 Proficiency in four domains

Parent/Guardian Form

Attendance in U.S. schools

Schooling in native country

Classroomequipment, books, and supplies in nativecountry

Types of assessmentsin native country

Grading and assignments

Test scores and experiences with testing

English and L1 proficiency (L1 includes ‘don’t know’ option)

Standardized score accuracy and judgments about reasons for inaccuracy

Student’s experience with standard test formats

Student’s perceived purpose of standardized testing

Classroom test condition options

Condition options that help student on classroom tests, evaluations

Teacher Form

Experience with standardized testing

Testing conditions used in the classroom

Beneficial testingconditions in the classroom

System Output

Individual Student Profile• English language proficiency, 4 domains • L1 proficiency, 4 domains • Cultural proximity, 3 variables:

1. Time in U.S.2. Experience with testing procedures3. Schooling proximity, native country to U.S. experience

System Output: STELLA, Teacher Recommended and SEA/LEA Allowed

Since different states allow different accommodations, STELLA program output is set up to identify the subsetof allowable accommodations for each student.

Non allowable accommodations selected for each student based on research and other best practice literature are also displayed in gray. This provides guidance to teachers and others about additional accommodations that could be useful for ELL students with specified needs.

Test Conversion Rules

Currently, output from 4 English language proficiency tests (including ELDA) is preloaded. There is also the opportunity for personnel to add results from another test.

In all cases score conversion rules place students on a common scale with four levels: Beginner ELL Low intermediate ELL High intermediate ELL Grade-level-competitive ELL

Student Information ConsolidationRules

In several cases, more than one piece of student information is used to make judgments about his or her level on relevant variables. Two types of consolidation rules are part of STELLA. These are:

Consolidation of data from related items Consolidation of information from more than one

source

Time in School = LOW

If the student has been in the US less than 1 academic year.OR

If the student has been in US between 1 and 2 years AND has missed more than two months of school per year for 1 or more years.

ORIf the student has been in US between 2 and 3 years AND has

missed more than two months of school per year for more than one year.

Combining Data from More Than One Question: An Example

Combining Data from More Than One Source: An Example

Consolidation rules are also used to determine L1 proficiency and English language proficiency.

For L1 proficiency, consolidation rules are applied to information from teacher, parent and records forms.

Parents describe student as performing above grade level in L1

Teacher describes student as performing below grade level in L1

Parents trump teacher’s knowledge of L1, but test scores of L1 proficiency trump parents.

What Trumps What? Consolidation Rules Example

Test Accommodation Decision-Making Rules

A beginning set of decision-making rules were developed and tested. These rules take relevant student information and pair it with relevant accommodation factors for individual students. The accommodation factors identify which student needs the specific accommodation was designed to remediate.

In this way, individual students are matched with accommodations appropriate to their particular needs.

LEP SCASS Validation and Enhancement of STELLA Activities

The LEP SCASS accommodations sub-group will validate existing accommodations decision-making rules by participating in a two-meeting and online expert panel process led by Barbara Plake.

Profiles of ELL students will be matched with specific sets of accommodations; accommodations for individual student profiles will be rated for validity and effectiveness; results will be compared to existing STELLA accommodations decision-making rules.

STELLA decision-making structure will be tweaked based on results of expert panel process.

STELLA interface and forms will be reviewed and revised.

Computer-based access-enhanced (CAE) science items review

CAE items aim to reduce barriers to access that English language learners may encounter in typical large-scale assessment items without altering the construct being measured or the rigor of the items

South Carolina will develop 20 prototypes of computer-based access-enhanced science items for grade four and place them online

LEP SCASS will review the CAE items and provide feedback

Benefits of Participation

LEP SCASS member states will have input into the design and development of STELLA

LEP SCASS member states (and any other interested parties) will have free access to the STELLA once it is completed and placed on the SDE Web site

LEP SCASS member states will be able to adapt the STELLA to best fit the policy and needs of their states

ASES SCASS: Parallel Activities The ASES SCASS will serve as an expert panel for the

development and review of the Accommodation Station (AS), an online decision-making model designed to assist IEP teams in assigning valid and appropriate accommodations to individual students with disabilities.

Current AS collects and consolidates information but does not include a decision-making algorithm.

The ASES manual and materials will serve as the basis for an algorithm that will be built and incorporated into the current AS.

A professional development video component will be developed to accompany existing ASES materials.

Sharon Hall of ASES will share additional details.

Questions?

Please contact Therese Gleason Carr at tcarr@ed.sc.gov or (803) 734-3747

Thank you very much for your collaboration!