jamal abedi university of california, davis/cresst validity, effectiveness and feasibility of...
TRANSCRIPT
Jamal AbediJamal AbediUniversity of California, Davis/CRESSTUniversity of California, Davis/CRESST
Validity, Effectiveness and Validity, Effectiveness and Feasibility of Feasibility of
Accommodations for Accommodations for English Language English Language
Learners With Disabilities Learners With Disabilities
(ELLWD)(ELLWD)
Accommodations Accommodations for ELLWDfor ELLWD
The process of provision of accommodations The process of provision of accommodations to ELLWD is more complex than the process to ELLWD is more complex than the process used for either SD or ELL students.used for either SD or ELL students.Care must be taken to choose Care must be taken to choose accommodations that are appropriate for this accommodations that are appropriate for this particular subgroup of students. particular subgroup of students. ELLWDs need accommodation addressing ELLWDs need accommodation addressing both their language needs and their both their language needs and their disabilities.disabilities.
Why Should English Why Should English Language Learners Be Language Learners Be
Accommodated?Accommodated?Their possible English-language Their possible English-language deficiency may interfere with their deficiency may interfere with their content knowledge performance.content knowledge performance.Assessment tools may be Assessment tools may be culturally and linguistically biased culturally and linguistically biased for these students.for these students.Linguistic complexity of the Linguistic complexity of the assessment tools may be a source assessment tools may be a source of measurement error.of measurement error.Language factors may be a source Language factors may be a source of construct irrelevant variance.of construct irrelevant variance.
Why Should Students Why Should Students With Disabilities be With Disabilities be
AccommodatedAccommodated??
Their disabilities Their disabilities put them at put them at disadvantage.disadvantage.
Accommodations Accommodations must be provided must be provided to level the to level the playing field.playing field.
Subgroup Reading Math Language Spelling
ELL Status
ELL
Mean 26.3 34.6 32.3 28.5
SD 15.2 15.2 16.6 16.7
N 62,273 64,153 62,559 64,359
Non-ELL
Mean 51.7 52.0 55.2 51.6
SD 19.5 20.7 20.9 20.0
N 244,847 245,838 243,199 246,818
SES
Low SES
Mean 34.3 38.1 38.9 36.3
SD 18.9 17.1 19.8 20.0
N 92,302 94,054 92,221 94,505
Higher SES
Mean 48.2 49.4 51.7 47.6
SD 21.8 21.6 22.6 22.0
N 307,931 310,684 306,176 312,321
Site 2 Grade 7 SAT 9 Subsection Scores
Reading Math Math Calculation
Math Analytical
Non-ELL/SWD
Mean 45.63 49.30 49.09 48.75
SD 21.10 20.47 20.78 19.61
N 9217 91.18 9846 92.50
ELL only
Mean 20.26 36.00 39.20 33.86
SD 16.39 18.48 21.25 16.88
N 692 687 696 699
SWD only
Mean 18.86 27.82 28.42 29.10
SD 19.70 14.10 15.76 15.14
N 872 843 883 873
ELL/SWD
Mean 9.78 21.37 22.75 22.87
SD 11.50 10.75 12.94 12.06
N 93 92 97 94
Site 4 Grade 8 Descriptive Statistics for the SAT 9 Test Scores by Strands
ReadingReading Science Science Math Math MM SDSD MM SDSD MM SDSD
Grade 10Grade 10SWD onlySWD only 16.416.4 12.712.7 25.525.5 13.313.3 22.522.5 11.711.7LEP onlyLEP only 24.024.0 16.416.4 32.932.9 15.315.3 36.836.8 16.016.0LEP & SWDLEP & SWD 16.316.3 11.211.2 24.824.8 9.3 9.3 23.623.6 9.8 9.8Non-LEP/SWD 38.0Non-LEP/SWD 38.0 16.016.0 42.642.6 17.217.2 39.639.6 16.916.9All studentsAll students 36.036.0 16.916.9 41.341.3 17.517.5 38.538.5 17.017.0
Grade 11Grade 11SWD OnlySWD Only 14.914.9 13.213.2 21.521.5 12.312.3 24.324.3 13.213.2LEP OnlyLEP Only 22.522.5 16.116.1 28.428.4 14.414.4 45.545.5 18.218.2LEP & SWDLEP & SWD 15.515.5 12.712.7 26.126.1 20.120.1 25.125.1 13.013.0Non-LEP/SWD 38.4Non-LEP/SWD 38.4 18.318.3 39.639.6 18.818.8 45.245.2 21.121.1All StudentsAll Students 36.236.2 19.019.0 38.238.2 18.918.9 44.044.0 21.221.2
Normal Curve Equivalent Means & Standard Deviations for Students in Grades 10 and 11, Site
3 School District
Accommodations for Accommodations for ELLs and SWDsELLs and SWDs
Can the same accommodations Can the same accommodations used for students with used for students with disabilities be used for ELLs?disabilities be used for ELLs?Can the same accommodations Can the same accommodations used for ELLs be used for used for ELLs be used for students with disabilities?students with disabilities?
How Are We Doing in How Are We Doing in Practice Nationally?Practice Nationally?
• Are the states and districts across the Are the states and districts across the nation cognizant of this important nation cognizant of this important principle of using accommodations that principle of using accommodations that are appropriate for a particular are appropriate for a particular subgroup?subgroup?
• Are there any objective national criteria Are there any objective national criteria to help states to select appropriate to help states to select appropriate accommodations for ELLWD students? accommodations for ELLWD students?
• Or, is the assignment of Or, is the assignment of accommodations to these students based accommodations to these students based on temporary and subjective decisions?on temporary and subjective decisions?
SD LEP Accommodation
l l 1. Extension of allotted time l l 2. Use of multiple shortened test periods l l 3. Simplification of directions l l 4. Reading of questions (Math and Science only) l 5. Translation of words and phrases on the spot l l 6. Decoding of words upon request l 7. Use of calculator l l 8. Gestures and nonverbal expression l 9. Allow students to mark answers in test booklet l 10. Allow students to point to response l 11. Allow student to answer orally l 12. Allow student to respond on audio tape l 13. Use of typewriter or personal computer l l 14. Use of graphic organizers and art work l 15. Use of large-print or magnification device l l 16. Testing in separate room l l 17. Small group setting l l 18. Use of study carrel
Accommodation in Stanford 9 Testing Used by a District
SY 2000-2001 Accommodations Designated for ELLs Cited in
States’ Policies
There are 73 accommodations listed:
N: Not Related
R: Remotely Related
M: Moderately Related
H: Highly Related
From: Rivera (2003) State assessment policies for English language learners. Presented at the 2003 Large-Scale Assessment Conference
N 1. Test time increased
N 2. Breaks provided
N 3. Test schedule extended
N 4. Subtests flexibly scheduled
N 5. Test administered at time of day most beneficial to test taker
N = not related; R = remotely related; M = moderately related; H = highly related
I. Timing/Scheduling (N = 5)
SY 2000-2001 Accommodations Designated for ELLs Cited in
States’ Policies
N 1. Test individually administered
N 2. Test administered in small group
N 3. Test administered in location with minimal distraction
N 4. Test administered in familiar room
N 5. Test taker in separate location (or carrel)
N 6. Test administered in ESL/Bilingual classroom
N 7. Individual administration provided outside school (home, hospital, institution, etc.)
N 8. Test taker provided preferential seating
N 9. Increased or decreased opportunity for movement provided
II. Setting (N = 17)
N 10. Teacher faces test taker
N 11. Special/appropriate lighting provided
N 12. Adaptive or special furniture provided
N 13. Adaptive pencils provided
N 14. Adaptive keyboards provided
N 15. Person familiar with test taker administers test
N 16. ESL/bilingual teacher administers test
N 17. Additional one-to-one support provided during test administration in general education classroom (e.g. instructional assistant, special test administrator, LEP staff, etc.)
SY 2000-2001 Accommodations Designated for ELLs Cited in States’
Policies
N = not related; R = remotely related; M = moderately related; H = highly related
R 1. Directions repeated in English
R 2. Directions read aloud
R 3. Audio-taped directions provided in English
N 4. Key words or phrases highlighted
M 5. Directions simplified
M 6. Audio-taped directions provided in native language
M 7. Directions translated into native language
N 8. Cues provided to help test taker remain on task
M 9. Directions explained/clarified in English
III. Presentation (N = 32)
M 10. Directions explained/clarified in native language
M 11. Both oral and written directions in English provided
M 12. Both oral and written directions in native language provided
M 13. Test items read aloud in English
H 14. Test items read aloud in simplified/sheltered English
N 15. Audio-taped test items provided in English
H 16. Test items read aloud in native language
H 17. Audio-taped test items provided in native language
SY 2000-2001 Accommodations Designated for ELLs Cited in States’
Policies
N = not related; R = remotely related; M = moderately related; H = highly related
N 18. Assistive listening devices, amplifications, noise buffers, appropriate acoustics provided
N 19. Key words and phrases in test highlighted
H 20. Words on test clarified (e.g. words defined, explained)
H 21. Bilingual word lists, customized dictionaries (word-to-word translations) provided
N 22. Enlarged print, magnifying equipment, Braille provided
N 23. Memory aids, fact charts, list of formulas and/or research sheets provided
N 24. Templates, masks or markers provided
III. Presentation (N = 32)
N 25. Cues (e.g. arrows and stop signs) provided on answer form
N 26. Acetate shield for page provided
N 27. Colored stickers or highlighters for visual cues provided
R 28. Augmentive communication systems or strategies provided (e.g. letter boards, picture communication devices, voice output systems, electronic devices)
H 29. Simplified/sheltered English version of test provided
H 30. Side-by-side bilingual versions of test provided
H 31. Translated version of the test provided
N 32. Test interpreted for the deaf or hearing impaired/use of sign language provided
SY 2000-2001 Accommodations Designated for ELLs Cited in States’
Policies
N = not related; R = remotely related; M = moderately related; H = highly related
N 1. Test taker marks answers in test booklet
N 2. Test administrator transfers test-taker’s answers
N 3. Test taker’s transferred responses checked for accurate marking
N 4. Copying assistance provided between drafts
N 5. Test taker types or uses a machine to respond (e.g. typewriter/word processor/computer)
N 6. Test taker indicates answers by pointing or other method
N 7. Papers secured to work area with tape/magnets
N 8. Mounting systems, slant boards, easels provided to change position of paper, alter test taker’s position
N. 9. Physical assistance provided
IV. Response (N = 17)
N 10. Enlarged answer sheets provided
R 11. Alternative writing systems provided (including portable writing devices, computers and voice-activated technology)
R 12. Test taker verifies understanding of directions
R 13. Test taker dictates or uses a scribe to respond in English
N 14. Test taker responds on audio tape in English
H 15. Oral response in native language translated into English
H 16. Written response in native language translated into English
H 17. Spelling assistance, spelling dictionaries, spell/grammar checker provided
SY 2000-2001 Accommodations Designated for ELLs Cited in States’
Policies
N = not related; R = remotely related; M = moderately related; H = highly related
N 1. Out-of-level testing provided
N 2. Special test preparation provided
V. Other (N = 2)
Preliminary Findings: State Assessment Policies for English Language Learners, SY 2000-2001
GW/CEEE, Large-Scale Assessment Conference 2003
SY 2000-2001 Accommodations Designated for ELLs Cited in
States’ Policies
N = not related; R = remotely related; M = moderately related; H = highly related
There are 73 Accommodations Listed
47 or 64% are not related
7 or 10% are remotely related
8 or 11% are moderately related
11 or 15% are highly related
The most important issue is the The most important issue is the concern over concern over
the validity of accommodation the validity of accommodation strategies:strategies:
Research findings suggest that providing Research findings suggest that providing accommodations may increase accommodations may increase performance of ELLs/SDs, while also performance of ELLs/SDs, while also benefiting non-ELLs/SDs.benefiting non-ELLs/SDs.There is not enough research support for There is not enough research support for many of the accommodations currently many of the accommodations currently being used in the national and state being used in the national and state assessments.assessments.The only way to make judgments about the The only way to make judgments about the efficiency and validity of accommodations efficiency and validity of accommodations used by states is to use them in used by states is to use them in experimentally controlled studies with both experimentally controlled studies with both ELL/SWD and non-ELL/SWD students. ELL/SWD and non-ELL/SWD students.
Some forms of accommodation strategies, such as the use of a glossary with extra time, raised the performance of both ELL and non-ELL students (Abedi, Hofstetter, Lord, and Baker, 1998, 2000)
ELL students’ performance increased by 13% when they were tested under glossary with extra time accommodation.
While this looks promising, it does not present the entire picture.
Non-ELL students also benefited from this accommodation, with an increase of 16%.
English and bilingual dictionary recipients may be advantaged over those without access to dictionaries. This may jeopardize the validity of assessment.
Concern over the validity of accommodation strategies: Concern over the validity of accommodation strategies: A research exampleA research example
• Linguistic modification of test items is among these accommodations.
• This accommodation also helped students with learning disabilities.
• Thus, an accommodation may have the potential to be effective and valid for both LEP and SD, consequently relevant for LEPWD.
There are, however, some accommodations that help ELL students with their English language needs without compromising the validity of assessment.
Concerns over the validity of accommodation strategies:Concerns over the validity of accommodation strategies:A research findingA research finding
Strategies that are expensive, impractical, or logistically complicated are unlikely to be widely accepted.
Validity: The goal of accommodations is to level the playing field for ELL/SWD, not to alter the construct under measurement.
Consequently, if an accommodation affects the performance of non-ELL/SWD, the validity of the accommodation could be questionable.
Feasibility: For an accommodation strategy to be useful, its implementation must be possible in large-scale assessments.
Concern over the validity of accommodation Concern over the validity of accommodation strategiesstrategies
How validity of How validity of accommodations can be accommodations can be
examined?examined?Only through experimentally controlled research where:
ELL/SWD and non-ELL/SWD students are randomly assigned to experimental and control groups
Both ELL/SWD and non-ELL/SWD students are observed under accommodated and non-accommodated assessments
Can existing data (from national Can existing data (from national and state assessments) be used for and state assessments) be used for examining the effectiveness and examining the effectiveness and validity of accommodations?validity of accommodations?
Effectiveness of accommodations can be Effectiveness of accommodations can be examined only if ELL/SWD students are examined only if ELL/SWD students are randomly assigned to the randomly assigned to the accommodated and non-accommodated accommodated and non-accommodated conditionsconditions
Validity of accommodations can be Validity of accommodations can be examined only if non-ELL/SWD students examined only if non-ELL/SWD students are randomly assigned to the are randomly assigned to the accommodated and non-accommodated accommodated and non-accommodated conditionsconditions
How the validity of How the validity of accommodations can be accommodations can be
examined?examined?
Using existing data?Using existing data?
Through experimentally Through experimentally controlled field study?controlled field study?
Grade 4 Grade 8 Grade 12
Accommodation Number of LEP students
Weighted percentage of assessed LEP
Number of LEP students
Weighted percentage of assessed LEP
Number of LEP students
Weighted percentage of assessed LEP
Total number of assessed LEP students
829 100 651 100 532 1000
Without accommodation
709 85 561 88 474 91
With accommodation 120 15 90 12 58 9 Primary Accommodations
Large print 0 0 0 0 0 0 Extended time 45 5 41 5 32 5 Read aloud 10 2 10 1 1 0 Bilingual Dictionary 5 1 14 2 8 1 Small group 56 7 22 3 15 2 One-on-one 2 0 2 0 0 0 Scribe or computer 0 0 1 0 1 0 Other 2 0 0 0 1 0
LEP Students Assessed With and Without Accommodations, 1998 NAEP Writing Assessment: National Sample, Public and Nonpublic Schools Combined
1 Source: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Writing Assessment
LEP Students Assessed With and Without Accommodations, 1998 NAEP Civic Assessment:
National Sample, Public and Nonpublic Schools Combined
Grade 4 Grade 8 Grade 12
Accommodation
Number of
LEP
students
Weighted
percentage
of assessed
LEP
Number
of LEP
students
Weighted
percentage
of assessed
LEP
Number
of LEP
students
Weighted
percentage
of assessed
LEP
Total number of assessed
LEP students 272 100 216 100 220 100
Without accommodation 240 89 192 90 201 92
With accommodation 32 11 24 10 19 8
Primary
Accommodations
Large print 0 0 0 0 0 0
Extended time 12 4 12 6 8 3
Read aloud 2 1 3 1 0 0
Bilingual dictionary 0 0 1 2 1
Small group 13 5 8 3 7 3
One-on-one 4 2 0 0 2 1
Scribe or computer 0 0 0 0 0 0
Other 1 0 0 0 0
Research findingsResearch findings
• Pitoniak, M., Lutkus, A., Pitoniak, M., Lutkus, A., Cahalan-Laitusis, C., Cook, L. & Cahalan-Laitusis, C., Cook, L. & Abedi, J. Abedi, J. (2005). Are Inclusion Policies and Practices for State Assessment (2005). Are Inclusion Policies and Practices for State Assessment Systems and NAEP State Assessments Aligned?Systems and NAEP State Assessments Aligned?
• Sireci, S. G., Li, S. & Scarpati, S. (2003). Sireci, S. G., Li, S. & Scarpati, S. (2003). The effects of test The effects of test accommodation on test performance: A review of the literatureaccommodation on test performance: A review of the literature (Center (Center for Educational Assessment Research Report No. 485). Amherst: for Educational Assessment Research Report No. 485). Amherst: University of Massachusetts. University of Massachusetts.
• Thompson, S., Blount, A., Thurlow, M. (2002). Thompson, S., Blount, A., Thurlow, M. (2002). A summary of research A summary of research on the effects of test accommodations: 1999 through 2001on the effects of test accommodations: 1999 through 2001 (Technical (Technical Report 34). Minneapolis: University of Minnesota, National Center on Report 34). Minneapolis: University of Minnesota, National Center on Educational Outcomes.Educational Outcomes.
• Thurlow, M. L., McGrew, S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., Thurlow, M. L., McGrew, S., Tindal, G., Thompson, S. J., Ysseldyke, J. E., & Elliott, J. L. (2000). & Elliott, J. L. (2000). Assessment accommodations research: Assessment accommodations research: Considerations for design and analysisConsiderations for design and analysis ( (NCEO NCEO Technical Report 26). Technical Report 26). Minneapolis: University of Minnesota, National Center on Educational Minneapolis: University of Minnesota, National Center on Educational Outcomes.Outcomes.
• Tindal, G., & Fuchs, L. (2000). Tindal, G., & Fuchs, L. (2000). A summary of research on test changes: A summary of research on test changes: An empirical basis for defining accommodations.An empirical basis for defining accommodations. Lexington, KY: Lexington, KY: University of Kentucky, Mid-South Regional Resource Center.University of Kentucky, Mid-South Regional Resource Center.
• Abedi, J., Hofstetter, C., & Lord, C. (2004). Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment Assessment accommodations for English language learners: Implications for accommodations for English language learners: Implications for policy-based empirical research. policy-based empirical research. Review of Educational Research, Review of Educational Research, 7474(1), 1-28(1), 1-28
How the validity of How the validity of accommodations can be accommodations can be
tested in an tested in an experimentally controlled experimentally controlled
condition?condition? LEP/SWD LEP/SWD Status/ Status/ AccommodationAccommodation
AccommodateAccommodatedd
Non-Non-AccommodationAccommodation
LEP/SWDLEP/SWD
Non-LEP/SWDNon-LEP/SWD
Conclusions and Conclusions and RecommendationRecommendation
Accommodations:Must be relevant to the subgroups of students Must be effective in reducing the performance gap between accommodated and non-accommodated studentsMust be valid, that is, accommodations should not alter the construct being measuredThe results could be combined with the assessments under standard conditionsMust be feasible in the national and state assessments
Conclusion
There is not enough research support for many of the accommodations that are currently used in national and state assessments.
The only way to make judgments about the efficiency and validity of these accommodations is to use them in experimentally controlled situations with both ELL/SWD and non-ELL /SWD students and examine their validity and effectiveness under a solid experimental condition.
The results of CRESST studies along with other studies nationwide have provided support for some of the accommodations used for ELL students.
Conclusion cont.Conclusion cont.
Providing a customized dictionary is a Providing a customized dictionary is a viable alternative to providing traditional viable alternative to providing traditional dictionaries. dictionaries. The linguistic modification of test items The linguistic modification of test items that reduce unnecessary linguistic that reduce unnecessary linguistic burdens on students is among the burdens on students is among the accommodations that help ELL students accommodations that help ELL students without affecting the validity of without affecting the validity of assessments.assessments.Computer testing with added extra time Computer testing with added extra time and glossary was shown to be a very and glossary was shown to be a very effective, yet valid accommodation (Abedi, effective, yet valid accommodation (Abedi, Courtney, Leon and Goldberg, 2003) Courtney, Leon and Goldberg, 2003)
Examples of research-supported accommodations:
Conclusion cont.
Without information on important aspects of accommodations such as validity, it would be extremely difficult to make an informed decision on what accommodation to use and how to report the accommodated and non-accommodated results.
It is thus imperative to examine different forms of accommodations before using them in state and/or national assessments.
For more information, For more information, please contact Jamal please contact Jamal
Abedi at:Abedi at:(530) 754-9150(530) 754-9150
or or
[email protected]@ucdavis.edu