considerations for transitioning students from operational ... · considerations for transitioning...

National Center on Educational Outcomes (NCEO)

Considerations for Transitioning Students from Operational 2% Assessments to General

Assessments: Challenges for States

Moderator: Martha Thurlow (National Center on Educational Outcomes)

Presenters: Sheryl Lazarus (National Center on Educational Outcomes) Wendy Stoica (Ohio Department of Education) Trinell Bowman (Maryland Department of Education) Susan Weigert (U.S. Department of Education, Office of

Special Education Programs)

Discussant: Marianne Perie (CETE, University of Kansas)

National Conference on Student Assessment National Harbor, MD

June 20, 2013

Lessons Learned Chapter Authors • Parker, Gorin, and Bechard • Perie, Fincher, Payne, and Swaffield • Elliott, Kettler, Zigmond, & Kurz • Lazarus & Thurlow** • Elliott, Rodriguez, Roach, Beddow, Kettler, and Kurz • Cohen, Danielson, Stoica, Wothke, and Zhang** • Bechard • Bowman** • Dean and Roberts • Loving-Ryder and Siler • Zigmond, Kloo, Lemons & Lupp • Nagle and Cameto


.

• Projects funded by the U.S. Department of Education in 2006-2007 under three funding sources (General Supervision Enhancement Grants, Enhanced Assessment Grants, and Supplemental Funding) compiled their findings in chapters that discussed the lessons learned from their studies on alternate assessments based on modified achievement standards (AA-MAS).

• This presentation highlights lessons from three chapters, plus perspectives from OSEP

Lessons Learned in Federally Funded Projects That Can Improve the Instruction and Assessment of Low Performing Students with

Disabilities (Thurlow, M., Lazarus, S., & Bechard, S. (Eds.))

Available at: www.nceo.info


Sheryl Lazarus

.

The Characteristics of Low Performing Students with Disabilities

Sheryl Lazarus

National Center on Educational Outcomes

(NCEO)

Email: [email protected]

Demographic Characteristics of Low Performing Students with Disabilities

• 4 projects looked at demographic characteristics of low performing students. More likely to be: – Male – From a racial/ethnic minority – From a low socio-economic background – Have ELL status

• Both students with and without disabilities were low performing.


Disability Categories • 4 projects looked at the disabilities categories of students

who might be candidates for an AA-MAS. • All found that a majority of the students had specific

learning disabilities. • Other categories:

o Intellectual disabilities o Speech and language impairments o Other health impairments o Emotional/behavioral disabilities o Autism o Other disability categories


Opportunity to Learn

4 projects looked at whether students who were candidates for the AA-MAS had the opportunity to learn the content. All found that some students may not have had

the opportunity to learn.


Learning Characteristics/Barriers 10 projects found one or more of these learning characteristics/barriers:

• Difficulty interacting with print • Difficulty solving problems that require multi-step solutions • Easily distracted • High vocabulary load • Lack of availability of needed accommodations • Limited meta cognition • Need clarification of instructions • Poor organizational skills • Self-monitoring skills • Slower work pace • Text structure (passage length and formatting) • Working memory capacity


Wendy Stoica


AA-MAS Test Development: Item Modifications

Wendy Stoica June 20, 2013

Research Questions 1. Do Target and GenEd group students

differ on the cognitive variables? 2. Are the cognitive variables accessible via

teacher judgment?

Item Modifications Fall 2008 Modifications of Reading Items

Bold/Underlined Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students.

Boxed

Questions are interspersed within passages and offset with a box.

Simplified Language Language is simplified beyond that which is typical for universal design.

Thought Questions Additional questions designed to focus students’ attention on particular aspects of a passage are interspersed with the passage. These questions are not answered but serve to help students focus attention.

Modifications of Mathematics Items

Bold/Underlined (1 level) Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students.

Modified Graphics Graphics may have been enlarged or simplified in order to increase readability.

Relevant Pictures Information in the stem of the item is given as a picture or table to help students organize and understand the information necessary to answer the question.

Scaffolding Complex items are decomposed into simpler parts.

Simplified Language Language is simplified beyond that which is typical for universal design.

Spring 2009 Modifications of Reading Items Bold or Underlined Important elements of the reading passage are bolded or underlined. This will facilitate

structured recall of the content passage by AA-MAS students.

Primed Items

A specially designed priming item is presented immediately before a test item. The priming item assists the students’ memory of the cognitive process so that they will more readily see the solution to the test item.

Cued Items Additional items break up complex questions into a series of simpler steps to reduce the planning load.

Graphic Organizers Graphic organizers provide a graphical structure to help students organize their thoughts.

Passage Primed Thought questions are introduced before a reading passage to help the students engage in the content of the passage.

Modifications of Mathematics Items Bold or Underlined Important elements of the problem are bolded or underlined. This will facilitate structured recall

of the content passage by AA-MAS students.

Primed Items

A specially designed priming item is presented immediately before a test item. The priming item assists the students’ memory of the cognitive process so that they will more readily see the solution to the test item.

Cued Items Additional items break up complex questions into a series of simpler steps to reduce the planning load.

Relevant Pictures Added icons help students visualize the problem at hand.

Results of interaction of item modifications with cognitive traits

for reading Working Memory Executive Function/

Planning Focused Attention Sustained Attention

Bold

Increases Influence Decreases Influence

Cue

Increases Influence

Graphic Organizers Decreases Influence

Prime

Passage Prime

Presenter

Presentation Notes

In general, test developers should be cautious utilizing item modification universally for students with disabilities since some modifications actually increase the difficulty for some students while they benefit other students.

Results of interaction of item modifications with cognitive traits

for mathematics

Working Memory Executive Function/ Planning

Focused Attention Sustained Attention

Bold

Increases Influence Increases Influence

Cue

Decreases Influence

Decreases Influence Increases Influence Increases Influence

Graphic Organizers

Decreases Influence

Increases Influence

Prime

Decreases Influence

More Research Questions 1. Do the revised items improve access? 2. Does performance exceed chance? 3. Do the revisions improve performance and

is the discrepancy reduced between the Target and GenEd group?

4. How do the revisions relate to the student’s cognitive traits?

Presenter

Presentation Notes

1. We assessed whether the modifications improved access by comparing whether Target students are less likely to omit responses to revised items than to base items. This analysis compares the percentage of parent items omitted by students in the Target group with the percentage of revised items the same students omitted. This within-students analysis calculates the percentage of each type of items omitted by each student and evaluates the difference. The attempted rates of the Revised items and the Parent items (unmodifi ed) approached 98%. There was no difference between the two types of items. This was likely because of the extremely high (almost ceiling) rates of attempting items in the two studies. 2. If the modifi cations effectively remove barriers to responding to the items, the Target students should not only respond to the revised items but respond meaningfully. They should have a reasonable probability of getting them correct—at minimum, above chance. The form of this analysis follows the form of the previous analysis. The results of both studies are consistent. Specifi cally, the GenEd group completed over 90% of the revised reading items above chance and over 60% of the mathematics items above chance. The Target Group completed about half the number of items above chance as the GenEd group. Importantly, the modifi cations did not increase chance performance. 3. bolding appears to increase the disparity between GenEd and the Target group for mathematics. There was no influence of Primed, Cued, Scaffolding, or Simplified Language. 4. Overall, the results indicate that working memory, executive functioning, and sustained attention all relate signifi cantly to both the parent and modifi ed items of the reading assessment. In addition, working memory, executive functioning, focused attention, and sustained attention all relate signifi cantly to both the parent and modifi ed items of the mathematics assessment. This result, in itself, validates the assessment of these four cognitive traits and reveals their importance in large scale assessment. The goal of the current analysis is to determine how these cognitive traits relate individually to the item modifications. That is, do the modifications reduce their influence or enhance their influence. It is the hope that we can create modifications that reduce the influence of these cognitive traits.

Conclusions/Recommendations 1. 17% of general education students not identified as having

disabilities actually had significant deficits in sustained attention and 11% of the general education students had significant deficits in executive functioning.

2. Students often had deficits only on one, or perhaps two, cognitive traits.

3. Question the ability of classroom teachers to identify those students with disabilities who have specific cognitive processing deficits which has implications for how appropriate data might be collected to inform IEP team decision making.

4. IEP teams require additional training to help them understand that the indiscriminate use of accommodations may actually make tests less accessible for some students with disabilities.

education.ohio.gov

Social Media

@OHEducation

ohio-department-of-education

Ohio Families and Education Ohio Teachers’ Homeroom

OhioEdDept

storify.com/ohioEdDept


Trinell Bowman

Maryland’s Alternate Assessment Based on Modified Achievement Standards (AA-MAS)

What did we learn?

2013 National Conference on Student Assessment

National Harbor, Maryland June 2013

Trinell Bowman Maryland State Department of Education

22

Presenter

Presentation Notes

Maryland administered the first AA-MAS online and paper in May 2008 for: Algebra/Data Analysis, English 2, Biology, and Government (included for graduation purposes only)‏ Maryland started with the High School end-of-course tests, which are both NCLB and graduation requirements Maryland developed and administered the AA-MAS for the Maryland School Assessment (MSA) Mathematics and Reading for grades 3-8, but 2012 was the last year of the Mod-MSAs in grades 3-8. The high school modified assessments will continue until the state transitions to the PARCC assessments.

Lessons Learned from the Development of the AA-MAS • Cognitive Labs were critical to the development process

and for the first time, students were asked to describe the “why” and “how” as they answered test questions.

• Online testing offers more standardization of certain

accommodations such as the use of the audio test and calculator.

• Online testing may offer additional support to students, which they normally may not receive when taking a paper and pencil test. Examples include: eliminating a choice, highlighting critical information or marking a test question for review.

23

Presenter

Presentation Notes

24

How does the student’s disability affect progress in the general curriculum? The student has difficulty with the following:

Percentage

Comprehending written materials 51.1%

Difficulty following multiple steps 51.1%

Overwhelmed with many sentences/words on a page

48.8%

Difficulty completing written tasks 44.1%

Maintaining time on tasks 39.5%

How are methods of instruction different for this student compared to special education students taking the regular assessments?

Percentage

Requires repetition of directions 62.7%

Extended time 58.1%

Scaffolding questions 55.8%

Slower pace of instruction 53.5%

Decreased level of content complexity 46.5%

Increased wait time 44.2%

More visual cues 32.5%

Greater use of assistive technology 30.2% 25

Lessons Learned from the Monitoring Process

1. Special educators must be included in professional development training to fully understand state standards.

2. All teachers must understand how to write and implement standards based IEPs.

3. All students must receive appropriate research-based

and/or evidenced-based interventions in order to address any gaps or deficit documented in an IEP.

Presenter

Presentation Notes

Curriculum framework guides should reflect differentiated instructional strategies for students with disabilities. IEP Teams must ensure that students received accommodations during instruction and assessment in order for students with disabilities to access the general education curriculum. As result of the state monitoring process, the MSDE developed Standard-based IEP Professional Development Training, Module and Facilitator Guide. The student continues to provide ongoing Technical Assistance to local school systems.

Contact Information

Trinell Bowman Program Manager

Division of Curriculum and Assessment (410) 767-2498

[email protected]

27

mailto:[email protected]


Susan Weigert

Perspectives of U.S. ED on Transition challenges and opportunities:

Students currently eligible for an AA-MAS

Transitioning to Improved Assessments

• AA-MAS Flexibility to states in 2007

• Students seen as unable to attain grade level proficiency

• Era of accountability reflected in NCLB State

percent proficient with add-on flexibilities.

• We can do better

Presenter

Presentation Notes

While the AA-MAS was conceived of as a flexibility for states unfortunately it represented just that—flexibility for states, but no real payoff for students. It promised simply a way to entrench a small segment of students in a world of lower standards—we felt we could do better for these students, because they can attain grade level standards with the right supports. With the move to common standards, we can create new education technologies to improve the efficiency of learning.

Changes in Accountability

• Permitted States to incorporate student growth • For students performing below grade level this

changed what was required of an assessment— • Features we asked for became part of the vision

for next generation assessment. • Moved from a simply easier test • To an assessment system more sensitive to lower

levels of performance, and capable of measuring progress toward grade level proficiency.

Presenter

Presentation Notes

From the AA-MAS we learned about student access profiles and the kind of supports that level the playing field for students with cognitive processing challenges. We can use student access profiles to ensure improved sensitivity to the performance ranges of students performing below grade level.

Improved Design of Test Items • AA-MAS research identified a key problem in

accessibility of items: non-construct cognitive load.

• Evidence Centered Design has led to improvement in measurement.

• Item accessibility features that do not undermine construct validity

• Accommodations that are linked to student access profiles.

Presenter

Presentation Notes

While the standards are higher and the challenge for low performing swds even greater, the new tests promise to be more accessible through ECD—containing less construct-irrelevant variance, and will be more sensitive to progress toward grade level proficiency.

Instructionally-Embedded Assessment

Next Generation Formative assessment Offer better evaluation of student’s problem-solving approaches Feedback to guide the student correctly along the instructional path. Instructors can get a detailed analysis not just of whether the student reached the final answer correctly, but how they solved the problem.

Presenter

Presentation Notes

From the studies on the 2% population we learned that opportunity to learn was a significant drawback for students—in response we have a better solution than lowering standards--We can embed formative assessment within targeted instruction to move students closer to grade level proficiency –we can use formative assessment to monitor the progress of low performing students with disabilities toward and against grade level achievement standards first by understanding more fully where they are on that trajectory. We can use formative assessments to help students keep pace toward grade level proficiency and also help their teachers ensure they have appropriate instructional resources to do so.

Focus on Continual Progress

Assessments sensitive to student performance range and growth toward standards. Provide students and their teachers with a far better understanding of where they are, where they need to go next Will Provide empirical evidence of how far they can actually go with targeted instruction.


Marianne Perie

considerations for transitioning students from operational ... · considerations for transitioning...

Documents