1 ideia, sld, rti, and cognitive assessment jack a. naglieri, ph.d. professor of psychology center...

1

IDEIA, SLD, RTI, and IDEIA, SLD, RTI, and Cognitive AssessmentCognitive Assessment

Jack A. Naglieri, Ph.D.Professor of Psychology

Center for Cognitive Development School Psychology Program Director

George Mason UniversityFairfax, VA 22030

http://[email protected]

Jack A. Naglieri, Ph.D. George Mason Univ, Fairfax, VA 22030. [email protected] 2

IDEA ReauthorizationIDEA Reauthorization

Topical outline IDEIA and forthcoming regulations

• A look at the Law

Response to intervention• Local comparison groups• Measurement of improvement• Research and reviews

A cognitive approach to assessment• Connecting LD definition with assessment of

“basic psychological processes”



Topical outline IDEIA and forthcoming regulations

• A look at the Law

Response to intervention• Local comparison groups• Measurement of improvement• Research and reviews




IDEIA 2004 LawIDEIA 2004 Law

Individuals with DisabilitiesEducation Improvement Act of 2004



The IDEIA and the No Child Left Behind Act have encouraged a reexamination of how school psychologists function

There have been many meetings of researchers, practitioners, and professional organizations and articles written which discuss how to improve the system, especially for LD diagnosis



Reexamination of the role of the school psychologists is

good for the evolution of the field how to improve diagnosis of Specific

Learning Disabilities (SLD) is particularly important

how to reduce over-representation of minority children in special education is also needed

7

IDEIAIDEIA

What are some of the details of the new Law?



RTI may be used AS A PART of the evaluation… but not as sole method

IQ achievement discrepancy no longer required


IDEIA 2004 LawIDEIA 2004 Law“use a variety of assessment tools”

“not use any single procedure”

“assess cognitive factors”



non discriminatory assessments

valid and reliable assessment


IDEIA Law SummaryIDEIA Law Summary Ability achievement discrepancy is no

longer required (not disallowed) A variety of assessment tools required The use of any single measure or

assessment as the sole criterion for determining SLD is not permitted

Assessments must not be discriminatory on racial or cultural basis

Definition of SLD remains


IDEIA 2004 LawIDEIA 2004 LawDefinition of SLD

remains the same


SLD Definition in Both BillsSLD Definition in Both Bills

The definition of SLD has not changed “The term ‘specific learning disability’

means a disorder in one or more of the basic psychological processes involved in understanding or in using language, spoken or written, which disorder may manifest itself in the imperfect ability to listen, think, speak, read, write, spell, or do mathematical calculations.”

14

Reactions to IDEIAReactions to IDEIA

Anticipation of the Regulations that follow…


Align the definition of SLD with the

classification criteria

Replace discrepancy with pattern of strengths and

weaknesses and cognitive abilities

impacting achievement

19

SLD & Cognitive Processing SLD & Cognitive Processing

Connecting the dots


Hale, Naglieri, Kaufman, & Kavale Hale, Naglieri, Kaufman, & Kavale (2004)(2004)


Hale, Naglieri, Kaufman, & Kavale Hale, Naglieri, Kaufman, & Kavale (2004)(2004) The definition of SLD is

“… a disorder in 1 or more of the basic psychological processes involved in understanding or in using language, spoken or written, which disorder may manifest itself in the imperfect ability to listen, think, speak, read, write, spell, or do mathematical calculations.”

Neither the discrepancy model or RTI evaluates basic psychology processes


Hale, Naglieri, Kaufman, & Kavale Hale, Naglieri, Kaufman, & Kavale (2004)(2004) The method of RTI is disconnected from

the definition of SLD “Establishing a disorder in the basic

psychology processes is essential for determining SLD”

Practitioners have ignored this approach because of limited availability of good measures of processing

Times have changed


Hale, Naglieri, Kaufman, & Kavale Hale, Naglieri, Kaufman, & Kavale (2004)(2004) Processing measures of today are

very different than those of the 1970s

Tests that we specifically developed to measure basic psychological processes should be used Kaufman Assessment Battery for

Children 2nd Edition Cognitive Assessment System



Defining intelligence as processing leads to smaller race and ethnic

differences than traditional IQ tests Yields excellent prediction to

achievement Provides sensitivity to the cognitive

disorders seen in many exceptional children

Has demonstrated relationships to intervention

Naglieri, J. A. (2003). Current advances in assessment and intervention for children with learning disabilities. In T. E. Scruggs and M. A. Mastropieri (Eds.) Advances in learning and behavioral disabilities Volume 16: Identification and assessment (pp. 163-190). New York: JAI.


What is Next?What is Next? Current status as of March 2005

Regulations which are interpretations of the law are being written

Input on the regulations was due by Feb 28, 2005

We need to learn about the strengths and weakness of the options RTI – which may be included Assessment of basic psychological

processes – which needs to be assessed


IDEA ReauthorizationIDEA Reauthorization Topical outline

IDEIA and forthcoming regulations•A look at the Law

Response to intervention•Local comparison groups•Measurement of improvement•Research and reviews



27

Alternatives to ability Alternatives to ability achievement discrepancyachievement discrepancy

Response to Intervention



Topical outline IDEA reauthorization The problem of LD identification Response to intervention

• Local comparison groups• Measurement of improvement

A cognitive approach to assessment

30

Response to Intervention Response to Intervention and SLDand SLD

A summary of the method and researchers’ response to

RTI


Kovaleski & Prasse (2004)Kovaleski & Prasse (2004)

The dual discrepancy format for SLD identification Part 1: Low academic performance Part 2: Poor response to appropriate

instruction


Kovaleski & Prasse (2004) Part Kovaleski & Prasse (2004) Part 11 Low academic performance

The student must be significantly below same-grade peers • Shinn (2002) notes that a 2.0 grade level

discrepancy is a typical index that identifies a significant academic deficiency

• This is based on a discrepancy from grade-level performance without reference to an assessment of the student’s ability level (i.e., IQ)


Assessing Academic Assessing Academic PerformancePerformance Low academic performance (continued)

BUT… Shinn’s 2.0 grade level discrepancy is also a “wait to fail ” model because you can’t be behind by two years until you are in at least second grade

BUT…The grade equivalent method has many well known psychometric problems

AND…differences in curriculum can influence who is behind


Kovaleski & Prasse (2004) Part Kovaleski & Prasse (2004) Part 11 Low academic performance

(continued) Advocates argue for RTI as a curriculum-

based measurement (CBM) approach Reading fluency is overemphasized The student’s discrepancy is determined

in relation to classmates by comparing the performance on CBM measures with norms from the student’s school or school district


Assessing Academic Assessing Academic PerformancePerformance Local Norms

Advantage• Local norms are good at telling where the

child is in relation to the smallest comparison group – the child’s classroom

Disadvantage• Local norms only tell where the child is in

relation to the smallest comparison group – the child’s classroom

• Change the classroom and the score changes• Change the school and the score changes

36

The Problem with Local The Problem with Local NormsNorms

Fairfax County Schools


Fairfax County ElementaryFairfax County Elementary Reading scores were obtained for all 69

elementary schools in Fairfax County Public Schools

The schools were ranked on reading scores The 10th, 35th, and 59th ranked schools were

selected School based standard scores were

computed Reading score – standard score (mean 100,

SD of 15) were computed and compared across schools


The same Reading score of 55 yields a standard score of 112 (above average) for a child in Herndon Elementary School but

a score of 92 (average) in Columbia Elementary School and a score of 84 (well

below average) in Flint Hill Elementary School.


Local Norms for Inner City SchoolsLocal Norms for Inner City SchoolsNational mean Local mean

68 78

69 79

72 82

76 86

78 88

80 90

85 95

88 98

92 101

95 105

97 106

99 109

103 113

109 119

115 124

119 128… …

90.3 100.0

Scores based on a national norm group indicate these children are low


National mean Local mean

68 78

69 79

72 82

76 86

78 88

80 90

85 95

88 98

92 101

95 105

97 106

99 109

103 113

109 119

115 124

119 128… …

90.3 100.0

Local Norms for Inner City Local Norms for Inner City SchoolsSchoolsStandard scores based on a national norm group indicate these children are low

Standard scores based on local mean falsely describe half the children with national scores below 85 as OK


Local NormsLocal Norms How effective is a local norm?

It calibrates a child based on a comparison to the classroom, school, or school district

Is it consistent across classrooms? By definition – NO

Is it consistent across schools? By definition – NO

Is it consistent across districts? By definition – NO

Local norms provide an inconsistent unit of measurement


Fairfax County ElementaryFairfax County Elementary Conclusions on Local Norms

Local norms are useful to determine how the child compares to the rest of the class and for instructional planning

A wide variety can exist between schools in the same school district

A child may be “failing” in one class but doing “well” in another

Determining SLD on local norms will yield considerable inequities especially for minority groups


Kovaleski & Prasse (2004) Part Kovaleski & Prasse (2004) Part 22 Poor response to appropriate

instruction The student performs poorly to carefully

planned and precisely delivered instruction The data are developed through ongoing

progress monitoring on a critical academic measure during the course of an individually designed intervention

The use of CBM as an ongoing performance measure (usually through data collected twice per week) is recommended

44

Does an increase in counts Does an increase in counts mean improvement is real?mean improvement is real?

Good news can be relative

45

Rates of ImprovementRates of Improvement

An Achievement Example



Vocabulary Growth Meaning vocabulary growth average

increase is 2,500 words per year (McCormick, 1998)

A child needs to learn about 2,500 words per year to keep up with peers

But a child can learn more words every year and still fall behind


Rates of Improvement-VocabularyRates of Improvement-Vocabulary

Meaning vocabulary growth average increase is 2,500 words per year (McCormick, 1998)

Normal Standard 50% Standard 100% Standard 167% StandardGrowth Score increase Score increase Score increase Score

n #1 #2 #3K 2,500 100 1,500 76 1,500 76 1,500 761 5,000 100 2,250 67 3,000 76 4,000 882 7,500 100 3,000 64 4,500 76 6,500 923 10,000 100 3,750 63 6,000 76 9,000 944 12,500 100 4,500 62 7,500 76 11,500 955 15,000 100 5,250 61 9,000 76 14,000 966 17,500 100 6,000 61 10,500 76 16,500 977 20,000 100 6,750 60 12,000 76 19,000 978 22,500 100 7,500 60 13,500 76 21,500 979 25,000 100 8,250 60 15,000 76 24,000 98

10 27,500 100 9,000 60 16,500 76 26,500 9811 30,000 100 9,750 60 18,000 76 29,000 9812 32,500 100 10,500 59 19,500 76 31,500 98

sd is 25% of mean % increases are based on initial levels


Improvement or Deterioration?Improvement or Deterioration?

0

2,000

4,000

6,000

8,000

10,000

12,000

50556065707580859095100

K 2 4 6 8 10 12

StSc50% increase in number of words per year expressed as standard scores in relation to normal growth rate

50% increase in number of words learned per year



Visual examination of changes in rate are only sufficient to demonstrate change from baseline

Changes over time are helpful for instructional decisions

Aim lines based on local norms are misleading

50

Conclusions Regarding Conclusions Regarding Kovaleski & Prasse (2004)Kovaleski & Prasse (2004)

The dual discrepancy format: Low academic performance

and Poor response to instruction


Conclusions on Dual Conclusions on Dual DiscrepancyDiscrepancy Changes over time do not necessarily

mean the child has reached a level that is consistent with normative expectations

Competency levels can be set so low, and “improvement” defined by such small steps, that apparent improvement may be better described as showing Illusory Validity


Conclusions on Dual Conclusions on Dual DiscrepancyDiscrepancy RTI may be a reasonable way to find

children who are doing poorly in class

RTI problems Local norms do not provide consistency Increases in performance can be

misleading

53

What do Researchers say What do Researchers say about RTI?about RTI?

Advantages, disadvantages, claims, and inconsistencies


NRCLD Conference on RTI (Dec NRCLD Conference on RTI (Dec ‘03)‘03)


RTI Conference Vellutino RTI Conference Vellutino (2003)(2003) RTI Advocates have argued that IQ

scores are unrelated to children’s response to intervention Therefore IQ tests are irrelevant

Vellutino especially has made this claim and provided a report of his research


Full Scale IQ scores (Vellutino)Full Scale IQ scores (Vellutino)

90

92

94

96

98

100

102

104

106

108

Diffi cult to

Remediate

Less Diffi cult

to Remediate

No Longer at

Risk

Average

Controls

Effect Size = .6

FSIQ


Full Scale IQ scoresFull Scale IQ scores

75

80

85

90

95

100

Not Dually

Discrepant

I nf requently

Dually

Discrepant

Frequetly Dually

Discrepant

Wech

sler

FSIQ

From: Case, Speece,& Molly (2004). Validity of Response to Intervention….School Psychology Review, 32,557-582.

Effect size =

1.0


RTI Conference - VaughnRTI Conference - Vaughn Sharon Vaughn described a study of the 3 Tier

model Reading fluency was measured From K to 3rd grade, 20% of the children did not benefit

from intervention – they are considered SLD School psychologist were not included at any level

It took THREE YEARS to identify these children Wait to fail !

The study identified 20% of the sample That is a large percentage of the sample

Conclusion: the hope that RTI offers faster identification of LD and reduces the numbers of children found was not achieved


RTI Conference - GerberRTI Conference - Gerber

Michael Gerber- The costs of RTI He estimated that the cost for TRI for

grades K-3 for the first year of a national scale implementation was $2,033,228,291

The 2003 Federal Appropriations for NCLB (Title II, Part A) is $1,780,825,000

The cost is $252,403,291 more than the allocation!


RTI Conference - KavaleRTI Conference - Kavale Success is not well defined in the RTI

model Increases in rate of learning alone are not

sufficient There are not clear definitions or cut

scores to indicate failure to respond to intervention

RTI is a good first step SLD is more than just reading failure but

RTI has been limited to reading fluency


RTI Conference - KavaleRTI Conference - Kavale The definition of LD has been ignored There is no connection between the

definition of SLD and the method of RTI RTI is not sufficient for identification of LD “a formal evaluation is absolutely

necessary” or inappropriate conclusions may be reached because reading failure can be caused by depression, emotional / behavioral disorders, anxiety disorders, ADHD, etc.


RTI ConferenceRTI Conference - Scruggs (2003) - Scruggs (2003)

RTI reading interventions should be used in general education to assure high-quality instruction and provide alternatives to special education placement

LD identification should be accomplished on the basis of strict adherence to criteria for identification which as commonly known, has not been followed in many parts of the country


RTI Conference - Mastropieri RTI Conference - Mastropieri The RTI model is too focused on

reading decoding and ignores other areas such as reading comprehension, math, science, etc.

Implementation of RTI at middle and high school levels seems unlikely This is particularly important given the

number of students who are currently identified at the middle and early high school level


RTI Conference - MastropieriRTI Conference - Mastropieri The RTI movement is an attempt to

change regular education Earlier attempts to change regular

education emanating from special education have failed

RTI is too limited to meet the diverse needs of SLD children

SLD is more than reading decoding or reading fluency

Too much emphasis on phonics


RTI Conference - Mastropieri RTI Conference - Mastropieri

She asked “Where is the solid research base providing scientific evidence for optimal instructional methods and materials across all grade levels and all curriculum levels?

The reading methods are described as “scientifically based” but where is the science?

66

Research on RTIResearch on RTI

Fuchs, D., Mock, D., Morgan, P, & Young, C. (2003). Responsiveness-to-intervention:

Definitions, evidence, and implications fro the learning Disabilities Construct. Learning

Disabilities Research & Practice, 18, 157-171.


Fuchs, et al., (2003)Fuchs, et al., (2003) Ohio’s Intervention Based Assessment

IBA combines a behavioral problem-solving approach with collaborative consultation• Behavioral definition of the problem• Baseline data are collected• Behavioral definition of the problem• Goals are set• Intervention plan implemented• Compare student to baseline


Fuchs, et al., (2003)Fuchs, et al., (2003)

Telzrow,et al., 2000 evaluated the IBA program 329 schools were involved “The present study suggests that

reliable implementation of problem solving approaches in schools remains elusive”

They did not find evidence of reliable and consistent implementation



Pennsylvania’s Instructional Support Teams (IST) Collaborative problem solving that

provides prereferral intervention like Ohio

Composition of team is a little different IST uses curriculum-based assessment

• Baseline, a goal is set, intervention planned• 50 day limit

– If no success, multidisciplinary evaluation possible



Pennsylvania’s Instructional Support Teams (IST) Kovaleski et al., (1999) studied the

impact of IST and found the students showed better progress than students in non-IST schools

But, according to Fuchs “time on task and task completion and comprehension are only indirect academic measures” (p. 162)



Heartland (Iowa) and Minneapolis four level model

1. Teacher and parents try to fix problem2. Teacher and Building Assistance Team

select, implement, & monitor intervention

3. Behavioral problem solving used to refine or redesign the intervention

4. Special educational assistance is considered


Fuchs, et al., (2003)Fuchs, et al., (2003) “Although the Minneapolis Public

Schools (2001) report that, “The problem-solving model has undergone considerable evaluation” we found few published or unpublished evaluations; none in peer-reviewed journals.

The report described school wide reading gains…but no data were presented



CONCLUSION #1 Evaluations of collaborative problem

solving “have generally failed to produce persuasive evidence that classroom-based interventions (1) are implemented with fidelity and (2) strengthen student’s academic achievement or improve classroom behavior” (p. 163)



CONCLUSIONS #2 & 3 There is insufficient evidence of the

effectiveness of RTI approaches in Ohio and Pennsylvania… and especially in versions used in Heartland and Minneapolis (p. 166)

“More troubling…is that these largely untested procedures are the basis of…RTI…which is being considered as a replacement for IQ-achievement discrepancy”



CONCLUSION #4 The absence of evidence weakens the

assumption that RTI provides feasible, timely, and effective interventions (p. 166).

Proponents of RTI as an alternative means of LD identification must sill prove that their problem-solving approach is worthy of the descriptor “scientifically based” (p. 167)


RTI – Pros and ConsRTI – Pros and Cons RTI may be a reasonable way to find

children who are doing poorly in class RTI problems

Local norms do not provide consistency Increases in performance can be

misleading There is no evidence that RTI is effective

for SLD identification RTI is inconsistent with the definition of

SLD RTI is not well supported by research

77

Information from NASPInformation from NASP

www.nasponline.org


IDEA Reauthorization & RTIIDEA Reauthorization & RTI Other issues…


IDEA Reauthorization & LDIDEA Reauthorization & LD

Assessment is defined as a process of gathering information from a variety of sources, using a variety of methods that best address the reason for evaluation.



Select assessment methods that include multiple sources of information…procedures…and settings



Not limited to any single methodology or theoretical framework

Comprehensive and address educational, cognitive, and mental health needs

Multidimensional

NASP endorses assessment practices that are:


IDEA Reauthorization & LDIDEA Reauthorization & LDAll approaches to assessment are used in ways consistent with their scientific base…

This includes norm and performance based, standardized and functional assessments; intelligence, cognitive processing, social-emotional, academic skills, etc


IDEA and Cognitive IDEA and Cognitive AssessmentAssessment Topical outline

IDEA reauthorization The problem of LD identification Response to intervention

• Local comparison groups• Measurement of improvement




IDEA and Cognitive IDEA and Cognitive AssessmentAssessment How do we connect the definition of LD

with methods to assess it? Measure “basic psychological processes”

using well validated standardized tests built on current views of processing

Two options to measure processing Planning, Attention, Simultaneous,

Successive (PASS) theory and CAS (Naglieri & Das, 1997)

The K-ABC II (Kaufman & Kaufman, 2004)



Measuring processing has advantages: smaller race and ethnic differences than

traditional IQ tests excellent prediction to achievement sensitivity to the cognitive disorders

seen in many exceptional children Connection to interventionNaglieri, J. A. (2003). Current advances in assessment and

intervention for children with learning disabilities. In T. E. Scruggs and M. A. Mastropieri (Eds.) Advances in learning and behavioral disabilities Volume 16: Identification and assessment (pp. 163-190). New York: JAI.


Naglieri, Kaufman, Kavale, Naglieri, Kaufman, Kavale, HaleHale specific learning disability “means a

disorder in one or more of the basic psychological processes ...” (§ 602(30)(A))

The law provides that, as part of the evaluation procedures in Sec. 614 of IDEIA, the local education agency shall use technically sound instruments that may assess the relative contribution of cognitive and behavioral factors… (§ 614(b)(2)(C)).


Naglieri, Kaufman, Kavale, Naglieri, Kaufman, Kavale, HaleHale The regulations should provide that a

comprehensive evaluation of these “basic psychology processes” must be part of the methods used to determine whether the child is learning disabled. In so doing, the definition of SLD and the procedures used to determine eligibility would be united.


Naglieri, Kaufman, Kavale, Naglieri, Kaufman, Kavale, HaleHale We also strongly support section §

614(b)(3)(A)(iii) of the law which states that assessment of these basic psychological processes must be conducted using well validated, reliable, norm-referenced cognitive measures that are not discriminatory on racial or cultural basis.


Naglieri, Kaufman, Kavale, Naglieri, Kaufman, Kavale, HaleHale we support the use of a response to

intervention method at the pre-referral stage.

We further support the fact that Congress has recognized (Sec. 614(b)(6)(B)) that the use of the RTI must include cognitive assessment conducted using well validated, reliable, norm referenced measures in accordance with paragraphs (2) and (3) of Sec. 614(b).



“assess cognitive factors”


Naglieri, Kaufman, Kavale, Naglieri, Kaufman, Kavale, HaleHale We hope that the regulations will not

favor the use of any particular methodology for eligibility determinations because, for example, we are not aware of any studies that show RTI can be conducted in a manner that is valid and reliable across state and local educational authorities nor that this method is nondiscriminatory.

93

IDEIA, SLD, RTI, and IDEIA, SLD, RTI, and Cognitive AssessmentCognitive Assessment

Conclusions and Closing Thoughts


SummarySummary School Psychology is a field that is diverse

and diversity is our strength There are many tools that we can use, and

this variety allows us to be more effective We should use methods that …

are consistent with the definition of SLD are consistent with the non discriminatory

requirement are an alternative to IQ achievement

discrepancy are related to instructional interventions


SummarySummary IDEIA law, CEC position, Roundtable

Consensus Report, and NASP documents all say use more than one methodology

The most defensible way to identify SLD is through a comprehensive evaluation conducted by a school psychologist that includes basic psychological processes (SLD definition) in addition to other data (e.g., RTI, achievement test data, measures of emotional status, etc.)


ReferencesReferencesHale, B., Naglieri, J. A., Kaufman, A. S. & Kavale, K. A. (2004). Specific

learning disability classification in the new Individuals with Disabilities Education Act: The danger of good ideas. The School Psychologist, 58, 6-13.

Kavale, K. A., Kaufman, A. S., Naglieri, J A., & Hale, J. B. Changing procedures for identifying learning disabilities: The danger of poorly supported ideas. The School Psychologist, 59, 16-25.

Naglieri, J. A. (2003). Current advances in assessment and intervention for children with learning disabilities. In T. E. Scruggs and M. A. Mastropieri (Eds.) Advances in learning and behavioral disabilities Volume 16: Identification and assessment (pp. 163-190). New York: JAI.

Naglieri, J. A. (2002). Best practices in interventions for school psychologists: A cognitive approach to problem solving. In A. Thomas & J. Grimmes (Eds.). Best practices in school psychology (4th Ed)(pp. 1373-1392). Bethesda, MD: NASP.

1 ideia, sld, rti, and cognitive assessment jack a. naglieri, ph.d. professor of psychology center...

Documents

achievement slide

cognitive assessment

reliable assessment

sld definition

law definition of sld

law rti

ideia anticipation

sld cognitive processing