valid, reliable & efficient:
DESCRIPTION
Valid, Reliable & Efficient:. A Psychometric Evaluation of “Flash” Word Recognition and “NSSI” Passage Reading Measures. Kathleen J. Brown Matthew K. Fields University of Utah R. Darrell Morris Appalachian State University. Impetus for Current Study. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/1.jpg)
Valid, Reliable & Efficient:
A Psychometric Evaluation of
“Flash” Word Recognition and
“NSSI” Passage Reading Measures
![Page 2: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/2.jpg)
Kathleen J. BrownMatthew K. FieldsUniversity of Utah
R. Darrell Morris Appalachian State University
![Page 3: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/3.jpg)
Impetus for Current Study
Need for valid, reliable, efficient instruments to determine instructional reading level
Flaws with current instruments DRA – no rate & time consuming DIBELS – screen only IRIs – psychometric evaluation often weak or
missing
![Page 4: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/4.jpg)
Impetus for Current Study
Growing use of “Flash” & selected graded passages a.k.a. “NSSI” (Virginia/ASU effect)
Initial psychometric evaluations positive (Frye, 2004; Frye & Trathen, 2004; Frye
Trathen, Olson, & Schlagal, 2002; Palmer, Trathen, Olson, & Schlagal, 2002)
![Page 5: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/5.jpg)
Theoretical Framework
Instrument Reliability Validity
Flash
(WRI)
Alternate Form
(manual vs. computer)
Content (rep of grade level?)
Concurrent (GORT)
NSSI
(IRI)
Alternate Form
(A passages vs. B passages)
Content (rep of grade level?)
Concurrent (GORT)
(Anastasi, 1988; APA, 1985)
![Page 6: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/6.jpg)
Methods
4 schools 2 = Title 1 1 = public, 1 = parochial 2 = non-Title 1 both = public & mixed SES
192 students in G2-G5 in March, 2006
Rank ordered DIBELS or QRI, then sampled 12 students per grade: 4 high, 4 average, 4 poor to achieve a representative distribution for testing
![Page 7: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/7.jpg)
Methods
135 minutes of assessment in 3 sessions
Presentation order counterbalanced
Flash item selection counterbalanced
9 on data team; 4 hours protocol training
Manual flash interrater differences = n.s.
![Page 8: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/8.jpg)
Alternate Form Reliability
measure of temporal stability for scores
measure of consistency of response for scores
![Page 9: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/9.jpg)
Alternate Form Reliability
To what extent are NSSI A passage scores equivalent to NSSI B passage scores?
To what extent are computer “Flash” scores equivalent to manual “Flash” scores?
![Page 10: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/10.jpg)
![Page 11: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/11.jpg)
![Page 12: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/12.jpg)
![Page 13: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/13.jpg)
Results: Alternate Form Reliability
NSSI Form B
Flash Manual
Flash Comp
NSSI A .878** .761** .803**
NSSI B Add
n.s. on t-tests
.807** .859**
Flash Manual
.847**
**p < .01
![Page 14: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/14.jpg)
Content Validity
provides evidence that items on test represent a specific domain
provides evidence that the format and response properties of the test represent the domain
![Page 15: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/15.jpg)
Content Validity
To what extent do the NSSI passages reflect/measure expected grade level benchmarks?
Maybe look at separate means for accuracy, rate, & comp & report those to show
To what extent does the Flash measure reading instructional level?
![Page 16: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/16.jpg)
NSSI Reading Level Criteria
Acc. (%)
Rate (wpm)
Comp (%)
Mid G1 95 30 60 End G1 95 40 60 Mid G2 95 65 60
End G2 95 or
97 90 60 or
75 End G3 95 90 60 End G4 95 110 60 End G5 95 120 60 End G6 95 130 60 End G7 95 150 60 End G8 95 150 60
![Page 17: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/17.jpg)
Performance Levelfor NSSI by Grade
NSSI Form A
NSSI Form B
Grade 2 M (SD)
2.14 (1.42)
2.28
(1.24)
Grade 3 M
(SD) 3.52
(1.09) 3.49
(1.11)
Grade 4 M (SD)
4.04
(1.43) 3.87
(1.34)
Grade 5 M (SD)
5.06
(1.85) 4.90
(1.57)
![Page 18: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/18.jpg)
Performance Levelfor Flash by Format & Criterion
Manual (70%)
Comp (70%)
Manual (85%)
Comp (85%)
Grade 2 M (SD)
3.26
(1.24) 3.21
(1.31) 2.58
(1.09) 2.48
(1.20)
Grade 3 M (SD)
4.82
(1.82) 4.59
(1.58) 3.50
(1.31) 3.80
(1.41)
Grade 4 M (SD)
5.81
(1.63) 5.91
(1.88) 4.28
(1.63) 4.57
(1.79)
Grade 5 M (SD)
6.99
(1.49) 6.97
(1.93) 5.46
(1.76) 5.66
(1.94)
![Page 19: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/19.jpg)
Concurrent Validity
To what extent are Flash scores and NSSI scores consistent with scores achieved on a “flagship” standardized reading measure (i.e., the GORT)?
![Page 20: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/20.jpg)
![Page 21: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/21.jpg)
Results: Concurrent Validity
NSSI Form B GORT
Flash Manual
Flash Comp
NSSI A .878** .823** .761** .803**
NSSI B .842** .807** .859**
GORT .808** .826**
Flash Manual
.847**
**p < .01
![Page 22: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/22.jpg)
Average Performance Levelfor NSSI & GORT by Grade Level
NSSI Form A
NSSI Form B GORT
Grade 2 M (SD)
2.14
(1.42) 2.28
(1.24) 3.92
(1.72)
Grade 3 M (SD)
3.52
(1.09) 3.49
(1.11) 5.63
(1.55)
Grade 4 M (SD)
4.04
(1.43) 3.87
(1.34) 6.34
(1.53)
Grade 5 M (SD)
5.06
(1.85) 4.90
(1.57) 6.78
(1.22)
![Page 23: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/23.jpg)
Conclusions: For G2-G5
NSSI A and NSSI B seem to have high validity for identifying students’ instructional reading levels
NSSI A and NSSI B can be considered equivalent forms
![Page 24: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/24.jpg)
Conclusions: For G2-G5
Manual Flash and Computer Flash seem to have high validity for identifying students’ instructional reading levels when the criterion is set at 85%
Manual Flash and Computer Flash seem to be equivalent forms
![Page 25: Valid, Reliable & Efficient:](https://reader036.vdocuments.site/reader036/viewer/2022062518/568145cb550346895db2d312/html5/thumbnails/25.jpg)
Conclusions: For G2-G5
The GORT does not seem to have high validity for identifying students’ instructional levels—at any grade level.
The GORT over-predicts instructional level—by approx. 2 years.
Note: most GORT comp questions are passage independent (Keenan & Betjemann, 2006)