using computerized assessment to identify profiles of … · 2016. 6. 23. · ll=log likelihood,...
TRANSCRIPT
Barbara Foorman and Yaacov Petscher - Florida Center for Reading Research at
Florida State University Liz Brooke and Alison Mitchell - Lexia Learning
USING COMPUTERIZED ASSESSMENT TO IDENTIFY PROFILES OF READING & LANGUAGE SKILLS IN ELEMENTARY AND SECONDARY STUDENTS
¡ The Structure of Reading
¡ Identifying Profiles
¡ Connections to Instruction
AGENDA
How does reading relate to language? How do we reconcile the structure of reading with profiles of strengths and weaknesses?
THE STRUCTURE OF READING
FLORIDA CENTER FOR READING RESEARCH (FCRR) READING ASSESSMENT (FRA)
As part of federal grants, FCRR developed the FRA • 2010-2014: Developed computer-adaptive K-12
component skills battery (item tryouts, IRT analyses, & linking studies. Called the FCRR Reading Assessment (FRA)
• 2010-2015: Conducted cross-sectional and longitudinal
study of reading & language development.
STRUCTURE OF READING K - 3
Foorman et al. (2015) Reading & Writing
Decoding fluency
Syntax
Phonological awareness
Vocabulary
Listening Comp
Oral language
Reading Comprehension
STRUCTURE OF READING 4 - 10
Foorman, et al. (2015) Journal of Educational Psychology
Decoding fluency
Syntax
Vocabulary
Oral language
Reading Comprehension
72% - 99% variance
BACKGROUND ON PROFILES
• Targeting instruction to profiles of students’ strengths & weaknesses is central to teaching
• Profiles often based on descriptive data of reading errors, text reading levels, or learning profiles.
• Regression-based techniques used to quantify profiles of good & poor readers and profiles within poor readers.
• Regression-based approaches use arbitrary achievement cut points (e.g., below 40th percentile on standardized test).
• A latent class approach (LCA) utilizes multiple measures to reduce measurement error and improve reliability and stability of classification.
• When the latent variable is continuous, the approach is often called latent profile analysis (LPA). LPA used here.
¡ What are the latent profiles of reading and language skills in a large, representative sample of Florida students in grades K-10?
¡ What are the relations among the latent profiles and a norm-referenced reading test in K-2 and a latent variable of reading comprehension in grades 3-10?
RESEARCH QUESTIONS
METHOD
• Participants: 7,752 students in K-10; 2295 in K-2 and 5,457 in 3-10. Representative of Florida demographics.
• Procedure: - K-2 FRA individually administered in two 45-min sessions in mid-year; 3-10 FRA administered in computer lab in two 45-min sessions in mid-year. - SESAT Word Reading administered in small groups in K; teachers administered SAT-10 and FCAT as usual.
• Design and analysis: FRA raw scores converted to Z scores. Latent profile and general linear modeling were conducted at each grade (with linear step-up correction to correct against false discovery rate).
Construct Task/Abbreviation Grade Phonological Awareness Phonological awareness (PA) K
Alphabetics
LS Letter Sounds (AP1/2; LS) K
Decoding Word Reading (WR) G1 + G2
Encoding Spelling (Spell) G2
Oral Language
Vocabulary Vocabulary Pairs (VOC) K-2
Syntax Sentence Comprehension (SC) K
Listening Comprehension Following Directions (FD) K-2
CONSTRUCTS/FRA SCREENING TASKS IN GRADES K-2
Construct Tasks/Abbreviation Word recognition Word Recognition (WRT)
Academic Language
Vocabulary (morphological awareness) Vocabulary Knowledge (VKT)
Discourse (verb tense, anaphora, connectives) Syntactic Knowledge (SKT)
Reading Comprehension Reading Comprehension (RCT)
CONSTRUCTS/FRA SCREENING TASKS IN GRADES 3-10
Grade(s) Test/Subtest Kindergarten Stanford Early Scholastic Achievement Test
(SESAT) Word Reading 1-10 Stanford Achievement Test (10th ed; SAT-10)
Reading comprehension 3-10 Florida Comprehensive Assessment Test
(FCAT 2.0) Reading
STANDARDIZED READING OUTCOMES IN GRADES K-10
Note. A latent factor score for Reading Comprehension was created from the developmental scale scores from the FRA’s RCT, SAT-10, and FCAT 2.0
¡ Print-related measures were moderately correlated in K (PA with LS, .48, and with SESAT WR, .58; and SESAT WR with LS, .51). OL measures were moderately correlated with each other.
¡ In G1-G2, print-related measures were more strongly correlated: SAT-10 with WR (.75 in first and .58 in G2); WR & Spell in G2 (.77). Oral language measures were moderately correlated with each other in all three grades and VOC was moderately correlated with SAT-10 in G1 (.58) & G2 (.62).
¡ The three RC measures were strongly correlated, with the RCT bivariate correlations ranging from .67 in G8 to .81 in G5. FCAT and SAT-10 correlations ranged from a low of .71 in G8 to a high of .81 in G3 & G5. VKT and SKT were moderately correlated in these grades (.31 to .46) as were the bivariate correlations of WRT (.29 to .51).
CORRELATIONS
What are the challenges associated with identifying profiles of student strengths and weaknesses? What are ways to do so that are meaningful, but remain reliable and valid and can be assessed efficiently?
IDENTIFYING PROFILES
LATENT PROFILE ANALYSIS – WHAT?
Hybrid Models
Continuous Measures
Categorical Measures
Continuous Latent Factor Analysis Item Response Theory
Categorical Latent Latent Profile Analysis
Latent Class Analysis
LATENT PROFILE ANALYSIS – WHAT?
Vocabulary
PPVT EVT SYN
Class
PPVT EVT SYN
LATENT PROFILE ANALYSIS – WHY?
WHY ADAPTIVE MEASURES?
3-6 hours!!!
Note. LL=log likelihood, AIC=Akaike Information Criteria, aBIC=sample adjusted Bayes Information Criteria, -2LL=log likelihood ratio test. Values in bold represent Final selected class. *p<.001. 19
Latent profile model fit for kindergarten through grade 5 and grade 8
Grade Profiles Parameters LL AIC aBIC -2LL K 2 19 -3255.01 6548.01 6624.87
3 22 -2681.62 5407.25 5496.24 1146.77*
4 28 -2653.41 5362.82 5476.08 56.42*
5 34 -2629.75 5357.51 5465.04 47.32*
6 40 -2618.44 5316.88 5458.68 22.63*
1 2 10 -2818.26 5656.52 5705.49
3 14 -2785.14 5598.28 5666.84 66.24*
4 18 -2768.46 5572.92 5661.01 33.36*
5 22 -2752.99 5549.98 5657.71 30.94*
6 26 -2743.43 5546.86 5674.17 19.12*
2 2 13 -3768.29 7562.59 7624.79
3 18 -3697.13 7430.26 7516.38 142.33*
4 23 -3669.02 7384.03 7494.08 56.22*
5 28 -3655.54 7367.07 7501.04 26.96*
6 33 -3642.95 7355.89 7513.78 25.17*
3 2 10 -2202.90 4425.81 4438.14
3 14 -2173.78 4375.56 4392.83 58.24*
4 18 -2154.42 4344.83 4367.04 38.73*
5 22 -2129.87 4303.74 4330.88 49.10*
6 26 -2104.72 4261.43 4293.51 50.30*
4 2 10 -2166.75 4353.49 4365.49
3 14 -2140.35 4308.69 4325.50 52.80*
4 18 -2112.79 4261.58 4283.18 55.12*
5 22 -2097.77 4239.54 4265.95 30.04*
6 26 -2087.22 4226.44 4257.65 21.10*
5 2 10 -2451.39 4922.79 4935.95
3 14 -2405.92 4839.83 4858.25 90.96*
4 18 -2383.61 4803.22 4826.91 44.61* 5 22 -2363.13 4770.25 4799.20 40.97*
Note. VOC=Vocabulary Pairs; FD=Following Directions; PA=Phonological Awareness; LS=Letter Sounds; SC=Sentence Comprehension
20
Latent Profile Analysis of FRA Measures in Kindergarten (N=422)
-3
-2
-1
0
1
2 VOC FD PA LS SC
Kind
erga
rten
Z-S
core
c1 c2 c3 c4 c5 c6
c6; 19% c3; 42% c4; 23%
c5; 7%
c1; 7%
c2; 2%
350
400
450
500
550
SESA
T
1 2 3 4 5 6
group
<.0001Prob > F37.11F
Distribution of SESAT
350
400
450
500
550
SESA
T
1 2 3 4 5 6
group
<.0001Prob > F37.11F
Distribution of SESAT
Note. The average absolute value of the standardized difference in SESAT WR performance across all classes was Hedge’s g = 1.10, indicating the magnitude of differences in FRA skill profile performance on standardized outcome
K SESAT WR BY LATENT CLASSES
22
Latent Profile Analysis of FRA Measures in Grade 1 (N=989)
Note. VOC=Vocabulary Pairs; FD=Following Directions; WR=Word Reading
-4
-3
-2
-1
0
1
2 VOC FD WR
Gra
de 1
Z-S
core
c1 c2 c3 c4 c5
c2; 35% c5; 43%
c1; 1% c2; 35%
c3; 3%
Note. The average absolute value of the standardized difference in SAT-10 RC performance across all classes was Hedge’s g = 1.43, indicating the magnitude of differences in FRA skill profile performance on standardized outcome
G1 SAT-10 RC BY LATENT CLASSES
24
Latent Profile Analysis of FRA Measures in Grade 2 (N=884)
Note. VOC=Vocabulary Pairs; FD=Following Directions; Spell=Spelling; WR=Word Reading
-2
-1
0
1
2 VOC FD Spell WR
Gra
de 2
Z-S
core
s
c1 c2 c3 c4 c5 c6
c5; 32%
c2; 10%
c4; 32%
c3; 15% c6; 5% c1; 52%
Note. The average absolute value of the standardized difference in SAT-10 RC performance across all classes was Hedge’s g = 1.48, indicating the magnitude of differences in FRA skill profile performance on standardized outcome
G2 SAT-10 RC BY LATENT CLASSES
Latent Profile Analysis of FRA Measures in Grade 5 (N=659)
Note. VKT=Vocabulary Knowledge Task; WRT=Word Recognition Task; SKT=Syntactic Knowledge Task
-4
-3
-2
-1
0
1
2 VKT WRT SKT
Gra
de 5
Z-S
core
c1 c2 c3 c4 c5
c5; 7 %
c4; 34 % c3; 53 %
c2; 4 %
c1; 1 %
-3
-2
-1
0
1
2
3
RC fa
ctor it
self
1 2 3 4 5
c
<.0001Prob > F132.73F
Distribution of frc
-3
-2
-1
0
1
2
3
RC fa
ctor it
self
1 2 3 4 5
c
<.0001Prob > F132.73F
Distribution of frc
Note. The average absolute value of the standardized difference in latent RC performance across all classes was Hedge’s g =2.53, indicating the magnitude of differences in FRA skill profile performance on standardized outcome
G5 RC FACTOR BY LATENT CLASSES
Latent Profile Analysis of FRA Measures in Grade 8 (N=629)
Note. VKT=Vocabulary Knowledge Task; WRT=Word Recognition Task; SKT=Syntactic Knowledge Task
-3
-2
-1
0
1
2 VKT WRT SKT
Gra
de 8
Z-S
core
c1 c2 c3
c2; 25%
c1; 72%
c3; 3%
-2
0
2
RC fa
ctor it
self
1 2 3
c
<.0001Prob > F196.02F
Distribution of frc
-2
0
2
RC fa
ctor it
self
1 2 3
c
<.0001Prob > F196.02F
Distribution of frc
Note. The average absolute value of the standardized difference in latent RC performance across all classes was Hedge’s g =2.19, indicating the magnitude of differences in FRA skill profile performance on standardized outcome
G8 RC FACTOR BY LATENT CLASSES
¡ LPA identified 5-6 classes in K-5 and only 3 in secondary grades.
¡ Latent profiles significantly related to standardized reading outcomes, accounting for 24% (in G3) to 61% (G9) of the variance, with the mode being 42%.
¡ Range of Hedges g (for average absolute values of the standardized difference in reading outcome across all latent classes) was 1.10 (in K) to 2.53 (in G5).
¡ Profiles above G5 fell into a pattern of low, medium, and high. ¡ The 5-6 reading and language profiles found in the elementary
grades reflect heterogeneity of skills.
SUMMARY OF RESULTS
¡ Fact that latent profiles accounted for substantial dif ferences in RC in a large diverse sample of students spanning 11 grades contributes to a field dominated by (a) unreliable or unstable classifications, and (b) LCA of clinical samples (e.g., Catts et al., 2012; Justice et al., 2015) or low-performing students (Logan & Petscher, 2010; Brasseur-Hock et al., 2011).
¡ Heterogeneity of skill profiles in the elementary grades in contrast to the low, medium, & high profiles in the secondary grades suggests focusing on differentiating instruction in K-5.
¡ Importance of FRA academic language (vocab and syntax) to RC, more than word recognition, apparent in G3 and above.
CONCLUSIONS
¡ Cross-sectional rather than longitudinal. [Replications of findings across grades helps.]
¡ Profiles and their relations to reading outcomes are limited to the measures used. [At least a latent variable of reading comprehension was used, which related strongly to the single measures in the FRA reading and language measures.]
¡ Next step is to test results of the heterogeneous profiles from this exploratory LPA with confirmatory latent class analysis.
LIMITATIONS/NEXT STEPS
Immediate and Actionable Data
CONNECTION TO INSTRUCTION
How do we make these types of results meaningful for educators?
HOW DO EDUCATORS MAKE SENSE OF ALL OF THIS DATA??
ONLY 16% OF EDUCATORS FEEL PREPARED
National Center for Literacy Education (2014)
16%
QUESTIONS TO DRIVE INSTRUCTION &
DECISIONS
¡ As a whole school or district, what percent of my students are at-risk for reading failure? What percent are on track for reading success?
¡ Have we seen growth in my students across the year? If so, how much growth?
¡ What subgroups of students are on track for reading success and which subgroups are at-risk for reading failure?
¡ Do I see any patterns of strengths and weaknesses across grades in my district/school? § This will help address PD and Curriculum needs
QUESTIONS ADMINISTRATORS NEED ANSWERED
¡ How many of my students are § Above grade level, § On grade level or § Below grade level?
¡ If they are above grade level, how far above? If they are below, how far below?
¡ Most importantly, how do I help those who are below, get to where they need to be?
QUESTIONS TEACHERS NEED ANSWERED
¡ What percent of my students are on track for reading success?
¡ What patterns of strengths and weaknesses do I see in my class? How can I adjust my instruction to meet the needs?
¡ What do these patterns mean for my small group instruction? Are there certain students who need the same type of instruction? If so, who are they and what is the instruction they need?
QUESTIONS TEACHERS NEED ANSWERED
“The goal (of assessment) is to gain enough information about student progress to make
effective decisions while minimizing the time spent administering assessments.”
- Torgesen, 2006, p.3
REMEMBER…
¡ Include oral language measures in screening, diagnostic, and progress monitoring of reading to accurately predict reading comprehension.
¡ Describe profiles of strengths and weaknesses in a valid, reliable, and efficient manner.
¡ Use profiles to differentiate instruction.
CONCLUDING THOUGHTS
