effectiveness of reading and math software products findings from the first student cohort mark...
TRANSCRIPT
Effectiveness of Reading and Math Software Products
Findings From the First Student Cohort
Mark Dynarski
May 2007
Effectiveness of Reading and Math Software Products
Findings From the First Student Cohort
Mark Dynarski
May 2007
2
Study Synopsis Study Synopsis
Design– Nine reading and six math software products in 132 volunteer
schools– Treatment teachers could use products, control teachers could
not
Implementation– Companies provided training and other types of support– Study purchased various upgrades and hardware components
Key Findings– Test scores at the end of the school year were not statistically
different– Few relationships between effects and implementation factors
Design– Nine reading and six math software products in 132 volunteer
schools– Treatment teachers could use products, control teachers could
not
Implementation– Companies provided training and other types of support– Study purchased various upgrades and hardware components
Key Findings– Test scores at the end of the school year were not statistically
different– Few relationships between effects and implementation factors
3
Study Size Study Size
Districts Schools Teachers Students Grade 1 14 46 169 2,619 Grade 4 11 43 118 2,265 Grade 6 10 28 81 3,136 Algebra 10 23 71 1,404 Total 45 140 439 9,424
Unduplicated 33 132 439
9,424
4
Implementation FrameworkImplementation Framework
Teacher training [O, R]
Amount of use [I, R]
Technical difficulties and teacher support [I]
Teacher training [O, R]
Amount of use [I, R]
Technical difficulties and teacher support [I]
Student and teacher roles [O]
– Teaching approach [O]– On-task behavior [O]– Consistency with
suggested use of products [I]
Student and teacher roles [O]
– Teaching approach [O]– On-task behavior [O]– Consistency with
suggested use of products [I]
Key: O indicates observations, R records, I interviews
5
General Implementation FindingsGeneral Implementation Findings
Nearly all teachers received training and believed it prepared them to use products
Difficulties using hardware mostly were minor
Total use of software products higher in treatment classrooms
When products are used– teachers more likely to be “facilitators” and students more
likely to work on their own– more students on task in math classrooms
Nearly all teachers received training and believed it prepared them to use products
Difficulties using hardware mostly were minor
Total use of software products higher in treatment classrooms
When products are used– teachers more likely to be “facilitators” and students more
likely to work on their own– more students on task in math classrooms
6
Difference in Technology Use in Treatment and Control Classrooms: First Grade
Difference in Technology Use in Treatment and Control Classrooms: First Grade
1
2222
3
3
4
55
5
6
6
6 7
7
8
8
888
9
999
9
9
10
10
10
1111
12
13
1313
1414
14
1414
14
-25
0
25
50
75
100
125
150
TeacherReportedHours
Numbers indicate districts and each point represents a school
7
Effects on Classroom Practices Effects on Classroom Practices
Percent Difference: Teacher as Facilitator
0%
25%
50%
75%
100%
First Grade
FourthGrade
Sixth Grade
Algebra
Note: * Significantly different from zero at the 0.05 level
*
*
* *
8
Effects on Classroom Practices Effects on Classroom Practices
Percent Difference: Students On Task
Note: * Significantly different from zero at the 0.05 level
0%
10%
20%
30%
**
XFirst Grade
FourthGrade
Sixth Grade
Algebra
9
Estimating EffectsEstimating Effects
Outcome: spring test score
3-level model (students, classrooms, schools)
3-level model extended to estimate effects of conditions and practices (implementation)– interactions of treatment effect and classroom
and school characteristics
Outcome: spring test score
3-level model (students, classrooms, schools)
3-level model extended to estimate effects of conditions and practices (implementation)– interactions of treatment effect and classroom
and school characteristics
10
Effect Sizes By School: First Grade Effect Sizes By School: First Grade
1
2
2
2
2
3
3
4
5
5
5
66
6
77
8
8
88
8
9
9
9
9
9
9
10
10
10
11
11
121313
13
14
14
14
14
14
14
-1.00
-0.50
0.00
0.50
1.00
Numbers indicate districts and each point represents a school
11
Test Scores: First Grade Test Scores: First Grade
SAT-9 Reading Score
-0.05
0.05
0.15
0.25
OverallScore
EffectSize
Note: * Significantly different from zero at the 0.05 level
Soundsand Letters
SentenceReading
WordReading
12
Test Scores: First Grade Test Scores: First Grade
Test of Word Reading Efficiency Score
-0.10
0.00
0.10
0.20
0.30
OverallScore
PhonemicDecoding Efficiency
Sight WordEfficiency
EffectSize
Note: None of the effect sizes is significantly different from zero at the 0.05 level
13
Interactions: First GradeInteractions: First Grade
Larger effects
– More experienced teachers
– Smaller student-teacher ratio
Larger effects
– More experienced teachers
– Smaller student-teacher ratio
No relationship
– Product usage– Problems getting access– Technical difficulties– Computer specialist in
school– Professional development
last year on using technology
– Poverty, urban area, African-American students, Hispanic, special education students
No relationship
– Product usage– Problems getting access– Technical difficulties– Computer specialist in
school– Professional development
last year on using technology
– Poverty, urban area, African-American students, Hispanic, special education students
14
Test Scores: Fourth Grade Test Scores: Fourth Grade
SAT-10 Reading Score
-0.10
0.00
0.10
0.20
0.30
Overall Score
Vocabulary Word Study Skills
Comprehension
Note: None of the effect sizes is significantly different from zero at the 0.05 level
X
EffectSize
15
Interactions: Fourth GradeInteractions: Fourth Grade
Larger effects
– Product usage
Larger effects
– Product usage
No relationship
– Problems getting access– Technical difficulties– Computer specialist in
school– Professional development
last year on using technology
– Poverty, urban area
No relationship
– Problems getting access– Technical difficulties– Computer specialist in
school– Professional development
last year on using technology
– Poverty, urban area
16
Test Scores: Sixth Grade Test Scores: Sixth Grade
SAT-10 Math Score
-0.100.00
0.10
0.20
0.30
Overall Score
Procedures Problem Solving
Note: None of the effect sizes is significantly different from zero at the 0.05 level
EffectSize
17
Interactions: Sixth GradeInteractions: Sixth Grade
No statistically significant relationships No statistically significant relationships
18
Test Scores: Algebra Test Scores: Algebra
ETS Algebra Exam
-0.30
-0.10
0.10
0.30Overall Score Concepts Processes Skills
Note: None of the effect sizes is significantly different from zero at the 0.05 level
EffectSize
19
Interactions: AlgebraInteractions: Algebra
Smaller effects when teachers had technical difficulties
Smaller effects when teachers had technical difficulties
20
Study TradeoffsStudy Tradeoffs
15 reading and math products studied– Many products and types of technology not in the
study
Precision to detect small effect sizes– Average effect reported
Experimental design – Teachers have not used these products in
current classrooms
15 reading and math products studied– Many products and types of technology not in the
study
Precision to detect small effect sizes– Average effect reported
Experimental design – Teachers have not used these products in
current classrooms
21
Second Year StudySecond Year Study
10 products, data from 77 schools, treatment and control teachers with new cohort of students
Will study relationship between teacher experience using products and effects
Effects reported for products
10 products, data from 77 schools, treatment and control teachers with new cohort of students
Will study relationship between teacher experience using products and effects
Effects reported for products