striving to link teacher and student outcomes: results from an analysis of whole-school...
TRANSCRIPT
Striving to Link Teacher and Student Outcomes: Results from an Analysis of
Whole-school Interventions
Kelly Feighan, Elena Kirtcheva, and Eric Kucharik
Research for Better Schools, Philadelphia, PA
American Evaluation Association Annual Meeting,
November 12, 2009 in Orlando, Florida
1
Study Purpose
Investigate which variables best explain student reading outcomes following teacher professional development
Explore the contextual reasons that help explain why no intervention “impact” was detected
Inform educational policy and improve rigor of educational research
2
Project Background
Federal Striving Readers program aimed at improving pedagogy and student achievementSchools were matched in pairs and then randomly
assigned to the treatment or control condition
Professional Development: four-semester course, onsite literacy coaching, leadership seminar, and curricular material
Developer’s hypothesis: integrating literacy strategies in content areas will yield student gains
3
Factors Affecting Student Learning
Student-level: SES, socio-demographic variables, family background, early development (Barton & Coley, 2009)
Teacher/classroom-level: expectations, preparation, experience, class size (Cohen, McCabe, Mitchelli, and Pickeral, 2009)
School-level: school climate - safety, student- adult and peer relationships, curriculum rigor (Cohen, McCabe, Mitchelli, and Pickeral, 2009)
4
Study Participants
• 30 ELA teachers taught at eight schools– 16 taught at intervention schools
– 14 taught at comparison schools
• 2,114 students linked to these teachers– state assessment reading scores (N = 2,064)
– ITBS scale reading scores (N = 1741)
5
Methodology
Quantitative data sources:RBS teacher surveySchool district school climate surveyDepartment of Education teacher HQT
statistics and student discipline dataStudents’ scores on state assessment and
ITBS
6
Methodology
Qualitative data sources:Observations
56 classrooms (Year 1)48 classrooms (fall of Year 2)10 paired observations (spring of Year 2)
Interviews 8 principals and 19 school improvement team
members in Years 1 and 2Focus groups: seven groups with 62 teachers
7
Research Hypotheses
Exposure to professional development participants will yield gains in reading achievement
Including contextual variables in impact analysis will increase explanatory power of results
8
Quantitative Analysis
Used Hierarchical Linear Modeling (HLM) to predict student performance based on student-, teacher-, and school-level characteristics
Fully unconditional model represents how variation in an outcome measure is allocated across the three different levels
9
Variables Included in the HLM
10
Outcome Variables: Reading ScoresRandom Effect Variance
Componentdf Chi-
squareP Value Variance
Decomposition (% by level)
State Test
Students 823.47 81.18
Teachers 178.82 22 322.32 0.000 17.63
Schools 12.14 7 9.50 0.218 1.19
ITBS
Students 519.68 77.8
Teachers 147.58 22 361.18 0.000 22.0
Schools 1.08 7 7.48 0.380 0.2
11
Student-Level Variation
Across multiple model specifications, the only predictors with statistical significance were the student’s
Pre-test scoreGender ELL status
Modeling teacher-level factors produced no significant results
12
Classroom Observation Results
No baseline differences in levels of engagement & cognitive demand, or in instructional strategies
Cognitive demand level of lessons was low in Year 2, irrespective of research condition
Intervention teachers tended to use more literacy strategies than comparison teachers in Year 238.5% of intervention teachers used multiple literacy
strategies vs 18.2% of comparison teachers
13
Why We May Not Find Impact
Low cognitive demand of lessons
Counterfactual situations may “water down” the treatment’s effect
Low implementation fidelity
Limitations in outcomes measures (just say measurement error)
14
Implications for Further Research
Better understanding of the relationship between a school-level intervention and its
potential to affect student achievement
Correlates of student achievement
Why an intervention that did not show impact may nevertheless be of value
15