value added for teacher evaluation in the district of columbia robin chait, office of the state...
TRANSCRIPT
Value Added for Teacher Evaluation in the District of Columbia
Robin Chait, Office of the State Superintendent of EducationAnna Gregory, District of Columbia Public Schools
Eric Isenberg, Mathematica Policy Research
Association for Education Finance and Policy 37th Annual Conference
March 16, 2012
Statistical model predicts student achievement
Account for pretests, student characteristics
Ranks teachers relative to an average teacher
Value Added
Teacher value added =
Students’ actual end-of-year test scores –
Students’ predicted end-of-year test scores
2
Value Added in DCPS Evaluation System
Implementation requires sufficient capacity
Communication strategy is vital
Value added is worth the investment
Key Points
4
Where We Were in 2007
District of Columbia Public Schools | Summer 20115
8th grade reading proficiency (2007
NAEP)
Teachers meeting or exceeding
expectations
12% VS 95%
Why Value Added for DCPS?
Fairest way to evaluate teachers
Objective, data-based measure
Focused on student achievement
6
Value Added in DCPS Evaluation System
Individual value-added measures: 50 percent of eligible teachers’ IMPACT scores
IVA: Individual value added
TLF: Teaching and learning framework (classroom observations)
CSC: Commitment to school community
SVA: School value added
7
Highly effective: performance pay
Ineffective (one year): subject to separation
Minimally effective (consecutive years): subject to separation
8
IMPACT Is High Stakes
100 175 250 350 400
Overall Performance DistributionPPEP vs. IMPACT
District of Columbia Public Schools | Summer 2011
n=3,469
9
Value Added in DC
Date Value Added
2009 DCPS (trial run)
2009-2010 First year of IMPACT in DCPS
2010-2011 Second year of IMPACT in DCPS
October 2011 - present Third year of IMPACT in DCPS
First year of Race to the Top for DCPS and DC charter schools
10
Help for DC Public Schools
11
Mathematica Policy Research
Technical Advisory Board [2012]– Steve Cantrell, Gates Foundation– Laura Hamilton, RAND Corporation– Rick Hanushek, Stanford University– Kati Haycock, Education Trust– David Heistad, Minneapolis Public Schools– Jonah Rockoff, Columbia Business School– Tim Sass, Georgia State University– Jim Wyckoff, University of Virginia
Mathematica’s Work with DC Schools
Challenges
Consider face validity, incentive effects
Teacher-student link data can be challenging
All data decisions shared with district
Timeline must allow DCPS to transition out poor performers, hire new teachers
13
14
No One-Size-Fits-All Value Added Model
Choosing student characteristics: communications challenge for race/ethnicity
Multiple years of data: bias/precision trade-off
Joint responsibility for co-teaching– Cannot estimate model of separate teacher effects– Can estimate “teams” model, but should team
estimates count?
Comparing teachers of different grades
15
Roster Confirmation
Teacher-student links critical for value added
Administrative data can be challenging– Specialized elementary school teachers– Co-teaching– Pull-out and push-in programs– Midyear student transfers
Teachers surveyed to confirm administrative roster data (Battelle for Kids)
16
Business Rules: Documenting Data Decisions
Every data decision defined, discussed, documented beforehand
Let OSSE, DCPS review all decisions
Document entire process
Make quick progress when final data arrive
17
Production: Meeting Timelines, Ensuring Accuracy
October data: formulate business rules
February data– Establish data cleaning programs– Begin trial runs from analysis file to final output
April data: Final student data in trial runs
June (test score) data: produce final results
Perspective of State Education Agency
Race To The Top
19
Federal competition between states
Required student achievement to contribute 50% of teacher evaluation score
Decision to use DCPS value-added model for all eligible DC teachers
Brought DCPS and charter schools together
Each charter school LEA has own evaluation system used to inform personnel decisions
Common Decision-Making
Need to make decisions on value added–Quickly to meet production schedule
–Informed by best available data
–Obtains buy-in from charter schools and DCPS
Technical Support Committee (TSC)–Six members: five charter, one DCPS
–Meets periodically
–Consensus decisions sought
20
Data Infrastructure
Most data elements for value added exist
. . . but not necessarily collected on right schedule
Student background characteristics–Collected twice a year for AYP purposes
– Need three-time-a-year collection, earlier schedule for value added
21
Need Capacity Within District
Do not just hire a contractor
Need dedicated staff to answer questions–Data team
– Technical Support Committee
22
Communicating Results to DC Teachers
Communication Strategy
Value added hard to understand–Requires a strong statistical background
– Final information is hard to connect to familiar test scores
– Different from other student achievement measures teachers commonly use
Communication tools–Guidebooks
– Information sessions
24
What Factors Affect a Student’s Achievement?
25
Teacher’s Level of Expectations
Teacher’s Pedagogical Expertise
Teacher’s Ability to Motivate
Teacher’s Content Knowledge
Student’s Prior Learning
Student’s Disability (If Any)
Student’s English Proficiency
Student’s Resources at Home
Student Achieveme
nt
As Measured by the DC CAS
Value-added isolates the teacher’s
impact on student achievement.
Initiatives Under Development
Student-level output for DC teachers– Would show pretest, predicted posttest, actual
posttest score for each student
– May be in graphical format
Intermediate value-added scores– Individual value-added scores based on intermediate
tests
– Could be given to teachers midyear
26
Conclusions
Implementing value added requires . . . – Availability and accessibility of current data
– Confirmation of teacher-student links
– Careful planning of production process
– Sufficient capacity within local and/or state education agency to interact with value-added contractor
Teacher buy-in is not a given – communication strategy is vital
Properly implemented, value added is worth the investment
–Fairest measure of teacher effectiveness
–Provides data for answering research questions
27