Informing Policy: Informing Policy: State Longitudinal Data State Longitudinal Data SystemsSystems

Jane Hannaway, DirectorJane Hannaway, DirectorThe Urban InstituteThe Urban Institute CALDERCALDERwww.caldercenter.org

State of U.S. EducationState of U.S. Education

½ of minority students graduate ½ of minority students graduate from high schoolfrom high school

4 grade level gap between white 4 grade level gap between white and minority students by 12and minority students by 12thth grade grade

15% of minorities earn BAs w/in 9 15% of minorities earn BAs w/in 9 years of 9years of 9thth grade grade

The The WILLWILL and the and the WAYWAY

The The WILLWILL– Left, Right, CenterLeft, Right, Center– Agreement on education crisisAgreement on education crisis– Strange bedfellowsStrange bedfellows

The The WAYWAY– Few, but growing, guidepostsFew, but growing, guideposts

Finding the Finding the WAYWAY with Evidence with Evidence-A New Day--A New Day-

Who has the evidence?Who has the evidence?– States have the makings of the evidenceStates have the makings of the evidence

Where are the makings?Where are the makings?– State administrative data systemsState administrative data systems

Why do states have it?Why do states have it?– Important effect of NCLBImportant effect of NCLB

Why important?Why important?– Address questions never before possibleAddress questions never before possible

Research Background: Research Background: What We KnowWhat We Know

Teachers matter- single most important Teachers matter- single most important schooling contributor to student outcomesschooling contributor to student outcomes

Wide variation in teacher effectiveness. Wide variation in teacher effectiveness. Some teachers are simply much better than Some teachers are simply much better than othersothers

Standard measures of teacher quality not Standard measures of teacher quality not much related to effectiveness, but directly much related to effectiveness, but directly related to spending.related to spending.

Research Background:Research Background:What We What We Don’tDon’t Know Know

What is it about What is it about teachers that matters?teachers that matters?

3 Research Probes3 Research Probes

Teacher MaldistributionTeacher Maldistribution

Teacher SelectionTeacher Selection

Teacher MobilityTeacher Mobility

Teacher Teacher Maldistribution 1Maldistribution 1 Comparison of VA of teachers in Comparison of VA of teachers in

high/ low poverty schoolshigh/ low poverty schools North Carolina and FloridaNorth Carolina and Florida FindingsFindings

– Low poverty - higher va, but not Low poverty - higher va, but not muchmuch

– High poverty – larger variation in High poverty – larger variation in schoolschool

Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (North Carolina-Math)Levels (North Carolina-Math)

North Carolina- Elementary Math

-0.4000

-0.3000

-0.2000

-0.1000

0.0000

0.1000

0.2000

0.3000

10 25 50 75 90

Percentile

Teacher Performance Percentile

Val

ue-

add

ed s

core

0-70% FRL

70-100% FRL

Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (Florida- Math)Levels (Florida- Math)

Florida- Elementary Math

-0.4000

-0.3000

-0.2000

-0.1000

0.0000

0.1000

0.2000

0.3000

10 25 50 75 90

Percentile

Teacher Performance Percentile

Val

ue-

add

ed s

core

0-70% FRL

70-100% FRL

Novice teachers are less effective Novice teachers are less effective than experienced teachers.than experienced teachers.

Returns to experience taper off 3-Returns to experience taper off 3-5 years.5 years.

Distribution of Value-Added of Distribution of Value-Added of Elementary Math Teachers in Elementary Math Teachers in High Poverty SchoolsHigh Poverty Schools

0.5

11

.52

Den

sity

-1 -.5 0 .5 1 1.5Value-added score

kernel = epanechnikov, bandwidth = 0.0885

>=70% Poverty SchoolsDistribution of Value-Add of Elementary Math Teachers

Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience

Distribution of Value-Added of Distribution of Value-Added of Elementary Math Teachers in Elementary Math Teachers in Lower Poverty SchoolsLower Poverty Schools

0.5

11

.52

Den

sity

-1 -.5 0 .5 1Value-added score

kernel = epanechnikov, bandwidth = 0.0698

0-70% Poverty SchoolsDistribution of Value-Add of Elementary Math Teachers

Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience

Teacher Teacher Maldistribution 2Maldistribution 2 New York CityNew York City

– Phasing out of emergency Phasing out of emergency certificationcertification

– Introduction of alternative route Introduction of alternative route teachersteachers

LAST Exam Failure Rate of LAST Exam Failure Rate of Elementary Teachers by Elementary Teachers by Poverty Quartile, 2000-2005Poverty Quartile, 2000-2005

10%

15%

20%

25%

30%

35%

40%

2000 2001 2002 2003 2004 2005

Per

cen

t o

f te

ach

ers

Low est quartile 2nd quartile 3rd quartile Highest quartile

LAST Exam Failure Rate of LAST Exam Failure Rate of New Teachers by Poverty New Teachers by Poverty Quartile, 2000-2005Quartile, 2000-2005

10%

15%

20%

25%

30%

35%

2000 2001 2002 2003 2004 2005

% o

f n

ew

te

ac

he

rs

Lowest quartile 2nd quartile 3rd quartile Highest quartile

Predicted Effectiveness For Predicted Effectiveness For Highest and Lowest Highest and Lowest Poverty SchoolsPoverty Schools

-0.06

-0.04

-0.02

0

0.02

0.04

0.06

0.08

0.1

Most Affluent Decile Poorest Decile Gap

2001

2005

Narrows by 25%

Can change predicted Can change predicted effectiveness by selection effectiveness by selection up-frontup-front

Mean VA Mean VA by by

QuintileQuintile(poor (poor

schools)schools)Passed Passed ExamExam

Not Not CertifieCertifie

dd

MatMath h SATSAT

VerbVerbal al

SATSAT

college college competitivenesscompetitiveness

MosMostt

SomSomee

LesLesss

NoNott

-0.068-0.068 0.460.46 0.730.73 355355 440440 0.040.04 0.070.07 0.550.550.30.3

55

-0.032-0.032 0.660.66 0.140.14 414414 467467 0.050.05 0.070.07 0.540.540.30.3

44

-0.010-0.010 0.780.78 0.080.08 423423 462462 0.090.09 0.130.13 0.440.440.30.3

44

0.0100.010 0.850.85 0.030.03 450450 470470 0.160.16 0.200.20 0.370.370.20.2

77

0.0450.045 0.910.91 0.010.01 512512 474474 0.250.25 0.250.25 0.350.350.10.1

55

Meaningful difference based only on attributes, though monitoring, development and selective retention also needed

Teacher SelectionTeacher Selection

Teach for AmericaTeach for America– North CarolinaNorth Carolina

– Secondary schoolSecondary school

– Mainly math and scienceMainly math and science

TFA Findings – high TFA Findings – high schoolschool

Student FE, Math subjects

TFA v. all TFA v. all othersothers

TFA v. in-TFA v. in-field non-field non-

TFATFA

TFA v. TFA v. traditional traditional

tracktrack

TFATFA 0.110.11 0.100.10 0.080.08

ExperienceExperience

3-5 yrs3-5 yrs 0.050.05 0.060.06 0.030.03

6-10 yrs6-10 yrs 0.050.05 0.060.06 0.020.02

> 10 yrs> 10 yrs 0.050.05 0.050.05 0.030.03

All TFA coefficients are significant at the .05 level.

Teacher MobilityTeacher Mobility

Mobility highest at most Mobility highest at most challenging schoolschallenging schools

The worst teachers are the first to The worst teachers are the first to leaveleave

General tendency to move to General tendency to move to more affluent schoolsmore affluent schools

Topic of the Day: Topic of the Day: Performance Performance

IncentivesIncentives Objective??Objective??

– Recruitment/ selectionRecruitment/ selection– Retention/ deselectionRetention/ deselection– Increase performance thru effortIncrease performance thru effort

IssuesIssues

How good are the measures?How good are the measures?

Individual vs school rewards?Individual vs school rewards?

Teachers without test scores?Teachers without test scores?

VA MeasuresVA Measures

ProblemsProblems– Year to year variabilityYear to year variability– Measurement errorMeasurement error– SortingSorting

How serious?How serious?– Less serious for policy researchLess serious for policy research– More serious for individual stakesMore serious for individual stakes

Predicting Predicting PerformancePerformance Using first 2 yrs of performance – top Using first 2 yrs of performance – top

to top/ bottom to bottom quintileto top/ bottom to bottom quintile

– Goldhaber and Hansen (NC): 46%/ 44%Goldhaber and Hansen (NC): 46%/ 44%– Koedel and Betts (SanDiego): 29%/ 35%Koedel and Betts (SanDiego): 29%/ 35%– Sass (Florida): 22-32%/ 24-24%Sass (Florida): 22-32%/ 24-24%

Policy ImplicationsPolicy Implications

Use VA freely for researchUse VA freely for research

Use VA carefully for individual teacher Use VA carefully for individual teacher judgmentsjudgments– Important informationImportant information– CorrorborationCorrorboration

More years are betterMore years are better– Move tenure decision out!Move tenure decision out!

Research QuestionsResearch Questions

Are teachers in high poverty schools more/ Are teachers in high poverty schools more/ less effective (value-added) than teachers in less effective (value-added) than teachers in lower poverty schools?lower poverty schools?

Do Do school factorsschool factors affect differences in the affect differences in the value-added of high poverty and lower value-added of high poverty and lower poverty teachers?poverty teachers?

Do Do teacher qualificationsteacher qualifications affect differences in affect differences in the value-added of high poverty and lower the value-added of high poverty and lower poverty teachers? poverty teachers?

DataData

Florida (2000/01- 2004/05)Florida (2000/01- 2004/05)– ElementaryElementary– Student achievement – FCAT-SSSStudent achievement – FCAT-SSS

Grades 3-10Grades 3-10– Teacher linksTeacher links

Assignment, certification, experience, educationAssignment, certification, experience, education

North Carolina (2000/1-2004/5)North Carolina (2000/1-2004/5)– ElementaryElementary– Student achievementStudent achievement

EOG – grades 3-8EOG – grades 3-8 EOC – secondary subjectsEOC – secondary subjects

– Teacher linked through proctor and verificationTeacher linked through proctor and verification Assignment, certification, experience, educationAssignment, certification, experience, education

DefinitionsDefinitions

High povertyHigh poverty elementary elementary schools (>70% FRL students) schools (>70% FRL students)

Lower povertyLower poverty elementary elementary schools (<70% FRL students) schools (<70% FRL students)

Very low povertyVery low poverty schools schools (<30% FRL students). (<30% FRL students).

NC Student-Teacher LinkNC Student-Teacher Link

EOC student-level records

Aggregate to EOC test classrooms by school,

year, subject, proctor id

Instructional Classroom records including school, year,

subject, a teacher id.

Decision Rules

Match if teacher and proctor id identical and  fit statistic < 1.5.

Sample RestrictionsSample Restrictions

Exclude charter schoolsExclude charter schools

Exclude schools that switch high poverty Exclude schools that switch high poverty to lower poverty statusto lower poverty status

Only classrooms w/ 10-40 studentsOnly classrooms w/ 10-40 students

Only self-contained elementary Only self-contained elementary classroomsclassrooms

Analytic SampleAnalytic Sample

0-30% 0-30% FRLFRL

30-70% 30-70% FRLFRL

70-100% 70-100% FRLFRL TotalTotal

FloridaFlorida(Elementary School (Elementary School Level)Level)

3, 0843, 084 6, 9756, 975 5,2325,232 14, 05214, 052

North CarolinaNorth Carolina(Elementary School (Elementary School Level)Level)

2,2072,207 5, 9455, 945 2, 3162, 316 9,2129,212

Note: We focus on elementary schools, grades 3-5 where poverty information is most reliable. We exclude teachers from charter schools and we exclude classrooms with <10 students or >40 students in our samples.

Methodological ChallengesMethodological Challenges

Non-random sorting of teachers and Non-random sorting of teachers and studentsstudents

Distinguishing teacher and school effectsDistinguishing teacher and school effects

Precision in Teacher Effects EstimatesPrecision in Teacher Effects Estimates

Sources of Teacher Effectiveness Sources of Teacher Effectiveness DifferentialsDifferentials

Descriptive Findings:Descriptive Findings:Elementary Student DemographicsElementary Student Demographics

Descriptive Findings: Descriptive Findings: Student PerformanceStudent Performance

FloridaFlorida North CarolinaNorth Carolina

0-30% 0-30% FRLFRL

30-70% 30-70% FRLFRL

70-100% 70-100% FRLFRL

0-30% 0-30% FRLFRL

30-70% 30-70% FRLFRL

70-100% 70-100% FRLFRL

Level Scores:Level Scores:

MathMath 0.49**0.49** 0.25**0.25** -0.16-0.16 0.43**0.43** 0.15**0.15** -0.32-0.32

ReadingReading 0.50**0.50** 0.26**0.26** -0.18-0.18 0.39**0.39** 0.14**0.14** -0.34-0.34

Gain Scores:Gain Scores:

MathMath -0.02**-0.02** -0.01**-0.01** 0.020.02 0.02*0.02* 0.010.01 0.020.02

ReadingReading -0.01-0.01 -0.01*-0.01* -0.01-0.01 0.02**0.02** 0.01**0.01** 0.000.00

* Differences between the given estimate and the corresponding estimates for schools with 70-100% FRL students significant at ≤ 5% and ** differences significant at ≤ 1%.

Descriptive Findings:Descriptive Findings:Teacher ExperienceTeacher Experience

Descriptive Findings:Descriptive Findings:Teacher QualificationsTeacher Qualifications

Distribution of Value-Added of Distribution of Value-Added of Elementary Reading Teachers in Elementary Reading Teachers in Lower Poverty SchoolsLower Poverty Schools

0.5

11

.52

2.5

Den

sity

-1 -.5 0 .5 1 1.5Value-added score

kernel = epanechnikov, bandwidth = 0.0517

0-70% Poverty SchoolsDistribution of Value-Add of Elementary Reading Teachers

Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience

Distribution of Value-Added of Distribution of Value-Added of Elementary Reading Teachers in Elementary Reading Teachers in High Poverty SchoolsHigh Poverty Schools

0.5

11

.52

2.5

Den

sity

-1 -.5 0 .5 1Value-added score

kernel = epanechnikov, bandwidth = 0.0709

>=70% Poverty SchoolsDistribution of Value-Add of Elementary Reading Teachers

Solid line: Novice teachersDash line: Teachers with 1-2 years of experienceDotted line: Teachers with 3-5 years of experience

FloridaFlorida North CarolinaNorth Carolina

Difference Difference DifferenceDifference

Math:Math:

FE Estimates FE Estimates -- ++

Student Covariate EstimatesStudent Covariate Estimates -- ++

Reading:Reading:

FE Estimates FE Estimates ++ ++

Student Covariate EstimatesStudent Covariate Estimates ++ ++

Differences in Estimates of Differences in Estimates of Teacher Value-AddedTeacher Value-Added

Magnitude of Differences in Magnitude of Differences in Value Added EstimatesValue Added Estimates

Differences in Standard Differences in Standard Deviations of Value-AddedDeviations of Value-Added

FloridaFlorida North CarolinaNorth Carolina

Difference Difference DifferenceDifference

Math:Math:

FE Estimates FE Estimates -- --

Student Covariate EstimatesStudent Covariate Estimates -- --

Reading:Reading:

FE Estimates FE Estimates -- --

Student Covariate EstimatesStudent Covariate Estimates -- --

Differences between Lower- and Differences between Lower- and High-Poverty by Percentile of High-Poverty by Percentile of Teacher Value AddedTeacher Value Added

Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (North Carolina- Reading)Levels (North Carolina- Reading)

North Carolina- Elementary Reading

-0.4000

-0.3000

-0.2000

-0.1000

0.0000

0.1000

0.2000

0.3000

10 25 50 75 90

Percentile

Teacher Performance Percentile

Val

ue-

add

ed s

core

0-70% FRL

70-100% FRL

Teacher Value-Added at Teacher Value-Added at Percentiles by School Poverty Percentiles by School Poverty Levels (Florida- Reading)Levels (Florida- Reading)

Florida- Elementary Reading

-0.4000

-0.3000

-0.2000

-0.1000

0.0000

0.1000

0.2000

0.3000

10 25 50 75 90

Percentile

Teacher Performance Percentile

Val

ue-

add

ed s

core

0-70% FRL

70-100% FRL

Correlation of Teacher Correlation of Teacher Qualifications and Value-Qualifications and Value-AddedAdded

-0.0200

0.0000

0.0200

0.0400

0.0600

0.0800

0.1000

1-2

yrs

3-5

yrs

6-12

13-2

0

21-2

7

28+

yrs

Gra

d.

Reg

.

1-2

yrs

3-5

yrs

6-12

13-2

0

21-2

7

28+

yrs

Gra

d.

Reg

.

1-2

yrs

3-5

yrs

6-12

13-2

0

21-2

7

28+

yrs

Gra

d.

Reg

.

1-2

yrs

3-5

yrs

6-12

13-2

0

21-2

7

28+

yrs

Gra

d.

Reg

.

Florida North Carolina Florida North Carolina

Math Reading

Co

effi

cien

t

0-70% FRL

70-100 % FRL

Sources of Difference in Teacher Value-Sources of Difference in Teacher Value-Added Between High-Poverty and Added Between High-Poverty and Lower-Poverty Elementary SchoolsLower-Poverty Elementary Schools

-0.005

0.005

0.015

0.025

0.035

Florida NorthCarolina

Florida NorthCarolina

Math Reading

Dif

fere

nc

e in

Te

ac

he

r V

alu

e A

dd

ed

Difference due to differencesin characteristics

Difference due to differencesin marginal effect ofcharacteristics

Difference due to interactionof characteristics andmarginal effects

Sensitivity AnalysisSensitivity Analysis

School EffectSchool Effect

Empirical Bayes AdjustmentEmpirical Bayes Adjustment

ConclusionsConclusions

Teachers in high poverty schools, on Teachers in high poverty schools, on average, are less effective than average, are less effective than teachers in lower poverty schools.teachers in lower poverty schools.– Changing schools (high poverty/lower Changing schools (high poverty/lower

poverty) does not affect teacher poverty) does not affect teacher effectivenesseffectiveness

There is greater teacher variation There is greater teacher variation within high poverty schools than within high poverty schools than within lower poverty schools.within lower poverty schools.

Conclusions (con’t)Conclusions (con’t)

Differences in teachers in High Differences in teachers in High Poverty and Lower Poverty Poverty and Lower Poverty schools:schools:– only weakly related to teacher qualificationsonly weakly related to teacher qualifications

– more strongly related to marginal effect of more strongly related to marginal effect of qualifications (experience)qualifications (experience)

– not explained by school poverty levelnot explained by school poverty level

Study LimitationsStudy Limitations

Issues with value-added measuresIssues with value-added measures– separating current teacher contributions separating current teacher contributions

from other current contributionsfrom other current contributions E.g., current family circumstancesE.g., current family circumstances

- - dynamic sortingdynamic sorting sorting on time variant characteristicssorting on time variant characteristics

– Instability of measuresInstability of measures E.g., measurement error, motivationE.g., measurement error, motivation