Estimating the Effect of Leaders on Public Sector Productivity: The Case

of School Principals

Eric Hanushek, Steven Rivkin, and Greg Branch

January 2013

Questions

• Does the quality of leadership explain a substantial share of the variation in organization outcomes?

• Does the variation in principal effectiveness differ by the share of low income students in a school?

• Is the pattern of teacher turnover consistent with the notion that raising the quality of teachers constitutes an important mechanism through which principals influence school quality?

• Are “effective” principals more likely to leave high poverty schools?

Methodological challenges

• Difficult to separate contributions of leaders from other factors– Observed characteristics explain little of the

variation in student performance– Typically behaviors cannot be observed or

directly related to outcomes– Semi-parametric analyses infer effectiveness

from contributions to student outcomes• Unlike the case for teachers, principal actions often

affect quality of school in future periods

Growing Literature on Principals

• Early work focused on observed characteristics– Clark, Martorell and Rockoff also consider

experience

• Other work builds on the panel data methods used by Bertrand and Schoar– raise methodological concerns

• Grissom and Loeb; Miller (2009)

Direct estimation of Variance in Principal Effectiveness

• Builds on Rivkin, Hanushek and Kain (2005) research on teacher quality– Variation in achievement increases with a

principal change

• Coelli and Green (2012) use this approach for Canada

Our Approaches

• Use value-added model with school fixed effects to estimate principal fixed effects– Compute variance of principal effectiveness

• Infer variance in principal effectiveness from relationship between year-to-year fluctuations in school value-added and principal turnover– Carefully examining sensitivity to timing

• Examine relationship between estimates of principal quality and changes in teacher quality during a principal’s tenure

• Describe principal quality differences by transition status

• Consider differences by poverty share throughout analysis

UTD Texas Schools Project

• Stacked panels of students and staff• Annual testing• Student demographic characteristics

– Divide schools by student poverty rate

• Information on staff– Role– Experience– school

• Can follow principals and students who switch schools and roles within Texas public schools

Estimation of Variation in Principal Quality-Broad Issues

• Non-random selection of principals and students into schools complicates analysis

• Tenure-quality relationship complex– Length of tenure unlikely to be monotonically related

to effectiveness• Principals likely learn from experience

– Skills– Behavior rewarded in school and district

• Principal effects on school quality likely grow in magnitude over time

• May be positive or negative

First empirical approach

• Principal by spell fixed effects based on first three years at a school

– Regress math score on lagged math score, student demographic variables, and grade by year fixed effects using aggregate data

– bias potentially introduced by unobserved school factors

Alternative Approach

• Ignore issue of tenure and use all spells

• Control for school fixed effects

• Potential bias if there are time-varying school factors not accounted for

• Estimate of variance includes differences due to tenure

Test Measurement Issues

• Random error inflates estimates of variation in principal quality

– Use Bayesian shrinkage estimator to mitigate effects of random error

– Unlike the case with the estimation of teacher quality, it is not a serious problem given adequacy of sample sizes even in small schools

• Tests focus on basic skills, so initial achievement differences may influence translation of principal quality into test score growth

• Create Z scores and re-weight observations such that average achievement in all schools aggregates over the same test distribution in terms of the share of students in each of ten deciles of the pre-test distribution

Table 3. Distribution of Principal by Spell Fixed Effects by Low Income Share

Share low inc

quartile

Standard deviation

10th 25th 75th 90th

all 0.21 -0.29 -0.15 0.11 0.22

lowest 0.16 -0.18 -0.06 0.13 0.22

2nd 0.18 -0.24 -0.14 0.09 0.19

3rd 0.21 -0.30 -0.16 0.10 0.21

highest 0.26 -0.38 -0.24 0.11 0.29

Sensitivity checks

• Shrink and reweight

• School fixed effects included in specification estimated over sample of all schools with multiple principals during period

Alternative, Test-Measurement Error Adjusted Estimates of the Variance in Principal

Effectiveness

Adjustment Neither Shrunk nor Reweighted

Shrunk Reweighted Shrunk and Reweighted

Standard deviation

0.207 0.200 0.270 0.241

Deficiencies of fixed effects

• Unobservables, even if orthogonal to principal quality, inflate variance estimate• These include changes over time in student

cohort quality and district curricula

• Not accounted for with shrinkage

Principal turnover based variance estimates

• Derive variance estimates from relationship between year-to-year changes in school average achievement and principal turnover– If principal quality matters, changes should be

larger in years in which there is a change in principal

– Builds on Rivkin et al (2005) estimates of variance in teacher effectiveness

• Variance in principal effectiveness equals additional year-to-year variation in transition years over non-transition years– Fluctuations between non-transition years

provide valid counterfactual for what would have taken place in transition years in the absence of a change in leadership

– Not valid if there is additional turbulence during transition years (e.g. Ashenfelter dip)

• If it is actually caused by principal quality, differences in school quality should be larger in non-adjacent years due to an increase over time in principal effects– Compare non-adjacent years around

transitions in order to investigate source of additional variation

specification

principal quality () in cohort y, quality of other school factors not under the control of the principal (γ)

Taking Expectation

)()(2)(2)( 22222''' s

ys

ys eEAAE y

syss

ys

yss

Assume cov(principal quality) = var(principal quality) if principal same Assume cov(principal quality) = 0 if principal different

• Regress squared year-to-year difference in school average test score gains on indicator for principal change

• Assumptions to identify within school variance in principal quality from turnover coefficient– Principal turnover orthogonal to other

unobserved changes that affect achievement– Schools draw principals from common

distributions during this period

Within School Covariance• Covariance between principal quality in

adjacent years with same principal in both years equals variance in principal quality

• Covariance between principal quality in adjacent years equals zero in schools that change principals

• Coefficient on the principal turnover indicator equals 2 times variance in principal quality

Sensitivity checks

• Add squared differences in demographic characteristics in some specifications

• Use non-adjacent years in some specifications

Results for entire sample

Timing of Comparison Adjacent year one year in between

Student Demographic and Mobility controls no yes no yes

Different Principal 0.0052 0.0048 0.0058 0.0056Coefficient (3.41) (3.16) (4.35) (4.28)

estimated standard deviation 0.051 0.049 0.054 0.053of principal quality(square root of 0.5*coefficient)

Estimated Standard Deviation by Poverty Quartile

Quartile Lowest 2nd 3rd highest

Adjacent years

0.029 0.037 0.049* 0.067

Non-adjacent years

0.027 0.035 0.057* 0.064

Principal quality and teacher turnover

• Principal may have limited control over entrants– Job security an issue, but can still exert

influence over who remains• Desirability of school for high quality teachers• Decision to move out lower performers

• Focus on effectiveness of exiting teachers and rate of turnover

Figure 1. Teacher Transitions by Principal Effectiveness and School Poverty Rate

0 .1 .2 .3 0 .1 .2 .3

Top

3rd

2nd

Bottom

Top

3rd

2nd

Bottom

Top

3rd

2nd

Bottom

Top

3rd

2nd

Bottom

Lowest Quartile Disadvantaged 2nd Quartile Disadvantaged

3rd Quartile Disadvantaged Highest Quartile Disadvantaged

Change School Change DistrictExit Sample

Quartiles Principle Effectiveness

Estimation

• Argument is that better principals are more likely to “dismiss” least effective teachers

• Data does not link students and teachers– Focus on differences in grade average value-

added within schools

• Grade with lower mean value-added is more likely to have a teacher below the “dismissal” threshold

Campus by year fixed effect regressions

• Regress share of teachers that exit grade g in school s in year y on controls and grade average value added interacted with principal quality quartile indicators

• Control for – student demographics– grade by year fixed effects– School by year fixed effects

• Potential problem of a mechanical relationship– In future plan to sever time periods

• Measure quality with data in 2nd year of spell• Examine link with teacher turnover in subsequent years

Poverty Quartiles all highest

Grade average gain*2nd quartile-0.018 -0.065

principal quality (0.89) (1.79)

Grade average gain*3rd quartile-0.029 -0.025

principal quality (1.35) (0.65)

Grade average gain*4th quartile-0.079 -0.102

principal quality (3.68) (3.16)

Table 8. Coefficients on Principal Quality Quartile-Grade Average Value-Added Interactions Using First Three Years Sample by School Poverty

Principal Transitions and Value Added

• Transitions categorized by new role and destination• New role

– Principal– Other position in school– Central office administrator

• Destination– same school– New school-same district– Central office-same district– New school-New district– Central office-new district– Exit Texas public schools

Probability Principal Remains in Same Position following 3rd Year in a School, by Quartile of Estimated Quality and

School Poverty Rate (<25 yrs ex) Principal Quality Quartile

Lowest 2nd 3rd highest

School Poverty Quartile

Lowest 59% 68% 73% 76%

2nd 52% 70% 81% 72%

3rd 44% 55% 64% 58%

Highest 63% 73% 72% 67%

Probability Principal with Fewer than 25 Years of Experience Becomes Principal in Different School following 3rd Year in a School, by Quartile of Estimated Quality and School Poverty

Rate (total probability of changing position)

Principal Quality Quartile lowest 2nd 3rd highest

School PovertyQuartile

lowest 7%(41) 6%(32) 8%(27) 9%(24)

2nd 5%(48) 8%(30) 3%(19) 12%(28)

3rd 12%(56) 11%(45) 11%(36) 15%(42)

highest 12%(37) 12%(27) 10%(28) 9%(33)

Future Work

• Estimate very flexible model with school by year fixed effects

• Use these estimates to examine whether variance in estimates of principal quality rises with tenure in a school

• Account for Ashenfelter dip in direct estimates of variance in principal quality

• Modify teacher turnover analysis

Summary

• Purposeful sorting and unobserved factors complicate estimates of leadership quality distribution

• We find substantial variation in estimates of principal quality– A one standard deviation increase in principal quality would

increase school average achievement by roughly 0.05 standard deviations (roughly half as much as a one std dev increase in teacher quality

• Least effective principals least likely to remain in a school– Often transition to other schools, particularly from a high

poverty school

• Details– Turnover based estimates ignore any between school

variation in principal quality– Find a higher quality variance in high poverty schools– Direct estimates of principal quality appear to

overstate variance, even in specifications that include school fixed effects

• Evidence is consistent with notion that the management of teacher composition is one channel through which principals influence school quality

Principal turnover

• Least effective principals least likely to remain in a school– Often transition to other schools, particularly from a

high poverty school

• Little or no evidence that the most effective principals are disproportionately likely to leave even high poverty schools