Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under NCLB

Randall RebackBarnard College, Columbia University

Jonah E. RockoffColumbia Business School

Heather L. SchwartzRAND

Excellent research assistant provided by Tamara Lalovic Cox & Elizabeth Davidson

Funding provided by the Spencer Foundation and the U.S. Institute of Education Sciences

October, 2012

Policy Background: No Child Left Behind

● First took effect during spring of 2003● Requires states to adopt school accountability systems that

determine whether public schools satisfy Adequate Yearly Progress (AYP)

● Schools’ AYP status based on – % of students taking and demonstrating proficiency on

statewide exams in math and reading– both overall and student subgroups pass rates– state-specific exams, standards, and rules

● Consequences of failing AYP– Escalating sanctions, including

• Inter-district public school choice • Funds redirected to after-school tutoring for students from

low income families– States are also required to publish annual school report cards,

so schools’ AYP status can also affect school prestige and local property values.

2

Overview of Our Study

● Assemble NCLB related data and outcomes for all schools nationwide for first 2 years of NCLB

● Find schools that were at substantial risk of failing AYP and therefore faced pressure under NCLB–Considerable variation across states

● Use several external data sets to investigate the impact of NCLB pressure on teachers and students

3

Preview of Main Results

● Teacher-level results: Accountability pressure– increases teachers’ concerns about their job security and

decreases their expected career length– decreases frequency of instruction in low-stakes subjects

(e.g., science) and decreases time on whole-class instruction

● Student-level results In schools facing the strongest short term accountability incentives, students…

• score higher on low-stakes readings exams• perform at least as well on low-stakes math and science

exams• do not experience any negative effects in terms of their

enjoyment of math & reading or their anxiety about testing

4

NCLB Policy Variation

Interaction of four features significantly influences the likelihood that a school fails AYP:

(1) state rules for the numerical significance of student subgroups

(2) within-school heterogeneity, which influences how many student subgroups are numerically significant

(3) the generosity of the state’s confidence intervals

(4) the generosity of the state’s safe harbor provisions

Several other idiosyncratic policies also matter

Methodology (Part 1)

● Predicting which schools were on the margin of making AYP– Use 2002 data on test scores and demographics (pre-

policy) to predict probability of making AYP in 2003 and 2004 using separate probit regressions for each state

● Define schools as on the AYP margin if……at least one group moderate probability of passing

…no group has a very low probability of passing

● Define schools as below the AYP margin if any group has a very low probability of passing

6

Difference-in-differences, An Example

● Take two pairs of schools, from NJ and PA

● Even though pairs are observably similar, differences in state rules create arguably exogenous variation in NCLB pressure

7

School name StateTitle I

eligiblePercent

poorPercent

whitePercent black

Percent Hispanic Size

Hamilton School PA yes 89% 5% 83% 10% 454

Richland Elementary PA no 18% 94% 1% 3% 472

School name StateTitle I

eligiblePercent

poorPercent

whitePercent black

Percent Hispanic Size

On margin of failing AYP?

Hamilton School PA yes 89% 5% 83% 10% 454 no

Richland Elementary PA no 18% 94% 1% 3% 472 no

Bradley Elementary NJ yes 87% 4% 81% 15% 418 yes

Upper Pittsgrove NJ no 16% 93% 4% 3% 419 no

School name StateTitle I

eligiblePercent

poorPercent

whitePercent black

Percent Hispanic Size

Hamilton School PA yes 89% 5% 83% 10% 454

Richland Elementary PA no 18% 94% 1% 3% 472

Bradley Elementary NJ yes 87% 4% 81% 15% 418

Upper Pittsgrove NJ no 16% 93% 4% 3% 419

Difference-in-differences

Regressions control for

– state fixed effects– various school-,child-,household- level

controls• including student-level and school-level

pre-NCLB test performance– simulated % of states where school would be

on AYP margin – simulated % of states would be below margin

8

Evidence from RAND-NCLB Survey of Math Teachers

9

0 10 20 30 40 50 60 70 80 90 100

I focus more effort on students who are close to proficient

I spend more time teaching general test-taking strategies

I focus more on topics emphasized in the state test

I focus more on the state standards

I spend more time teaching content

I search for more effective teaching methods

% Agreement Among Math Teachers

High (N=1074) Marginal (N=224) Low (N=19)

Evidence from RAND-NCLB Survey of Principals

10

0 10 20 30 40 50 60 70 80 90 100

Distributed commercial test preparationmaterials

Distributed released copies of the state testor test items

Encouraged or required teachers to spendmore time on tested subjects and less time

on other subjects

My school can attain the AYP targets for2003-04

My school can attain the AYP targets forthe next five years

My school’s AYP status accurately reflectsthe overall performance of our school

% Agreement Among Principals

Marginal (N=21) High (N=104)

Data Used in Second Stage Analysis

● Early Childhood Longitudinal Survey (ECLS): students in the kindergarten cohort of 1998-99, followed up at 1st, 3rd, 5th (and 8th) grade– Nationally representative, but not from all 50 states– Rich data set with low stakes exams. Used adaptive

testing to capture wide range of abilities.

● SASS 2003-04: nationally representative sample of schools and teachers within schools

● Restricted-use data allow use to link ECLS/SASS observations with variables on NCLB pressure

11

Effects of NCLB on Teachers

12

Job Security Concerns for

Relatively Inexperienced

Teachers

Plan to Teach until

Retirement

Work Hours in a Typical Week for

“Generalist” Teachers

Work Hours in a Typical Week

for “Specialist” Teachers

Main Sample: NCLB Sample Wave

On AYP Margin 0.101* -0.057* -1.84** 4.15**(0.057) (0.033) (0.84) (1.81)

Below AYP Margin

0.170** -0.154*** -2.65** 4.89**

(0.076) (0.048) (1.13) (2.31)

Falsification Sample: Pre-NCLB Sample WaveOn AYP Margin -.071 0.039 0.28 0.63

(.068) (0.036) (0.66) (2.16)Below AYP

Margin-.025 0.059 0.79 -0.40(.093) (0.049) (0.86) (2.73)

Effects of School being on AYP Margin on Student Learning

13

Reading Score Math Score Science Score Anxiety About

Testing

   

           0.08 ** 0.00 0.04 -.09***

  (0.04)     (0.04)   (0.03)  (.04)

Reading Enjoyment Math Enjoyment

0.03 0.13 *

(0.07) (0.08)

ECLS: Other Findings

● Effects are not very different for– Students in the subgroups under pressure– “Bubble students”– Students from low-income families

● Stronger positive effects for math score gains in states without strong accountability prior to NCLB (related to Dee and Jacob, 2011)

14

Conclusions

● States vary widely in rates of making AYP– Cross-state variation in student academic

aptitude or in exam difficulty explains relatively little of this variation

● Short term NCLB pressure…– Threatens perceived job security of teachers– Influences teachers’ time use – Has positive net effects on average student test

score growth on low-stakes reading exams– Does not have any short-term negative effects

on outcomes for the average student!

15

No Data Left Behind!

Our NCLB data is publicly available from our “No Data Left Behind” website:

http://www7.gsb.columbia.edu/nclb/

16

17

First stage: Pr(Made AYP in ’03 & ’04)

● 1 in 5 marginal; 1 in 10 low probability● Large differences in actual AYP outcomes

18

● Large variation across groups in contributions to risk of AYP failures

Table 3: First Stage Predictions of AYP Outcomes

Panel A: School-wide Outcomes On

Percent of Schools 22.9%

Above Below

10.1% 67.0%

Table 3: First Stage Predictions of AYP Outcomes

Panel A: School-wide Outcomes On

Percent of Schools 22.9%

Percent Actually Made AYP 2003 and 200436.4%

Below

10.1%

6.9% 86.0%

67.0%

Above

Panel B: Subgroup Outcomes

Math ReadingOverall School Population 92.5% 7.5% 9.3%

Actually made AYP in subject in '03 and '04 52.0% 52.7%Economically Disadvantaged 61.5% 14.9% 18.5%

Actually made AYP in subject in '03 and '04 54.2% 53.7%Limited English Proficient 24.0% 19.4% 37.6%

Actually made AYP in subject in '03 and '04 58.3% 49.8%Disabled 32.0% 31.0% 33.3%

Actually made AYP in subject in '03 and '04 51.0% 51.7%White 71.8% 1.4% 0.9%

Actually made AYP in subject in '03 and '04 56.7% 61.4%Black 33.9% 27.4% 24.9%

Actually made AYP in subject in '03 and '04 51.6% 53.5%Hispanic 30.6% 11.5% 19.9%

Actually made AYP in subject in '03 and '04 55.8% 54.7%Asian/Pacific Islander/Filipino 11.7% 5.6% 11.2%

Actually made AYP in subject in '03 and '04 39.2% 47.3%Native American 5.2% 11.0% 12.9%

Actually made AYP in subject in '03 and '04 53.23% 44.74%

Predicted Marginal ChanceConditional on Numerical Significance

Numerically Significant Subgroup

Panel B: Subgroup Outcomes

Math Reading Math ReadingOverall School Population 92.5% 7.5% 9.3% 2.2% 2.6%

Actually made AYP in subject in '03 and '04 52.0% 52.7% 11.0% 8.6%Economically Disadvantaged 61.5% 14.9% 18.5% 3.7% 4.6%

Actually made AYP in subject in '03 and '04 54.2% 53.7% 12.6% 13.1%Limited English Proficient 24.0% 19.4% 37.6% 4.9% 10.5%

Actually made AYP in subject in '03 and '04 58.3% 49.8% 14.2% 19.2%Disabled 32.0% 31.0% 33.3% 14.6% 17.6%

Actually made AYP in subject in '03 and '04 51.0% 51.7% 14.5% 12.7%White 71.8% 1.4% 0.9% 0.1% 0.0%

Actually made AYP in subject in '03 and '04 56.7% 61.4% 15.8% 33.3%Black 33.9% 27.4% 24.9% 9.8% 7.8%

Actually made AYP in subject in '03 and '04 51.6% 53.5% 16.5% 14.6%Hispanic 30.6% 11.5% 19.9% 1.3% 2.7%

Actually made AYP in subject in '03 and '04 55.8% 54.7% 12.3% 15.8%Asian/Pacific Islander/Filipino 11.7% 5.6% 11.2% 6.6% 9.1%

Actually made AYP in subject in '03 and '04 39.2% 47.3% 43.6% 11.5%Native American 5.2% 11.0% 12.9% 11.3% 10.4%

Actually made AYP in subject in '03 and '04 53.23% 44.74% 8.06% 14.75%

Predicted Marginal Chance Predicted Low ChanceConditional on Numerical Significance

Numerically Significant Subgroup

Panel B: Subgroup Outcomes

Math Reading Math ReadingOverall School Population 92.5% 7.5% 9.3% 2.2% 2.6%

Actually made AYP in '03 and '04 52.0% 52.7% 11.0% 8.6%Economically Disadvantaged 61.5% 14.9% 18.5% 3.7% 4.6%

Actually made AYP in '03 and '04 54.2% 53.7% 12.6% 13.1%Limited English Proficient 24.0% 19.4% 37.6% 4.9% 10.5%

Actually made AYP in '03 and '04 58.3% 49.8% 14.2% 19.2%Disabled 32.0% 31.0% 33.3% 14.6% 17.6%

Actually made AYP in '03 and '04 51.0% 51.7% 14.5% 12.7%White 71.8% 1.4% 0.9% 0.1% 0.0%

Actually made AYP in '03 and '04 56.7% 61.4% 15.8% 33.3%Black 33.9% 27.4% 24.9% 9.8% 7.8%

Actually made AYP in '03 and '04 51.6% 53.5% 16.5% 14.6%Hispanic 30.6% 11.5% 19.9% 1.3% 2.7%

Actually made AYP in '03 and '04 55.8% 54.7% 12.3% 15.8%Asian/Pacific Islander/Filipino 11.7% 5.6% 11.2% 6.6% 9.1%

Actually made AYP in '03 and '04 39.2% 47.3% 43.6% 11.5%Native American 5.2% 11.0% 12.9% 11.3% 10.4%

Actually made AYP in '03 and '04 53.23% 44.74% 8.06% 14.75%

Predicted Marginal Chance Predicted Low ChanceConditional on Numerical Significance

Numerically Significant Subgroup

Which Children Have Been Left Behind?

19

Failed AYP Made AYP Failed AYP Made AYP

Total Number of Schools 19,483 65,332 19,483 65,332Average Enrollment 727 500 1,216 800

Student/Teacher Ratio 17.4 16.1 18.4 17.2Percent of Schools… School Title I Eligible 39.9% 29.4% 34.8% 27.6%

Located in City 36.5% 23.1% 39.2% 27.7%

Located in Urban Fringe 31.9% 33.3% 37.8% 41.1%

Located in Town or Rural Area 31.5% 43.5% 23.0% 31.2%

Serving Primary Grades 36.0% 64.1% 27.1% 56.3%

Serving Middle Grades 27.4% 15.5% 29.4% 17.0%

Serving High School Grades 27.9% 17.1% 39.1% 24.5%

Ungraded/Other 8.5% 3.2% 4.4% 2.2%Percent of Students… Eligible for Free/Reduced Lunch 54.2% 39.9% 49.3% 37.0%

White 46.5% 67.7% 46.2% 63.6%

Black 25.6% 13.2% 24.1% 14.1%

Hispanic 20.8% 13.6% 23.7% 16.3%

Asian 3.4% 3.4% 4.3% 4.5%Note: Includes data from 50 states. Data on school and student characteristics taken from the Common Core of Data, 2003-04. For schools in Tennessee, data on student ethnicity taken from 1998-99 and data on free/reduced price lunch eligibility is unavailable.

Weighted by EnrollmentUnweighted