the stanford education data archive: measuring academic ... · how do we pick a reference cohort?...

the stanford education data archive: measuring academic performance and learning rates

in every school in america

sean f. reardon stanford university

november, 2019

© 2019 sean f. reardon

the educational opportunity project


We’re measuring educational opportunity in every community in America


The Stanford Education Data Archive (SEDA)(http://edopportunity.org)

• Based on 350 million standardized test scores• 50 million students• 13,000+ school districts, 70,000+ elementary schools• Grades 3-8•Math & Reading/English Language Arts (ELA)• 2009-2016 (13 cohorts, entering kindergarten in 2000–2012)• By race/ethnicity, gender, and economic disadvantage• All scores placed on a common scale


How did we build this?


• Provide estimates of test score distributions for all school districts on state accountability tests that are:• Comparable across states (and years and grades)• Standardized to interpretable units for both researchers and

practitioners• Publicly releasable

• Use these to construct 3 key statistics for each district/school:• Average test scores (over all grades and years)• Trend in average test scores (across years within grades)• Growth in average test scores (across grades within cohorts)

Goals


Achievement Source Data• EDFacts Data Initiative:• Proficiency count data for each US public school & district• Available by grade-year-subject-subgroup• Available for:• 8 Academic Years: 2008/09 – 2015/16• 6 Grades: 3 – 8• 2 Subjects: Math and Reading/English Language Arts• 350 million test scores from 13,000+ public school districts

• EDFacts data have three shortcomings:• Aggregated (not student longitudinal file)• Coarsened (proficiency counts, not means/standard deviations)• State student longitudinal data would be much better, but not easily

available for all states• Based on >1000 different (non-comparable) tests (tests vary by state,

grade, year, subject)© 2019 sean f. reardon

Solution• Link each test’s proficiency thresholds to common national scale

(NAEP)• Assumes tests measure similar constructs in the aggregate• Validated using TUDA data• Relies (weakly) on interpolating NAEP score distributions in non-tested

grades & years• Estimate mean and standard deviation of scores in each

district/school-grade-year-subject from proficiency counts• Assume scores are normally distributed in each cell (after a monotonic

transformation of the test scale)• Validation analyses show that estimates are robust to this assumption• Mean and standard deviations are on NAEP scale (bc thresholds are)


Constructing SEDA: Overview

1. Link state cut scores to a common national scale (National Assessment of Educational Progress)

2. Estimate means and SDs on the common scale for schools, districts, district-subgroups, and more…

3. Pool the estimates for a summary school measure across grades and years.

4. Interpret the resulting district and school estimates.

2008 2009 2010 2011 2012 2013 2014 2015

Mea

ns

8 280.1 281.4 282.7 280.47654 238.1 239.2 240.4 239.13

SDs

8 37.6 37.1 37.1 37.57654 29.8 29.7 30.3 30.53

How do we compare state scores? The NAEP scale.

10

2008 2009 2010 2011 2012 2013 2014 2015

Mea

ns

8 279.0 280.1 280.8 281.4 282.0 282.7 281.5 280.47654 238.0 238.1 238.6 239.2 239.8 240.4 239.8 239.13

SDs

8 37.7 37.6 37.3 37.1 37.1 37.1 37.3 37.57654 29.8 29.8 29.8 29.7 30.0 30.3 30.4 30.53

11

Interpolating between years, then grades.

2008 2009 2010 2011 2012 2013 2014 2015

Mea

ns

8 279.0 280.1 280.8 281.4 282.0 282.7 281.5 280.47 268.8 269.6 270.2 270.9 271.5 272.1 271.1 270.16 258.5 259.1 259.7 260.3 260.9 261.5 260.6 259.85 248.2 248.6 249.2 249.8 250.4 251.0 250.2 249.44 238.0 238.1 238.6 239.2 239.8 240.4 239.8 239.13 227.7 227.6 228.1 228.6 229.2 229.8 229.3 228.8

SDs

8 37.7 37.6 37.3 37.1 37.1 37.1 37.3 37.57 35.7 35.6 35.4 35.3 35.3 35.4 35.6 35.86 33.8 33.7 33.5 33.4 33.5 33.7 33.8 34.05 31.8 31.8 31.7 31.5 31.8 32.0 32.1 32.34 29.8 29.8 29.8 29.7 30.0 30.3 30.4 30.53 27.9 27.8 27.9 27.9 28.2 28.6 28.7 28.8

Interpolating between years, then grades.

12

Linking state cut scores to the NAEP scale

State A NAEP Distribution (grey) State B NAEP Distribution (black)

Cut 1

Cut 2Cut 3

Cut 4

Repeat this process for every state in every subject-grade-year cell. Now all state cut scores are on the NAEP scale, in each subject-grade-year cell. See Reardon, Kalogrides, Ho (2019) for details

Cut 1Cut 2

Cut 3

2008 2009 2010 2011 2012 2013 2014 2015

Mea

ns

8 282.77 271.56 260.35 249.24 238.13 227.7

SDs

8 37.17 35.36 33.45 31.74 29.83 27.9

How do we pick a reference cohort?

14

The cohort scale (CS): Comparing to the mean and SD of the national 2005 Kindergarten cohort, within each grade

Now, all state cut scores are on a SD-unit scale for the national 2005 Kindergarten cohort, within each grade.

Recap

• Defined a national within-grade, across-years scale, the “CS” scale based on NAEP, for 3rd-8th grades.• Need a method to estimate the distribution (mean and

standard deviation) of achievement test scores relative to the CS scale in each unit across grades, years, and subjects.

• Unit: a single school, district, or district-by-subgroup.• Cell: a single year-grade-subject’s data for a given unit. Can be

up to 8 years * 6 grades * 2 subjects = 96 cells per unit.

15

Source: EDFacts Data Initiative

•Aggregate “coarsened” proficiency count data for each US public school, by grade, subject, year, and subgroup•Available for• 8 Academic Years: 2008/09 – 2015/16• 6 Grades: 3 – 8• 2 Subjects: Math and Reading/English Language Arts

16

“Coarsened” proficiency data

Level 1

Level 5

Level 3Level

2Level 4

farbelowbasic

below basic

basic proficient

advanced

15 12 42 12 19 17

Data

•We have two pieces of information for each cellof each unit that we use to estimate distributions:

18

C1 C2 C3

Level 1 Level 2 Level 3 Level 421 37 33 17

Proficiency counts (for a given grade, year, and subject):

Location of the cut scores separating proficiency categories, relative to CS scale, based on the state-grade-year-subject of the cell:

higher scoreslower scores

Estimating the mean ( ) and standard deviation ( )

19



C1 C2 C3 C1 C2 C3

𝜇Unit A Unit B

𝜎 𝜇 𝜎

The resulting data

• This process yields an estimated mean and standard deviation for each cell (grade-year-subject) in each unit(school, district, or district-subgroup) if sufficient data.• SDs are sometimes constrained within schools or districts.

• Estimates are on the “CS” scale.• Are converted to the GCS or other scales.

• Estimates are then used to estimate pooling parameters (overall averages, grade slopes, etc.) in HLM.• Only pooled data is released for schools; pooled and long for

districts.

20

Is this linking method valid?

21© 2019 sean f. reardon

Three measures of school/district performance

•Average test scores•Trend in average test scores • (across years within grades)

•Growth in average test scores • (across grades within cohorts)

© sean f. reardon, 2017

Average test scores, by grade and year


Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27

Year (spring)Chicago Math Test Scores, 2009-2014

Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27


Average test scores, by grade and year


Average is -0.32 over all grades and years (0.32 SDs below national average)


Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27


Test score trend (across years w/in grades)


Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27


+0.24 from 2009 to 2014

+0.27 from 2009 to 2014



Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27


+0.054 average annual change


Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27



Test score growth (across grades w/in cohorts)

Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27



+0.40 from 3rd to 8thgrade


Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27





+0.16 from 6th to 8thgrade


Grade 2009 2010 2011 2012 2013 20148 -0.38 -0.32 -0.26 -0.22 -0.10 -0.147 -0.42 -0.36 -0.29 -0.23 -0.12 -0.176 -0.42 -0.41 -0.34 -0.29 -0.18 -0.225 -0.48 -0.43 -0.35 -0.31 -0.24 -0.214 -0.48 -0.45 -0.36 -0.29 -0.23 -0.253 -0.54 -0.49 -0.42 -0.41 -0.34 -0.27



+0.08 per grade on average


Estimating Pooled Parameters (average, trend, and growth)• Pooled parameters are estimated using the model:𝑦 = 𝛽 + 𝛽 𝑐𝑜ℎ𝑜𝑟𝑡 − 2007 + 𝛽 𝑔𝑟𝑎𝑑𝑒 − 5.5 +𝛽 𝑀 − .5 + 𝑢 + 𝑒• is the estimated mean for district , year , grade and subject from the

long form data.• Random coefficients model: all coefficients subscripted are multivariate normally

distributed; variance of is used for precision-weighting.

• The estimated pooled average for a district/school is , the mean of the scores in the middle year of the middle cohort (grade 5.5 for 2007 cohort, i.e. 2012.5).• is the estimated growth rate; is the estimated trend for district/school .

32

What do test scores tell us about educational opportunity?


What do average test scores tell us about educational opportunity?• Average scores largely measure educational opportunity. (Individual students’

scores measure both individual characteristics and educational opportunity.)

• Average test scores differences are not solely the result of differences in schools; they are the total result of children’s home, neighborhood, pre-school, after-school, and K-12 schooling experiences.

• They are not measures of intelligence, but of performance (so are affected by what students have been taught and have learned and how motivated they are to perform on standardized tests).

• Test performance is not the only educational outcome we care about; but it is a reasonable proxy for the extent of average opportunity.


Using average test scores to characterize opportunity structures

•Average scores in grade 3 describe opportunities prior to 3rd grade (early childhood experiences and early elementary school experiences)

• The average growth rate of scores from grades 3-8 largely reflects opportunities provided by schools


Are learning rate measures based on aggregated repeated cross-sectional data valid?• School effect estimates• Generally need randomization (lottery)• Only available for a small number of schools

• School average student score growth• A good, but imperfect, proxy for school effects• Require student-level longitudinal data

• School growth in average student scores• Can be estimated from grade-x-year aggregate data • Inaccurate measure of average student score growth if there is

systematic student in- and out-mobility© 2019 sean f. reardon

Are learning rate measures based on aggregated repeated cross-sectional data valid & reliable?•We validate the SEDA school/district growth rate measures

against measures of average student score growth•We use longitudinal data from 3 states (MA, MI, TN)•We assess reliability by examining the volatility of growth

estimates across years


Are learning rate measures based on aggregated repeated cross-sectional data valid?• SEDA-style and longitudinal growth measures are highly

correlated (0.87 for districts, 0.80 for schools)• Discrepancies are generally small or modest for districts, but

can be larger for schools.• The discrepancy has a mean of 0: on average the SEDA-style

growth measures do not over/understate the target measure.• In charter schools, the SEDA-style measures systematically

overstate the longitudinal growth measures.• Growth estimates based on 2 years of data are very unreliable

relative to estimates based on 4 or more years of data.


The value of population-level data


a. Place-specific information

b. Place-by-age specific information


c. Place-by-age-by-subgroup specific information


What we learning about educational opportnuity?


What role does schooling play in educational inequality?

• It depends on where you are:• Schools are equalizing in some places;• In other places, schools exacerbate inequality;• And in some inequality changes little during the schooling

years.• Segregation (economic segregation) is the strongest

predictor of how unequally schools provide opportunities


School poverty and academic performance

• These analyses do not identify segregation mechanisms• They indicate that school poverty is the best proxy for, or is

most proximal to, the operative mechanisms of segregation• Other forms of segregation (residential, racial, between-district)

may operate through differential exposure to school poverty• These results do not imply “peer effects” (though they might):

High-poverty schools may be lower-quality for many reasons:• hard to attract most skilled teachers; • less parental social/political capital, • lower peer achievement may affect curriculum/instruction, etc.)


Educational Opportunity and Socioeconomic Mobility

edopportunity.orgwe’re measuring educational opportunity in every community in America

built by hyperobjekt


Stanford Education Data Archive (SEDA)• Available at http://edopportunity.org

• These data exist thanks to the following people:Ross Santy, Michael Hawes, Marilyn Seastrom, Jennifer Davies (US Dept. of Education)Andrew Ho (Harvard University)Demetra Kalogrides, Kenneth Shores, Ben Shear, Erin Fahle, Richard DiSalvo, Jenny Buontempo, Belen Chavez (Stanford University)

• Funding support fromInstitute of Education SciencesSpencer FoundationWilliam T. Grant FoundationBill and Melinda Gates FoundationOverdeck Family Foundation


the stanford education data archive: measuring academic ... · how do we pick a reference cohort?...

Documents