School-level Correlates of Achievement: Linking NAEP,

State Assessments, and SASS

NAEP State Analysis Project

Sami Kitmitto

CCSSO National Conference on Large-Scale Assessment

June 2006

Create a valuable data set for policy analysis by adding achievement scores to a comprehensive school survey

School and Staffing Survey (SASS) Extensive information from a national survey of

schools, but no achievement scores National Assessment of Educational Progress (NAEP)

Nationally representative scores comparable between states

State Assessment Database (NLSLSASD) Collection of all available school-level state

assessment data Scores comparable within states

Overview of the Study Overview of the Study

What are the important school characteristics that correlate with achievement?

Do the results of Don McLaughlin and Gili Drori (2000) compare to the results from a larger and more recent set of data? 2000 SASS vs. 1994 SASS

36-38 states vs. 20 states

Research QuestionsResearch Questions

NAEP 1998, 2000 and 2002

Used 2000 Math Grades 4 & 8 and 1998 & 2002 Reading scores for Grades 4 & 8

Used full population estimates

Mean and standard deviation at the school level

Mean and standard deviation at the state level

Replicate weights used

Data AssemblyData AssemblyNAEP DataNAEP Data

NLSLSASD 2000

Selected two scores for each grade/subject:

Grade 4 Math, Grade 4 Reading

Grade 8 Math, Grade 8 Reading

Remove between state variation

Create standard score within each state:

Data AssemblyData AssemblyNLSLSASD 2000 DataNLSLSASD 2000 Data

s

sisis StateStd

StateMeanSchoolMeanX

NAEP and NLSLSASD Correlation

Using only schools in both NAEP and NLSLSASD:

Calculated correlation between NAEP and NLSLSASD scores at the state level for matched schools

Data AssemblyData AssemblyNAEP and NLSLSASD School-LevelNAEP and NLSLSASD School-Level

Used NAEP to introduce between state differences and variation to standardized scores

Rescaled to mean of 50 and standard deviation of 10

Data AssemblyData AssemblyNAEP State-Level and NLSLSASDNAEP State-Level and NLSLSASD

sssisis NaepMeanNaepSaCorrNaepStdXY )(

)(

10))((50

YStd

YMeanYACHIEVE is

is

School Level Information

From school, principal, teacher and district surveys

Social Background

Organizational Characteristics

School Behavioral Climate

Teacher Characteristics

Data Preparation Step 2Data Preparation Step 2SASS 2000SASS 2000

Analysis Sample Dropped schools with less than 50 students

Did not include schools that were combinations of elementary, middles and or high schools

Missing values: list-wise deletion of observations

Teacher Qualifications Dropped Teacher sample is not random or representative at

the school level

High percent of variation was within schools not between schools

Results indicated that these measures were mostly noise

Data Set Used for AnalysisData Set Used for Analysis

Number of Schools With Two Valid Scores

Number of Schools in Analysis Sample

Data NumbersData Numbers

NAEP/

NLSLSASD SASS Schools

# Schools # Schools # States

Math 34,106 2,287 38 Elementary School Reading 34,099 2,273 37

Math 17,524 1,414 38 Middle School

Reading 15,707 1,333 36

# Schools

Math 1,885 Elementary School Reading 1,883

Math 723 Middle School

Reading 698

Structural Equation Modeling

Similar to multiple regression analysis

Allows for multiple measures of concepts

Models measurement error

Observed variables = Measures

Conceptual factors = Latent Variables

Analysis MethodologyAnalysis Methodology

Path Model Relating Latent Variables

ModelModel

Poverty

Limited English

Proficiency

Race

Normative Cohesion

Student Behavioral

Climate

Class Size

Teacher Influence

Student Academic

Achievement

School Size

Measurement Model

ModelModel

Poverty

Limited English

Proficiency

Race

% Free Lunch Eligible

Poverty a Problem

% LEP

% Non-White

Teacher Attitudes and

Opinions

School Size

Enrollment

Class Size

Average Class Size

Student/ Teacher Ratio

Class Size a Problem

Normative Cohesion

Clear Norms Parcel

Cooperation Parcel

Teacher Influence

Control of Classroom

Parcel

Influence on School Policies Parcel

Student Behavioral

Climate

Climate Problems Parcel #1

Climate Problems Parcel #2

Score #2

Score #1

Student

Academic Achievement

Fit Statistics

Replication ResultsReplication Results

Elementary School Middle School

Math Reading Math Reading

GFI 0.976 0.973 0.973 0.973

AGFI 0.948 0.941 0.942 0.942

RMR 0.029 0.032 0.030 0.030

Chi-Square 381 446 156 149

Chi-Square DF 63 63 63 63

RMSEA Estimate 0.052 0.057 0.045 0.044

90% Lower Limit 0.047 0.052 0.037 0.035

90% Upper Limit 0.057 0.062 0.054 0.053

Bentler's CFI 0.979 0.978 0.986 0.988

Estimated Coefficients for Achievement Equation

Replication Results (cont)Replication Results (cont)

Elementary School Middle School

Math Reading Math Reading

Class Size -0.230 * -0.242 * -0.181 * -0.487 *

School Climate Problems -0.257 * -0.085 -0.743 0.082

Normative Cohesion 0.149 * 0.039 0.153 -0.125

Teacher Influence -0.017 0.022 0.038 0.120

School Size 0.011 -0.052 * 0.042 0.033

Poverty -0.473 * -0.346 * -0.324 -0.415 *

Race -0.184 * -0.347 * -0.106 -0.455 *

Limited English 0.068 * -0.032 0.106 * 0.082 *

R-squared 0.625 0.533 0.738 0.637

Latent variables are scaled to one of their measures

‘Class Size’ is scaled to student/teacher ratio

Coefficients are standardized

A one standard deviation increase in ‘Class Size’ is correlated with a -.23 standard deviation difference in math achievement in elementary schools

Standard deviation of student/teacher ratio in the sample is ~ 4 students/teacher

Mean is 15.5 students/teacher

Interpretation of CoefficientsInterpretation of Coefficients

Reported Estimated Effects of Student/Teacher Ratio and Class Size

Literature on ‘Class Size’Literature on ‘Class Size’

Variable No Effect or Not Significant Small Effect Sizeable Effect

Student/Teacher Ratio

Hanusheck 1986 Prais 1996 Hedges and Greenwald 1996

Eide & Showalter 1998

Todd and Wolpin 2004

Class Size Hoxby 2000 Nye, Hedges & Konstantopoulos 2000

Ferguson 1991

Krueger and Whitmore 2001 Boozer & Rouse 1995

Coates 2003 Krueger 1999

Angrist & Lavy 1999

Fertig & Wright 2005

Add principal responses to school climate questions

Add additional controls: urbanicity, % IEP, magnet school indicator

‘Principal Leadership’

‘Resources’

Per pupil expenditures (district level)

Number of computers

‘Parent Involvement’

Teacher and principal reports of parent involvement being a problem

School programs to involve parents

Avenues for Future ResearchAvenues for Future Research

Linking NAEP, NLSLSASD and SASS provides a powerful national sample of schools matched to achievement scores

SASS provide multiple measures of key conceptual factors

SEM provides a methodology to take advantage of the depth of SASS information

Class size found to be correlated with achievement

In middle schools, more important for reading than math

Results on achievement are similar to McLaughlin and Drori 2000 with improved fit

ConclusionsConclusions