the impact of no child left behind (nclb) on school achievement and accountability

The Impact of No Child Left Behind (NCLB) on School Achievement and

Accountability.

Glenn MaleykoWayne State University Detroit, MI

October 17, 2011, Dissertation Defense

No Child Left Behind (NCLB)

The No Child Left Behind (NCLB) reform may be the most significant legislation affecting public education that has been enacted by the federal government during the past 35 years (Peterson & West, 2003).

The Implementation of public education legislation has historically and constitutionally been the responsibility of individual states; however NCLB educational policy at the federal level sets mandates for performance standards with required consequences if they are unmet (Hess & Finn, 2004).

Statement of the Problem

The purpose of this study was to evaluate the effectiveness of AYP at measuring school success in order to establish the conditions for school reform at the school level and classroom level.

This included an analysis of both quantitative data and qualitative data.

Research Questions Instruments Data Analysis Tools

Is there a significant correlation between the schools that meet proficiency standards on the NAEP and the schools that make AYP in the sample states? Are there significant differences among the subgroups?

State AYP data in the sample states along with NAEP data from grades 4 a nd 8 in the sample states.

Pearson’s Product-Moment Correlation

Is there a significant correlation between the schools that meet proficiency standards on the NAEP and schools that meet proficiency standards on the state accountability assessments (STAR, MEAP, End of Grade Test, and TAKS) in the sample states? Is there a c ombination of factors that best predicts proficiency status on the NAEP and state accountability assessments?

NAEP restricted school level data in mathematics and reading at the 4th and 8th grade level. State proficiency assessment data from the sample of schools where NAEP data was collected in the sample states for the years 2005 and 2007 available through state databases and websites.

Pearson’s Product-Moment Correlation Logistic Regression

Are the demographic (categorical) characteristics and educational resources significant predictors of the schools that make AYP and fail to make AYP in the sample states?

State AYP school data and demographic data that is available through state database websites and the National Center for Education Statistics (NCES).

Logistic Regression

What impact is AYP having on school improvement initiatives and classroom instruction?

Qualitative research techniques through interviewing principals and teachers in four schools in one of the sample states.

Semi-structured interview

protocol and the triangulation of interview data.

Refer to Handout Packet #1

Sample Population

Quantitative: All schools that took the NAEP assessment in reading and mathematics at the 4th and 8th grade level during the years 2005 and 2007 from the sample states:

1. California 2. Michigan 3. North Carolina 4. Texas

Qualitative: 4 schools from the Quantitative Dataset.

4 teachers and one administrator were selected from each school for the semi-structured interview protocol.

National Assessment of Educational Progress

(NAEP) AssessmentComplicated formula (Plausible Values)This data was difficult to use and it is where I spent the majority of my time with data analysis in this study

A formula was created to develop school level proficiency on the NAEP which is the first of its kind

Refer to Handout Packet #2 for variable definitions.

Pearson Correlation Analysis r value defined

Terminology Approx. RangeMinor (+ or -) .000 to .099Moderate (+ or -) .100 to .399Large, Strong, or Sizable (+ or -) .400 to .549Extremely large or Strong (+ or -) .550 to 1.0

Research Question One Findings

California, Michigan, some of the NC datasets, and the 8th Grade Texas dataset --

--Provided a stronger relationship with AYP and the basic level NAEP scale vs. AYP and the proficient level NAEP scale

Please refer to Handout Packet #3 and #4 for NAEP basic and proficient level definitions.

Michigan 8th Grade 2007 AYP

STATEAYP

NAEP BAS IC

level

NAEP PROF

level

TOTAL

REVENUE EDPER NATA MPER

1 .429** .055** -.155** -.396** .064**

.429** 1 .079** -.261** -.666** .074**

.055** .079** 1 .093** -.202** -.045**

ASIANPER BLACKPER HISPANICPER WHITEPER ELLPER SPECIALEDPER

STATEAYP

NAEP BASIC level

NAEP PROF level

STATEAYP

.139** -.627** .027** .522** -.023** -.057**

NAEP BAS IC level .126** -.717** -.256** .665** -.343** -.256**

NAEP PROF level

.053** -.068** -.055** .082** -.048** -.058**

Please refer to Handout Packet #5

North Carolina and Texas

Most of the North Carolina datasets produced a stronger relationship with AYP and the proficient level NAEP scale vs. the basic Level NAEP scale

The Texas datasets had a very minor relationship with AYP and the NAEP proficiency at the proficient and basic levels

North Carolina 2007 4th Grade Dataset

STATEAYP

NAEP BASIC

level

NAEP PROF

level

TOTAL


1 .336** .418** -.048** -.445** -.124**

.336** 1 .214** .141** -.548** -.394**

.418** .214** 1 .079** -.581** -.106**


STATEAYP

NAEP BASIC level

NAEP PROF level

STATEAYP

.021** -.426** -.215** .475** -.255** -.204**

NAEP BASIC level .103** -.229** -.238** .475** -.242** -.087**

NAEP PROF level

.264** -.375** -.273** .432** -.199** -.103**


Research Question 2 Results:

Michigan, California, and some of the NC datasets:

Produced Pearson Correlation results and Logistic Regression results showing a stronger relationship with the basic level NAEP scale and state accountability

assessments vs. the proficient level NAEP scale and state accountability assessments

Research Question 2 Results

California had the greatest increase from the constant model vs. the predicted model showing that the logistic model had a strong influence

Thus the logistic model was useful in California.

It is important to emphasize that the logistic regression is a more powerful tool of analysis for predicting the probability that a school will meet proficiency status.

Constant Model 2005 California 8th Grade

Predicted

did not meet

Proficiency

met

Proficiency

Observed

.00 1.00

did not meet proficiency

status

0 148293 .0 STATEMATHprofstatus

met proficiency status 0 186618 100.0

Step 1

Overall Percentage 55.7

a. The cut value is .500


Predicted Model 2005 California 8th Grade

Predicted

did not meet

Proficiency

met

Proficiency

Observed

.00 1.00


status

115866 32427 78.1 STATEMATHprofstatus


Step 1





Texas and California had similar sample sizes and population demographics including a higher level of the HISPANICPER variable

However, the Texas results were not very useful and there was a very minor relationship between the state accountability assessments and the NAEP as measured by the Logistic Regression and the Pearson Correlation results


Explanation: An extremely high percentage of schools in Texas met proficiency status resulting in low variability with the dependent variable.

Texas had the weakest relationship with NAEP at both the basic and proficient level among the 4 sample states

This suggests less rigor with the Texas state accountability assessments

Texas 2007 4th Grade Mathematics: Pearson

Results

STATE Math

NAEP BASIC

level Math

NAEP PROF

level Math

TOTAL


1 -.008** .046** -.069** -.109** .052**

-.008** 1 .061** .001 -.001 .069**

.046** .061** 1 .005* -.608** .220**


STATE Math

NAEP BASIC Math

NAEP PROF Math

STATE Math

.047** -.048** -.035** .062** -.015** .043**

NAEP BASIC level .054** -.272** .079** .090** .014** .026**

NAEP PROF level

.506** -.214** -.545** .654** -.441** -.037**


Texas 2007 4th Grade Math Logistic: Constant Model

Predicted

did not meet

Proficiency

met

Proficiency

Observed

.00 1.00


status

0 979 .0 STATEMATHprofstatus


Step 1



Please refer to Handout Packet # 9

Research Question 2 Findings

The California and NC logistic regression results were the most useful in this study

The Pearson results in Michigan were useful showing that that was a strong association with the NAEP scale basic level and the state accountability assessment results

Texas Results were the least useful

Michigan 2005 8th Grade Mathematics

State Math

NAEP BASIC

level Math

NAEP PROF

level Math

TOTAL


1 .640** .150** -.280** -.666** .098**

.640** 1 .194** -.267** -.705** .085**

.150** .194** 1 .166** -.355** -.123**


STATE Math

NAEP BASIC Math

NAEP PROF Math

STATE Math

.187** -.704** -.020** .682** -.137** .128**

NAEP BASIC level .167** -.760** -.091** .749** -.098** .074**

NAEP PROF level

.401** -.177** -.082** .155** -.051** -.082**


Research Question 2 Results: NC

North Carolina produced some of the most intriguing results

DUMMYREV had a positive impact in NC2007 4th and 8th grade mathematics results produced the largest r value in relationship to the proficient level NAEP scale

The Logistic Regression results also showed that there was a strong relationship with the 2007 NC state accountability assessments at the proficient level and the NAEP assessments in mathematics.

2007 NC 4th Grade Math

STATE Math

NAEP BASIC

level Math

NAEP PROF

level Math

TOTAL


1 .079** .596** .186** -.635** -.098**

.079** 1 .091** .016** -.248** -.124**

.596** .091** 1 .127** -.679** -.179**


STATE Math

NAEP BASIC Math

NAEP PROF Math

STATE Math

.156** -.485** -.300** .536** -.190** -.163**

NAEP BASIC level .092** -.193** .058** .177** .044** -.018**

NAEP PROF level

.190** -.441** -.350** .563** -.259** -.112**


2007 North Carolina Math

95% C.I.for EXP(B)

B S.E. Wald df Sig. Exp(B) Lower Upper

DUMMYREV 1.314 .024 2901.049 1 .000 3.721 3.547 3.903

EDPER -9.461 .120 6220.284 1 .000 .000 .000 .000

NATAMPER 15.782 .858 338.117 1 .000 7148837.835 1329370.4

90

3.844E7

ASIANPER 36.833 1.190 957.663 1 .000 9.913E15 9.618E14 1.022E17

BLACKPER 4.540 .937 23.483 1 .000 93.682 14.935 587.631

HISPANICPER 10.850 .921 138.735 1 .000 51541.625 8473.310 313518.44

9

WHITEPER 13.419 .866 240.022 1 .000 672812.255 123200.24

9

3674313.4

48

ELLPER 6.355 .216 862.377 1 .000 575.175 376.362 879.012

SPECIALEDPER -5.825 .188 957.691 1 .000 .003 .002 .004

NAEPMATHPRO

Fbasic

12.064 871.625 .000 1 .989 173541.145 .000 .

Step 1

NAEPMATHPRO

Fprof

1.144 .028 1626.099 1 .000 3.138 2.969 3.318

Constant -24.457 871.625 .001 1 .978 .000


Research Question 2However, NC changed the reading cut scores in 2007, thus there was a very weak relationship with the reading assessments in 2007

Cut score manipulation was one of the findings in the literature review (Guilfoyle, 2006; Harris, 2007; Sunderman, et al. 2005)

Other cut score manipulations, Michigan 2011 example, College Readiness

Research Question 3 Logistic Regression

results were similar to research question 2

results NC and California Logistic Results were the most useful

The Texas logistic regression results were less useful supporting the finding that there was a low level of rigor with the Texas assessments as a high percentage of schools met AYP in Texas

This was an interesting finding in itself as almost all schools in Texas met AYP during the 2005 and 2007 school years

Economically Disadvantaged (EDPER)

variable: All 3 Research questionsThe EDPER variable produced a

consistent negative association with AYP proficiency. The results were generally found to be moderate to sizable

This was the most consistent finding in the study with both the Pearson correlation results and the logistic regression.

California 2007 8th Grade Mathematics

STATE Math

NAEP BASIC

level Math

NAEP PROF

level Math

TOTAL


1 .567** .242** -.299** -.516** .029**

.567** 1 .270** -.300** -.688** .100**

.242** .270** 1 -.192** -.516** -.004*


STATE Math

NAEP BASIC Math

NAEP PROF Math

STATEAYP

.301** -.193** -.463** .465** -.275** -.203**

NAEP BASIC level .408** -.131** -.665** .626** -.543** -.056**

NAEP PROF level

.279** -.145** -.443** .440** -.278** -.129**


NC 2005 8th Grade Mathematics

95% C.I.for EXP(B)

B S.E. Wald df Sig. Exp(B) Lower Upper

DUMMYREV .797 .031 676.152 1 .000 2.220 2.090 2.357

EDPER -6.638 .147 2042.765 1 .000 .001 .001 .002

NATAMPER -15.537 .387 1607.696 1 .000 .000 .000 .000

ASIANPER 44.504 .824 2920.013 1 .000 2.127E19 4.235E18 1.069E20

BLACKPER -11.101 .198 3141.020 1 .000 .000 .000 .000

HISPANICPER -6.538 .331 390.921 1 .000 .001 .001 .003

DUMMYWHITE -.903 .041 475.658 1 .000 .406 .374 .440

ELLPER -10.689 .273 1534.364 1 .000 .000 .000 .000

SPECIALEDPER -2.480 .138 324.600 1 .000 .084 .064 .110

NAEPMATHPROFb

asic

19.292 515.240 .001 1 .970 2.390E8 .000 .

Step 1

NAEPMATHPROFp

rof

15.960 354.704 .002 1 .964 8534996.5

61

.000 .

Constant -6.200 515.240 .000 1 .990 .002


Demographic Variables 3 Research Questions

SPECIALEDPER and ELLPER variables produced inconsistent results: ELL in California a greater negative impact

NATAMPER, BLACKPER, and HISPANICPER variables were not consistent among the different datasets.

However, BLACKPER in Michigan produced a consistent negative influence on the probability that a school met proficiency status

Demographic Variables 3 Research Questions

TOTALREVENUE produced minor to moderate positive associations with State AYP and the accountability assessments

TOTALREVNUE in NC had a strong positive association with the NAEP proficient level scale in comparison to State AYP

Demographic Characteristic

Findings from 3 Research Questions

The ASIANPER and WHITEPER were inconsistent but generally had a positive influence on state accountability results

Research Question 4: School Population:

Each School had different demographics

Blue School, low ED populationRed school high ED populationOrange school high ED and ELL population

Purple School high ED located in an urban setting. Experience with restructuring.



AYP identification and sanctions will not lead to improvement based on participant responses

Sanction provisions lacking scientific evidence of effectiveness. Literature review aligned with participant responses

Ex. Red School principal “Dropping down the hammer will not work” Additional support is needed.


Purple School went through the NCLB restructuring process.

Did not feel restructuring led to improvement.

The removal of one grade provided change and actually gave the school a better statistical chance of making AYP as identified in the literature review.


EDPER variable and Triangulation: The literature review, quantitative dataset and the qualitative data established that this variable had the greatest impact on student achievement as measured by AYP and the standardized assessments.

Blue School principal: One of the only respondents that felt the ED population did not make a difference with AYP

Blue School had less than 3% ED.

Research Question 4 Findings: Social

CapitalSocial Capital InfluenceResearchers (Elmore, 2002; Harris, 2007; Mathis, 2004: Mathis, 2006) contend that AYP is measuring school demographics and the social capital that students bring to the school instead of school effectiveness. If this is the case, then the external validity of the AYP measurement formula is questionable.

Orange School Felt that ELL students had the greatest impact but their ELLPER population matched the EDPER population

Research Question 4 Findings: School

FundingMost participants felt that school funding was key.

Title One funds and additional resources were important when servicing ED students.

However, interesting that Blue School did not feel this way. Funding in their district was lower than others. However, they had a greater level of social capital in the school.

Probable that Social Capital with an Effective SIP was key to their success.

Research Question 4: School Funding a two

tiered variableTOTALREVNUE had a positive impact or a mild negative impact with the quantitative dataset

The qualitative analysis helped to create a finding listing TOTALREVENUE as a two tiered variable

Funding is important but step two is how the funds are used: Effective or not effective

Research Question 4 Findings: Positive

ConsequenceFocus on SIP plan and increased focus on scientific strategies and research based best practices (ex. Differentiated instruction)

Increased Collaboration with faculty

Increased focus on analyzing data and electronic systems

Increased focus on subgroup populations and individual student datasets at the classroom level

Research Question 4: Philosophical Intent vs.

Negative Impact or Unintended Consequences

Increased pressure on administration: Exception Blue school

Additional paperwork and tasks for administration

Stress on teachers and administratorsPossibility that some educators would not want to work in schools labeled as failing.

Merit pay, etc.

Red School principal response

Well, I think it’s narrowing the curriculum. And its…. Taking away some of the choices kids have because they have to, you know, have to do all math. It kind of bothers me because I, you know, I think the purpose of education is to also help a child become more well rounded. But they also need to know math.

Research Question 4: Negative Impact or

Unintended Consequences

Low Morale. Ex. Orange school Implemented Effective SIP based on my triangulation of data but not getting the AYP results. One subgroup had a negative impact

Less focus on higher level thinking skills needed for success after leaving the public school system

Research Question 4 : Internal Capacity

No support at the school level for increased internal capacity and development

Many researchers in the field posit that the AYP formula is faulty because it relies on sanctions and punishments and fails to provide schools with the internal capacity to make change (Abdelmann & Elmore, 1999; Elmore, 2002; Fullan, 2006; Schoen & Fusarelli, 2008).

Practical Implications

Data analysis is a positive outcome of AYP.

There was an increased focus on subgroups.

More support is needed at the building level to support schools with implementing effective SIP plans

Potential to make a great difference via accountability reform.

However: We need to create an effective

measurement system to measure school progress This study shows that the current AYP measurement system is not effective at measuring school success and improvement.

Researchers maintain there is a lack of consistency with AYP that results in reliability issues with using AYP in order to measure school effectiveness (Elmore, 2002; Guilfoyle, 2006; Kane and Douglas, 2002; Linn & Haug, 2002; Porter et. al, 2005; Scheon & Fusarelli, 2008; Wiley et al. 2005).

Theoretical Implications

Single Measure Standardized Accountability systems result in unintended consequences

If a single annual test were the only device a

teacher used to gauge student performance, it would indeed be inadequate. Effective teachers assess their students in various ways during the school year. (Guilfoyle, 2006, p. 13)

Theoretical Implications

The current system has a heavy reliance on standardized assessments (one size fits all measure)

A multiple Measures approach to evaluating school effectiveness would be more effective

Growth Data formulas

Use of the NAEP

National Curriculum and use of NAEP

A move towards a National Curriculum and the Common Core Standards is gaining momentum

As mentioned, This was the first study that could be found which came up with a school level proficiency measure using the NAEP

Summary and Questions

the impact of no child left behind (nclb) on school achievement and accountability

Documents

sample of schools

sample states

naep data

state ayp data

state ayp school data

demographic data

qualitative data

school reform