NAEP 2007– What About NCLB?

Marshall S. Smith

October 10, 2007

Table of Contents

• Methodology

• Summary and Conclusions

• Has NCLB increased the rates of growth (gain) of student achievement? – (1) Compared to the rates for the 5 years

before NCLB started in 2002?– (2) Compared to the rates during the years of

Clinton reforms prior to NCLB?

Methodology (1)• All of the data are from the 2007 NAEP release of Math and

Reading Assessment results on September 25. See http://nces.ed.gov/nationsreportcard/

• NAEP Scale scores are used. One way to interpret is that10-11 points = roughly one grade level

• The data used are from the NAEP national public school sample. The public school sample was chosen because NCLB is a public school intervention program. It does not directly influence private schools, which are in the total sample. There are small differences in test scores between the two samples.

• The NAEP data are always presented disaggregated by ethnic/racial subgroup, specifically white, black and Hispanic. Because the relative sizes of the subgroups change over time and the mean scores of the subgroups are substantially different, the disaggregated scores better approximate schooling effects over time than would total aggregated scores.

Methodology (2)• The logic of the analyses for estimating “effects” of NCLB is simple.

– (1) The tables contain test scores from years before and after the beginning of NCLB in 2002 for 3 subgroups. Changes in scores over time provide information about student gains for each subgroup. If the test scores grow (gain) at the same rate before and after there is no discernable effect of NCLB. If gain rates are greater before 2002, then NCLB might have a negative effect. If gain rates are greater after 2002, then NCLB might have had a positive effect.

– (2) For some analyses the rates of gain of a particular subgroup over time are contrasted with the rates of gain of other subgroups over the same time period.

– (3) I use a difference in rate of gain of at least 0.3 points per year as a benchmark indicating that there might be something real going on. Over a ten year period this would be about 1/3rd of a grade level. Most differences are substantially larger than 0.3 points per year.

– Unless the second decimal point in an estimate of rate was five or was important in an adjustment in the math analyses, I have rounded the results

– Rate of gain in test scores, rate of growth in test scores and rate of change in test scores are used interchangably and mean the same thing. For NAEP, rate of change is probably the best term to use. .

Methodology (3)• Survey assessment data such as NAEP can only provide weak

causal claims since there are many other factors influencing test scores in the 80,000 schools, 14,000 school districts and 50 states of United States.

• The federal government and its laws and regulations do not teach students. The 2.5 million teachers teach and the 60 million students learn in settings far away from Washington DC. Yet the federal government’s laws and regulations and rhetoric can influence the climate and environment of the classroom. When a leader says that NCLB works he/she is saying that NCLB is having such an effect on the climate and environment that it is felt in the classroom. Such an inference is hard to make since the influence has to penetrate through many layers of government and overcome many traditions. Thus, every statement made in these pages that suggests that NCLB is having a positive or negative effect should be taken with many grains of salt, though no more skeptically than those made by others based on the NAEP data.

Summary and Conclusions(1) • The paper uses the NAEP data to explore whether the rate of gain

per year in NAEP test scores before NCLB was passed in 2002 is less, equal or greater, than during the NCLB years. In effect, it explores the issue of whether NCLB is effective in increasing the rate of improvement in student test scores. Two analyses address this question. One uses a period of roughly 5 years before 2002. The second uses a period starting roughly in 1994, the first year of the Clinton school reforms. Each analysis compares the rate of growth before NCLB to the rate from 2002-2007 and is carried out for twelve comparisons: 2 grade levels (4th and 8th) for 2 content areas (reading and math), for 3 subgroups (white, black and Hispanic students). Differences less than 0.3 points per year are considered to be no difference.

Summary and Conclusions (2)• The results are separated for reading and math.

– For reading, for 11 of the 12 comparisons across the two analyses, the rates of gain for test scores are greater before 2002 than after. Fourth grade whites show no difference in rate of gain in analysis 2 while they gain more before 2002 in Analysis 1. No results from either analysis favored the NCLB years.

– This suggests that NCLB has had a consistent negative effect on student rates of gain on the NAEP reading assessments for all groups of both 4th and 8th grade students, except 4th grade whites.

– For math at the fourth grade, in 4 of the six comparisons, the rates of gain is greater before 2002 and in two comparisons there is no difference. At the eighth grade in math, 4 of the six comparisons show no difference, one comparison shows a greater gain before 2002, and one shows a greater gain after 2002.

– The data indicate a modest, negative effect on student growth for NCLB for 4th grade math and no consistent effect of NCLB for 8th grade math.

Has NCLB improved the rate of gain of NAEP test scores compared to the rates in the five years

before NCLB was signed into law in 2002? • Does NCLB add value or productivity in terms of increasing the rate

of increase of achievement test scores? Increasing the rate of increase of test scores, particularly among low income and minority children, is clearly seen as the purpose of the Act by policy makers and journalists around the nation.

• The analyses compare rates of increase before and after NCLLB started. The dates before NCLB were chosen to approximate a period of about 5 years, the amount of time it has been since NCLB. Since NCLB started in 2002 the date for five years before would be 1997. No NAEP was given during 1997 so I used 1996 for Math and 1998 for Reading. Reading assessments were administered in 1992 while math assessments were not. Consequently I used 2003 as the starting date for NCLB for math.

Analysis 1: Has NCLB increased the rate of NAEP Reading Scores

• The next two tables show 4th and 8th grade NAEP reading scores for three subgroups (white, African American and Hispanic American) and for four dates (1998, 2002, 2005, 2007). NCLB was signed into law in 2002 so the starting date for NCLB is 2002. The period 1998-2002 gives us a chance to look at the gains made during the four years before NCLB. The five year period from 2002-2007 is the period during which NCLB has existed. By comparing the rates of gain before and after 2002 we can estimate whether the introduction of NCLB has had a positive or negative or no effect.

Calculating a rate of gain.• First, an Example: Table 1 shows the 4th grade NAEP reading

scores. In the bottom horizontal line labeled Hispanic American, there are four NAEP scores for four different years and two gain scores that contrast the gains before and after 2002. Directly underneath each gain score is a rate of gain measure. For the 4th grade Hispanic students, the gain before 2002 is 7 points over four years (1998-2002) and the gain after 2002 is 5 points over five years (2002-2007). The rates of gain for the time period before 2002 is 7/4 = 1.75 points per year and after 2002 is 5/5 = 1.0 points per year for a difference of 0.75 points per year. This rate differences converts to difference of roughly 70% of a grade level over a decade.

• Now the data -- next!

Table 1: 4th Grade Reading Scale Scores and

gains before and after NCLB was initiated in 2002.

1998 2002 2005 2007 2002-1998

2007-2002

Gain

(Rate)

Gain

(Rate)

White 223 227 228 230 4 (1/yr)

3 (0.6)

African Am

192 198 199 203 6 (1.5)

5 (1)

Hisp

Am

192 199 201 204 7 (1.75)

5 (1.0)

Table 2: 8th Grade Reading Scale Scores and gains before and after NCLB was initiated in 2002.

1998 2002 2005 2007 2002-1998

2007-2002

Gain

(Rate)

Gain

(Rate)

White 268 271 269 270 3 (0.75)

-1 (-0.20)

African Am

242 244 242 244 2 (0.50)

0

(0.0)

Hisp

Am

241 245 245 246 4

(1.0)

1 (0.20)

Analysis 1 Results and Interpretation: What do the two tables indicate about whether NCLB increased

the rates of growth of NAEP reading scores?

• For all three subgroups, for both 4th and 8th graders, the reading scores increased at least 0.3 points per year faster before NCLB, from 1998 to 2002, than after NCLB began, from 2002 to 2007. Many of the gain rate differences are considerably greater than 0.3 points per year. In fact, for blacks in both grades the gain rate differences are 0.5 points per year or roughly ½ grade level per decade and the Hispanic gain rates differ in both grades by 0.75 points per year or more, about 3/4ths of a grade level per decade. These are substantial differences. (See Tables 1-2)

• The data in these tables indicate that NCLB appears to have had a

negative effect on the rates of gain of NAEP reading scores for students in all three groups at both grades.

Analysis 1. Math NCLB 2007• The next two tables show math scores for the three subgroups for

4th and 8th grades. The groups are arrayed against four dates when Math NAEP was administered. The dates that the Math NEAP was administered do not allow as clean a picture of the gains before and after 2002 as the reading dates did because math was not tested in 1998 or 2002. Thus1996 is the starting date for pre-NCLB gains. Since math was not measured in 2002, the next best date is 2003. One analysis compares the rates of gains during those periods (1996-2003 and 2003-2007) using a time period of 7 years for the estimate for before 2002 and 4 years after NCLB.

• Another way of looking at the data is to allocate 1/7 of the gain between 1996 and 2003 and grant it to NCLB since NCLB started in 2002. Thus the gain before NCLB would be 6/7 of the difference between 1996-2003 over a period of six years and the NCLB gain would be the sum of one seventh of the 1996-2003 gain plus the 2003-2007 gain over five years. This alternative approach is called the “adjusted” analysis.

Table 3: 4th Grade Math Scale Scores and gains before and after NCLB was initiated in 2002.

1996 2003 2005 2007 2003-1996

2007-2003

Gain

(Rate)

Gain

(Rate)

White 231 243 246 248 12

(1.7)

5

(1.25)

African Am

198 216 220 222 18

(2.6)

6

(1.5)

Hisp

Am

207 221 225 227 14

(2.0)

6

(1.5)

Table 4: 8th Grade Math Scale Scores and gains before and after NCLB was initiated in 2002.

1996 2003 2005 2007 2003-1996

2007-2003

Gain

(Rate)

Gain

(Rate)

White 279 287 288 290 8

(1.14)

3

(0.75)

African Am

239 252 254 259 13

(1.9)

7

(1.75)

Hisp

Am

249 258 261 264 9

(1.3)

6

(1.5)

Results 1: Math NAEP results for unadjusted and adjusted scores.

• At the 4th grade, for all three groups, the unadjusted and adjusted gain rates for the period before NCLB started in 2002 exceeded the rate of gains after 2002. The unadjusted gain rates are in the table. The adjustment of adding 1/7 of the pre NCLB gain to the NCLB gain and making the time periods six years and five years produced the same general result. The adjusted gain rates for 4th grade for whites are 1.7 before and 1.34 after NCLB started in 2002, for blacks 2.57 before and 1.71 after, and for Hispanics 2.0 before and 1.6 after. In each instance the rate for the gains before 2002 exceeded the rate after 2002 by at least 0.3 points per year.

• At the 8th grade, the unadjusted gains in Table 4 and the adjusted gains show that whites gained slightly faster before 2002 than after. Blacks and Hispanics, for unadjusted and adjusted scores, both show no differences in their rates of gain before and after NCLB started in 2002. In each case for blacks and Hispanics the difference in rates of gain is less than 0.3 points per year.

Analysis I.1. Summary of Evidence about Reading and Math rates of gain for before and after NCLB began in

2002. • The question addressed in charts 1-4 is whether the NCLB

intervention improved the effectiveness of schools in producing achievement gains beyond those made over the five or so years just before NCLB. The two time periods are roughly 1997-2002 and 2002-2007. Because NAEP was not given at exactly the right dates other approximate dates had to be used. The analysis compares the rates of gains of the two time periods for three groups of student for math and reading.

• In all comparisons, for all three subgroups of students, for 4th and 8th grade reading and for 4th grade mathematics unadjusted and adjusted scores, the rates of gain were greater by at least 0.3 points per year before 2002 than after.

• In both the unadjusted and the adjusted comparisons for 8th grade mathematics the rates for whites very slightly favored before 2002 and the rates for blacks and Hispanics showed no differences exceeding 0.3 points per year.

Study 1. Conclusions about the comparisons of rates of gain for roughly five years before NCLB started in 2002 with the rates of gain

during the NCLB years.

• Reading: For 4th grade and 8th grade reading, the NAEP data indicate that NCLB has had a negative effect on the rates of test score gains for all sub-groups for both grades. All student groups for both grades appear to have lost momentum when NCLB was implemented.

• Math: For 4th grade math all three subgroups of students gained less rapidly during the NCLB years than during the earlier years. For 8th grade math the NCLB scores indicate a tiny negative effect on the gain rate for whites and no effect for blacks and Hispanics. In short, NCLB appears to have had a consistent, small negative effect on 4th grade math achievement and no effect on 8th grade math.

Analysis 2. A second approach to estimating effects of NCLB

• Background: • In 1994, President Clinton signed the Improving America’s School Act, the

fore-runner of NCLB. It along with Goals 2000 created a powerful incentive for states to create state standards based reforms, with standards, aligned resources, aligned assessments and state developed accountability systems. The focus of the 1994 law was on improving teaching and learning and fostering professional accountability rather than relying only on top-down performance accountability. The state assessments and accountability provisions were modeled from the Reagan era predecessor law passed in 1988. The challenge for the 1994 law was to get state reforms systems implemented in all 50 states. This required massive changes in many states, which took substantial time – it was not until 1998-2000 that the core elements of standards, assessments and accountability were in place.

• The NCLB, signed by President Bush in 2002, ramped up greatly the number of tests, put into place a stringent federal accountability system on top of the state systems, increased transparency requirements, and required all teachers to be credentialed.

Analysis 2 Introduction: Rates of NAEP gains

during Clinton era compared to NCLB era.

• This next four charts address the question: Is NCLB on average more effective than the prior reform which started under Clinton in 1994 and extended to 2002? Are the rates of growth of NAEP test scores different under the two reforms? The time periods for reading are 1994 to 2002 compared with 2002-2007. In math the comparison periods are 1992-2003 and 2003 to 2007. For math I also look at “adjusted” scores similar to the earlier analyses.

Table 5. 4th Grade Reading Scale Scores comparing rates of gain

over Clinton reform years with rates during NCLB years.

1994 2002 2005 2007 2002-1994

2007-2002

Gain

(Rate)Gain

(Rate)

White 222 227 228 230 5

(0.6)

3

(0.6)

African Am

184 198 199 203 14

(1.75)

5

(1.0)

Hisp

Am

186 199 201 204 13

(1.6)

5

(1.0)

Table 6. 8th Grade Reading Scale Scores: Comparing rates of gain over Clinton reform years with rates during NCLB years.

1994 2002 2005 2007 2002-1994

2007-2002

Gain

(Rate)

Gain

(Rate)

White 265 271 269 270 6

(0.75)

-1

(-0.2)

African Am

235 244 242 244 9

(1.1)

0

(0.0)

Hisp

Am

239 245 245 246 6

(0.75)

1

(0.20)

Table 7. 4th Grade Math Scale Scores: Comparing rates of gain over Clinton reform years with rates over NCLB years.

1992 2003 2005 2007 2003-1992

2007-2003

Gain

(Rate)

Gain

(Rate)

White 227 243 246 248 16

(1.5)

5

(1.25)

African Am

192 216 220 222 24

(2.2)

6

(1.5)

Hisp

Am

201 221 225 227 20

(1.82)

6

(1.5)

Table 8. 8th Grade Math Scale Scores: Comparing rates of gain over Clinton reform years with rates over NCLB years.

1992 2003 2005 2007 2003-1992

2007-2003

Gain

(Rate)

Gain

(Rate)

White 276 287 289 290 11

(1.0)

3

(0.75)

African Am

236 252 254 259 16

(1.5)

7

(1.75)

Hisp

Am

247 258 261 264 11

(1.0)

6

(1.50)

Analysis 2 Summary of rates of Gain during

Clinton reform era with rates during NCLB years • Reading Results: In 4th and 8th grades, for all groups except 4th

grade whites, the differences in rates of gain favored the Clinton era (1994-2002) over the NCLB era (2002-2007), by greater than 0.3 points/year. 4th grade whites had similar rates of gain before and after 2002.

• Math Results: 4th grade unadjusted and adjusted results show no difference in rates of gain between the two eras for whites. For 4th grade blacks both unadjusted and adjusted rate gains favor the Clinton era. For 4th grade Hispanics, the unadjusted rates of gain favor the Clinton era and the adjusted gain is less than 0.3 points per year indicating no difference in rates between the two eras. 8th grade unadjusted and adjusted results show no differences in rates of gain between eras for two groups, whites and blacks and show a small positive effect favoring the NCLB era for Hispanics.

Analysis 2 Interpretation

• Reading: When compared to the Clinton era, the NCLB appears to have a negative effect on overall reading achievement at both the 4th and 8th grades for all groups but 4th whites, where there is no effect.

• Math: Compared to the Clinton era, the NCLB appears to have a very slight negative effect on 4th grade NAEP math achievement and a small positive effect on 8th grade NAEP math achievement.

Another way of interpreting the data

• Table 9 looks at both Analysis 1 and 2. It reduces the numbers to symbols for differences in rates of gain. A (-) means the rate of gain (rog) before 2002 was larger than the rog after 2002 by at least 0.3 points per year. A (+) means the rog was greater by at least 0.3 points after 2002 than before and a (0) means a difference of less than 0.3 points per year.

• Table 10 uses data from Analysis 1 to estimate how much gain or loss, compared to the rog from 1998-2002, students would make under NCLB if they gained at the same rate as they did from 2002-2007 for a full decade (2002-2012).

Table 9: Gain Rate Comparison Summary: Rate differences < [0.3] points per year = 0:

Rate differences >0.3 favoring Post NCLB = +. Rate diff. > 0.3 favoring Pre NCLB = -. Analysis 1:

5 years Pre NCLB v 5 yrs post NCLB

Analysis2:

Clinton era v NCLB era

Wh BL Hisp Wh BL HispReading 4th - - - 0 - -Reading 8th - - - - - -

Math 4th - - - 0 - 0

Math 8th - 0 0 0 0 +

Table 10. Estimated gains or losses in NAEP scores from 2002-2012 derived by extrapolating from 2002-2007 data compared to NAEP scores rates of gain from 1998-2002. The numbers are grade levels (gl). The number -0.4 gl means that

students would gain 4/10s of a grade level less from 2002-2012 if they continued at

their 2002-2007 gain rates than they would if they gained at the 1998-2002 rates.

Reading Math4th Grade 8th Grade 4th Grade 8th Grade

White -0.4 gl -0.95 gl -0.4 gl -0.3 gl

Black -0.5 gl -0.5 gl -1.0 gl 0

Hispanic -0.75 gl -0.8 gl -0.4 gl 0

Interpretation of Tables 9 and 10

• In Table 9 there are 24 cells representing combinations of content area, race, grade level and period of analysis. Of the 24 cells, 16 have a (-) indicating that the rate of gain in test scores during the period before 2002 (the start of NCLB) is at least 0.3 points greater than the rate from 2002-2007. Seven (7) cells have a 0 representing differences smaller than 0.3 points per year. One (1) cell has a + indicating a greater rate of gain after 2002. This table shows clearly that, on average, NCLB appears to have had a negative effect on student achievement.

• In Table 10 the differences in rates of gain between the periods of roughly five years before 2002 and five years after 2002. These differences are extrapolated over a ten year period (2002-2012). The extrapolations are converted into grade levels. For example, in this analysis eighth grade Hispanic students would gain 80% of a grade level less over the 2002-2012 decade if they continue at the NCLB 2002-2007 rates of gain compared to the before NCLB 1998-2002 rates of gain. Some of the possible losses are impressively large.