1

Children Left Behind in AYP and Non-AYP Schools: Using Student Progress and the Distribution of Student Gains to Validate AYP

Kilchan ChoiMichael Seltzer

Joan Herman Kyo Yamashiro

UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)

2

Research Questions

Are there schools that meet AYP yet still have children who are not making substantial progress? i.e., leaving some children behind?

Are there schools that do not meet AYP yet still enable students to make substantial progress?

Do AYP schools achieve a more equitable distribution of student growth? Are students at all ability levels making progress in AYP schools?

Are there non-AYP schools that are reducing the achievement gap?

3

Sample

Large, Urban District in WA

2,524 students

2 time-point ITBS reading scores (Grade 3 in 2001 & Grade 5 in 2003)

Standard Errors of Measurement (SE) on ITBS reading scores (Bryk, et.al., 1998)

72 schools

• Average # students/school: 35

• Average % qualifying for FRPL: 36.4%

• Average % Minority (African American, Native American, or Latino): 68.6%

4

AYP vs. Non-AYP schools In WA

School AYP decision made based on 4th grade performance on WA Assessment of Student Learning (WASL)

51 schools made AYP; 21 did not make AYP in baseline year (2002), according to WA State Dept of Ed

Our study re-evaluates AYP and non-AYP schools with a new value-added model (an advanced hierarchical Modeling technique)

5

A New Methodology for School Effect / Accountability: Latent Variable Regression

in Hierarchical Model

Additional Questions and Interest using LVR-HM Move beyond school mean growth rates

and examine hidden/underlying process

How equitably is student achievement distributed? (The distribution of student growth: Children Left Behind or No Child Left Behind)

Why is it that student achievement is distributed in a more equitable fashion in some schools than in other schools?

6

Distribution of Student Growth(Relationship between initial status and

rate of change)

Y Y Y t t+1 t t+1 t t+1 Figure 1. Figure 2. Figure 3. High initial low gain High initial medium gain High initial high gain Low initial high gain Low initial medium gain Low initial low gain

7

Why a New Value-Added Model (LVR-HM)?

Gains or Growth might be highly dependent upon a status at certain point of time (i.e., initial status)

Initial status can be a strong and important factor to “valued-added gain or growth”

New value-added gain or growth: Adjusting student intake characteristics PLUS

student initial difference Adjusting school intake characteristics, policies

and practice PLUS school initial difference

Thus, providing value-added gain or growth PLUS revealing the distribution of student achievement

8

Latent Variable RegressionHierarchical Model (LVR-HM)

Level 1: Time series within student

Yti = 0i + 1i Timeti + ti ti ~ N (0, 1)

Estimating initial status and gain for each student i with standard errors

Level 2: Student level

0i = 00 + r0i r0i ~ N (0, 00)

1i = 10 + b(0i - 00) + r1i r1i ~ N (0, 11)

Cov(r0i , r1i ) = 0

Gain for student i is modeled as function of his or her initial status

9

Different Levels of Initial Status

Many ways to define performance subgroups based on initial status

Examined gains for 3 performance subgroups within each school

Defined by initial status

Hi Performers: 15 pts above the school mean initial status

Mean: School mean initial status

Low Performers: 15 pts below the school mean initial status

10

Estimating Expected Gains for Different Levels of Initial Status

We estimate expected (predicted) gain for each of the performance subgroups using LVR-HM

Model-based estimation, not separate group analysis

Point estimate of gain & its 95% confidence interval (statistical inferences)

Possible to estimate expected gains after controlling for factors that lie beyond school’s control (e.g., student SES, school compositional factors)

11

Only 12 of 52 AYP schools have 95% interval above the district avg.1 AYP school’s 95% interval includes 0

Expected mean gain in ITBS reading scores for AYP schools

12

Expected mean gain in ITBS reading scores for non-AYP schools

2 Non-AYP schools have 95% interval above district avg.

13

7 AYP schools’ 95% interval 303 AYP schools’ 95% interval includes 0 (low performers make no gains)

Expected gain for low-performing students (AYP schools)

14Expected gain for low-performing students

(non-AYP schools)

5 Non-AYP schools have gains for low performers >20

15

Expected gain for high-performing students (AYP schools)

9 AYP schools’ 95% interval 303 AYP schools’ 95% interval < 10 (high performers make little or no gains)

16

Expected gain for high-performing students (non-AYP schools)

5 Non-AYP schools’ 95% interval 303 Non-AYP schools’ 95% interval < 10 (high performers make little or no gains)

17

Distribution of Gains Within A School

Type I: Substantial gain across all performance subgroups (e.g., no child left behind – ex: AYP school #8, non-AYP school #26)

Type II: No adequate gain for high performers; substantial gain for low performers (ex: AYP schools #19, non-AYP school #27)

Type III: No adequate gain for low performers; substantial gain for high performers (ex: AYP schools, non-AYP school #6 )

1815pts above the school mean Equal to the school mean 15pts below the school mean

Estimate 95% interval Estimate 95% interval Estimate 95% interval

AYP SchoolType I

Sch. #8Sch. #22 Sch. #25

37.136.337.7

( 30.7, 43.8 )( 30.0, 43.2 )( 30.8, 42.8 )

38.736.936.4

( 35.2, 42.2 )( 31.7, 42.0 )( 32.2, 40.5 )

40.337.436.0

( 35.0, 45.7 )( 31.1, 43.9 )( 31.1, 41.1 )

Type IISch. #19Sch. #63

22.128.2

( 6.9, 35.9 )( 18.2, 39.0 )

33.534.1

( 22.8, 44.1 )( 27.8, 40.3 )

44.840.0

( 30.7, 60.3 )( 30.5, 49.6 )

Type IIISch. #28Sch. #65

41.235.4

( 33.0, 51.2 )( 31.4, 39.5 )

31.231.3

( 27.4, 35.1 )( 28.5, 34.0 )

21.227.1

( 12.1, 28.7 )( 23.7, 40.4 )

Non-AYP school

Type ISch. #26

32.8 ( 24.1, 41.7 ) 32.2 ( 27.6, 36.9 ) 31.6 ( 22.5, 40.4 )

Type IISch. #27

18.7 ( 9.0, 27.2 ) 24.6 ( 17.6, 31.5 ) 30.5 ( 21.5, 40.1 )

Type IIISch. #6

Sch. #38Sch. #64

40.739.237.3

( 31.5, 50.8 )( 30.0, 49.9 )( 31.2, 44.2 )

32.429.930.6

( 26.2, 38.5 )( 25.5, 34.2 )( 26.9, 34.4 )

24.120.524.0

( 13.6, 33.7 )( 9.3, 30.2 )( 17.0, 30.2 )

19

Distribution of student gain for 3 AYP schools

20

Distribution of student gain for 3 non-AYP schools

21

Comparing Features: AYP & the CRESST Approach

AYP CRESST Approach

Data Structure Cross-sectional (follow grade levels, e.g., 4th graders in a school, over time)

Longitudinal (follow individual students over time)

Performance Measure (Outcome)

Proficiency levels (using cut scores)

Individual gains or growth

Subgroup Demographic characteristics Performance-level groups Plus Demographic characteristics

Adjustments / Controls for Student or School Characteristics

No controls or adjustments, just disaggregations – loss of advantages when comparing against other schools

Can adjust for differences between schools and students in the model

Type of Growth Examined

Percent Proficient may mask different underlying growth patterns: Even flexibility given to schools through Safe Harbor option is only for movement around the proficiency cut score

More complete picture of growth PLUS growth distribution

22

Different Growth By Performance Subgroups & Demographic Subgroups

40

50

60

70

80

90

grade7 grade8 grade9 grade10

Grade

Math

Ach.

exp. traj. of change forboys 12 points abovethe mean

exp. traj. of change forgirls 12 points abovethe mean

exp. traj. of change forgirls at the mean

exp. traj. of change forboys at the mean

exp. traj. of change forgirls 12 points belowthe mean

exp. traj. of change forboys 12 points belowthe mean

23

Conclusions

Analyses using our alternative approach:

• More informative picture of growth using individual, longitudinal student gains

• More complete picture of how student growth is distributed within a school

Stimulate discussion among teachers and administrator to identify students in need earlier (Seltzer, Choi & Thum, 2003)

Encourage educators to think about achievement levels rather than (or in addition to) current subgroup categories - may be more productive and actionable