Evaluation Results2002-2004

Missouri Reading InitiativeMissouri Reading Initiative

MRI’s Evaluation Activities:MRI’s Evaluation Activities:

SurveysSurveys*Teacher Beliefs and Practices (pre/post)*Teacher Beliefs and Practices (pre/post)Annual Participant QuestionnaireAnnual Participant Questionnaire

*Data Collection*Data CollectionTest ScoresTest Scores

Standardized TestsStandardized TestsClassroom Assessments (DRA)Classroom Assessments (DRA)MAPMAP

DemographicsDemographicsSpecial Education InformationSpecial Education Information

MAP AnalysesMAP Analyses*For schools beginning in 2002*For schools beginning in 2002

MAP ANALYSES:MAP ANALYSES:Map analyses compare schools that have Map analyses compare schools that have

finished the MRI program with randomly chosen finished the MRI program with randomly chosen samples of non-MRI elementary schoolssamples of non-MRI elementary schools

Results indicate MRI schools generally Results indicate MRI schools generally outperform non-MRI schoolsoutperform non-MRI schools

Notes for MAP AnalysesNotes for MAP AnalysesNote: With all the following MAP Analyses charts the numbers are not as important as the comparative performance between MRI and non-MRI schools. This is because:

1. There is a lot of variation in the data from year to year and school to school

2. The calculation of the baseline changes as more data becomes available:

– For the 2002 schools 1999 was the baseline– For the 2003 schools an average of 1999/2000 was the baseline– For the 2004 schools an average of 1999/2001 was the baseline

Notes for Chart 1Notes for Chart 1

1.1. This chart compares MRI and non-MRI schools performance This chart compares MRI and non-MRI schools performance on the 2002 MAP Communication Arts Indexon the 2002 MAP Communication Arts Index

2.2. Baseline year is 1999; Outcome year is 2002Baseline year is 1999; Outcome year is 2002

3.3. Each sample has 15 schools = number of schools that Each sample has 15 schools = number of schools that finished MRI Spring 2002finished MRI Spring 2002

4.4. Total random sample = 150 (large enough number to satisfy Total random sample = 150 (large enough number to satisfy statistical significance, high confidence levels)statistical significance, high confidence levels)

% Change in MAP Commuincation Arts Index 1999-2002 for MRI and 10 Samples

1.6

3.7

3.0

-0.1

1.3

3.1

1.8 1.9

0.5

1.2

1.8

6.5

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

Samples:1-10

Sample Average N=150

MRI N=15

Notes for Chart 2Notes for Chart 2

1.1. This chart compares MRI and non-MRI schools This chart compares MRI and non-MRI schools performance on the 2002 MAP Reading Indexperformance on the 2002 MAP Reading Index

2.2. Baseline year is 1999; Outcome year is 2002Baseline year is 1999; Outcome year is 2002

3.3. Each sample has 15 schools = number of schools that Each sample has 15 schools = number of schools that finished MRI Spring 2002 finished MRI Spring 2002

4.4. Total random sample = 150 (large enough number to Total random sample = 150 (large enough number to satisfy statistical significance, high confidence levels)satisfy statistical significance, high confidence levels)

% Change in MAP 3rd Grade Reading Index 1999-2002

MRI and 10 Samples

3.0

4.5

8.5

2.83.4

6.5

5.35.6

2.8

3.9

4.6

13.1

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Samples:1-10

Sample Average N=150

MRI N=15

Notes for Chart 3Notes for Chart 3

1.1. This chart compares MRI and non-MRI schools performance This chart compares MRI and non-MRI schools performance on the 2003 MAP Communication Arts Indexon the 2003 MAP Communication Arts Index

2.2. Baseline year is an average of 1999/2000 (smoothes out Baseline year is an average of 1999/2000 (smoothes out variations); Outcome year is 2003variations); Outcome year is 2003

3.3. Each sample has 20 schools = number of schools that Each sample has 20 schools = number of schools that finished MRI Spring 2003finished MRI Spring 2003

4.4. Total random sample 200 (large enough number to satisfy Total random sample 200 (large enough number to satisfy statistical significance, high confidence levels)statistical significance, high confidence levels)

MAP Communication Arts Index 1999/2000 to 2003

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

Samples

Total sample: N=200Mean = 0.9

MRI: N=20Mean = 4.8

0.2 0.3

1.4

-0.3

1.21.6

3.0

2.3

-0.1

-1.1

95% Confidence Level=/- 1.0%

Samples: 1-10

Notes for Chart 4Notes for Chart 4

1.1. This chart compares MRI and non-MRI schools This chart compares MRI and non-MRI schools performance on the 2003 MAP Reading Indexperformance on the 2003 MAP Reading Index

2.2. Baseline year is an average of 1999/2000 (smoothes Baseline year is an average of 1999/2000 (smoothes out variations); Outcome year is 2003out variations); Outcome year is 2003

3.3. Each sample has 20 schools = number of schools that Each sample has 20 schools = number of schools that finished MRI Spring 2003 finished MRI Spring 2003

4.4. Total sample 200 (large enough number to satisfy Total sample 200 (large enough number to satisfy statistical significance, high confidence levels)statistical significance, high confidence levels)

MAP Reading Index 1999/2000 to 2003

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Samples

Total sample: N=200Mean = 1.6

MRI: N = 20Mean = 7.3

1.4

-0.3

3.8

-0.7

-0.1

2.8

3.6

4.3

0.3 0.4

95% Conficence Level +/- 1.5%

Samples: 1-10

Notes for Chart 5Notes for Chart 5

1.1. This chart compares MRI and non-MRI schools performance This chart compares MRI and non-MRI schools performance on the 2004 MAP Communication Arts Index on the 2004 MAP Communication Arts Index

2.2. Baseline year is an average of 1999/2001 (smoothes out Baseline year is an average of 1999/2001 (smoothes out variations); Outcome year is 2004variations); Outcome year is 2004

3.3. Each sample has 27 schools = number of schools that Each sample has 27 schools = number of schools that finished MRI Spring 2004finished MRI Spring 2004

4.4. Total random sample 270 (large enough number to satisfy Total random sample 270 (large enough number to satisfy statistical significance, high confidence levels)statistical significance, high confidence levels)

1999-2004 MAP MRI and Non-MRI Communication Arts Index Comparison

1.9 1.9

2.3

-0.2

3.3

2.7

1.7

3.4

0.1

2.3

1.9

3.9

-0.5

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

Avera

ge %

In

cre

ase/S

ch

oo

l

10 Randomly Selected Samples of 27 Schools

Total Random Samplen=270

MRI Samplen=27

Baseline=Average Index Score 1999-2001Outcome Year=2004

Notes for Chart 6Notes for Chart 6

1.1. This chart compares MRI and non-MRI schools This chart compares MRI and non-MRI schools performance on the 2004 MAP Reading Index performance on the 2004 MAP Reading Index

2.2. Baseline year is an average of 1999/2001 (smoothes Baseline year is an average of 1999/2001 (smoothes out variations); Outcome year is 2004out variations); Outcome year is 2004

3.3. Each sample has 27 schools = number of schools that Each sample has 27 schools = number of schools that finished MRI Spring 2004finished MRI Spring 2004

4.4. Total sample 270 (large enough number to satisfy Total sample 270 (large enough number to satisfy statistical significance, high confidence levels)statistical significance, high confidence levels)

1999-2004 MAP MRI and Non-MRI Reading Index Comparison

3.4

2.3

3.9

4.6

5.45.7

2.4

4

2.5

4.13.8

7.4

0

1

2

3

4

5

6

7

8

Avera

ge %

In

cre

ase/S

ch

oo

l

10 Randomly Selected Samples of 27 Schools

Total Random Samplen=270

MRI Samplen=27

Baseline=Average Index Score 1999-2001Outcome Year=2004

Adequate Yearly ProgressAdequate Yearly Progress

As mandated by federal law, Missouri schools must make yearly progress goals in MAP scores

For 3rd Grade Communication Arts those goals were defined as 19.4% of students achieving levels of Proficient or better in 2003, and 20.4% for 2004.

The following Table provides a comparison between MRI schools and state-wide results.

MRI State

2003

81%

(60 / 74)

50.9%

(1,046 / 2,053)

2004

100%

(50 / 50)

57.4%

(1,167 / 2,053)

Percentage of Schools Meeting AYP Levels2003=19.4% 2004=20.4%Proficient and Advanced

Participant Survey

Participants rate the usefulness of component utilization,practice change, "buy in", attitudes toward the program and trainer, etc.

Results drive program change; e.g., Orientation

Notes for Participant Survey Slide

This slide introduces the survey and its uses. This slide introduces the survey and its uses. The table in the next slide demonstrates how the The table in the next slide demonstrates how the survey is often used. In this case:survey is often used. In this case:

1.1. 2002 survey respondents identified the problem 2002 survey respondents identified the problem of being “overwhelmed” of being “overwhelmed”

2.2. Program responded by redesigning orientation Program responded by redesigning orientation and other detailsand other details

3.3. Program satisfaction improved from 02 to 04Program satisfaction improved from 02 to 04

"Reflecting on the effectiveness of the Missouri Reading Initiativeas a whole, how would you rate it?" (1-Poor to 5-Excellent)

Sour c e: M i s s our i R eadi ng I ni ti ati ve A nnual P ar ti c i pant Sur vey 2002- 2004

0%

10%

20%

30%

40%

50%

60%

70%

80%

2002 (n=63) 20.60% 25.40% 54%

2003 (n=74) 14.90% 23.00% 62.10%

2004 (n=50) 4.00% 28.00% 68.00%

<3.5 3.5-3.99 >4.0

Special EducationSpecial Education

We track the effects of MRI on Special Education We track the effects of MRI on Special Education in two ways:in two ways:

1.1. Beginning with schools that started MRI in Beginning with schools that started MRI in the Fall of 2002, all students with IEPs are the Fall of 2002, all students with IEPs are identified and the type of IEP is described identified and the type of IEP is described (Reading, Math, Speech, etc.)(Reading, Math, Speech, etc.)

2.2. Annual reports are made from schools about Annual reports are made from schools about their IEP evaluation process: referrals, their IEP evaluation process: referrals, evaluations, and IEPs.evaluations, and IEPs.

Notes for Special Education Chart

This Chart is for schools that were in their 3This Chart is for schools that were in their 3rdrd year 2003-2004 year 2003-2004Many schools do not have this data, or it is not easily accessed Many schools do not have this data, or it is not easily accessed 9 of 18 in 20029 of 18 in 2002 5 of 23 in 20035 of 23 in 200310 of 34 in 200410 of 34 in 2004The data points to a decrease in referrals, evaluations, and The data points to a decrease in referrals, evaluations, and assignment of IEP over the time schools participate in MRI.assignment of IEP over the time schools participate in MRI.Data collection began from the onset of schools in 2002-2003 Data collection began from the onset of schools in 2002-2003 with a complete report in 2005.with a complete report in 2005.

Missouri Reading Initiative 2004 Special Education Results

2000-2001 to 2003-200410 of 34 Schools Reporting

0

20

40

60

80

100

120

140

160

180

TEAMS EVAL IEP

2001 2002 2003 2004

Developmental Reading Assessment Developmental Reading Assessment (DRA)(DRA)

The following Table describes the changes in the percentages of cohorts of students who scored At or Above Grade Level according to the DRA at 15 MRI schools who have completed two years in the

program..

Key:Key:=% At or Above GL=% At or Above GL

S031=1S031=1stst Graders in Spring 2003 Graders in Spring 2003S042=2S042=2ndnd Graders in Spring 2004 Graders in Spring 2004

F02 =2F02 =2ndnd Graders in Fall 2002 Graders in Fall 2002S043=3S043=3rdrd Graders in Spring 2004 Graders in Spring 2004

= Percentage Change= Percentage Change

Change in DRA Grade Level School S031 S042 F022 S043

1 66.5 80.1 20.45% 58.8 70.7 20.24%

2 56.8 63.2 11.27% 54.7 45.5 -16.82%

3 56.4 76.3 35.28% 52.2 72.5 38.89%

4 67.5 92.1 36.44% 42.3 72.2 70.69%

5 53.2 76.3 43.42% 25.8 92.1 256.98%

6 19.2 35.4 84.38% 35.1 53.6 52.71%

7 43.5 68.0 56.32% 38.5 72.4 88.05%

8 25.5 46.5 82.35% 13.3 49.1 269.17%

9 48.0 74.1 54.38% 53.3 72.2 35.46%

10 50.0 73.6 47.20% 46.5 72.9 56.77%

11 40.0 55.4 38.50% 40.0 43.3 8.25%

12 76.8 93.0 21.09% 60.4 92.1 52.48%

13 71.7 89.4 24.69%

14 76.5 66.2 -13.46% 78.2 89.8 14.83%

15 73.8 88.2 19.51% 54.2 78.9 45.57%

TOTALS 53.8 70.6 38.37% 48.3 71.1 67.9%