DYS StARR ProjectDYS StARR ProjectMarch 2010March 2010

SREE Presentation, Washington DCSREE Presentation, Washington DC Ohio Department of Youth ServicesOhio Department of Youth Services

Striving Readers Grant Funded by IESStriving Readers Grant Funded by IESProject Evaluation By Ohio State UniversityProject Evaluation By Ohio State University

Issues on Design and Impact from a Issues on Design and Impact from a Randomized Control Trials StudyRandomized Control Trials Study

William E. Loadman, PIWilliam E. Loadman, PI Raeal MooreRaeal Moore Jing ZhuJing Zhu Richard LomaxRichard Lomax

LocalLocal ContextContext

Incarcerated youth ages 12-21Incarcerated youth ages 12-21 7 High Schools (6 Male 1 Female)7 High Schools (6 Male 1 Female) Highly restrictive environmentsHighly restrictive environments Students with limited reading abilityStudents with limited reading ability

– 2/3 below grade level2/3 below grade level Average length of stay 10.5 monthsAverage length of stay 10.5 months Students with concomitant problemsStudents with concomitant problems No consistency on entry and exit of No consistency on entry and exit of

studentsstudents

Mobility IssuesMobility Issues ODYS release and return ODYS release and return

– 16% of Ineligible youth left and returned16% of Ineligible youth left and returned– 8% of Read 180 youth left and returned8% of Read 180 youth left and returned– 7.5% Traditional youth left and returned7.5% Traditional youth left and returned

Inter School movementInter School movement– 24% Ineligibles moved across schools at least once24% Ineligibles moved across schools at least once– 27% of Read 180 and Traditional youth moved across 27% of Read 180 and Traditional youth moved across

schools at least onceschools at least once– Ineligible youth moved as many as 5 times and Read Ineligible youth moved as many as 5 times and Read

180 and Traditional youth as many as three times180 and Traditional youth as many as three times

At any point in time there are approximately At any point in time there are approximately 1400 - 1700 youth across all DYS schools1400 - 1700 youth across all DYS schools

Targeted Intervention Logic ModelTargeted Intervention Logic ModelProgram

Inputs/ActivitiesClassroom Practices:

Intermediate Outcomes Short-Term Long Term

Student Outcomes

Leadership Training provided for Principals, Literacy Coaches, and District Staff [Initial and follow-up; 5 hours total

in Y1]

R180 Training for Teachers/Teacher Aides [initial 2 days and Semi-annual

follow-up; 15 hours total in Y1. Two 5 hour PD sessions (10 hrs

total) in Y2]

Scholastic Software for classroom including

supplemental materials. [including

Classroom supplies/materials]

Scholastic R180 Software Adapts instruction for

Students needs

A Maximum of 15 students are scheduled for 2-45

minute periods consisting of the 5 components of

Read180: whole group, individualized learning,

computer activities, small group (each 20 min), wrap

up (10 min)

Targeted Strategies Used include:

-Sequencing-Identifying main idea/detail

-Summarizing-Identifying cause/effect

-Making Inferences

Positive impact upon return to

home/community

Use of strategies to increase

comprehension/vocabulary

Reduce recidivism

Increased literacy, fluency, comprehension and confidence.

Reading at Grade Level

Increased engagement in

educational tasks.

Students are assessed using R-skills and Scholastic

Reading Inventory

Literacy Coach on classroom practices

Increased student self-efficacy

Implementation Data CollectionImplementation Data Collection

Daily implementation logsDaily implementation logs Weekly observation data (OSU)Weekly observation data (OSU) Quarterly feedback from Scholastic Quarterly feedback from Scholastic

observations observations Professional development attendanceProfessional development attendance

– Literacy coachesLiteracy coaches– Read 180 teachers/aidesRead 180 teachers/aides– All teachersAll teachers– PrincipalsPrincipals

Targeted Intervention Targeted Intervention Implementation ResultsImplementation Results

FacilityPD Attendance

Year 1PD Attendance

Year 2Instruction

Year 1Instruction

Year 2

1 Moderate High Needs Improv. Moderate

2 Moderate High Moderate Needs Improv.

3 High High Moderate High

4 High High High High

5 High High Moderate Moderate

7 High High Moderate Moderate

8 High High Moderate High

Total High High Moderate High

Targeted Intervention Targeted Intervention Implementation ResultsImplementation Results

OSU Observation ResultsOSU Observation Results– Read 180 implemented on ModelRead 180 implemented on Model– Variation across schoolsVariation across schools– Youth interested and engagedYouth interested and engaged– Differentiated instruction observedDifferentiated instruction observed– Youth disrupted class on 60% of early Youth disrupted class on 60% of early

visits, but youth not removed from visits, but youth not removed from classroomclassroom

– When either aide or teacher not in When either aide or teacher not in classroom, instruction very difficultclassroom, instruction very difficult

Scholastic observationsScholastic observations– Reported as on model/protocolReported as on model/protocol– Better able to meet 90 minute model in Better able to meet 90 minute model in

year two compared with year oneyear two compared with year one– Encouraged use of more strategies to Encouraged use of more strategies to

help keep youth engaged (year two)help keep youth engaged (year two)– Encouraged use of SAM reports to better Encouraged use of SAM reports to better

inform instruction (year two)inform instruction (year two)

Targeted Intervention Targeted Intervention Implementation ResultsImplementation Results

Impact Design Impact Design Targeted InterventionTargeted Intervention

Random assignment of student to Random assignment of student to condition (100% reading below grade condition (100% reading below grade level, 6 month stay, no GED/diploma)level, 6 month stay, no GED/diploma)– Condition 1 Read 180Condition 1 Read 180– Condition 2 Traditional English classCondition 2 Traditional English class

Project start-up baseline testingProject start-up baseline testing– CAT Reading and MathCAT Reading and Math– SRI Reading Lexile scoresSRI Reading Lexile scores

End of each term assessments on SRI (4 End of each term assessments on SRI (4 per year)per year)

Targeted Intervention Sample: Targeted Intervention Sample: Classrooms/TeachersClassrooms/Teachers

8-12 students per classroom across 8-12 students per classroom across facilities and quartersfacilities and quarters

One-on-one interaction was greater in One-on-one interaction was greater in Read180 relative to Traditional classesRead180 relative to Traditional classes

One Read 180 teacher and aide per One Read 180 teacher and aide per building/classroombuilding/classroom

All teachers certified in English/Language All teachers certified in English/Language ArtsArts

Targeted Intervention Sample:Targeted Intervention Sample:StudentsStudents

0

70.1

2.8

24

3.1

55

45

0.2

68.3

1.7

26.1

3.7

96.5

3.5

57.8

42.2

95.2

4.8

0

10

20

30

40

50

60

70

80

90

100A

sia

n

Bla

ck

His

pa

nic

Wh

ite

Mu

ltir

acia

l

Ma

le

Fe

ma

le

No

Ye

s

Race Gender  SpecialEducation

R180 Traditional

Targeted Intervention Sample:Targeted Intervention Sample:StudentsStudents

0.3 0.3 15

0.3 0.3 1 0.3 0.30.4 0

42.2

0.84.5

0 0 0

29.1

0.4 0

17.2

30.1

43.7

22.5

0

10

20

30

40

50

60

70

80

90

100

R180 Traditional

12.8

13.1

2123.8 22.3

12.6

2.3 1 0.5 1.1

41.7

28.2

9.3

4.6

14.6

1.35

14.8

23.118.9

9.4

1.1 0.4 0.4 1.8

45.8

24.2

7.35.3

15.1

25.9

0

10

20

30

40

50

60

70

80

90

100

14 15 16 17 18 19 20 21 22

0 (Pr

e-enro

lled)*

*

8 9

10 11 12

gradu

ated

Age * Grade

R180 Traditional

Targeted Intervention AnalysesTargeted Intervention Analyses Treatment of the Treated (TTT)Treatment of the Treated (TTT)

– This study defines TTT based on being present This study defines TTT based on being present in class for at least 5 weeks in each of 2 or in class for at least 5 weeks in each of 2 or more quartersmore quarters

– Estimated 5 HLM analyses using the number of Estimated 5 HLM analyses using the number of quarters of actual treatment receivedquarters of actual treatment received

– The base model (grand mean centered) The base model (grand mean centered) included the following variables (white, age, included the following variables (white, age, Math CAT, Read CAT, disability, grade level, Math CAT, Read CAT, disability, grade level, institution and mobility) institution and mobility)

– The dependent variable was the SRI scoreThe dependent variable was the SRI score– Variables with p values of .08 or less were Variables with p values of .08 or less were

retained in each final model. retained in each final model.

TTT Descriptive ResultsTTT Descriptive Results

720

740

760

780

800

820

840

0 1 2 3 4 5 6 7 8

Number of Terms

SR

I Sc

ore

Comparison

READ 180

Overall

Time Plot of the Mean Responses for the READ 180 Group, the Comparison Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall Group, and the Overall including All Youthincluding All Youth

TTT Descriptive ResultsTTT Descriptive Results

500

550

600

650

700

750

800

850

900

0 1 2 3 4 5 6 7 8

Number of Terms

SR

I Sc

ore

Comparison

READ 180

Overall

Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall for and the Overall for Youth with Youth with Less Than Two Quarters of TreatmentLess Than Two Quarters of Treatment

TTT Descriptive ResultsTTT Descriptive Results

720

740

760

780

800

820

840

0 1 2 3 4 5 6 7 8

Number of Terms

SR

I S

co

re

Comparison

READ 180

Overall

Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, Time Plot of the Mean Responses for the READ 180 Group, the Comparison Group, and the Overall for and the Overall for Youth with Youth with At Least Two Quarters of TreatmentAt Least Two Quarters of Treatment

Overall Read 180 Comparison

n col % n col % n col %

Slope > 0 405 62.40% 248 72.09% 157 51.48%

Slope 0 244 37.60% 96 27.91% 148 48.52%

Total 649 100% 344 100% 305 100%

TTT Descriptive ResultsTTT Descriptive Results

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

0 2 4 6 80

500

1000

1500

# of Terms

SR

I S

core

Overa

llTra

dit

ion

al

Read

18

0

TTT HLM ResultsTTT HLM Results

Level 1:Level 1:

Level 2: Level 2:

Age is continuous; grand mean centeredAge is continuous; grand mean centeredMath CAT is continuous; grand mean centeredMath CAT is continuous; grand mean centeredRead CAT is continuous; grand mean centeredRead CAT is continuous; grand mean centeredDisability Status is dichotomized (0= not disabled; 1 = Disability Status is dichotomized (0= not disabled; 1 =

disabled)disabled)Grade level is continuous and ranges from 8-12; grand Grade level is continuous and ranges from 8-12; grand

mean centeredmean centered

iijiiij Njnijy 0,1,2,... and 1,2,..., for ,

iiii

iii

b0543

210

).GRDLVLGRDLVL()DISB().READCATREADCAT(

.)MATHCATMATHCAT().AGEAGE(

iiii b121 )TRTGRP().READCATREADCAT(

Youth (n = 542) who received between Youth (n = 542) who received between two and eight two and eight quarters of treatment. quarters of treatment.

TTT HLM ResultsTTT HLM Results

Fixed Effect Estimate SE t-ratio p-value Effect Size

Intercept α0 782.9 10.253 76.36 <.0001 --

Age α1 15.9294 5.4662 2.91 0.0037 0.02

MathCAT α2 10.4414 3.8851 2.69 0.0074 0.01

ReadCAT α3 37.6965 3.7074 10.17 <.0001 0.18

Disability α4 -28.3319 14.8666 -1.91 0.0572 0.01

Grade Level α5 17.3026 7.2463 2.39 0.0173 0.01

ReadCAT*Time β1 3.458 1.1505 3.01 0.0028 0.02

TRTGroup*Time β2 22.342 3.4899 6.4 <.0001 0.12

Random Effect b0 b1

b0 18440*

b1 -846.83 2181.18*

18820*

Estimates for the Fixed & Random Effects Estimates for the Fixed & Random Effects Using 2- 8 Quarters of TreatmentUsing 2- 8 Quarters of Treatment

Regression Adjusted Means Regression Adjusted Means 2006-2007 (Year 1)2006-2007 (Year 1)

2 Quarters or more 3 Quarters or more

READ 180

Comparison READ 180

Comparison

Baseline Mean

784.34 773.27 783.88 773.36

Estimated Post-test Mean

856.95 827.95 879.99 820.12

Difference score

+72.61 +54.68 +96.11 +46.76

Regression Adjusted Means Regression Adjusted Means 2006-2008 (Year 1 & 2)2006-2008 (Year 1 & 2)

2 Quarters or more

3 Quarters or more

4 Quarters or more

READ 180

Comparison

READ 180

Comparison

READ 180

Comparison

Baseline Mean

766.98 753.64 764.71 757.11 756.17 759.64

Estimated Post-test Mean

813.02 768.34 823.28 766.79 842.02 757.46

Difference score

+46.04 +14.7 +58.57 +9.68 +85.85 -2.18

a

Summary of IssuesSummary of Issues Staff turnoverStaff turnover Release datesRelease dates Time in treatmentTime in treatment Student attendanceStudent attendance Student mortalityStudent mortality Local building level perturbationsLocal building level perturbations Student cooperationStudent cooperation Variability of SRI measureVariability of SRI measure Psychometric properties of SRIPsychometric properties of SRI Use of CAT as an additional outcome Use of CAT as an additional outcome

measuremeasure

Summary of Methodological IssuesSummary of Methodological Issues

Tracking student entry, movement, exit Tracking student entry, movement, exit and returnand return

PowerPower SRI measure quality/variabilitySRI measure quality/variability Missing dataMissing data Losing dataLosing data Maintaining student confidentialityMaintaining student confidentiality Reporting issuesReporting issues Use of dataUse of data

Summary of Issues Cont’dSummary of Issues Cont’d

Student disruptive behavior in classStudent disruptive behavior in class Student’s placed in isolationStudent’s placed in isolation Student’s removed from classStudent’s removed from class Student receipt of treatmentStudent receipt of treatment Student assessment and cooperationStudent assessment and cooperation Teacher cooperationTeacher cooperation Student release dateStudent release date

Summary of FindingsSummary of Findings Read 180 being implemented on model and according Read 180 being implemented on model and according

to protocolto protocol Randomized control trials able to verify causal Randomized control trials able to verify causal

relationshiprelationship Approximately 72% of Read 180 students demonstrate Approximately 72% of Read 180 students demonstrate

a positive slope of increasing reading scores over a positive slope of increasing reading scores over time, compared with about 51% for the traditional time, compared with about 51% for the traditional group group

Read 180 adds about 22 Lexile points per quarter of Read 180 adds about 22 Lexile points per quarter of treatment over and above the traditional group treatment over and above the traditional group performance performance

Students in Read 180 significantly out perform Students in Read 180 significantly out perform students assigned to the traditional group students assigned to the traditional group

Effect sizes vary according to model, but most are Effect sizes vary according to model, but most are trivialtrivial

Most Read 180 students gain, but are still not reading Most Read 180 students gain, but are still not reading at grade levelat grade level

Future ResearchFuture Research Student perception of Read 180 content Student perception of Read 180 content

and structureand structure Impact analysis with CAT as an outcome Impact analysis with CAT as an outcome

measure.measure. Sub-analysis determining the influence of Sub-analysis determining the influence of

disability status on program intervention disability status on program intervention impacts. impacts.

Analysis of the influence of program on Analysis of the influence of program on recidivism.recidivism.

Sub-analysis of reason for incarceration, Sub-analysis of reason for incarceration, reading improvement, and recidivism. reading improvement, and recidivism.