school reform and beyond brian rowan burke a. hinsdale collegiate professor in education research...

School Reform and Beyond

Brian RowanBurke A. Hinsdale Collegiate Professor in

EducationResearch Professor, Institute for Social

ResearchProfessor of Sociology

Part I:The Problem

ECLS Estimates of Expected Literacy Learning by Risk Status: Kindergarten to Grade 3

0

20

40

60

80

100

120

140

0 10 20 30 40 50

Months

IRT

Scale

Sco

re

High Risk

Average for Population

Low Risk

How Big Are the Gaps?

At first testing, high risk students cannot identify letters, low risk can. By the end of 3rd grade (month 40), high risk students can identify common sight words, low risk students can make basic inferences from connected text and are moving toward extrapolation.

High-Risk vs. Average-Risk

High-Risk vs. Low-Risk

SD Months SD Months

Beginning of Kindergarten 0.36 1.6 0.64 2.8

End ofKindergarten

0.32 2.1 0.55 3.6

End of 1st Grade

0.34 1.8 0.61 3.2

End of3rd Grade

0.45 4.7 0.74 7.8

Approaches to Narrowing Gaps

High quality pre-schooling:

can increase achievement at entry into schooling

High quality elementary schooling:

accelerates academic growth during schooling

Out-of-school reforms

address factors “beyond the control of schools” that limit academic growth before and during schooling

Projected Outcomes:High Quality Pre-School + Effective Early Grades

Schooling

Model-Based Estimates of Reading Achievement Growth from ECLS Data

Gaps in Achievement for High Risk vs. Low Risk Students Under Different Conditions

Hi Risk ChildHi Start in K

Hi Risk ChildHi Start in KEffective School

Beginning of Kindergarten 0.72 months 0.25 months

End of Kindergarten 0.56 months -0.65 months

End of 1st Grade 0.72 months -0.54 months

End of 3rd Grade 2.76 months -0.68 months

Part II: Participation in R&D

Development

Full Scale Effectiveness Trial Exploratory Research

Efficacy Trial

The Cycle of R&D

Development Projects

Schools and researchers have interest in addressing/resolving a common “problem of practice.”

Researchers and school professionals work together in a few schools to:

conceptualize problem(s) design and test “tools/strategies” to address problem(s) conduct “informal” research to improve/evaluate

Products:

new curriculum units new assessment instruments new teaching practices new administrative practices/arrangements

Development Projects: An Exemplar at Michigan

The Henry Ford Museum/Michigan SOE Partnership

Team: Museum curator, UM faculty and graduates students, classroom teachers.

Initial Problem: Could you help us develop technology to improve access to museum resources for students and enhance the quality of their learning?

Initial Findings: Students have a “one shot” engagement at museum.

The tools: “Virtual Curator” and “Virtual Explorer” keep students engaged before and after visit.

The cycle: Six revisions over two years with Allen Park schools.

Development Projects: Features of District Participation

Small scale project: a few schools a limited focus

Intensive collaboration: multiple partners assumption of shared expertise mutual learning mutual benefit

Multi-year cycle of development and testing


Development


Efficacy Trial

The Cycle of R&D

Exploratory Research

Often emerges from a development project in a handful of schools.

Additional schools now recruited into project.

Developers conduct more systematic research to examine:

theory of action (are things working as they should?) effects on outcomes (are we getting the results we want?)

Research findings:

can serve as basis for further development work can serve as basis for efficacy trial

Exploratory Research: An Exemplar at Michigan

Project-based science (PBS) in the Detroit Public Schools middle schools.

Project begins with development in pilot sites.

In 1998, project moves to 13 teachers in 10 schools.

Six years later: 63 teachers in 26 schools.

Researchers found:

among “matched” students, the more PBS units a classroom used, the higher the MEAP science scores.

among “matched students”, those using PBS show higher levels of science motivation.

successful scaling required more “specified” implementation procedures.

Exploratory Research: Features of District Participation

District agrees to allow spread of initial project

Students, teachers, administrators now become:

“implementors” of project (not co-developers) “subjects” of research (not co-researchers)

Research becomes more intensive and rigorous:

matched samples formal instruments (e.g., surveys, observations)


Development


Efficacy Trial

The Cycle of R&D

Efficacy/Effectiveness Trials

Aim is to evaluate the effectiveness of well-developed interventions as implemented under “normal conditions of practice.”

Efficacy trials examine effectiveness of intervention in favorable conditions of implementation (e.g., in medicine, a clinic)

Effectiveness trials examine effectiveness of intervention under broad conditions of use (e.g., in medicine, regular practice setting)

Researchers often use “random assignment” to treatment vs. control conditions (RCT).

Researchers will want to gather detailed data on:

degree of implementation effects of treatment on outcomes, where control group is counterfactual

To assure adequate statistical power, number of schools/classrooms involved in research can be quite large.

70- 80 classrooms (40 treatment/40 control) 70 – 80 schools (40 treatment/40 control)

Efficacy/Effectiveness Trials: An Exemplar at Michigan

A Study of Instructional Improvement examined the effectiveness of three of America’s largest CSR programs:

Accelerated Schools Project (ASP) America’s Choice (AC) Success for All (SFA)

This was a 4-year, quasi-experiment conducted in ~120 high poverty elementary schools:

Matched samples of ~30 schools working with each of the three interventions, plus a matched sample of ~30 schools not implementing one of the interventions

Inside all 120 schools, data were gathered annually on school improvement processes, instruction, and student achievement

Looking at these programs, researchers examined:

whether and how the CSR programs altered instruction in schools whether and how the instruction inside CSR schools affected student achievement

Efficacy/Effectiveness Trials: An Exemplar at Michigan (SII)

ComparisonAcceleratedSchools

Success for AllAmerica Choice

1.0

0.5

0.0

-0.5

-1.0

Values-basedDecision Making

Innovative Climate

Teacher Autonomy

Press for Innovation & Teacher Autonomy

ComparisonAcceleratedSchools

Success for AllAmerica Choice

1.0

0.5

0.0

-0.5

-1.0

InstructionalGuidance

Monitoring for Fidelity

Press forStandardization

Instructional Guidance & Standardization

The CSR Programs used different strategies to implement their proposed instructional changes

ASP emphasized adaptation/discovery AC and SFA emphasized fidelity


ASP did not produce changes in instructional practice

Figure 1: Instructional Differences Between ASP and Comparison Schools in Literacy Topic Focus Across All Lessons (N=39,720) Topic Focus Mean

(Confidence Interval) Grade Slope Sig.

(t-ratio)

▬ (1.12) ▬ (.85)

▬ (-1.07)

▬ (.13)

▬ (.85)

▬ (.09)

▬ (-1.32)

0.2 0.5 1 2 5 10

Reading Fluency 0.77 (0.58, 1.03)

Vocabulary 0.80 (0.59, 1.08)

Word Analysis 0.89 (0.62, 1.27)

Comprehension 0.90 (0.68, 1.19)

Spelling 0.97 (0.70, 1.34)

Grammar 1.00 (0.70, 1.42)

Writing 1.09 (0.79, 1.52)

Odds Ratio


SFA succeeded in producing “skill based” reading instruction

Figure 5: Instructional Differences Between SFA and Comparison Schools in Literacy Topic Focus Across All Lessons (N=34,182) Topic Focus Mean


(t-ratio)

▬ (.81) ▬ (1.61)

▬ (.18)

▬ (-1.32)

▲ (1.78)

▬ (-.72)

▼ (-2.54)

0.2 0.5 1 2 5 10

Spelling 0.33 (0.23, 0.48)

Grammar 0.48 (0.34, 0.68)

Writing 1.00 (0.73, 1.37)


Vocabulary 1.23 (0.94, 1.62)

Word Analysis 1.50 (1.13, 1.99)



AC produced “literature-based” reading instruction

Figure 3: Instructional Differences Between AC and Comparison Schools in Literacy Topic Focus Across All Lessons (N=40,701) Topic Focus Mean


(t-ratio)

▬ (.59) ▬ (1.30)

▬ (-.27)

▬ (-.89)

▼ (-2.18)

▼ (-1.68)

▼ (-1.73)

0.2 0.5 1 2 5 10

Vocabulary 0.57 (0.42, 0.77)


Spelling 0.71 (0.51, 0.98)

Grammar 0.75 (0.55, 1.03)

Word Analysis 0.78 (0.55, 1.10)


Writing 1.95 (1.33, 2.88)

Odds Ratio


SFA’s “skill-based” program improved reading achievement at the early grades

Student Starting K at 30th Percentile

Comparison Percentile

CSR Percentile

Spring 2nd Grade (ASP) 44th 44th Spring 2nd Grade (AC) 43rd 43rd Spring 2nd Grade (SFA) 39th 53rd Student Starting K at 50th Percentile Spring 2nd Grade (ASP) 66th 66th Spring 2nd Grade (AC) 65th 65th Spring 2nd Grade (SFA) 62nd 74th Student Starting at K 70th Percentile Spring 2nd Grade (ASP) 79th 79th Spring 2nd Grade (AC) 78th 78th Spring 2nd Grade (SFA) 76th 84th


AC’s literature based program improved reading achievement in the upper grades

Student Starting at 30th Percentile

Comparison Percentile

CSR Percentile

Spring 5th Grade (ASP) 19th 22nd Spring 5th Grade (AC) 19th 27th Spring 5th Grade (SFA) 19th 22nd Student Starting at 50th Percentile Spring 5th Grade (ASP) 35th 39th Spring 5th Grade (AC) 35th 46th Spring 5th Grade (SFA) 34th 39th Student Starting at 70th Percentile Spring 5th Grade (ASP) 54th 58th Spring 5th Grade (AC) 53rd 65th Spring 5th Grade (SFA) 52nd 58th

Efficacy/Effectiveness Trials: Features of District Participation

Districts as “subjects” of research.

Districts “recruited” through:

Direct contact from researchers

Support from intervention programs

Districts allowed to serve as gatekeepers to schools

District research office requires proposals District provides access to schools School choice to participate

Schools are given “incentives” for participation

Schools receive $2000/year Teachers receive incentives ($20/q’naire, $300-600 for logging)

Parental consent carefully managed (passive vs. active)

Benefit defined as “cosmopolitan” not “local”

These strategies had varying degrees of success in promoting instructional change. Two of the three programs improved student achievement in reading, but not at all grades.

Summary of District Participation in Research

Development Efficacy/Effectivness

Symmetric Collaboration Assymetric Collaboration

Intensive but localized impact

Less intensive but broader impact

Informal/flexible reckoning of benefits

Formal reckoning ofBenefits (i.e., incentives)

“School Reform and Beyond” in the Research Cycle

Pre-school research:

Ready to Learn: Developing new pre-school literacy program (development phase, going to exploratory research, then to efficacy trial)

Early elementary:

CARSS: Combining effective reading interventions with effective behavioral interventions (moving from design to development stage)

K-12:

UM SOE: Education Leadership Center (development partners needed)

Working at “system” level to meld school, family, community supports for student success.

Based on principals of Kellogg Foundation Spark Initiative

school reform and beyond brian rowan burke a. hinsdale collegiate professor in education research...

Documents

end kindergarten7

quad var months0

linear var months0

educationresearch professor

hinsdale collegiate

beyondbrian rowanburke

risk status

ecls readingmath se