Similar English LearnerSimilar English Learner Students, Different Results Students, Different Results

Why Do Some Schools Do Why Do Some Schools Do Better?Better?

EdSourceEdSourcewww.edsource.orgwww.edsource.org

Trish Williams, Kenji Hakuta, Ed Haertel, Mike Trish Williams, Kenji Hakuta, Ed Haertel, Mike Kirst, Jesse Levin, Robert Linquanti, Noli Brazil, Kirst, Jesse Levin, Robert Linquanti, Noli Brazil,

Mary Perry, Isabel OregonMary Perry, Isabel Oregon

Castañeda v. PickardCastañeda v. Pickard

Sound theory

ImplementationResults

examine

evaluate

reform

revise

Castañeda v. PickardCastañeda v. Pickard

Sound theory

ImplementationResults

examine

evaluate

reform

revise

Reading Comprehension

0 1 2 3 4 5 6GRADE

3

4

5

6

7

8

9

10

11

12A

ge E

quiv

alen

t Sco

re

English-OnlyBilingual

Program

0 1 2 3 4 5 63

6

9

12

Opportunity CostsOpportunity Costs

AIR / WestEd StudyAIR / WestEd Study Staff capacity to address EL Staff capacity to address EL

needs;needs; Schoolwide focus;Schoolwide focus; Shared priorities and Shared priorities and

expectations; andexpectations; and Systematic assessment and data-Systematic assessment and data-

driven decision-making.driven decision-making.

The Research QuestionThe Research QuestionWhy do some schools serving similarly Why do some schools serving similarly

challenged student populations vary challenged student populations vary in their performance on the API by in their performance on the API by

as many as 250 points? as many as 250 points?

Extended analysis in 2006-07:Extended analysis in 2006-07: Why do some schools serving similarly Why do some schools serving similarly

challenged EL student populations challenged EL student populations vary in their performance on the EL-vary in their performance on the EL-

API by as many as 250 points?API by as many as 250 points?

2005 English Learner Base 2005 English Learner Base Academic Performance Index (API) Academic Performance Index (API)

vs. School Characteristics Index vs. School Characteristics Index (SCI)(SCI)

300

400

500

600

700

800

900

1000

120 130 140 150 160 170 180 190 200 210

School Characterstics Index (SCI)

2005

EL

Bas

e A

cade

mic

Per

form

ance

Inde

x (A

PI)

25th-35th Percentile

Number of Schools with EL API: 485

EL API Scores Max: 811 Median: 663 Min: 508

Schools with most challenges Schools with fewest challenges

Survey DevelopmentSurvey Development Review of effective schools literature Review of effective schools literature

updated for current CA standards-based updated for current CA standards-based policy environmentpolicy environment

Focused on concrete, actionable items, Focused on concrete, actionable items, neutrally phrasedneutrally phrased

Initial surveys reviewed by academics, Initial surveys reviewed by academics, state policymakers, K-12 educatorsstate policymakers, K-12 educators

Piloted study in 11 elementary schools Piloted study in 11 elementary schools

Size of SSDR Sample is Size of SSDR Sample is ImpressiveImpressive

257 elementary schools from 145 districts257 elementary schools from 145 districts 80% of K-5 teachers at each school80% of K-5 teachers at each school 5,500 total completed teacher surveys5,500 total completed teacher surveys All principals from each schoolAll principals from each school About 50% of the schools from the 25-35 About 50% of the schools from the 25-35

bandband Teacher survey was 31 pages with 63 multi-Teacher survey was 31 pages with 63 multi-

part questions, totaling 370 items eachpart questions, totaling 370 items each The principal survey had 442 itemsThe principal survey had 442 items

Geography of SSDR Sample Geography of SSDR Sample SchoolsSchools

AreaArea # of Districts # of Districts # of Schools # of Schools

Northern CaliforniaNorthern California 4747 7474

Central ValleyCentral Valley 3434 5353

Southern CaliforniaSouthern California 6464 129129

SSDR Sample Student SSDR Sample Student DemographicsDemographics

Average Values for:Average Values for:

SampleSampleSchoolsSchools

All CA All CA Elem. Elem. SchoolsSchools

English LearnersEnglish Learners 42%42% 32%32%Low IncomeLow Income 78%78% 57%57%

African AmericanAfrican American 8%8% 8%8%AsianAsian 6%6% 8%8%HispanicHispanic 66%66% 50%50%WhiteWhite 15%15% 29%29%

Restricting the Sample to Look at EL:

Restricting the Sample to Look at EL:

Survey Research Survey Research DomainsDomains

1.1. Prioritizing student achievementPrioritizing student achievement2.2. Implementing a coherent, standards-based Implementing a coherent, standards-based

instructional programinstructional program3.3. Using assessment data to improve student Using assessment data to improve student

achievement and instructionachievement and instruction4.4. Ensuring availability of instructional Ensuring availability of instructional

resourcesresources5.5. Enforcing high expectations for student Enforcing high expectations for student

behaviorbehavior6.6. Encouraging teacher collaboration and Encouraging teacher collaboration and

providing professional developmentproviding professional development7.7. Involving and supporting parentsInvolving and supporting parents

FindingsFindingsWhich schoolwide and district Which schoolwide and district

practices differentiated schools with practices differentiated schools with higher EL-APIs from schools with higher EL-APIs from schools with lower EL-APIs, serving similar EL lower EL-APIs, serving similar EL

student populations?student populations?

Findings: Use of Findings: Use of Assessment DataAssessment Data

Teachers report receiving CST data in variety Teachers report receiving CST data in variety of formatsof formats

Principals report directly using data from CST Principals report directly using data from CST and other tests in many ways related to and other tests in many ways related to student learning and instructional student learning and instructional improvementimprovement

Principals report that district evaluates them Principals report that district evaluates them upon and provides support for site level upon and provides support for site level planning related to improving student planning related to improving student achievementachievement

Findings: Ensuring Findings: Ensuring Availability of Instructional Availability of Instructional

ResourcesResources Teacher competence is strong:Teacher competence is strong:

teachers report 5 years experience and teachers report 5 years experience and a standard credentiala standard credential

principal reports staff have strong skills principal reports staff have strong skills and positive attitudesand positive attitudes

Both report school has current texts Both report school has current texts for every student, adequate for every student, adequate materials, maintained facilitymaterials, maintained facility

Findings: Curriculum and Findings: Curriculum and Instructional Program is Instructional Program is

CoherentCoherent Based on CA academic content standards, Based on CA academic content standards,

especially math and ELAespecially math and ELA

Consistent within each grade and aligned Consistent within each grade and aligned from one grade to the nextfrom one grade to the next

District provides the grade by grade District provides the grade by grade curriculum, principal provides the curriculum, principal provides the oversight, and teachers make it happenoversight, and teachers make it happen

Findings: Prioritizing Findings: Prioritizing Student AchievementStudent Achievement

School vision = student learning School vision = student learning outcomesoutcomes

Principal makes expectations clearPrincipal makes expectations clear Teachers embrace responsibilityTeachers embrace responsibility Plans for improvement are well definedPlans for improvement are well defined API/AYP goals are a priorityAPI/AYP goals are a priority Measurable goals for exceeding the Measurable goals for exceeding the

API subgroup growth targetsAPI subgroup growth targets

Effect SizeEffect SizeDomain (#)Domain (#) Predicted Predicted

variation in variation in EL-APIEL-API

Predicted Predicted variation in variation in schoolwide schoolwide

API*API*Using assessment data (3)Using assessment data (3) 20.420.4 16.716.7Availability of resources (5)Availability of resources (5) 17.917.9 16.916.9Coherent standards-based Coherent standards-based instruction (1)instruction (1)

17.317.3 17.617.6

Prioritizing student Prioritizing student achievement (8)achievement (8)

14.714.7 16.316.3

High expectations for High expectations for student behavior (6)student behavior (6)

10.310.3 12.312.3

Involving and supporting Involving and supporting parents (2)parents (2)

10.110.1 9.99.9

Teacher collaboration (4)Teacher collaboration (4) 9.49.4 11.011.0

School Sample (N)School Sample (N) 237 (high EL)237 (high EL) 257 (original 257 (original sample)sample)

Predicted variation in API scores (based on standard deviations) for each domain in order of importance.

*Based on findings published in Similar Students, Different Results, 2005

Observation: Vertical Observation: Vertical Alignment Alignment

District to ClassroomDistrict to Classroom Strong district leadership, accountability, Strong district leadership, accountability,

and support appear to influence schoolwide and support appear to influence schoolwide and EL APIand EL API

Principal leadership is being redefined to Principal leadership is being redefined to focus on effective management of the school focus on effective management of the school improvement processimprovement process

Teachers get clear signals and support, Teachers get clear signals and support, work together to implement curriculum, and work together to implement curriculum, and take responsibility for student achievementtake responsibility for student achievement

Analysis of EL-Analysis of EL-Specific Survey Specific Survey

QuestionsQuestions

FindingsFindings Higher school EL–API was correlated with Higher school EL–API was correlated with

principals who affirmed that in the last principals who affirmed that in the last four years, their school has implemented four years, their school has implemented a new program for EL students.a new program for EL students.

Also more positively correlated with Also more positively correlated with higher EL-API was response by a school’s higher EL-API was response by a school’s teachers that explicit English Language teachers that explicit English Language Development instruction was delivered to Development instruction was delivered to the teacher’s EL students through a pull the teacher’s EL students through a pull out program (e.g. resource teacher). out program (e.g. resource teacher).

FindingsFindings The response by the teachers at a school that The response by the teachers at a school that

their EL students were taught mathematics their EL students were taught mathematics using ESL or immersion techniques (SDAIE) using ESL or immersion techniques (SDAIE) was also more positively correlated with a was also more positively correlated with a higher schoolwide weighted mean scale score higher schoolwide weighted mean scale score for math on the CST.for math on the CST.

In addition, as was also found in our original In addition, as was also found in our original analysis, schools with higher EL–API more analysis, schools with higher EL–API more often had teachers who agreed strongly that often had teachers who agreed strongly that their district addresses the instructional needs their district addresses the instructional needs of English Learner students at their school.of English Learner students at their school.

Positive with AMAOPositive with AMAO• • School principal responses School principal responses

indicating that a new EL program indicating that a new EL program had been implemented at the had been implemented at the school within the past four years school within the past four years was positively correlated with was positively correlated with AMAO.AMAO.

Negative with AMAONegative with AMAO Negatively and significantly correlated with Negatively and significantly correlated with

AMAO 1 and 2 were strong school-level AMAO 1 and 2 were strong school-level teacher responses that explicit ELD teacher responses that explicit ELD instruction was delivered to EL students by instruction was delivered to EL students by the classroom teacher herself, or by ELD the classroom teacher herself, or by ELD level through teacher teaming.level through teacher teaming.

Also negatively correlated were teacher Also negatively correlated were teacher reports that they receive CELDT reports that they receive CELDT assessment data and that, besides CELDT, assessment data and that, besides CELDT, teachers use no other type of ELD teachers use no other type of ELD assessment for their EL students.assessment for their EL students.

An An InterpretationInterpretation

We interpret these results to complement those We interpret these results to complement those discussed elsewhere in this report. Namely, discussed elsewhere in this report. Namely,

having in place an explicit program of having in place an explicit program of instruction for ELs is fundamentally instruction for ELs is fundamentally important to their English language important to their English language

development needs, and enhancing EL development needs, and enhancing EL students’ access to and comprehension of students’ access to and comprehension of core academic subjects such as math may core academic subjects such as math may

also yield benefits to their academic language also yield benefits to their academic language development in English.development in English.

Some SurprisesSome Surprises We were surprised that the We were surprised that the

presence in a school of more presence in a school of more teachers with CLAD/BLCAD teachers with CLAD/BLCAD certification did not have a certification did not have a relationship, in our analysis, to relationship, in our analysis, to schools’ EL–API scores.schools’ EL–API scores.

Some SurprisesSome Surprises Another surprise was that the Another surprise was that the

number of minutes teachers number of minutes teachers reported devoting to explicit reported devoting to explicit ELD instruction at the school ELD instruction at the school did not correlate with EL–API.did not correlate with EL–API.

An An InterpretationInterpretation

Quality trumps Quality trumps quantity.quantity.

Bigger Picture: Castañeda Bigger Picture: Castañeda v. Pickardv. Pickard

Sound theory

ImplementationResults

examine

evaluate

reform

revise

The Daunting The Daunting ChallengeChallenge

The rich get richer.The rich get richer.

THANK YOUTHANK YOU

For more information:For more information:

hakuta@stanford.eduhakuta@stanford.eduwww.edsource.org

EdSourceEdSource520 San Antonio Road, Ste. 200520 San Antonio Road, Ste. 200Mountain View, CA 94040-1217Mountain View, CA 94040-1217

(650) 917-9481 (650) 917-9481