pennsylvania charter schools: what’s working and what’s not

Pennsylvania Charter Schools: What’s Working and What’s Not

Christopher Nelson, Ph.D.The Evaluation CenterWestern Michigan [email protected]

http://www.evaluation.wmich.edu

The Evaluation Center Western Michigan University

2

Purposes of the Talk Identify key strengths and

weaknesses Identify key policy issues Generate discussion of options for

addressing weaknesses and shoring up strengths


3

Outline Background and data sources Evaluative criteria Key findings Questions and discussion

Background and Data Sources


5

Background on the Evaluation Act 22 mandates evaluation after 5 years Contract awarded through competitive

bidding process Current report is culmination of 4 years of

field research and secondary data analysis One of the longest sustained charter school

evaluations Evaluation report focuses mainly on

aggregate trends and patterns


6

Data Sources Site visits to nearly all charter

schools Interviews Focus groups Observations


7

Data Sources (cont’d) Analysis of secondary data

PSSA Demographic information Financial data

Charter school annual reports In-depth information on school missions

and practices


8

Data Sources (cont’d) Documentation provided by schools

Mission statements Internal evaluations

Interviews with district officials Interviews with other stakeholder

groups

Evaluative Criteria


10

Factual Claims +

Criteria of Merit & Worth

=

Evaluative Judgments

Evaluative Criteria

Evaluative judgments combine facts and values

Thus, data alone cannot fully address questions of merit and worth


11

Evaluative Criteria . . .According to Act 22

“Improve pupil learning . . .”

“Provide expanded choices . . .”

“Increase learning opportunities for all pupils . . .”

“Encourage . . . different and innovative teaching practices . . .”

“Create new professional opportunities for teachers . . .”

“Hold charter schools accountable . . .”

Key Findings

Student Achievement


13

Key FindingsStudent Achievement

Student achievement as an evaluative criterion

Key issues Aggregate trends Consistency of the charter school effect Correlates of success


14

1000

1100

1200

1300

1400

1500

1600

Charter schools All PA public schools


Charter school students have typically scored well below the average PA student

But score levels say more about student composition than school effectiveness

150 point difference


15


Charter school students have scored only slightly below those in demog. similar noncharter schools

But matching on demographics might mask less tangible differences (e.g., “interventionist” parents)

1000

1100

1200

1300

1400

1500

1600

Charter schools Demographically & geographicallysimilar schools

36 point difference


16


Average charter school student has gained 15 points per year, net of changes in student composition

-100

-50

0

50

100

1998 1999 2000 2001 2002

Ave

rage

filte

red

scor

e

1997 cohort

1998 cohort

1999 cohort

2000 cohort

Zero indicates parity with comparison schools


17

Key FindingsStudent Achievement Yet, there is considerable variation

among schools – some positive

Com

posi

te F

ilter

ed S

core

Year1999 2000 2001 2002

-400

-200

0

200

400


18


among schools – others negative

C

om

posi

te F

iltere

d S

core

Year1998 1999 2000 2001 2002

-400

-200

0

200

400


19


among schools

C

ompo

site

filt

ered

sco

re

Year1998 1999 2000 2001 2002

-400

-200

0

200

400


20

Key FindingsStudent Achievement – the Bottom Line

Achievement appears to be a source of modest strength

But considerable school-to-school variation

Not yet clear what the drivers of success are

Key Findings

Choice and Satisfaction


22

Key FindingsChoice and Satisfaction

Consumer satisfaction as an evaluative criterion

Key issues Satisfaction with charter schools Number of charter school options Are charter school options any different than those

in noncharter public schools? – the question of “innovation”


23


Parents, students, and teachers are generally satisfied

However, unable to make comparisons with noncharter schools

I am satisfied with the instruction offered

4% 5%14%

24%

53%

0%

20%

40%

60%

80%

100%

Stronglydisagree

Disagree Neitheragree ordisagree

Agree Stronglyagree

The choice “halo” effect?


24


Typical charter school has a self-reported waiting list of 28% of current enrollment

Yet, schools report turnover rates of 0 - 40% per year Dissatisfaction, counseling out, or

realization of poor fit?


25

Key FindingsChoice and the “Supply” of Options

The number of charter school options has grown considerably

0

10

20

30

40

50

60

70

80

1997-98 1998-99 1999-2000 2000-01 2001-02

Num

ber

of S

choo

ls

0

5,000

10,000

15,000

20,000

25,000

30,000

Num

ber of Students

SchoolsStudents


26

Key FindingsChoice and the “Supply” of Options

Yet, those options remain concentrated in a few regions of the Commonwealth

Only 18 of 67 counties (27%) have charter schools

Of those, charter enrollment exceeds 1% in only 10

County2001/02

Enrollment(% total)

Philadelphia 7.9

Clinton 4.4

Beaver 4.2

Bucks 3.8

Chester 2.5

Charter EnrollmentTop 5 Counties


27

Key FindingsChoice and Innovation

Many charter schools offer new choices to particular education markets

Non-traditional grade groupings (67%)

Unique mission foci Non-traditional use of

time and calendar Smaller schools &

classes Cyber schools

Non-traditional grade groupings (67%)

Unique mission foci Non-traditional use of

time and calendar Smaller schools &

classes Cyber schools


28

Key FindingsChoice and Innovation

However, there is little evidence that charter schools are creating wholly new educational approaches

Barriers to innovation Slack time Slack resources Challenges of start-up Consumer preferences Problems with CAB

definition


29

Key FindingsChoice and Innovation – the Bottom Line

Parents, students, and teachers appear to be quite satisfied with their charter schools

Number of charter school options has grown considerably but remains geographically concentrated

Many charter schools offer new options in particular markets, but few appear to be producing wholly new educational approaches

Key Findings

Equity and Access


31

Key FindingsEquity and Access

School choice . . . Engine of desegregation? Sorting machine?

Key issues Types of communities served Types of students selected (self-

selected?) from these communities


32

Key FindingsEquity and Access – School Location

Charter schools tend to be in areas with larger schools, lower test scores, and higher concentrations of low income and nonwhite students.

Variable Nonchartering Chartering

Median total district enrollment (2001-02) 1,749 3,446

Median per-pupil expenditure (2000-01) $8,319 $8,304

Median aggregate PSSA score (2001-02) 1341 1221

Median % low income students (2001-02) 20.3 63.9

Median % nonwhite students (2001-02) 4.9 67.0


33

Key FindingsEquity and Access – (Self?) Selection

Declining relative percentage over time

Considerable school-by-school variation

Charter-Host Demographica Differences

-20

-10

0

10

20

30

1997-98 1998-99 1999-00 2000-01 2002-02

Ch

arte

r-H

ost

(%

)%Nonwhite

More nonwhite students


34


Slightly more low income students

Relatively stable over time


Charter-Host Differences

-20

-10

0

10

20

30

1997-98 1998-99 1999-00 2000-01 2002-02

Ch

arte

r-H

ost

(%

)%Nonwhite

%FRL


35


Fewer students with IEPs

IEP students in charter schools tend to have less severe disabilities


Charter-Host Demographica Differences

-20

-10

0

10

20

30

1997-98 1998-99 1999-00 2000-01 2002-02

Ch

arte

r-H

ost

(%

)%Nonw hite

%FRL

%IEP


36

Key FindingsEquity and Access – The Bottom Line

On the whole, charter schools have target communities of need

In the aggregate, little systematic evidence of cream-skimming

Possible exception of special education Outlier schools might raise some red flags Negative trend on race Some anecdotal evidence of cream-skimming


37

Key FindingsEquity and Access – The Bottom Line

But, findings limited by limitations in using enrollment data

Family self-selection vs. school selection

Key Policy Issues

Achievement


39

Key Policy Issues & OptionsAchievement

What are the correlates of success?

How can we reduce variability on the low end?

What is the potential for success?

Study correlates of success

Facilitate identification and dissemination of best practices


40

Key Policy Issues & OptionsAccountability as Possible Correlate

Accountability lies at the heart of the charter concept

PA has done better than most states in reporting and technical assistance

Yet, there is still room for improvement


41


Continue improvements to charter school annual report format

Make information public to facilitate market accountability

Continue providing technical assistance in developing clear, measurable goals


42


Consider assessment of LEA capacity for oversight (wills, skills, & bills) Consider the impact of “just-in-time”

oversight by many LEAs Consider targeted state oversight

based on past performance & compliance

Key Policy Issues

Choice and Equity


44

Key Policy Issues & OptionsChoice & Equity

Is choice good in itself or as tool for increasing achievement?

Is innovation reasonable for charter schools?

What counts as “innovation”?

Possible trade-off between choice and equity

Slippery relationship between satisfaction and achievement


45

Slippery Relationship Between Achievement & Satisfaction

r = -0.08 (p = 0.64)

Ave

rage a

nnual

gain

(filter

ed PS

SA sc

ores)

Mean satisfaction level (parents, teachers, and students)3 3.5 4 4.5

-100

-50

0

50

100

Key Policy Issues

Cross-Cutting Issues


47

Key Policy Issues & OptionsCross-Cutting Issues

Balancing achievement, choice, and equity – the challenges of multi-attribute evaluation systems

Cost-effectiveness How do charter schools stack up against other

investments in educational improvement? Actual vs. potential charter school

performance Addressing questions of fact and value


48

Final Thought “By providing important data and

raising key policy issues, this report has sought to make a sound contribution to the debate over the Commonwealth’s charter schools”

pennsylvania charter schools: what’s working and what’s not

Documents

mandates evaluation

school missions

pennsylvania charter

secondary data analysisone

similar schools1

pupil learning

talkidentify key strengths

new professional opportunities