1

The Ball-RUSD PartnershipFinal Evaluation Report

September 23, 2011

Catherine Awsumb Nelson, Ph.D.

2

Key QuestionsI. What did the partnership look like?

a) Designb) Participationc) Quality

II. What impacts did it have?a) Non-participantsb) Personal transfer of participantsc) Broader impacts

III. What does Ball leave behind?IV. What is the potential for sustainability?V. What was learned about investing in district

transformation?

3

2010-11 Data Collection

DATA SOURCE n Details

Individual interviews

22 Executive Cabinet (5), SIL design team (2), CoP2 (2), CoPs (7), IC (6)

SIL case studies 5 schools30 interviews

At each of 5 schools, fall and spring interviews with principal, new team member, and returning team member, plus document analysis

Event observations

9 SIL (3), Literacy Network (3), IC (1), Efficacious Instruction workgroup (1), Sensing team (1)

Event feedback surveys

10 SIL (5), Literacy Network (4), IC (1)

Milestone monitoring meetings

8 Design teams from LN, SIL, and IC, full Executive CabinetMiddle and end of year

Staff survey 364 131 partnership participants (58% response)233 non-participants (35% response)

Student survey 4949 Overall response rate 56% for Grades 4-12

Participant tracker

895 Certificated staff only

4

Ball approach to district transformation Transform, not reform Focus is on systems change Assumption that most necessary expertise is

already in the district Key strategies:

Capacity building Coherence making

5

Why is “coherence” so important? “I hope we are strong enough to

keep it going. I worry about it. Not a reflection on the work Ball has done but the district…we do amazing things in this district, but we tend to dabble. You have to keep things alive yourself if they are working for you because the district moves on. We do a lot of good things but don’t stick with anything long enough to make it great.” (CoP member)

“The ‘here’s another binder’ mentality is what we fight here all the time. So many things they want you to know and learn about. We dip our toes into so many things and they all blur together. I don’t think we need to try so many things at once. With kids, when you bombard them with stuff, it doesn’t work. When you teach for depth, spend the quality time on a unit, that is when learning happens. (SIL member)

Fewer than 10% of respondents agreed with the statement “Once we start a new program in this district we follow up to make sure it

is working.”

6

Critical Features of the Rowland Partnership Ball provided structures and time for inquiry

and collaboration around instruction at three levels Classroom School District

Collaboration supported through design and coaching

Focus on changing adult learning to change student learning

Working with the willing

7

Flipping the adage

In education, money buys time, which creates opportunity for learning. That is the great gift Ball has given Rowland, along with design and coaching that help the professional time align with what we know about what creates impact in the classroom.

The old adage is time is money.

8

U.S. Schools Lag International Competitors in Providing Professional Learning Time The United States is far behind in providing public school

teachers with the kind of high intensity, job-embedded collaborative learning that research shows is most effective in changing practice and improving learning

U.S. teachers report little professional collaboration in designing curriculum and sharing practices, and the collaboration that occurs tends to be weak and not focused on strengthening teaching and learning.

Compared to other nations that outperform the United States on international assessments, American teachers spend much more time teaching students and have significantly less time to plan and learn together.

“Professional Learning in the Learning Profession: A Status Report on Teacher Development in the U.S. and Abroad.”

(National Staff Development Council, 2009)

9

Building Instructional Capacity in RUSDStructure Function Ball supports #

Communities of Practice

Small groups of educators collaborating around a specific literacy topic

•$ for professional books and training•Literacy Network Days•Cluster days•CoP Garden•Developmental framework

120-150

School Instructional Leaders

Teams of 6-8 administrators and teachers tasked with leading professional learning at each site

•Design and facilitation for monthly cross-district meetings•Learning Walks

110-150

Instructional Cabinet

Representatives across district and role groups charged with identifying and supporting district wide instructional priorities

•Meeting design and facilitation•Support for workgroups

25+

Executive Leadership

Existing district leadership structure

•Coaching 5

10

District Context Budget cuts Accountability pressure Demographic shifts Hollowing out of district instructional support

capacity

11

Partnership Participation, 2009-2011

Never; 65.9%2011 only;

8.4%

2010 only; 8.9%

Both years; 16.8%

12

Intermediate and secondary participation increased

Elementary Intermediate High0%

5%

10%

15%

20%

25%

30%

35%

40%

35%

21%

14%

33%

28%

19%

2010 2011

% o

f cla

ssifi

ed

em

plo

ye

es p

art

icip

at-

ing

in

Part

ne

rsh

ip

13

Quality of the Work

14

Ratings for Quality of Professional Learning remarkably stable

% of participants rating partnership professional learning as “Good “ of “Excellent”

2011 2010

Ensuring that all voices are heard 86% 86%

Having a positive impact on student learning 78% 77%

Making it safe to raise difficult issues 78% 74%

Building on existing professional expertise within the district 78% 76%

Focusing on issues directly relevant to my practice 76% 75%

Being grounded in data and/or other evidence of student learning 71% 67%

Striking a good balance between content and process 69% 71%

Using time well 57% 58%

15

Measures of overall intervention quality increased across the board, with the biggest increase in potential for sustainability

Important aspects of the work will continue once Ball personnel and funds are no longer in the district.

Working with Ball helped RUSD respond more strategically to Program Improvement status and budget cuts than we otherwise would have.

Ball Partnership activities help us deeply examine our approach to teaching and learning.

Ball Partnership work is well integrated into the day-to-day work of the district.

District leaders have demonstrated that they are committed to the Partnership work.

Ball Partnership work has helped RUSD move different parts of the system toward a common focus.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

65%

59%

78%

45%

61%

71%

42%

49%

69%

33%

61%

57%

2010 Partic-ipants

% agree/strongly agree

16

Impacts on non-participants

17

All avenues for non-participant awareness increased this year, with informal communication about CoP work growing the most

I haven't heard anything about the Partnership this year

Work from a teacher Community of Practice group was shared in a staff development meeting

I was told about the work of the Instructional Cabinet

Participated in a Learning Walk @ my site led by my school's School Instructional Leadership team

Read about Ball Partnership activity on the district website or in printed materials

Participated in a meeting led by my school's School Instructional Leadership team

I heard informally about the work a teacher Community of Practice was doing

A colleague talked to me about work he/she was doing with Ball

Heard about Ball Partnership activity in a staff meeting

0% 10% 20% 30% 40% 50% 60%

6%

20%

21%

29%

33%

36%

39%

47%

56%

10%

18%

13%

25%

24%

31%

22%

42%

46%

20102011

% of non-participants who heard about partnership through…

18

40-50% of non-participants noticed improvements in site meetings

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

29% 30% 32% 35% 38% 40% 36%

12% 13%14% 12%

14% 15% 20%

Much more so More so

% o

f non-p

art

icip

ants

sayin

g s

ite m

eetings

over

the last

tw

o y

ears

have c

hanged

19

Participant Impacts

20

Reported levels of personal transfer are unchanged from last year

0%10%20%30%40%50%60%70%80%90%

100%

9% 9% 9% 9% 9% 7%

31% 32% 37% 37% 34% 26%

40% 38% 36% 34% 33%38%

Revolutionized my practice Major transferModerate transfer

% o

f part

icip

ants

report

ing level of

transfe

r

21

Participant ratings of broader district impacts

Co

llab

ora

tio

n in

pro

fe..

.

Qu

alit

y o

f p

rofe

ssio

na

...

Co

Ps

as

a m

od

el o

f co

...

No

rms

of

refl

ect

ive

p..

.

Op

po

rtu

nit

ies

for

tea

...

Co

he

ren

t vis

ion

of

ef.

..

Use

of

eff

ect

ive

/re

sea

...

SIL

te

am

s d

eve

lop

ing

...

Imp

rove

d d

eci

sio

n-m

ak..

.

Dis

tric

t b

ett

er

stru

ctu

...

IC d

eve

lop

ing

dis

tric

t...0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

43% 40% 38% 34% 33% 32% 30% 26%20% 16% 14%

43%41% 44%

41% 46% 48% 49%

41%

34% 47%44%

Mod-erate

% o

f p

art

icip

an

ts r

ati

ng

im

pa

ct

ma

jor/

mo

de

rate

22

For all impact areas with a direct comparison from last year, ratings of Major impact increased

2011-Participants rating impact "Major"

Increase from 2010

Increased collaboration in professional learning 43% +17%

Increased quality of professional learning 40% +12%

Establishing norms of reflective practice and rigorous dialogue about instruction 34% +17%

Use of effective/research-based practices for instruction 30% +13%

Improved decision-making processes in the district 20% +10%

District better structured to support effective instruction 16% +6%

23

District instructional support capacity

District-level decisions are made with adequate input from school-based personnel

It is clear where and how decisions about instruction get made

Key resources of time, money, and personnel are clearly connected to instructional improvement priorities

District decisions are grounded in data

The district has a coherent vision of quality instruction

District priorities are clearly focused on supporting and improving instruction

Staff are held accountable for realizing the district's vision of quality instruction

0% 20% 40% 60% 80% 100%

13%

20%

28%

30%

34%

35%

39%

% of participants who agree/strongly agree

24

Capacity Building Structures

25

CoPs took off in 2nd half of year

January March June0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

71%

88% 89%

18%

76%

96%

43%

82%

94%

36%

67%69%

58%

81%

93%

Used a cycle of plan/act/reflect

Jointly examined artifacts of student learning

Put in place ways to capture our learning

Shared learning with colleagues not in our CoP

Moved beyond sharing to true group collaboration (joint work, development of shared mean-ing)

% o

f m

em

bers

sayin

g t

heir

CoP

had d

one

"Mostl

y"

or

"Com

ple

tely

"

26

Communities of Practice participant ratings of rigor up sharply over last year

My CoP had a clear question or purpose to focus our inquiry

We routinely agreed to try specific things in our classroom and then discuss with the

group how they worked

We routinely looked at evidence of student work from our own classrooms as we talked

about how well a specific practice worked

We pushed each other to be rigorous about what works and why in the literacy practice

we were focusing on

All of our members stayed engaged and accountable to each other

We routinely used Process Learning Circles and/or specific conversational processes

like ordered sharing

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

84%

76%

54%

57%

63%

59%

90%

90%

81%

81%

77%

66%20112010

% of CoP members agree/strongly agree

27

Greatest SIL accomplishments were in team development; whole staff learning also significantly impacted

Used learning walks to reflect on instruction in our school

Built shared understanding among the whole staff about what efficacious instruction looks like

Used cultural proficiency as a lens to analyze instruction in our school

Become seen by all staff as leaders of learning in the school

Built shared understanding among our team about what efficacious instruction looks like

Made staff meeting time more collaborative and inquiry based

Made staff meeting time more focused on learning

Built our own competencies as instructional leaders

Learned to function well as a team

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

27%

12%

13%

19%

16%

29%

36%

23%

48%

29%

44%

49%

48%

57%

48%

42%

63%

43%

CompletelyMostly

% of members rating extent to which their team accomplished goal (May SIL, n=95)

28

SIL impact ratings up across the board this year

% of SIL participants who agree/strongly agree 2011

Change from last

year

Because of the SIL work, people across the district are starting to use more similar language about instruction 65% 26%

The SIL work has significantly influenced our site-level professional development approach and agenda 57% 11%

Our SIL team had enough representation to effect change in our site 49% 17%

Expectations for implementing the SIL work at our site were clear 43% -3%

Our SIL team will be a driving force in our school's instructional improvement efforts going forward 56% 11%

SIL has given teachers more of a leadership role over instruction in this school 51% 8%

We made progress this year in making instruction more public in this school 59% 11%

Instructional Capacity assessments of their own effectiveness vary widely across goals

29

Monitoring the effectiveness of PI plan implementation

Increasing district wide awareness of IC

Supporting staff in implementing instructional priorities

Monitoring the implementation of the PI plan

Understanding IC's role

Setting district wide instructional priorities

Setting a direction for RTI district wide

Determining training needed for new data system

Building our own understanding of RTI (within IC)

Launching a system wide pilot of the new data system

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

16%

23%

31%

35%

55%

63%

67%

74%

91%

95%

% of IC rating group group “Effective” or “Extremely Effective”

30

Challenges to district-level coherence-making: Imbalance between mandate and resources

Mandate: Build

instructional coherence

through developing and

supporting instructional

priorities

Capacity

Authority

Lines of communicat

ion

31

Summary of Impacts: What does Ball leave behind?

32

Areas of most and least change this year

Greatest progress Least progress

Rigor/depth of learning for partnership participants

Quality of professional learning for ALL district staff

Concrete agreements around Efficacious Instruction

District ownership, confidence in sustainability

Cross strand connections

New structure for district coherence continued to struggle

Staff assessment of district instructional capacity

Levels of personal transfer stayed flat

33

Summary of Impacts

Transitions

with

momentum

Idea of design Expectation that decisions will be

collaborative and inclusive Expectation that professional

learning will be collaborative CoPs as a vehicle for teacher

directed inquiry into practice Learning Walks starting to de-

privatize practice in some sites SIL starting to re-shape site-based

learning

34

“No going back” “One of the most striking places I saw the impact

was when we had this presenter from county on EL issues. The way they presented was just so foreign from how we do things- it showed how far we have come. It was just, throw up a power point and then we will take your questions- boom. Instead of taking a piece and really working it the way we do now. (Principal)

“I think many of us have passed the point of no return this year. We don’t want to go back. There is no way we are going back to professional development that is not collaborative and self-initiated.” (Teacher)

35

Summary of ImpactsC

hallenges to the emerging system

Haven or silo? Struggle to balance accountability

with “learning as a journey” Divergent conceptions of

assessment Traditional conceptions of teacher

“leadership” Search for plug-in solution still

evident in some areas

36

What does Ball leave behind? Morale maintained during difficult time Cuts made with more intentionality Capacity for the design and facilitation of

adult learning (in a much broader base of staff)

District owns new structures for learning and leadership

Norms about adult learning Agreements about efficacious instruction

37

Looking Forward: Sustainability Potential

38

Majority of participants are optimistic that most impacts will be lasting

Improved decision-making processes in the district

IC developing district-level coherence in instruction

CoPs as a model of collaborative inquiry

Quality of professional learning

Norms of reflective practice/rigorous dialogue

Collaboration in professional learning

33%

41%

44%

53%

54%

56%

58%

59%

59%

62%

65%

% of participants saying impact will "Definitely" or "Probably" last

39

Large increase in confidence that impacts are sustainable

Change in Sustainability Rating

Communities of Practice as a model of collaborative professional inquiry +36%

Increased collaboration in professional learning +35%

School Instructional Leadership (SIL) teams as a means to develop site-level coherence in instruction +34%

Increased quality of professional learning +31%

Improved decision-making processes in the district +14%

The Instructional Cabinet as a means to develop district-level coherence in instruction -1%

40

Moving Forward in RUSD:Efficacious Instruction

41

RUSD Learning Paradigm

Linked paradigm shifts in adult and student learning

Both grounded in brain-mind principles

Process of strands is Learning for Effective Teaching

Content for strands is Teaching for Effective Learning

ADULTS:Learning for

Effective Teaching

InquiryCollaboration

DataOwnership

STUDENTS:

Teaching for Effective Learning:Democratic

RelationshipsClarity

Invested CognitionFeedback

Expert learners

Learn

er ce

nte

red

prin

ciple

s

43

Process for creating and enacting the framework embodies “capacity” Bottom up Incorporated research and practitioner

knowledge Back and forth between the strands “Not a thing” Ongoing opportunities for meaning-making vs.

“Rollout”

44

Baseline findings about Efficacious Instruction in RUSD Teacher clarity is the strongest domain,

relationships and engagement are the weakest

No significant differences in how Hispanic students experience instruction

Quality of instruction as experienced by students drops slowly from 4th to 8th grade, bottoms out in 9th, then climbs again until 12th

Compared to students, teachers overestimate the quality of relationships, underestimate quality of feedback

45

What was learned about district transformation? Slower than reform People want to “thingify” Monitor the balance between ownership and

coherence Changing power relationships at the top is

hardest Broadening the teacher role isn’t easy either

People do need to be taught skills of collaboration and inquiry

Capacity and buy-in are easier to build in the process of doing authentic work