tony bryk - bristol - joining improvement science to nics
DESCRIPTION
A public lecture by Prof Tony Bryk (President of the Carnegie Foundation for the Advancement of Teaching) on the Design-Educational Engineering and Development (DEED) approach to systemic school improvement, through the use of Improvement Science concepts and Networked Improvement Communities.TRANSCRIPT
Anthony S. Bryk
University of Bristol May 21, 2014
2
Triple Aims of Educational Improvement
EFFICIENCY
EFFECTIVENESS
ENGAGEMENT
Be0er Use of Resources
Deeper Learning For All Students
More Relevant
Last Decade: Performance Management Using Accountability to Drive Change
– Set targets
– Create incentives
– Collect data/dashboards
– Hold individuals accountable “Go figure it out or else…”
Last Decade: Evidence-based Practice Movement
An academic has an idea
He/she design and fine tunes an intervention
An RCT field trial (5 years later)
Evidence it can work
Reviewed by What Works Clearing House Goes on an “approved list”
Districts required or “incented” to buy only from approved list
Educators “Implement with Fidelity”
Practice Improves!
But there is a problem, actually many problems.
Last Decade: Communities of Practice
5
Great Dynamism, But PuLng All of This Together
6
Resembles a Tower of Babel
We have to find a be0er way to accelerate learning to improve.
I. Analogical Scavenger in hot pursuit!
9
We can accomplish more together, than
even the best of us can do alone.
Complex systems problems that we now seek to solve
A Better Way: Integrating Two Big Ideas
• Improvement Science Disciplines Efforts
joined to
• Structured Networks that Accelerate the Learning
A Shi& Toward Learning Fast to Implement Well.
II.
Two Networked Improvement Communities
! 1. Extraordinary high failure rates in developmental mathematics courses in community colleges—a gatekeeper to opportunity. – Can colleges dramatically increase the success of
these students to and through college-level math in one year of instruction?
! 2. The weak and incoherent supports by which individuals learn to teach in public school settings. – Can districts learn how to support new teachers to
learn faster, better and hold on to them?
13
III. Forming As a Community: Building Education’s Capacity to Improve
The Six Core Principles
14
! 1. Make the work problem-specific and user-centered.
! 2. Variation in performance is the core problem to address.
! 3. See the system that produces the current outcomes.
! 4. We cannot improve at scale what we cannot measure.
! 5. Anchor practice improvement in disciplined inquiry.
! 6. Accelerate improvements through networked communities.
Taken Together
15
• Disciplined Inquiry
• Rudiments a scientific community
• Aim: systematic practice improvement
The Six Core Principles
16
! 1. Make the work problem-specific and user-centered.
! 2. Variation in performance is the core problem to address.
! 3. See the system that produces current outcomes.
! 4. We cannot improve at scale what we cannot measure.
! 5. Anchor practice improvement in disciplined inquiry.
! 6. Accelerate improvements through networked communities.
How Do We Heal Medicine? Atul Gawande April, 2012
See the System
18
Gawande’s Closing Observation
Making systems work is the great task of my generation of physicians and scientists. But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.
And at the heart of making systems work is the problem of complexity…
When you are confronted by any complex social system with things about it that you’re dissatisfied with and anxious to fix, you cannot just step in and set about fixing with much hope of helping.
This realization is one of the sore discouragements … You cannot meddle with one part of a complex system without the almost certain risk of setting off disastrous events that you hadn’t counted on.
If you want to fix something you are first obliged to seek to understand it…the whole system.
-‐ Lewis Thomas, 1974 Biologist and Essayist
What We Should Work On:
Task complexity Organiza\onal complexity
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
The Invisible Complexity Schooling
22
The Invisible Complexity of Schooling
What We Should Work On:
Task complexity Organiza\onal complexity
Consequences: • Breakdowns • Wide variability
in outcomes
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
What We Should Work On:
Task complexity Organiza\onal complexity
Consequences: • Breakdowns • Wide variability
in outcomes
Need to iden\fy high-‐leverage problems
Design/develop/refine quality work processes
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
I. Problem-‐ & User-‐Centered
• What we tend to do now: a general issue comes into view and
we jump on solu\ons.
! What is the specific problem we’re trying to solve?
60-‐70% Students assigned to developmental math
course.
80% Percent of these
students that never get past this gate.
500,000 students
in every cohort will never complete a college math
requirement.
26
A Specific High-Leverage Problem to Solve
What We Should Work On: How We Work On It:
Task complexity Organiza\onal complexity
Consequences: • Breakdowns • Wide variability
in outcomes
Need to iden\fy high-‐leverage problems
Design/develop/refine quality work processes
Inherent indeterminism
Must learn our way into improvement; “Change it to understand
it.”
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
V. Engage in Disciplined Inquiry
• Design-‐development orienta\on, itera\ve cycles
– The Driving Principle: Quick, minimally intrusive, and an empirical warrant
– Mantra: Learn Fast, Fail Fast, Improve Fast!
Framing the Learning to Improve Challenge
Current Situation Resistant Indifferent Ready
Low confidence: Good idea, But how to
make it work ????
Limited Capacity
Very Small Scale Test
Very Small Scale Test
Very Small Scale Test
Good base Capacity
Very Small Scale Test
Very Small Scale Test
Small Scale Test
High
confidence: Good idea
èè
Can execute
Limited Capacity
Very Small Scale Test
Small Scale Test
Large Scale Test
Good base Capacity
Small Scale Test
Large Scale Test
Implement
Framing the Learning to Improve Challenge
Current Situation Resistant Indifferent Ready
Low confidence: Good idea, But how to
make it work ????
Limited Capacity
Very Small Scale Test
Very Small Scale Test
Very Small Scale Test
Good base Capacity
Very Small Scale Test
Very Small Scale Test
Small Scale Test
High
confidence: Good idea
èè
Can execute
Limited Capacity
Very Small Scale Test
Small Scale Test
Large Scale Test
Good base Capacity
Small Scale Test
Large Scale Test
Implement
Framing the Learning to Improve Challenge
Current Situation Resistant Indifferent Ready
Low confidence: Good idea, But how to
make it work ????
Limited Capacity
Very Small Scale Test
Very Small Scale Test
Very Small Scale Test
Good base Capacity
Very Small Scale Test
Very Small Scale Test
Small Scale Test
High
confidence: Good idea
èè
Can execute
Limited Capacity
Very Small Scale Test
Small Scale Test
Large Scale Test
Good base Capacity
Small Scale Test
Large Scale Test
Implement
Framing the Learning to Improve Challenge
Current Situation Resistant Indifferent Ready
Low confidence: Good idea, But how to
make it work ????
Limited Capacity
Very Small Scale Test
Very Small Scale Test
Small Scale Test
Good base Capacity
Very Small Scale Test
Small Scale Test
Modest Scale Test
High
confidence: Good idea
èè
Can execute
Limited Capacity
Small Scale Test
Modest Scale Test
Large Scale Test
Good base Capacity
Modest Scale Test
Large Scale Test
Implement
Wide-spread
33
Of Educational Reform
Going Fast, but Learning Slow.
34
We Are in Good Company
Shil to Learning Fast to Implement Well
What We Should Work On: How We Work On It:
Task complexity Organiza\onal complexity
Consequences: • Breakdowns • Wide variability
in outcomes
Need to iden\fy high-‐leverage problems
Design/develop/refine quality work processes
Inherent indeterminism
Must learn our way into improvement; “Change it to understand
it.”
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
Developing a Quality Process Reliably at Scale
Develop A Change
Test under mul\ple condi\ons
Test under increasingly varied
condi\ons
Make the change permanent
Ini@al Hunches
System Changes
1 school 1 administrator
5 schools Many administrators
En\re ver\cal team A more diverse group of administrators
District Wide All administrators
Seeing Task Complexity
Seeing Organiza\onal Complexity
Learning to improve feedback conversa\ons between
principals and new teachers PLAN DO
ACT STUDY
What We Should Work On: How We Work On It:
Task complexity Organiza\onal complexity
Consequences: • Breakdowns • Wide variability
in outcomes
Need to iden\fy high-‐leverage problems
Design/develop/refine quality work processes
Inherent indeterminism
Must learn our way into improvement; “Change it to understand
it.”
Can’t always see all the
consequences of what we
do.
Centrality of measurement: How will you know Δ is an improvement?
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
You Cannot Improve at Scale What You Cannot Measure
• Need measureable targets – But, you just can not stand at the end of the line.
• We need process measures \ed to intermediate targets and key process changes.
Trad
ition
al S
eque
nce
Stat
way
Effects: Time to Complete a College Level Math Course 1 Year 2 Years
Triple the
success rate in half the time.
6%
51%
15%
Produc\ve Persistence
Belonging Uncertainty: Suppor\ve social \es
Target: How do we measure it? Mindsets about the
value of math
Mindsets about poten\al to learn
math
Anxiety Regula\on
Study Skills Conceptual Task: reduce 40+ concepts to 5 core ideas focus on underlying malleable causes + change evidence
Prac@cal Measurement:
reduce 900 items to 26 “you have 3 minutes”
A Primary Driver: key intermediate outcome
41
!
Testing a Change Idea: A Starting Strong Package to Enhance Productive Persistence
What We Should Work On: How We Work On It:
Task complexity Organiza\onal complexity
Consequences: • Breakdowns • Wide variability
in outcomes
Need to iden\fy high-‐leverage problems
Design/develop/refine quality work processes
Inherent indeterminism
Must learn our way into improvement; “Change it to understand
it.”
Can’t always see all the
consequences of what we
do.
The Goal: Quality reliably at scale
• Can we get an idea to work? • What will it take to make it
work in many other contexts?
Centrality of measurement: How will you know Δ is an improvement?
Complexity of work in 21st century ins\tu\ons
Need to “see the system”
II. Variation in Performance Is the Problem to Solve
• Cri\cal ques\on is not: “What Works?” But rather:
“How to advance quality among diverse teachers engaging varied popula\ons of students and
working in different organiza\onal contexts?”
Goal: Achieve quality reliably at scale.
Belonging Uncertainty and Stereotype
12% 13% 14%
28%
40%
7% 11% 14%
50%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Never Hardly Ever Some\mes Frequently Always
Pathways Drop
out
All students Black students
“How olen, if ever, do you wonder: ‘Maybe I don't belong here?’”
N = 714 math students
Variation in Pathways Success Rates by College (n=19)
45
1
2 3
4 5
6 7
8
9
11
12 13
14
15
17
18 19
-50%
0%
50%
100%
0% 50% 100%
Stat
way
Stu
dent
s
Non-Statway Matched Comparisons
No improvement line
We also have a failure, why? What can we learn?
Triple success rate line
VI. Accelera@ng Improvement: Tap the Power of Networks
• A source of innova\on
• Mul\ple fast tests/refinements
• Improvement diffusion A Learning Educa/onal System
A A
Improvement Networks: Accelerate Learning in Prac\ce for Improvement
A
B
A A A B
A A A B
A A A B
C
(Englebart,1994)
A System of Social Learning to Improve: Embracing Disciplined Inquiry
Transla@onal Research
Interven@ons (Alpha Labs)
Will it work in our
context and with out students, and if so,
how?
Expert Prac@@oner Knowledge (Subnet)
Building robust clinical
knowledge about effec\ve materials and instruc\onal prac\ces
(PDSA).
Learning from Network Data (Hub Analy@cs)
Learning from observed variability.
Discerning the unseen.
The Network Improvement Paradigm
49
Researchers vs Users “Knowers” “Doers”
All Improvers
What Works!
How to Make It Work! Replicability as the new
Gold Standard.
“Script it” vs. “Every situation is unique”
Develop Quality Processes to
Support Complex Work
Individual Autonomy As Most Prized Norm
Working Together We Can Accomplish More
"
Implement Fast and Scale Wide
Learn Fast to Implement Well
"
Focus on Standard Effect Size
Focus on Sources of Variability in Performance
"
"
"
"
Developing Practice-Based
Evidence
The Network Improvement Paradigm
“The problem that is managing quality is not just an intellectual endeavor; it is a pragmatic one. The point is not just to know what makes things better or worse; it is to make things actually better.”
–Dr. Don Berwick, Founder Institute for Healthcare Improvement
Learning Fast to Implement Well to Achieve Quality Reliably at Scale.
50
It is all about accelerating how we learn in and through practice to improve.
Our thanks to our foundation partners in this work: Carnegie, Gates, Hewlett, Kresge and Lumina.
Also the Institute of Education Sciences and the National Science Foundation