northwestern dbir brown bag
DESCRIPTION
Brown Bag presentation by Barry Fishman and Bill Penuel at Northwestern University School of Education and Social Policy on Design-Based Implementation Research (DBIR), presented on Thursday, May 23rd, 2013TRANSCRIPT
Design-Based Implementation Research:
Working in Partnership(s) to Transform the Relationship of Research and Practice
Bill Penuel (@bpenuel)University of Colorado Boulder
Barry Fishman (@barryfishman)University of Michigan
May 23, 2013
Acknowledgments• National Science Foundation 1054086
• Annie Allen, University of ColoradoBritte Haugan Cheng, SRI InternationalNora Sabelli, SRI InternationalAndy Krumm, SRI International
Where We Begin• Many promising educational interventions have been
developed, validated, tested... Then what?
• The majority fade away as funding ends or attention turns elsewhere
• A few are sustained in (or near) the contexts where they were developed
• A very few are brought to scale and are used across settings/contexts
• Why is the rate of “success” so low?
A Validity Problem• Interventions are usually developed in “hothouse” environments
• Researchers seek to reduce sources of variation in evaluations to increase internal validity
• Funded research focuses more on developing and validating interventions from basic research (Type I translation) than on understanding or closing gaps between research and practice (Type II translation)
Type I Translation
Type II Translation
What is being translated?
Translating principles from basic learning
research into interventions
Translating interventions
developed for one or a few settings into
interventions that are scalable to many
settings
What kind of research is involved?
Design-based researchEfficacy and
effectiveness trials
Implementation research
(for example, DBIR)
Type I Translation
Type II Translation
What kinds of questions does translational
research answer?
What/how do people learn from this design?
What do problems in learning or implementation
suggest about redesign?
What kinds of capacities are required to implement
this design?What supports are needed for people implementing the design to adapt it in ways congruent with the design’s core principles?
Who is involved?
Learning scientists, classroom teachers,
subject matter experts, often also software
developers
Learning scientists, organizational researchers, teacher leaders, school and
district administrators, often also publishers and
enterprise software engineers
Design-Based Research• The objective of DBR is to develop theory, not
just empirically tune “what works”• “Design experiments create the conditions for
developing theories yet must place these theories in harm’s way.”
• The theory is expected to do “real work in practical educational contexts”
(Cobb, Confrey, diSessa, Lehrer, Schauble, 2003)
Design-Based Implementation Research• Blends learning sciences and policy
implementation research traditions and methods
• Learning sciences: iterative, collaborative, guided by and informing theories of learning/teaching
• Policy implementation: focus on conditions for implementation effectiveness, guided by and informing theories of institutional change and organizational learning. Focus on the design of systems and infrastructure
DBIR Principles1. A focus on jointly-defined problems of teaching and
learning practice
2. A commitment to iterative, collaborative design
3. A concern with developing knowledge and theory through disciplined inquiry
4. A goal of developing capacity for sustaining change in systems
Source: Penuel, W. R., Fishman, B., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331–337.
Example: The Center for Learning Technologies in Urban Schools
Commitment to Iteration & Collaboration
• All work unfolded over multiple phases of pilot testing and revision, starting with a small core of teachers and then expanding
• Design teams formed “work circles” that included university and school personnel
• Project leadership worked to forge new relationships within and across institutions
– E.g., Bringing curriculum and technology leadership together in Detroit Public Schools
– E.g., Convening technology leadership across districts
Concern for Developing Knowledge and Theory Through Inquiry• The goal throughout was not only to create better
materials
• Goal was to develop new knowledge by developing the theory of educative curriculum materials, with a range of subgoals, e.g.:
– Study of teacher learning through professional development that linked PD to student outcomes
– Research on cognition supported by technology
– Evolving theory of system capacity for supporting reform, and how to design for systemic change
A Concern with Developing Capacity to Sustain Change
Evidence Standards for DBIR
Design and Development
EfficacyTrials
EffectivenessStudies
Involvement of R&D Team
Involvement of Evaluators & Practitioners
Design and Development
EfficacyTrials
EffectivenessStudies
Involvement of R&D Team
Involvement of Evaluators & Practitioners
Sustainability?
+0.50
+0.00
Replication… Really?
• Organizational replication is often thought of as a process that yields reliable results at the expense of local and professional control
• Peurach and Glazer (2011) argue for a “knowledge-based logic” in which program developers collaborate with schools to produce, use, improve, and retain practical knowledge
Key Differences from (old) IES Model
• Teams plan for scaling and sustainability from the start
• Designers maintain their involvement throughout, because design is ongoing and iterative
• Powerful innovations are not expected to survive in the wild without changes at the system level
Generating Forms of Evidence in an Era of “Fast Science”
• Get it mostly right, fast.– How do you know you “got it right”?– What’s a “minimum viable” innovation to test?
• Fail early, fail often.– How can you put your theory in harm’s way from the
start?– What documentation can benefit your own and other
teams?– When do you do the “big test”?
Matching Phase of Development to Phase of Research
Phase of Development
Driving Questions Sources of Evidence
Problem Negotiation What problem of practice should be the focus of our joint work?
Available data from multiple sectorsResearch evidencePerspectives and values of stakeholders (including nonschool actors)
Co-design What should be the focus of our work?To what extent do teams leverage the diverse expertise of stakeholders?
Design RationalesEthnographic accounts of design processes
Matching Phase of Development to Phase of Research
Phase of Development
Driving Questions Sources of Evidence
Early implementation How do implementers adapt the innovation to their local contexts?How do implementers use the innovation to reconstruct their practice?What are the appropriate measures of impact?
Observations of implementationInterviewsAssessment design
Efficacy What is the potential impact of the innovation on teaching and learning?What mediates impacts on learning?
Randomized Controlled TrialsInterrupted Time Series DesignsExplanatory Case Studies
Matching Phase of Development to Phase of Research
Phase of Development
Driving Questions Sources of Evidence
“Translation” (Type II) What supports are needed to implement the program effectively?What are the conditions for sustainability?
Experimental comparisons of different means of supportExplanatory comparative case analysis
Challenges in DBIR
• Establishing shared relevance• Balancing rewards across partners• Collaborating across disciplines• Synchronizing time tables and cycles• Earning trust
Funders Interested in DBIR
• National Science Foundation• US Department of Education– RELs– Research-practice partnerships– Continuous Improvement Research in
Education
• William T. Grant Foundation– Use of research evidence RFP– Funding of work in research-practice
partnerships (white paper, Coburn-Penuel study, Carnegie study)
Coming Soon
Forthcoming volume on DBIR:
Fishman, B., Penuel, W. R., Allen, A., & Cheng, B. H. (Eds.). (2013). Design-based implementation research: Theories, methods, and exemplars. National Society for the Study of Education Yearbook, Vol. 112(2). New York: Teachers College Record.
Also: http://tinyurl.com/dbir-aera