from model demonstration to sustained implementation of evidence-based practices pat gonzalez, ph.d....
TRANSCRIPT
From Model Demonstration to Sustained Implementation of
Evidence-Based Practices
Pat Gonzalez, Ph.D. MDCC Project Officer, OSEP
Debbie Shaver, Ph.D. MDCC Co-Principal Investigator, SRI International
Ed Shapiro, Ph.D., Lehigh University
Suzanne Robinson, Ph.D., University of Kansas
Mary Wagner, Ph.D. MDCC Principal Investigator, SRI International
Model Demonstration Coordination Center (MDCC)
Debbie ShaverMDCC Co-Principal
InvestigatorSRI International
• Since 1970’s OSEP has funded model demonstrations projects (MDPs) to better understand how to improve outcomes for children and youth with disabilities.
• Since 2005, OSEP has funded MDCC to document and learn from the model development and implementation process and outcomes of eight cohorts of MDPs.
OSEP's model demonstrations and MDCC
3
MDCC works with cohorts of MDPs to:• Facilitate collaborative partnerships to learn and
share.• Help establish consistent design elements to deepen
evidence base for models.• Synthesize findings across MDPs to identify factors
that lead to high-quality implementation, sustainability, and wider adoption of evidence-based practices
• Identify key issues in translating research to practice to improve outcomes for children and youth
MDCC Roles
4
Cohort 4: Tiered writing approaches for
secondary students
Cohort 5: Tiered interventions for
English language learners
Cohort 6: Assistive technology for
young children
Cohort 7: Reentry of students from
juvenile justice facilities
Cohort 8: Stepping up technology implementation
Active Cohorts of MDP Grantees
5
Cohort 1: Progress monitoring in elementary reading
Cohort 2: Tertiary behavior interventions
Cohort 3: Early childhood language development
Previous Cohorts of MDP Grantees
6
7
Framework for understanding model implementation and outcomes
INTERVENTIONIMPLEMENTATION
INTERVENTIONOUTCOMES
SourceThe Model
Core intervention components―e.g.,▪ Methods to develop child-specific AT
plans for infants, toddlers, and preschoolchildren with disabilities
▪ Methods for developing effective localprograms for re-use of AT
▪ Methods to modify local policies andprocedures to facilitate the use of AT
FeedbackModel evaluationFidelity dataSocial validity data
InfluencesState/localOther external
factors
DestinationParticipating Organizations and Staff
Characteristics of participatingorganizations, programs, and staff
Implementation outcomes
Changes in:▪ Staff knowledge, attitudes, skills, and
actions
▪ Organizational structures, processes, andculture
▪ Community and other peripheralrelationships
Sustained implementation
PurveyorThe MDP Grantee
Core implementation components▪ Partnering with programs and services
in the community
▪ Providing professional development,training and support including coaching
▪ Ongoing assessment of implementa-tion quality
▪ Staff selection and staffing strategy
Systems Outcomes▪ Positive changes to
policies and procedures
▪ Positive changes toresource allocations (e.g.,increased access to AT)
Family Outcomes▪ Increased ability to help
their children develop andlearn
Child Outcomes▪ Increases in positive
social-emotional skills
▪ Increases in acquisitionand use of knowledge andskills
▪ Increases in use ofappropriate behavior tomeet needs
Family/childcharacteristics
Note: Adapted from Implementation Research: A Synthesis of the Literature (Fixsen et al., 2005).
Fixsen, D. L., Naoom, S. F., Blase, K. A. Friedman, R. M., and Wallace, F. (2005). Implementationresearch: A synthesis of the lite
Intervention Components
1. How do core intervention components change over time in response to implementation experiences, and how does this process differ across MDPs and cohorts?
2. How do variations in the core intervention components relate to fidelity of intervention, acceptance of the model, and establishment of conditions for sustaining the intervention?
Implementation Components
3. How do variations in core implementation components relate to fidelity of implementation, acceptance of the model, and establishment of conditions for sustained implementation?
MDCC evaluation questions [1 of 3]
8
Destination Organizations4. How do variations in the destination organizations (e.g.,
population served, organizational functioning, staff and leadership, climate/culture, resources, support for the model) relate to fidelity of implementation, acceptance of the model, and establishment of conditions supportive for sustained implementation?
Context5. How do variations in contextual factors relate to fidelity of
implementation, acceptance of the model, and establishment of conditions for sustained implementation?
MDCC evaluation questions [2 of 3]
9
Outcomes6. How do variations in core intervention and implementation
components, destination organizations, and contexts relate to:– Implementation outcomes (changes in practice/functioning
in the destination organizations, and changes in the skills, attitudes, and beliefs of staff)
– Child/youth outcomes– Systems-level outcomes?
MDCC evaluation questions [3 of 3]
10
MDCC:• Analyzes data provided by MDPs to address
evaluation questions throughout model implementation.
• Reports results of analyses and lessons learned to OSEP at implementation conclusion.
Then OSEP asks,• What happens to the models after the MDPs exit
their sites? • What promotes and hinders model sustainability
there and spread to other sites?
Following up on implementation experiences
11
• OSEP commissioned MDCC to facilitate early-cohort MDPs in returning to their sites to document:– The core intervention components of the
models that were still in place in their original schools and districts• as originally implemented,• adapted, or • abandoned.
– The extent to which the models, in whole or in part, had been implemented outside the original schools and districts.
Follow-up studies
12
• Develop qualitative templates for MDPs to use in reporting what they found.
• Document MDPs’ data sources and collection methods.
• Compile follow-up data collected by MDPs.• Facilitate conversations about the factors MDP
staff believed helped and hindered model sustainability and spread.
• Synthesize the findings and report to OSEP.
MDCC follow-up activities
13
• Follow-up reports from the first two cohorts can be found at:Model Demonstration Center Website's 'Products and Reports' page: http://mdcc.sri.com/prod_serv.html
• Also available:– Descriptions of cohorts and
projects– Cohort summary reports– “Briefs” on implementation-
related topics– Contact information
MDCC information and products
14
What Stays, What Goes, and Why?
Sustainability of Model Demonstration Project MP3
EDWARD S. SHAPIRO, PH.D.DIRECTOR, CENTER FOR PROMOTING
RESEARCH TO PRACTICEPROFESSOR, SCHOOL PSYCHOLOGY
LEHIGH UNIVERSITY
Agenda
Brief description of Project MP3What remained two years later.What changed (enhanced/diminished)Key factors identified by the site in
sustainability Technical factors (e.g., instructional
practices, financing systems), Human factors (e.g., skill levels,
communication patterns, staffing changes), Contextual factors (e.g., policy shifts,
demographics)
MP3 Objectives
Project MP3 documents the implementation of a school-wide progress monitoring model in five elementary schools across Pennsylvania.
The model provides general and special education teachers (K- 4) with the necessary support and expertise to use progress monitoring data to determine students’ needs for reading intervention and to develop a Response-to-Intervention framework for special education referral.
Objectives
Result in increased reading achievement for students in participating schools
Equip schools to prevent reading failure by “Catching” students falling behind Introducing intensive, differentiated, scientifically-
based interventions that alter students’ trajectories
Document school adoption of new practices
Project MP3
Large district (by PA standards), 11,750 students 13 elementary, 4 middle, 2 high schools
Funded in fall 2005, 9 month planning periodImplementation began fall 2006 in one schoolImplementation in all 3 schools in 2007All project support ended in June 2009Revisited for sustainability in April 2011Continued interaction with district currently
Participating Schools
School A
K-4 Enrollment
318
% Free/Reduced
32%
%Minority
30%
% 3rd Graders Passing
Reading PSSA
48%
K-4 Enrollment
302
% Free/Reduced
50%
%Minority
55%
% 3rd Graders Passing
Reading PSSA
77%
K-4 Enrollment
257
% Free/Reduced
32%
%Minority
45%
% 3rd Graders Passing
Reading PSSA
76%
TIER 1: All students in core program(Everyone taught reading from H-M)
Fall Benchmark
Student benchmarkscore = BENCHMARK
(90% will do fine)
TIER 1 Enrichment30 min 4-5x week
PM 3x/year
Student benchmarkscore = STRATEGIC
(Might be at risk)
TIER 2 Intervention30-45 min 4-5x weekPM every other week
Student benchmarkscore = INTENSIVE(Definitely at risk)
TIER 3 Intervention30-45 min 4-5x week
PM 1x week
TIER 1: All students in core program(Everyone taught reading from H-M)
Winter Benchmark
When we were finished….
Benchmarking & progress monitoring processes
Data based decision making team processes at school (administrative) and grade (teacher) level
Multi-tiered system of supports (MTSS)Tools for managing MTSSTools for analyzing MTSSFidelity of instruction and intervention
assessmentsParent engagement
Enhancements two years later: Technical factors
Fall benchmark for kindergarten push from September to October
Jump start program – interventions began before fall benchmark for students identified from previous spring in need of intensive intervention.
Grade level meetings in one school alternate between all grades meeting together at the same time one month, followed by focused and individual grade levels meeting the alternate month.
Data collection tools enhanced
Enhancements two years later: Technical factors
Evaluation of fidelity continued, mostly by principalAdded period of intervention for intensive studentsExpansion to all 13 elementary schoolsExpansion to 1 middle school (selected as pilot site)Two schools added PBIS modelSwitch from DIBELS to AIMSwebContinuation of parental engagement process
(letters)Desigated point of contact (teacher) for each grade
levelAll schools applied for and approved for using RTI
for SLD determination (state approval process)
Enhancements two years later: Human factors
Replication of model at all schoolsModel embraced as part of school cultureMinimal teacher turnoverSchools examined data outcomes and
focused on enhancing core instruction
Challenges: Contextual factors
Resource reductionsAdministrative staff changes, key leaders
retired Superintendent (twice) Director of C & I- elementary Director of Special Ed Director of Pupil Services Several supervisors Principals at two of three MP3
schools
General comments: Sustainability
Leadership and commitmentCulture shift, embedded perspectivesApproval at state level for SLD
determination useMaintenance of resources in light of budget
cutsPlanned from the start“What will be there when we leave?”“Can the staff continue without our
support?”Fade and scaffold
Two years later
Continuation despite staff changesMaintenance and routine part of school
culturesSupport at all levels of district from central
to localHiring includes knowledge and experience
with RTI
Thanks
Dr. Ed ShapiroEmail: [email protected]
RtI and Writing Instruction in High Schools
Suzanne RobinsonAssociate Professor of Special Education
University of Kansas
• Background for this “story”• Model demonstration project funded by OSEP 2010-2013• RtI focus on 9th grade, expanded to 10th grade• Focus on writing instruction
– Strategic writing program developed at KU-CRL for core instruction; Tier 2 adapted, focused instruction; Tier 3 intensive, personalized instruction
• Multiple baseline design• Year 1 in one urban district (ended relationship). Year 2 in
district with diverse population in different state. Year 3 added another high school. Year 4 focus on building sustained practices.
Secondary school RtI (tiered support) context
31Cohort 6 Model Demonstration Projects
Tier 1
1. Strategic Writing as 50% of the 9th English Language Arts curriculum (supported in the other
disciplines)
Tier 2
1. Small groups (some writing groups, some literature groups)
2. Co-teaching classes (2 teachers = 2+ groups)
Tier 3 (trials still underway on what works)
3. Writing class/lab taught by ELA teachers
4. Writing class/lab taught by Special Education teachers resurrected
5. SPED Resource classes with students meeting IEP goals/Tier 3 needs resurrected
6. Self-contained special education classes and ELL classes
7. Co-teaching special education teacher (during resource period) sometimes yes/sometimes
no
8. Peer tutors (yet to be implemented)
Impact of program on student performance (2011-2012)
33Cohort 6 Model Demonstration Projects
Impact of program on student performance (2011-2012) [1 of 7]
34Cohort 6 Model Demonstration Projects
Impact of program on student performance (2011-2012) [2 of 7]
35Cohort 6 Model Demonstration Projects
Impact of program on student performance (2011-2012) [3 of 7]
36Cohort 6 Model Demonstration Projects
Impact of program on student performance (2011-2012) [4 of 7]
37Cohort 6 Model Demonstration Projects
Impact of program on student performance [ 5 of 7]
38Cohort 6 Model Demonstration Projects
Impact of program on student performance (2011-2012) [6 of 7]
39Cohort 6 Model Demonstration Projects
Impact of program on student performance (2011-2012) [7 of 7]
40Cohort 6 Model Demonstration Projects
Common Implementation Challenges in Secondary School Tiered Support Models
• Staff capacity
• Fidelity
• Accountability
• Scheduling
• Resources
• Beliefs about roles and instructional responsibility
_____________ Negotiated contract constraints Other school reform initiatives
Sustaining … threats and strategies
1. District leadership, building leadership, and school staff must want the change and must recommit as often as needed as their understanding of what change entails grows and reconsider what systemic adoption entails (scaling up).
2. The students in the school have to need what the reform offers in ways that matter.
3. The district, school, and teachers must be able to assist in implementation and set up structures to practice, maintain, and sustain the reform (model components, roles on leadership teams, PD/coaching, accountability management).
4. Solutions need to be systemic and they need to be as simple as possible.
• Must come to consensus about what is absolutely necessary for accelerating achievement for all students in terms of:– Type of instruction needed– Necessity and possibility of mastery– How to facilitate mastery– Duration of instruction in the context of other demands– How to provide Tier 3 supports in the context of credit/scheduling
demands– Who will provide Tier 3 support and how to facilitate timely
communication between Tier 1 and 3 teachers
• How to engage all teachers (across departments) and support or specialized service personnel (ELL, SpEd, remedial, etc.)
43Cohort 6 Model Demonstration Projects
Lessons learned in tiered support systems in academic arenas with secondary schools. . .
• Technical considerations in sustaining effective practices• Contextual considerations in sustaining effective practices• Human considerations in sustaining effective practices
Almost always, stakeholders in school-change projects have different, sometimes competing, as well as compatible interests in the outcomes of any reform effort. The interplay between compatible interests and different and/or personal interests can cause tension; these tensions compete for attention and can cause relational problems among partners in change.
In conclusion
44Cohort 6 Model Demonstration Projects
What Promotes and Hinders Model Sustainability?
Mary WagnerMDCC Principal Investigator
SRI International
Model demonstration projects (MDPs) with strong implementation had:
– “Rolled” with the surprises, working with their sites where they found them.
– Focused on changing:• The knowledge, attitudes, skills, and behaviors of key
adults at all levels of the system.• Formal and informal organizational structures and cultures.• Relationships with stakeholders.
– Built capacity for change at all levels.– Sold their sites on the importance of collecting and using
data.– Produced evidence of impacts ASAP.– Actively learned from their experiences and adapted as
they went.
Implement well
46
• Multi-level leadership• Compatibility with ongoing systems; adapting to fit.• Improved practice and outcomes
– They also improve administrator and practitioner efficacy.
– Staff stayed committed over time.• Leaving behind useful, affordable tools (e.g.,
assessment materials, meeting protocols)• Increasing and relying on system-level supports
– District policies and resources drove sustainability. “It’s just the way we do things now.”
– State-level TA systems were critical for spread.
Supports for sustainability and spread
47
• Staff turnover; capacity doesn’t stay built – The antidote: a culture of good practice.
• Competing initiatives. – The antidote: use data to make the case; don’t forget
the power of parents.• Low salience of the problem being solved
– The antidote: use data to make the case; don’t forget the power of parents.
• Budget cuts. – The antidote: creative repurposing of resources;
“strong fingernails.”
Challenges to sustainability and spread
48
• Cohort 3 (early childhood language promotion models) final report coming soon.
• Cohort 3 submits data from its follow-up study to MDCC in August; report will follow.
• MDCC is preparing a “brief” on issues involved in MDPs replicating and disseminating promising models.
• Continued facilitation of cohorts 4 through 8.• Cohort 9 technology grantees starting up in the
fall.• Learn more at http://mdcc.sri.com !!
What’s next?
49
Questions??