building implementation capacity to improve youth outcomes allison metz, ph.d. associate director...
Post on 18-Dec-2015
220 Views
Preview:
TRANSCRIPT
Building Implementation Capacity to Improve Youth Outcomes
Allison Metz, Ph.D.Associate Director
National Implementation Research Network University of North Carolina at Chapel Hill
Agenda• The Challenge• Implementation Science• Plan for Change• Active Implementation Frameworks
The Challenge: Recognizing the Gaps• Science to Service Gap
– What is known to be effective is not what is selected to help students
• Implementation Gap– What is selected is not used with fidelity and good
outcomes– What is used with fidelity is not sustained for a useful
period of time– What is used with fidelity is not used on a scale
sufficient to broadly impact youth outcomes
Implementation Gap
RESEARCHPRACTICE
Implementation is defined as a specified set of activities designed to put into practice an activity or program of known dimensions.
IMPLEMENTATION
RESEARCH PRACTICEGAP
Why Focus on Implementation?
IMPLEMENTATION
“Children, youth, and families cannot benefit from interventions they do not experience.”
Effective NOT Effective
Effective
NOT Effective
IMPLEMENTATION
INT
ER
VE
NT
ION Actual Benefits
(Institute of Medicine, 2000; 2001; 2009; New Freedom Commission on Mental Health, 2003; National Commission on Excellence in Education,1983; Department of Health and Human Services, 1999)
Inconsistent; Not Sustainable; Poor outcomes
Unpredictable or poor outcomes;
Poor outcomes; Sometimes harmful
from Mark Lipsey’s 2009 Meta-analytic overview of the primary factors that characterize effective juvenile offender interventions – “. . . in some analyses, the quality with which the intervention is implemented has been as strongly related to recidivism effects as the type of program, so much so that a well-implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”
“Implementation science is the systematic study of variables and conditions that lead to full and effective use of evidence-based programs and other effective innovations in typical human service settings.”
—Blase and Fixsen, 2010National Implementation Research
Network
“Implementation Science”
Download at: http://www.fpg.unc.edu/~nirn/resources/publications/Monograph/
Best data show these methods, when used alone Do not Result in Implementation as Intended– Diffusion/ Dissemination of information– Training – Passing laws/ mandates/ regulations– Providing funding/ incentives– Organization change/ reorganization
5 to 10% return on investmentNECESSARY BUT NOT SUFFICIENT
Plan for Change: “Making It Happen”Letting it happen
– Recipients are accountable
Helping it happen– Recipients are accountable
Making it happen– Purposeful use of implementation practice and science– Implementation teams are accountable
—Based on Hall & Hord (1987); Greenhalgh, Robert, MacFarlane, Bate, & Kyriakidou (2004);Fixsen, Blase, Duda, Naoom, & Van Dyke (2010)
To successfully implement and sustain evidence-based and evidence-informed youth violence prevention interventions, we need to know:
WHAT to doWhat is the intervention (e.g.identified prevention, intervention and enforcement strategies that are feasible and relevant)?
HOW to do itActive and effective implementation and sustainability frameworks (e.g. strategies to change and maintain behavior of adults)
WHO will do itOrganized, purposeful, & active implementation support from linked implementation teams
Q. How?
A. Effective Implementation• Changing the behavior of practitioners and administrators • Creating the setting conditions to facilitate these changes• Creating the processes to maintain and improve these
changes in both setting conditions and behavior of well-intentioned adults
• So that youth benefit
Active Implementation Frameworks
Implementation Drivers Implementation Stages
Improvement Cycles Implementation Teams
IMPLEMENTATION DRIVERS
Common features of successful supports to help make full and effective use of a wide variety of innovations
Core Implementation
Components
© Fixsen & Blase, 2008
Positive Outcomes for Youth
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Effective Youth Violence Prevention Practices
Capacity to provide direction and vision
Staff capacity to support youth with the selected practices
Institutional capacity to support practitioners in implementing practices with fidelity
Performance Assessment(Fidelity)
Coaching
Training
Selection
Systems Intervention
Facilitative Administration
Decision Support Data System
Adaptive
Technical
Integrated & Compensatory
Com
pete
ncy
Driv
ers
Com
pete
ncy
Driv
ers O
rganization Drivers
Organization D
rivers
LeadershipLeadership
Improved youth outcomes
Interventions meet
Implementation
Consistent Use of
Prevention Innovations
© Fixsen & Blase, 2008
IMPLEMENTATION STAGES
Purposeful matching of critical implementation activities to the stage of the process
Stages AND Drivers
Implementation Takes
Time: 2 – 4 Years
EXPLORATIO
N
IN
STALL
ATION
INIT
IAL
IMPLE
MENTA
TION
FULL
IMPLE
MENTA
TIONDrivers
Drivers
Drivers
“DRIVERS”
Stages of Implementation
EXPLORATIONC
ompe
tenc
y D
river
s Organization D
rivers
Leadership Drivers
Integrated & Compensatory
—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
“Pay now or Pay later”
Goals of Exploration
• Create readiness for change• Changing hearts and minds
• Examine degree to which the proposed strategies and practices meet the needs of our community and our youth
• Determine whether the strategies, practices, and implementation are desirable and feasible
The HexagonAn EBP Exploration Tool
NEED
FIT
RESOURCES
EVIDENCE
CAPACITY
READINESS
Fit with current Initiatives• School, district , state priorities• Organizational structures
Community values
Need in school, district, state• Academic & socially significant Issues• Parent & community perceptions of need• Data indicating need
Resources and supports for:• Curricula & Classroom• Technology supports (IT dept.)• Staffing• Training• Data Systems• Coaching & Supervision• Administration & system
Evidence• Outcomes – Is it worth it?• Fidelity data• Cost – effectiveness data• Number of studies• Population similarities• Diverse cultural groups• Efficacy or Effectiveness
Capacity to Implement• Staff meet minimum qualifications• Able to sustain Imp Drivers
• Financially • Structurally
• Buy-in process operationalized• Practitioners • Families
Readiness for Replication• Qualified purveyor• Expert or TA available• Mature sites to observe• Several replications• How well is it operationalized?• Are Imp Drivers operationalized?
The “Hexagon” can be used as a planning tool to evaluate evidence-based programs and practices during the Exploration Stage of Implementation.
Download available at:www.scalingup.org/tools-and-resources
EBP:
5 Point Rating Scale:High = 5; Medium = 3; Low = 1.Midpoints can be used and scored as a 2 or 4.
High Med Low
Need
Fit
Resource Availability
Evidence
Readiness for Replication
Capacity to Implement
Total Score
© National Implementation Research Network 2009-2012 Adapted from work by Laurel J. Kiser, Michelle Zabel, Albert A. Zachik, and Joan Smith at the University
of Maryland
Stages of Implementation
Com
pete
ncy
Dri
vers
Organization D
rivers
Leadership Drivers
Integrated & Compensatory
INST
ALL
ATIO
N
—Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
“If you build it, they will come”. . .
but you actually have to build it!
Goals of Installation
• Structural and functional changes are made to support implementation
• Staff selection protocols developed
• First ‘practitioners’ selected
• Define and initiate training of first cohort of practitioners
• Develop coaching system and plans
• Evaluate readiness and sustainability of data systems (e.g. fidelity, outcomes)
Stages of Implementation
Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
EXPLORATION
INSTALLATIO
N
INITIAL
IMPLEMENTATION
“Get Started, then Get Better.”
Com
pete
ncy
Dri
vers
Organization D
riversLeadership Drivers
Integrated & Compensatory
Goals of Initial Implementation
• Work through the Awkwardness
• Provide training and coaching on the evidence-based practice, re-organization of school roles, functions and structures
• Make use of improvement cycles to resolve systems issues
• Continue buy-in efforts and manage expectations
• All the components of the program or innovation are at least partially in place and the implementation supports begin to function
Stages of Implementation
2 - 4
Years
Fixsen, Naoom, Blase, Friedman, & Wallace, 2005
FULL
IMPLEMENTATION
“The only thing worse than failing and not
knowing why you failed, is succeeding and
not knowing why you succeeded.”
Goals of Full Implementation
• Maintaining and improving skills and activities throughout the system
• Components integrated, fully functioning
• Skillful practices by front line staff, supervisors, administrators (50% meet performance criteria)
• Changes in policy are reflected in practice at all levels
• Ready to be evaluated for expected outcomes
IMPROVEMENT CYCLES
Changing on purpose to support the new way of work
PDSA CyclesImprovement Cycles
Plan
DoStudy
Act
• New practices do not fare well in existing
organizational structures and systems
• Effective innovations are changed to fit the system, as opposed to existing systems changing to support effective innovations.
• People, organizations, and systems. . .• Cannot change everything at once (too big;
too complex; too many of them and too few of us)
• Cannot stop and re-tool (have to create the new in the midst of the existing)
• Cannot know what to do at every step (we will know it when we get there)
• Many outcomes are not predictable (who knew!?)
• Plan-Do-Study-Act Cycles Rapid cycle problem solving
(Shewhart; Deming) Transformation Zone Usability testing (Neilson; Rubin) Practice-policy communication loops
Practice-Policy Communication Cycle
Policy
Practice
Po
licy En
ables P
ractices
Plan
DoEx
tern
al I
mp
lem
enta
tio
n S
up
po
rt Policy
Practice
Structure
Procedure
Pra
ctic
e In
form
s P
oli
cy
Fee
db
ack
Stu
dy - A
ct
FORM SUPPORTS FUNCTION
IMPLEMENTATION TEAMS
Organized, expert assistance to develop and sustain an accountable and effective structure
Implementation Teams• Provide accountable and effective structure to move
intervention through stages of implementation • Scope of the initiative determines the number of teams
and the linked communication protocols needed• Focus is on
– Ongoing “buy-in” and readiness– Installing and sustaining the Implementation Drivers – Fidelity & Outcomes– Systems Alignment and Stage-based work– Problem-solving and sustainability
“We tend to focus on snapshots of isolated parts of the system and wonder why our deepest problems never seem to get solved.”
—Senge, 1990
Linked Team StructuresImplementation Teams
Site-basedImplementation Team
Site-basedImplementation Team
Community-basedImplementation Team
Community-basedImplementation Team
Regionally-basedImplementation Team
State-basedImplementation Team
State-basedImplementation Team
Who Makes Change Happen?
Implementation Teams
• INDICATORS CORE COMPETENCIES
• Representative• Accountable
Team Structure
• Formal/Practice Knowledge• Model fluency • Fully operationalize Intervention
Know the Intervention
• Best practices for Implementation Core components
• Stage-appropriate Work
Know Implementation
• Communicate Change• Use of data for decision making,
problem solving and feedback loops
Know Improvement Cycles
•Knowledge and skills for system building and components
Know Systems Change
SO THAT...
…can effect change throughout the system and
make decisions
…can promote implementation of core components,
adaptations, and infrastructure
…can guide implementation and build capacity throughout the organization and system
…can develop and follow-through on action planning
…can support efforts to improve access, reach or scale, improve connection, influence decision-
making
Examples of SuccessImplementation results (higher fidelity) and intervention results (improved outcomes) improve when Active Implementation Frameworks are used purposefully and systematically• Catawba County Child Wellbeing Project- 5 year period to develop, implement
and evaluate post-care service system for children exiting foster care; high fidelity and early indicators of success
• Colorado EPIC Project- Scale-up of MI using Active Implementation Frameworks in corrections
• EPIS Center at Penn State University - 419 age-grade cohorts over a 5-year period: youth in CTC communities using EBPs had significantly lower rates of delinquency, greater resistance to negative peer influence, stronger school engagement and better academic achievement
• Evidence-Based Model Purveyors – NFP, MST, FFT use many components of active implementation frameworks to achieve sustainable outcomes
Summary: “Making it Happen” for youth Purposeful selection of an effective and feasible “What”
Conceptualize a change process so that effective interventions for children and families can become embedded and sustained in socially complex settings “stage-matched activities” to guide the process “implementation drivers” to build the infrastructure
Improvement processes are critical the work is never done because the environment is in motion
Invest in the development of organized, “expert” implementation support
Stay Connected!
nirn.fpg.unc.edu www.scalingup.org
www.implementationconference.org
Allison.metz@unc.edunirn@unc.edu
top related