Leveraging Implementation Practice and Research

For The Greater Good

Sonja K. Schoenwald Professor of Psychiatry & Behavioral Sciences

Medical University of South Carolina April 8, 2014

The Fourth Annual CiMH Evidence-Based Practice Symposium

Sacramento, CA

Brought to You By •California Institute for Mental Health 2 NIMH-Funded Centers for Services and Intervention Research • Implementation Methods Research Group (IMRG); John Landsverk, PI

– NIH 1P30MH074678 (Advanced Center) • LINKS Center, Marc Atkins, PI

– NIH 1P20MH0784458 (Developing Center) 2 Philanthropies • The Annie E. Casey Foundation • The Research Network on Youth Mental Health, John Weisz, PI

– John D. and Catherine T. MacArthur Foundation Several NIMH and NIDA funded studies, including • National Institute of Mental Health, S. Schoenwald & J. Chapman Co-PIs

– 1 R21 MH097000-01, 06/11/12 – 05/31/2014 • Transportability of New Treatments… 1R01MH59138, S. Schoenwald, PI • Testing Context Effects… R01 DA018107-01, S. Schoenwald, PI • Several effectiveness and implementation studies conducted by Scott Henggeler And, I am a Board Member and Stockholder, MST Group, LLC, which has the exclusive licensing agreement through MUSC for the dissemination of MST technology

This Morning

• Why care about implementation? • What research tells us about implementation

strategies and tools; a selective review • What we can do to increase the impact of

practice and research in implementation; some ideas, some examples

Why Care About Implementation? • People cannot benefit from innovations they

do not experience (courtesy of Dean Fixsen) • How well an innovation works and how well it

is implemented are two different things

Someone left the cake out in the rain. I don’t think that I can take it. Cause it took so long to bake it. And I’ll never have that recipe again, Oh, no!

Why is Implementation Research Relevant? Dissemination Step Concept % Impacted

50% of Clinics Use Adoption 50%

50% of Practitioners Recommend Adoption 50%

50% of Patients Accept Recommendation/Attempt Change

Reach 12.5%

50% Follow Regimen Advice Correctly Implementation 6.2%

50% of Those Implementing Correctly have Substantial Benefit

Effectiveness 3.1%

50% Continue to Adhere and Benefit after 6 Months

Maintenance 1.6%

Courtesy of Russell Glasgow, Re-Aim http://www.re-aim.org/

Source: Aarons, Hurlburt & Horwitz, 2011

ClLIENT OUTCOMES

• Implementation research examines “the use of

socio-behavioral strategies to adopt, integrate, and scale-up evidence-based interventions and change practice patterns within specific settings” Chambers, Ringeisen, & Hickman, 2005

Implementation Research Leverages The

Value and Impact of Both Treatment Research and Clinical Practice

Model for Implementation Science

8

Implementation Strategies

Outcomes

Service Outcomes*

Efficiency Safety Effectiveness* Equity Patient-centeredness Timeliness

Client Outcomes

Satisfaction Function* Symptomotology*

Implementation Outcomes

Feasibility Fidelity* Penetration Acceptability Sustainability Uptake

Costs

Intervention Strategies

Evidence-Based Practices

Implementation Research Methods

Systems Environment

Organizational*

Group/Learning*

Supervision*

Individual * Providers/Consumers *IOM Standards of Care

Red bold font* = Today’s talk

Source: Proctor, Landsverk, Aarons, Chambers, Glisson, & Mittman, 2009

Quality Tools for Implementation

• Information-centered work requires quality tools that are designed and refined on the basis of relevant research (e.g., educational, cognitive, informatics) and user input (Wandersman, Chien, & Katz, 2012)

• The same guidance applies to human services (with domains of relevant research varying)

Training

• Adequate training can reduce variations in provider behavior, improve fidelity, and increase service delivery quality.

• In what settings, under what training and support conditions, can this be achieved?

• Research on training and support in EBP implementation contexts is in early stages

• But, we do know from research on learning, adult education, and professional training. . .

Evidence-Based Training ?

“Telling Ain’t Training Training Ain’t Performance”

Harold Stolovitch

Learning and Performance Solutions American Society for Training and Development and the International

Society for Performance Improvement Courtesy of the

Annapolis Coalition on the Behavioral Health Workforce http://www.annapoliscoalition.org/

Performance Based Training

• Know: Knowledge acquisition • Know How: Skill Acquisition • Do: Skill performance • Performance-based training includes

o Deliberate practice o Experiential learning o Feedback o Support

Toward Evidence-Based Training • One-time workshops can increase knowledge and

attitude toward change but not behavior • Building proficiency appears to require repetitions and

active learning o One estimate: 20 – 25 implementation attempts to achieve

consistent professional behavior change (Joyce & colleagues, 2002)

• Approaches & techniques recently reviewed o academic detailing o coaching o interprofessional learning o reminders o self-regulated learning o problem-based learning

Lyon, Wiltsey Stirman, Kerns, & Bruns (2011)

Coaching and Consultation

Common Types1 • Expertise-based • Peer-to- peer

Common Methods1 • Direct feedback following observation • Coach reflection on own practice

Emerging Type? • Peer - Expert

1Lyon, Wiltsey Stirman, Kerns, & Bruns (2011)

Coaching and Consultation

• Expertise-Based Coaching Examples o Trauma-Focused Cognitive Behavioral Treatment o Parent-Child Interaction Training o Multisystemic Therapy Consultation effects on adherence and outcomes

• Peer Coaching Example o Triple P Positive Parenting Program encourages

peer support groups One study underway to evaluate advantages and

challenges of using these groups

Lyon, Wiltsey Stirman, Kerns, & Bruns, 2011

Coaching and Consultation

• Specific functions, processes, of consultation o Engagement o Problem solving implementation barriers o Direct case application o Appropriate adaptation o Accountability o Mastery skill building o Sustainability planning

*Nadeem, Gleacher, & Beidas (2013). Consultation as an implementation strategy for evidence-

based practices across multiple contexts: Unpacking the black box. Administration & Policy in Mental Health and Mental Health Services Research

Toward Evidence-Based Consultation • Dosage, delivery-method, collaboration, &

proactive nature matter (Wandersman, Chien, & Katz, 2012)

• Less is known about the specific functions and processes that are most effective

• Study underway: o Stirman & colleagues (2013) RCT: 3 consultation conditions for clinicians treating PTSD

in military veterans with Cognitive Processing Therapy (CPT) Outcomes: CPT adherence, competence; client symptoms

Stirman et al., (2013)

Clinical Supervision What Constitutes Effective Supervision?

• Clinician behavior change • Effects on client outcomes

“The “acid test” of good supervision is client outcomes” (Ellis & Ladany, 1997)

• The “acid test” is rarely conducted • Quality of EBT coaching in clinical

supervision associated with: o Decreased emotional exhaustion & turnover intention

(Knudsen & colleagues, 2008) o Decreased turnover (Aarons & colleagues 2009)

Toward Evidence-Based Supervision

Examples from EBT Implementation Research

• “Old”News: MST Transportability Research o 9-site pilot included supervisor adherence

measurement development o 43-site study

• “New” News: • Research Network on Youth Mental Health,

System and Treatment Enhancement Projects o Prospective observational study of supervision o During RCT of treatment effectiveness o Valid, reliable measurement of supervision

Toward Evidence-Based Supervision

MST Transportability Study

• Two of four MST SAM scales predicted one-year post-treatment changes in youth behavior problems

• One scale predicted therapist adherence • Direct effects of supervisor adherence and

therapist adherence were observed

Schoenwald, Sheidow, & Chapman, 2009

Clinical Supervision, MST Transportability Study

SAM Structure & Process Predicting Post-Treatment Criminal Charges

0.86 (+1 SD)1 (Max.)

0 (Min.)

0.76 (Mean)0.66 (-1 SD)

1

1.5

2

2.5

3

3.5

4

0 (Min.) 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 (Max.)

Supervisor SAMSP

Num

ber o

f Pos

t-Tre

atm

ent C

harg

es

*MST Transportability Study, funded by NIMH and NIDA

Results reported in Schoenwald, 2008

Toward Evidence-Based Supervision

New News: “More Practice, Less Preach?”

• 57 community-based therapists trained in EBPs • 12 doctoral level supervisors with EBP expertise • 136 youths and families • Modeling and role-play higher use than discussion • Modeling practice use in next session • Greater effects for older clinicians • Discussion predicted use for male therapists (n = 18)

Bearman, Weisz, Chorpita et al & The Research Network on Youth Mental Health, 2013

Toward Evidence-Based Supervision

Newest News – Study In Progress • Randomized trial of supervision in public mental

health in Washington State (Dorsey & colleagues, in press)

o Descriptive study of supervision of those trained in TF-CBT via WA public mental health initiative

o RCT of supervision elements from efficacy trials (session review, model fidelity, outcome monitoring, skill-building – behavioral rehearsal) Both conditions: Symptom and fidelity monitoring One condition: SFM plus Behavioral Rehearsal in

supervision

Feedback To Practitioners Health Care

• Audit-and-feedback systems • Effective for:

o Relatively straightforward tasks o Physicians demonstrating low baseline adherence o When the feedback is sufficiently detailed and intensive.

(Grimshaw et al, 2001; Jamvedt et al., 2006)

Mental Health/Behavioral Health • In mental health/behavioral health, there are fewer,

and fewer rigorous, studies • Evidence from randomized trials appears promising

Feedback To Practitioners

• Feedback to school-based providers about fidelity to Coping Power as rated on observational measures improved fidelity and outcomes

(Lochman, Boxmeyer, Powel, Qu, Wells, & Windle, 2009.)

• Feedback to therapists on standardized measures of client progress and outcome

(Bickman, Kelley, Breda, De Andrade, & Riemer, 2011).

o Outcomes for children were better when practitioners had access to weekly feedback

o Positive association between practitioners’ frequency of use of outcome feedback and the extent of child improvement

From Practitioner Feedback to Implementation Feedback Systems

“Implementation feedback systems could support the reach, quality, and sustainability of evidence-based treatments”

McLeod, Southam-Gerow, Tully, Rodriguez, & Smith (2013). Making a case for

treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice, 20, 14-32.

Implementation Feedback Systems • Distinguish closer to real time the failure of

implementation from the failure of intervention

• Refine the target of implementation support o Practitioners: Intervention- specific skills and performance o Organization: policies, procedures, leadership,

implementation climate, climate-and-culture profile o Service system: Regulations, procedures, norms, inter-

organizational interactions

• Support “learning organization” process and culture

• Contribute “shared yardstick of accountability”* amongst the multiple participants in implementation *Yeaton & Sechrest, 1981

You Get What You Measure Measurement Focuses Attention, Behavior, & Resources Some Examples

o Hospital Accreditation o No Child Left Behind o Child Welfare

Potentially Perverse Effects o No Child Left . . . Untested? o Child Welfare Paperwork

“Driving with Roadmaps & Dashboards…”

Fig. 3 Individual client summary example from the child STEPs clinical trial (Chorpita et al. 2008 ) Springer Science and Business Media

Help?

Data Dashboards

Characteristics of

Effective Measurement

Characteristics of

Efficient Measurement

Balancing characteristics of effective and efficient adherence measurement on the basis of measurement purpose(s)

Purpose(s)

Stakes (Higher/Lower)

Number (Single, Multiple)

Schoenwald, Garland, Chapman, Frazier, Southam-Gerow, & Sheidow, 2011

Quality Tools for Implementation

• Quality tools alone are not enough Wandersman, Chien, & Katz, 2012

Multicomponent Strategies

Examples • Availability, Responsiveness, Continuity (ARC;

Glisson & colleagues) • Community Development Teams (CDT; CIMH) • Interagency Collaborative Teams (ICT; Aarons

& colleagues, under evaluation) • Learning Collaboratives (see Nadeem et al., in press,

Millbank Quarterly)

Availability, Responsiveness, Continuity (ARC; Glisson & Colleagues)

Objective • “Help community based service settings improve their social contexts

by creating the capacity to identify & address contextual deficits and barriers that impede service and treatment outcomes” (Glisson, Hemmelgarn, Green, & Williams, 2013, p. 494)

Means • ARC facilitator • Team-based clinician and management focused activities • Three broad strategies

o Embed 5 principles into the organization’s processes & practices o Use 12 component tools to identify and address service barriers

(within organization and in inter-organizational relations) o Cultivate service provider attitudes and behaviors (e.g., openness

to change • Implement over an 18-month period

Rural Appalachian Project (RAP) Randomized Trial of MST & ARC

in a 2-Level EBT Implementation Strategy Glisson, Schoenwald, Hemmelgarn, Green, Dukes, Armstrong, & Chapman, 2010

Questions • Are MST model fidelity and outcomes better

in counties that receive ARC? • Are youth outcomes better in ARC counties

than non-ARC counties? • Are youth outcomes better for youth who

receive MST compared to youth who receive the usual services?

RAP Implementation Results

• MST Implementation in ARC and non-ARC o No differences in adherence (therapist or supervisor)

o Therapists in ARC counties rated progress with extra-familial systems more highly and spent fewer minutes within the family system

o But, no differences in the odds a therapist addressed a particular system in natural or service ecology

RAP Outcomes Results **

Out of home placement* 18-months after baseline

• ARC reduced the odds of placement by 44% • MST reduced the odds of placement by 53% • Together, MST and ARC reduced the odds of out of

home placement from 34% to 16% • Mechanism of action for combined effects not yet

evaluated (e.g., is effect in part accounted for by ARC-induced change in team climate/ culture?)

*Parent-reported

MTFC Implementation Trial*

• Randomized trial of Multidimensional Treatment Foster Care (MTFC)

• 40 counties in California; 11 counties in Ohio • Randomized to 2 implementation conditions Standard Implementation Community Development Teams (CDT)

• Both conditions received standard MTFC training, clinical consultation and QA procedures

• Randomized to cohorts for start time • Total of 51 sites consented

* Courtesy of Patti Chamberlain and Lisa Saldana; Research Funding from NIMH, Children’s Bureau, WT Grant Foundation

8 Stages: Involvement: 1. Engagement System Leader 2. Consideration of Feasibility System Leader, Agency 3. Readiness Planning System Leader, Agency 4. Staff Hired and Trained Agency, Practitioner 5. Adherence Monitoring Practitioner, Client Established 6. Services and Consultation Practitioner, Client 7. Ongoing Services, Consultation Practitioner, Client Fidelity Monitoring, Feedback 8. Competency (certification) System Leader, Agency,

Practitioner, Client

Stages of Implementation Completion (SIC)

Pre

Imp

Sus

Chamberlain, P., Brown, C.H., & Saldana, L. (2011).

• Meaningful prediction of implementation milestones

• Pre-implementation SIC behavior predicts successful program start-up

• Pre-implementation SIC behavior predicts discontinuing program

• Thus far, pre-implementation SIC behavior does not predict competency but does predict penetration

Emerging SIC Findings

Findings reported in Brown, Chamberlain, Saldana & Wang, submitted

Aims • Adapt the SIC for other EBPs • Evaluate SIC for other EBPs (predictive ability, which activities

are common across practices and systems, psychometrics) • Conduct cost-mapping using the COINS strategy to determine

resource allocation needed across the different practices to conduct different implementation activities

Method • 3 evidence-based treatments, 3 service sectors (JJ, CW,

Education) • Retrospective assessment of activities in 15 sites per

treatment • Prospective assessment of activities in 15 sites per treatment

Utility of SIC for Other Treatments ?

“Stages of Implementation Completion for Evidence Based Practices” L. Saldana, PI, NIH R01MH097748

Infrastructure and Implementation

. Koerth-Baker, M. (2012, October). Why your car isn't electric. The New York Times Magazine.

• The feasibility of technology use is a contextual experience •Infrastructure supporting current use of a technology is well developed and often invisible at the point of use •When use of a new technology requires adjustments to infrastructure, feasibility of use appears limited •As infrastructure to support the technology becomes commonplace, use is experienced as feasible

Implications for Larger Scale

Implementation of Effective Interventions

• “Practical” constraints on implementation may reinforced by “invisible infrastructure”

• That infrastructure can be revealed and altered

• Alternative infrastructures can be developed

Harnessing Technology Infrastructure For Implementation

Examples From Currently Ongoing Research • Broad-band enabled tablets for coach observation of

SafeCare ® fidelity during home visits (G. Aarons, PI)

• Distance digital recording and web-based feedback of group supervision sessions (Chapman & Schoenwald, Co-PIs)

• Internet-based fidelity monitoring for MTFC pilot study (Feil & colleagues, 2012).

• Computer-assisted and e-learning modalities for training (see review and discussion by Beidas, et al., 2011)

• Evidence-Based Implementation Support Systems (EBSIS; Wandersman, Chien, & Katz, 2012)

• “The D&I research field has advanced from examining barriers and facilitators to investigating the uptake of effective interventions and conducting comparative studies of systematic approaches to implementation” (Chambers & Azrin, 2013).

From Implementation Strategies to Implementation Systems

Leveraging Practice And Research for the Greater Good

• Ongoing, Collaborative Partnerships o For strategic and opportunistic purposes o Conjoint discernment of relevancy of goals (but, beware status quo) o For comparative studies of multicomponent, systematic strategies

• Synergies & Complementarities: of purpose, expertise, methods o When designing a project for one purpose, consider which data and methods

can legitimately accomplish additional purposes. o If none, are there data and methods that could be added with minimal

burden and fundraising (grant writing) to accomplish those purposes?

• Some Examples: o Development of the SIC in the context of MTFC Implementation Trial o Child STEPs, Research Network on Youth Mental Health o MST adherence instrument language mini-studies o NIMH FOA on fidelity assessment in practice contexts

References

Aarons, Hurlburt & Horwitz, 2011 Aarons, G. A., Hurlburt, M., & Horwitz, S.M. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4-23. DOI 10.1007/s10488-010-0327-7. Aarons, Green, Palinkas, Self-Brown, Whitaker, Lutzker, Silovsky, Hecht, & Chaffin (2012). Dynamic adaptation process to implementation of an evidence-based child maltreatment intervention. Implementation Science, 7:32. Aarons, GA, Sommerfeld, DH, Hecht, DB, Silovsky JF, Chaffin MJ: The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for a protective effect. Journal of Consulting and Clinical Psychology 2009, 77:270-280. Beidas, R.S., & Kendall, P.C., Training therapists in evidence-based practice: A critical review of studies from a systems-contextual perspective. Clinical Psychology: Science and Practice, 17 (1). 1 – 30. Boxmeyer, C.L., Lochman, J.E., Powell, N.R., Windle, M.,& Wells, K. (2008, Fall). School counselors’ implementation of Coping Power in a dissemination field trial: Delineating the range of flexibility with in fidelity. Emotional and Behavioral Disorders in Youth, pp. 79-84, 94-5 Chambers & Azrin (2013). Research and services partnerships: Partnerships: A fundamental component of dissemination and implementation research. Psychiatric Services, 64, 509-513. Chamberlain, P., Brown, C.H., & Saldana, L. (2011). Observational measure of implementation progress: The Stages of Implementation Completion (SIC). Implementation Science Dorsey, S., Pullmann, M.D., Deblinger, E., Berliner, L., Kerns, S.E., Thompson, K., et al. (in press). Improving practice in community-based settings: A randomized trial of supervision – study protocol, Implementation Science. Glasgow R & Steiner J. (2012). In Dissemination and Implementation Research. Brownson, R, Colditz, G, and Proctor, E (Eds.). Oxford University Press. Grimshaw, J.M., Shirran, L., Thomas, R., Mowatt, G., Fraser, C., Bero, L., Grilli, R., Harvey, E., Oxman, A., & O’Brien, M.A. (2001). Changing provider behavior: An overview of systematic reviews of interventions. Medical Care, 39, 8 (Suppl 2), pp. II-2 – II-45. Knudsen HK, Ducharme LJ, Roman PM: Clinical supervision, emotional exhaustion, and turnover intention: A study of substance abuse treatment counselors in the Clinical Trials Network of the National Institute on Drug Abuse. Journal of Substance Abuse Treatment 2008, 35:387-395

References

Knudsen HK, Ducharme LJ, Roman PM: Clinical supervision, emotional exhaustion, and turnover intention: A study of substance abuse treatment counselors in the Clinical Trials Network of the National Institute on Drug Abuse. Journal of Substance Abuse Treatment 2008, 35:387-395 Lochman, J.E., Boxmeyer, C., Powell, N., Qu, L., Wells, K., & Windle, M. (2009). Dissemination of the Coping Power program: Importance of intensity of counselor training. Journal of Consulting and Clinical Psychology, 77, 397-409. Lyon, Wiltsey Stirman, Kerns, & Bruns (2011). Developing the mental health workforce: Review and application of training approaches from multiple disciplines. In Becker & Stirman, Guest Editors: Special Issue, Training Clinicians in Evidence-Based Interventions, Administration and Policy in Mental Health and Mental Health Services Research, 4. McLeod, B.D., Southam_Gerow, M.A., Tully, C.B., Rodriguez, A., & Smith, M.M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology, Science and Practice, 20, 14 - 32 MST Transportability Study References Available from the Family Services Research Center, Medical University of South Carolina, at:

http://www.musc.edu/psychiatry/research/fsrc/pubs.htm Nadeem, Gleacher, & Beidas (2013). Consultation as an implementation strategy for evidence-based practices across multiple contexts: Unpacking the black box. Administration & Policy in Mental Health and Mental Health Services Research Proctor, E.K., Landsverk, J., Aarons, G., Chambers, D., Glisson, C., Mittman, B. (2009). Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health and Mental Health Services Research, 36, 24-34 Saldana, L., Chamberlain, P., Wang, W., & Brown, H. (2012). Predicting program start-up using the stages of implementation measure. Administration and Policy in Mental Health Research, 39, 419-425. Schoenwald, S.K., (2011). It’s a bird, it’s a plane, it’s . . . Fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18 , 142-147. Schoenwald, S.K., Garland, A.F., Chapman, J.E., Frazier, S.L., Sheidow, A.J., & Southam-Gerow, M.A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32-43. Stirman, Shields, Deloriea, Lnady, Belus, Maslej, & Monson, (2013) A randomized controlled dismantling trial of post-workshop consultation strategies to increase effectiveness and fidelity to an evidence-based psychotherapy for Posttraumatic stress disorder. Implementation Science, 8:82 Wandersman, A., Chien, V.H., & Katz, J. (2012). Toward an Evidence-Based System for Innovation Support for Implementing Innovations with Quality: Tools, Training, Technical Assistance, and Quality Assurance/Quality Improvement. American Journal of Community Psychology, 50, 445-459.