Supervision Strategies to Enhance Implementation and
Fidelity to EBP
Kelly Pitocco, LISW-S, LICDC
University of Cincinnati
Corrections Institute
Putting Supervision in
Context of Implementation
Process
To Do . . . PractitionerTraining
State Policy
Funder
Supervisor
Training and Support
Supervisor
Training and Support
The Four Phases of Learning
Preparation
Practice
Presentation
Performance
OftenSkipped
Preparation
Assessing and Readying for Change
Preparation
• Get them interested
• Encourage positive feelings about new learning
• Create Learning Environment
Survey of AOD Professionals
83%
Past Year use ofCBTMETMI
12 Step Facilitation
75% Reported Currently Using EBPs
However, Many have Negative View of Curriculum-based Treatment
Attitudes About EBP
Most Clinicians eitherBelieve they are Using EBP
Or Want to Use EBP
Positive Attitudes
• Using manual help counselor evaluate and improve skills
• Treatment manual will enhance outcomes
• Counselors are ethically obliged to use EBP
Negative Attitudes
• EBP make staff more like technician than caring people
• Treatment manuals appropriate for research but not real life
Readiness Factor
Prepare for Implementation
• Determine if the organization and program are ready to adopt model
• Assess staff readiness to change
• Set up a multidisciplinary change committee
• Program leadership and change committee develop the training and follow-up plan
Areas for Readiness
• Strategic Plan
• Readiness Assessment
• Project Management
• Change Management
• Staff Training
• Staff Supervision– Rating– Coaching
• Monitoring Fidelity– CQI– Evaluation
•Staff Training
•Staff Supervision•Rating•Coaching
Goals for Preparation
• Get staff out of a passive or resistant mental state
• Remove learning barriers• Arouse interest• Give staff positive feelings about, and an
incentive to learn• Create active learners• Establish a learning group
Techniques
• Provide positive suggestion
• Discuss benefits to clinician and client
• Set clear, meaningful goals
• Raise curiosity
• Create a safe and positive environment
• Calm staff fears
• Identify and remove barriers to learning
Techniques
• Raising questions and posing problems
• Getting staff involved in implementation
Learner Barriers
NO WIIFM
Need to Save Face
If it Ain’t Broken,Don’t Fix it
This is the WayWe’ve Always
Done it
The way I do itIs Good Enough
I don’thave Time
PersonalIssuesI already do this
Learner Benefits
• Brainstorm benefits
Curiosity Arousal
• Give people problems to solve in teams
• Send staff on fact-finding missions
• Play question and answer games
• Self-discovery activities
Presentation
Broader than Training
• Need a mechanism to accomplish:
– Acceptance of change
– Means to incorporate change
– Reinforced at all levels of system
Typically underestimate the time and effort needed to:
• Train
• Implement
• Achieve fidelity to the model
Occasional Quotes from Trainees
“I was told I have to be at this training.
I have no idea why I am here.”
“I’m just here to get my 30 hours/CE’s/
mandatories done”
“That won’twork at my
Site/with ourclients/withinthe time we
have.”
Training Culture
Successful training experience wanes
with each disengaged participant
Typical Reaction of Trainees
• Excited to learn a new clinical intervention
• Enthusiastic and committed to trying it with clients
• Although there wasn't much time for skill-building during the session, they have their notes and want to try it out
Reality Returns
• Then they get back to the site– to the routine– to the caseload– to the demands by courts– to UR or required contacts– to the supervisor and co-workers who didn't
take the training– to the clients who aren't prepared for
something new or different
Not Prepared
• Notes now seem incomplete
• Can’t recall details from the training
• Enthusiasm and new knowledge begin to fade
Adopting new
practices in the context
of everyday work is
difficult and frustrating
Do One-Shot Trainings Work?
• 15 hour training on MI
• Pre-training baseline audiotape
• Helpful Responses Questionnaire
Self-Report
• Participants over-inflated their skills of using MI after the training
• Use of MI declined with time following the training (about 50%)
• Skills were about ¼ of proficient
Maximizing Gains from Classroom Training
• Use of knowledge-based pre/post tests
• Use of knowledge-based proficiency tests
• Use of skill-based rating upon completion of training
• Mechanism for use of data– Rated competent or continued development
until reach competent
Role of Trainer
• Training is a means to an end – not an end in itself
• Trainer is Performance Consultant• Trainer partners with the learner• Link to business need• Proactive and reactive • Front end assessment and evaluation of
performance
Evaluation
• Knowledge Test – pre/post
• Competency – skill-building sessions– Skill check off– Structure through policy and procedure– Program Integrity Evaluation– CQI or outcome evaluation
Supervisory Strategies for Enhancing Training Transfer
• Communicate expectations prior to training
• Demonstrate involvement of training content – integrated (clinical and staff meetings, paperwork, etc)
• Hold learner accountable for applying content in work (rewards/sanctions)
Supervisory Strategies for Enhancing Training Transfer
• Demonstrate and Model the skills
• Provide learning and practice opportunities
• Integrate learning objectives into performance appraisal
• Observe and provide feedback and coaching
• Booster training sessions
What would the Supervisor need to be able to do those items?
Small Group Discussion
Remember Two Part Process
Change Management
And
Skill Development
Areas to Address
• Skill Deficit
• Resistance
• Both
Role of Clinical Supervisor
• Safety
• Quality
• Effectiveness
• Compliance
Oh yeah, and . . . .Oversee servicesAdministrative TasksDaily Crisis ManagementPrepare ReportsManage CaseloadProvide coverageHire StaffAll other duties no one else wants to do
Evidence Supporting Supervision
• Conditions– Manual Only– 14 hour Workshop– Workshop + Feedback– Workshop + Coaching– Workshop + Feedback + Coaching
All had initial Skill Acquisition
Miller, et. al., A Randomized Trial of Methodsto Help Clinicians Learn Motivational Interviewing.Journal of Consulting and Clinical Psychology (2004)
Four Months Later
• Conditions– Manual Only– 14 hour Workshop– Workshop + Feedback– Workshop + Coaching– Workshop + Feedback + Coaching
Could not Detect Who had Training
Only condition that maintainedbenefit after 4 Months
CBT Study• Conditions
– Manual Only
– Manual + Web-based Training (40 hours)
– Manual + Training + Supervision (observation and feedback)
Sholomskas, et. al., We don’t Train in Vain: ThreeStrategies of Training Clinicians in CBT .Journal of Consulting and Clinical Psychology (2005)
CBT Study
• Conditions
– Manual Only – No Transfer
– Manual + Web-based Training (40 hours) - Modest Transfer
– Manual + Training + Supervision (observation and feedback) - Proficient
Sholomskas, et. al., 2005
“Face to face training followed by supervision may be essential for effective technology transfer and raises questions about whether practitioners should feel competent to administer an empirically-
supported treatment on the basis of reading a manual alone.”
Current State
• What is your current assessment of staff in providing evidence-based practices?
• What are the challenges in achieving staff proficiency?
• What changes could facilitate improvement in clinical supervision?
Observed Contacts
• On a scale of 1 - 10
– How important is it for the clinical supervisor to have direct observation to effectively provide supervision?
– If you rated high – why?
– If you rated low – why not?
What do youthink your staff
would say?
Behind Closed Doors
Ever Makeyou Nervous???
Types of Interventions Used
Score of 4 Considered Proficient
Types of InterventionsNever Used
Types of InterventionsRarely Used
What Were They Doing?
Those not Trained in Model
Significantly More “Chat”
What was the Informal Conversation?
• Common Experiences
• Opinions not related to Treatment
• Current Events/News
• Personal Feelings about Client
• Work-Related Problems
• Professional Background
• Other
Without Observation
• No chance to reinforce good work
• No correcting mistakes or inconsistencies
• No provision for gaps in skill
• No assurance of fidelity with model
Practice
Practice
• Integrate and incorporate the new knowledge or skill
• Use a variety of methods and scenarios
• Providing coaching and feedback
Practice - Goals
• Integrate and incorporate new skill in direct practice situations
• Encourage transfer of skill to variety of situations
Techniques
• Observations of current skill
• Together identify skill gaps
• Consider team or pair learning partnerships
• Contextual learning experiences
• Problem-solving exercises
Techniques
• Develop activities for the staff to process learning
• Hands-on trial/feedback/reflection/retrial
• Real-world simulations
• Individual reflection and articulation
Techniques
• Partner and team-based discussion• Collaborative teaching and review• Skill-building activities• Teachbacks• Interview staff member as if s/he was client• Best and worst critique• Tell a story that illustrates how skill was effective• Memory or job aids
Strategies
• Role play• Trial and error• Microskills training • Written vignettes – if client says . . . you
respond _______.• Role play and make common mistakes –
have them identify mistakes and how to replace
Strategies
• Clinician develops a pictogram or flow chart of the methods
• Imagery• Cards with vignettes – what would you
say? Or cards with skills – respond to the vignette
• Project client responses• Group brain role play
Performance
Performance
• Help apply and extend their new knowledge or skill to the job
• Create integrated skill
• Continuous improvement of performance
Performance - Goals
• Make sure the learning sticks and is applied successfully
• Make sure the skill is applied in appropriate situations
• Continuous improvement
• Is the skill having the desired impact or outcome?
Techniques
• Immediate real-world application• Creating and executing plan for clients• Follow-through reinforcement activities• On-going coaching• Performance evaluation and feedback• Peer support activities• Supportive organizational and environmental
changes
Threats to Transfer of Learning
• No immediate need to apply the skill
• No support system for reinforcing the learning on the job
• A culture or work setting that is antithetical to the new learning
• No rewards for applying the skill
• No consequences for not applying the skill
• No time to integrate the new skill
Early Attempts
• What worked well? What would have made it better?
• What were the problems with implementation and/or strategy
• How would you do it different?• What do you need to improve your
performance?
Reinforcement Strategies
• Buddy System
• Rewards and Consequences– Evaluation system – objective measure– Self-evaluation– Review to see if the benefits are realized
Current Measures
How do we determine performance?
Specific Measures - Examples
• Uses EBP language during interactions
• Goals address criminogenic needs
• Avoids power struggles with clients
• Consistently applies protocol for rewards and sanctions
• Helps client identify thinking errors and correctives in neutral manner
Fidelity
Fidelity
Implementing the intervention as closely as possible to the way it was designed and
delivered during the research stage
Drift
• Accidental adaptation can pose significant problems
• Too much adaptation might decrease an intervention's effectiveness
Evaluation
• Effective programs utilize Continuous Quality Improvement
• Assess clinical targets and outcome data
The Role of QA/QI in Community Corrections (based on UC Halfway House and CBCF study)
Every major study shows strong relationship between program
integrity and recidivism
INTEGRITY
RECIDIVISM
Program Integrity and Treatment Effect in Residential Programs
As Scores for Integrity RiseProgram Lowers Recidivism
Therapist Competency Ratings and Recidivism
Having Unskilled CliniciansWorst than Providing No Treatment
Rating Systems
Individual adherence to EBP
Fidelity to Model
Not a discussion of
RIGHT or WRONG
Identifying what is
CONSISTENT OR INCONSISTENT
Process of Supervision
(1) Direct observation of treatment sessions
(2) Structured feedback about adherence and competence
(3) Coaching to improve implementation with proficiency
(4) Evaluate
Training and Supervision Model
• Intensive training by expert for clinicians and supervisors
• Program-based supervisors proficient – based on rating
• Staff turn in tapes or observed with feedback and coaching
Training and Supervision Model
• Continued intensive supervision until achieve proficiency rating in 3 sessions
• Ongoing individual or group supervision
• Consultant monthly contact with supervisors
Anxiety with Rating
• Emphasis on learning process – don’t tie to evaluation until a preplanned learning curve period
• Emphasis on improving rather than criticism
• Utilize model in feedback
Rating Process - Example
Motivational Interviewing – MIA:STEP
Rating Items
MI Consistent Items• MI Style• Open Ended ?’s• Affirmations• Reflections• Foster Collaboration• Motivation Change• Discrepancies• Pros/Cons• Change Planning• Client-Centered FB
MI Inconsistent Items• Unsolicited Advice• Emphasize Abstinence• Direct Confrontation• Powerlessness• Asserting Authority• Closed-Ended Questions
Adherence
• Indicate when the person demonstrated the Skill
Competence
• Higher Quality – 4 through 7
• Lower Quality – 1 through 3
Agreement About Occurrence
• Supervisor and Clinician rated similarly with Clinician rating higher
• Independent rater rated lower in most cases
Feedback
• Specificity about performance strengths and weaknesses
• Routine and formal discussions
• Opportunities for counselor self-evaluation and input
• Help counselor develop discrepancy between current interventions and goal
Coaching Format
• Start with strengths• Show them the rating sheet• Listen to tape or give specific examples• Ask clinician for area they would like
coaching • Suggest role play• Summarize• Plan for next time
Development
• Establish Standards and Expectations
• Monitor Clinical Services Regularly
• Provide Feedback
• Collaborate on Professional Development Plans
• Facilitate Knowledge and Skill Acquisition
Reduce Burnout
CLINICAL SUPERVISION
EXHAUSTIONAND
TURNOVER
NIDA study reveals that clinical
supervision was negatively
associated with emotional
exhaustion and turnover
Creating Lesson Plans
• Identify a skill-based learning goal for each supervisee
• Develop a learning plan– Preparation– Presentation– Practice– Performance
Summary and Review
• Design with the Learning Cycle– What are the four P’s?
• Appeal to the Learning Style of your learner– How can you be SAVI?
• Get the learner involved• Create a Learning Environment and Community• 30/70 Rule
30% preparation and presentation70% practice and performance
• Be flexible to the needs of the learner
Closure
Be the change you want to see in the world
Ghandi