practical guide for ebp implementation in public mental health cbt/pages/1...cbt+ ebp practical...
TRANSCRIPT
Practical Guide for EBP Implementation
in Public Mental Health
Washington State Division of Behavioral Health and Recovery
and
Harborview Center for Sexual Assault and Traumatic Stress
and
University of Washington, School of Medicine, Public Behavioral Health and
Justice Policy
Lucy Berliner, LCSW
Shannon Dorsey, PhD
Laura Merchant, LCSW
Nathaniel Jungbluth, PhD
Georganna Sedlar, PhD
CBT+ EBP Practical Guide 2
Table of Contents
Practical Guide for EBP Implementation in Public Mental Health ................................................ 1
Practical Guide Goals ...................................................................................................................... 3
Background ..................................................................................................................................... 4
Public Mental Health Context Considerations ................................................................................ 7
I. Organizational Climate and Leadership .................................................................................. 9
II. EBP Training .......................................................................................................................... 13
III. EBP Supervision ................................................................................................................... 18
IV. Standardized Trauma Screening and Assessment of Clinical Targets ................................. 23
V. Quality Assurance ................................................................................................................. 27
VI. Technology .......................................................................................................................... 31
Summary and Recommendations ................................................................................................. 34
Appendices .................................................................................................................................... 35
References ................................................................................................................................. 36
CBT+ Team ................................................................................................................................. 39
CBT+ Senior Leader Meeting Participants ................................................................................. 40
Acknowledgements………………………………………………………………………………………………………………37
CBT+ EBP Practical Guide 3
Practical Guide Goals
The goals of this guide are to provide:
A background on EBP implementation and sustainment within the context of public
mental health.
Practical strategies for addressing common challenges to EBP implementation and
sustainment.
A foundation for continuing to build knowledge and develop additional approaches for
expanding EBP availability within public mental health in Washington State and beyond.
We start by providing a background on EBP, including definition, characteristics, and research
on EBP. We think this background will help organizations make the case for what they are
doing, and explain what they are doing to others, as well as place the practical
recommendations within a context.
CBT+ EBP Practical Guide 4
Background
Evidence-based practice (EBP) can be thought of as both a general approach and as a descriptor
of a specific program. EBP, as an approach, means embracing evidence-based principles and
practices as the standard of care. EBPs or evidence-based programs are those that have been
tested in scientific studies and shown to have outcomes that are overall more favorable than
alternatives to which they have been compared such as waitlists, usual care or non-specific
interventions, or the program is as effective as another EBP for the same outcome. There are
many different classification schemes for assigning the level of “evidence” that a program has
achieved. Of particular relevance to public mental health practice is the California Evidence-
Based Clearinghouse for Child Welfare (http://www.cebc4cw.org) and the Washington State
Institute for Public Policy Inventory of programs for children in public mental health
(http://www.wsipp.wa.gov/pub.asp?docid=E2SHB2536).
The characteristics of the EBP general approach starts with a genuine belief that the scientific
method can show that some programs work better overall than others and that clients are best
served when they have access to the most effective programs. There is recognition that a shift
to EBP means that providers will have to make some changes in how they have traditionally
practiced. EBP is not seen as fad or as an unachievable goal for public entities even though
there are many practical challenges to full implementation. Organizations that take on EBP
principles recognize that EBP adoption is more than training providers in specific EBPs; in
addition to training, EBP adoption involves making changes in how clinical practice is
conceptualized and carried out and entails a commitment to installing organizational
procedures that reinforce the principles of EBP and support sustainment of specific EBPs.
There is a science of “dissemination and implementation” that specifically focuses on how
organizations can adopt EBPs and how providers can best learn, practice, and sustain EBPs over
time. This science applies in many contexts including medicine, education, and criminal justice
as well as mental health. There are numerous models that describe the levels and stages that
are typically involved (Aarons, Hulburt, & Horowitz, 2011; Damschroeder et al., 2009; Fixen,
Naaom, Blase, Friedman, & Wallace, 2005; Palinkas & Soyden, 2012; Proctor, 2012). Some
dissemination and implementation scientists have specifically focused on public mental health
and child welfare (Aarons et al., 2011; Glisson et al., 2008).
All models address at least two basic stages. The first stage is deciding to adopt one or more
EBPs and the initial installation of the EBP within the organization. The second stage is keeping
the EBP(s) going within the organization after the initial training and provider competence has
occurred. Aarons et al. (2011) describe four stages: Exploration, Preparation, Implementation,
CBT+ EBP Practical Guide 5
and Sustainment (EPIS). The first three EPIS elements focus on getting EBP going within an
organization and the last stage focuses on keeping it going (i.e., sustainment). This is often a
challenging stage, and we know the least about sustainment from research studies (see Wiltsey
Stirman et al., 2012). A number of factors have been identified across studies that are
associated with organizational change and provider uptake of the skills that are necessary for
adoption and sustainment of EBP.
The Washington State CBT+ Initiative forms the basis for this Guide. The state mental
health division (Division of Behavioral Health and Recovery) has invested in training in CBT,
consultation and sustainment since 2009. CBT+ is the name used to describe an approach to
teaching providers four models that cover the presenting diagnoses and clinical problems of
about 80% of all children served in public mental health systems (Burley, 2009). Providers are
taught CBT for anxiety, CBT for depression, Trauma-Focused CBT for the impact of trauma
including Posttraumatic Stress or Posttraumatic Stress Disorder (PTS/PTSD), and Parent
Management Training for behavior problems (PMT; also known as Behavioral Parent Training).
All of these models appear on EBP lists (e.g., http://www.cebc4cw.org) and on the WSIPP
Inventory (Evidence Based Practice Institute & Washington State Institute for Public Policy,
2012).
The Guide recognizes that there are various specific EBP models for these clinical targets-
behavior problems, anxiety, depression, PTS and as well EBP for other targets (e.g., infant
mental health, juvenile delinquency, substance abuse). Many of them are “brand name” EBPs,
meaning those with a specific title and associated manual (e.g., Trauma-Focused CBT, Coping
Cat, Positive Parenting Program, Parent-Child Interaction Therapy, Incredible Years,
Multidimensional Treatment Foster Care, Functional Family Therapy). Brand name programs
often have implementation infrastructures that provide training and varying degrees of expert
consultation, ongoing supervision and quality assurance (QA). It is a benefit when
organizational resources or external funding are available to support the full package of
implementation and QA services for non- or brand name EBPs. Many brand name EBP
companies have developed comprehensive, sophisticated implementation support services and
a cadre of expert consultants.
However, a characteristic of most EBPs is that they target a specific clinical condition, problem
area, or diagnosis and are evidence-based only for that specific condition. Public mental health
organizations serve clients who present with the full array of mental health conditions and
disorders; many clients have significant co-morbidity. This means that even when organizations
select one or more brand name EBPs for a particular outcome or subgroup of clients, they still
need to address the larger organizational context and find ways to install mechanisms to
support EBPs for clients served throughout the organization.
CBT+ EBP Practical Guide 6
This Guide is developed for public mental health organizations that have made the decision to
adopt at least one EBP and have invested in training staff. The organizations are interested in
making the necessary organizational changes so that the initial investment in EBP training pays
off in terms of actual delivery of the EBP and positive client outcomes. According to
dissemination and implementation science, the training of providers is only the beginning—the
hard work, and the important aspects of increasing provider adoption, come after the training
(Beidas & Kendall, 2010; Herschell, Kolko, Baumann, & Davis, 2010). Many organizations have
had the experience of discovering that they invested considerable time and financial resources
into training with limited perceived benefit. Research shows that training is necessary and the
first step, but not sufficient, for provider adoption and organizational sustainment (Beidas &
Kendall, 2010, Herschell et al., 2010). Training alone is not enough, but yet that is where most
resources and time are often spent. The research suggests that investing in training, without an
investment in necessary post-training supports, is probably not worthwhile.
The Guide intends to assist organizations that are prepared to institutionalize processes to
support sustainment by providing practical and feasible strategies. The emphasis is on
situations where the organizations are mostly restricted to accomplishing EBP sustainment
within existing resources. The Guide is also designed to be helpful for organizations that have
both non- and brand name EBPs. Brand name companies may set the specific elements of
training, supervision, and QA but the providers are typically situated within an overall
organizational context.
The practical suggestions contained in this Guide are primarily based on the experiences and
recommendations of public mental health centers in Washington State, all of whom have
participated in the CBT+ Initiative and many of which also have also adopted other brand name
EBPs within their organizations (e.g., Cognitive Processing Therapy, Parent-Child Interaction
Therapy, Triple P Parenting Program).
CBT+ EBP Practical Guide 7
Public Mental Health Context Considerations
Public mental health organizations often have a mission that encompasses far more than
delivering discrete mental health interventions to children and families. Installation and
sustainment of EBPs is occurring in a context in which specific mental health interventions are
typically one of many services being provided to children and families. Successful adoption of
EBPs and their sustainment must take into account the larger organizational context.
Public mental health refers to mental health services that are paid for by Medicaid. A majority
of children receiving specialty mental health care in the US have Medicaid as the insurance. The
Affordable Health Care Act that goes into effect in 2014 will expand Medicaid eligibility in most
states. Typically, public mental health services are available through community-based
organizations that have contracts with the state government or where the providers are state
employees (http://www.medicaid.gov/Medicaid-CHIP-Program-Information/By-
Topics/Benefits/Mental-Health-Services-.html).
The good news is that Medicaid is a comprehensive mental health insurance that is often
generous by comparison to what is covered by many commercial insurance packages. It usually
covers a broad array of services beyond the limited number of outpatient sessions or inpatient
days that is typical for commercial insurance. For example, it can cover ancillary activities such
as case management. It is common for public mental health organizations to be required or
expected to provide case management, case aides, emergency crisis services, medication
management, day treatment, and hospitalization. Intensive team-based approaches are
common for youth with the most severe problems (e.g., Wrap Around, Systems of Care).
Services are often delivered in multiple settings including clinics, home, school, juvenile justice,
and the community.
In addition, many organizations have a plethora of other programs that may be available to
clients to supplement mental health services. These programs may include prevention services,
early interventions, educational support and tutoring, peer outreach, mentoring, skills classes,
after school home work and recreational programs, etc. Organizations often also have
specialized programs for specific populations that are supported by United Way, private fund
raising, local, state or federal grants.
Public mental health organizations frequently encompass substance abuse services as well,
although the funding streams and program structures may be separate. These services similarly
include a broad array of types and intensity of services including outpatient individual or group
therapy of various kinds, day programs, inpatient or residential services, hosting AA/NA type
CBT+ EBP Practical Guide 8
services, urinalysis or other biological monitoring. The degree of coordination with the mental
health services varies.
Each organization must consider how to install specific EBPs within their own organizational
context and culture. Public mental health organizations vary in size and comprehensiveness of
service array. Some organizations are large and have multiple offices and settings where they
deliver services. In some cases organizations serve only children, but many serve both adults
and children. For those organizations that serve adults, child services typically have fewer staff
and are smaller than the adult programs. The opportunities and challenges will be different
depending on the organizational context and size.
CBT+ EBP Practical Guide 9
I. Organizational Climate and Leadership
The importance of organizational climate and leadership is identified by many dissemination
and implementation researchers, a number of whom have addressed the issues specifically in
the context of public mental health or child welfare (Aarons, Sommerfeld, Hecht, Silovsky, &
Chaffin, 2009; Glisson et al., 2011; Kolko et al., 2012; Palinkas et al., 2009). A great deal of
research has documented that characteristics of organizations are very influential in attitudes
that providers have toward EBPs and the success of EBP implementation efforts, including the
clinical outcomes for clients. Leadership from top administrators as well as mid-level managers,
such as clinical supervisors, has been shown to make a big difference (Aarons & Sommerfeld,
2012).
It is important to note that adoption of EBP in an organization can actually be beneficial to the
organization and the organizational climate. Aarons et al. (2009a; 2009b) found decreased staff
burnout and decreased staff turnover associated with the implementation of EBP in a child
welfare services context when providers received ongoing coaching and support in delivering
the intervention (but not when providers were trained in the EBP and did not receive coaching
and support).
Challenges
While the evidence shows that bringing EBP into an organization, with support, can actually
lead to higher staff satisfaction and lower burnout, how the implementation is accomplished
and the nature of the organizational climate is critical. Where implementation goes awry is
when it is seen as yet another management band wagon or fad that is being imposed on
providers with additional work load and no clear benefit. When administrators and supervisors
are perceived as simply carrying out an externally imposed requirement (without buy-in or
interest), buy-in at the provider level will be low. New initiatives and program imperatives are
common in public human service systems. It can seem that there is an unending flow of
directives mandating a new program, new way of doing business, new standards or new
requirements. It is not so surprising in this context that adoption of EBP is seen as yet another
initiative that will eventually go by the wayside.
A key challenge for senior leadership in organizations is to call out EBP as a change that is
critical to improved client outcomes and one that will be enduring even if it takes different
forms over time. It is essential that senior leadership and clinical supervisors provide an
explanation of why such a move is beneficial to clients. In addition, the organizations must
CBT+ EBP Practical Guide 10
ensure that providers have the necessary supports for learning new skills and delivering EBPs
with fidelity in the context of the complex public mental health environment.
Practical Strategies
1. Senior Leaders such as the CEO, division directors, program managers and other
organizational leaders make a specific decision to prefer or emphasize use of EBPs. When
new opportunities or mandates arise, an EBP approach or specific EBPs are discussed. All
programming is evaluated through the lens of EBP principles (evidence, measurement of
change over time, outcomes).
2. Senior leaders emphasize that the reason WHY EBPs are being adopted is because they
produce better outcomes overall for clients. This rationale will resonate most for providers,
versus legislation or other imperatives, because helping clients is the primary professional
motivation for mental health professionals.
3. Senior leaders and managers seek out opportunities to bring EBPs into the organization and
speak positively about them. When EBPs are discussed there is open support expressed for
the EBP approach as well as for individual EBPs. A proactive problem solving approach is
modeled for how to fit EBPs into organizational practice.
4. The EBP approach and availability of EBPs in the services array is prominently mentioned in
organization promotional materials, publicity efforts, fundraising activities and other
community oriented initiatives.
5. Senior leaders use EBP outcomes data in annual reports, in advertizing organization services
and within the organization to show the benefit to staff.
6. EBP outcomes data are used to advocate for funding, enhanced reimbursement, expansion,
adding new EBPs.
7. Availability of EBPs for all eligible clients is identified as an important organization goal. It is
not considered acceptable for only some eligible clients to have access to EBPs when there
are trained providers within the organization.
8. Organizational leadership cultivates, encourages and supports clinical supervisors to acquire
EBP supervision expertise; they become the “champions” for EBPs with clinical staff.
CBT+ EBP Practical Guide 11
9. Hiring practices and announcements for staff positions—both management and clinical—
specifically mention a preference or requirement for knowledge/experience in EBPs and
active support for EBPs.
a. During interviews
i. Specifically ask about EBP training, certifications, experience.
ii. Specifically convey that EBP orientation is required and includes
acceptance of ongoing feedback and coaching, openness to learning and
doing new interventions.
b. Getting Staff Up to Speed
i. Require providers to read selected EBP books and manuals; take online
learning courses (e.g., TF-CBTweb).
ii. Begin orientation and coaching to EBP principles and practices
immediately.
10. When possible, senior leaders make some direct contact with providers who are starting the
EBP process (attending a training) or in the process (recently returned/just started CBT
Consultation groups) to show attention to, and support of their involvement in EBP
activities. Often, we wait until people are failing/not meeting expectations. Give some
attention early.
a. This can be an email, brief meeting, or a mention in an all staff meeting
b. A quick email during the training/consultation process inquiring how it is going
c. Brief meeting or email to providers after key stages are met (finished training,
completed consultation)
d. Senior leader “pops in” for part of in-house EBP supervision or consultation (for
example, comes to a portion of a CBT+ consultation call)
e. Shows the senior leader is paying attention and has interest in what providers
are doing at the individual level
11. Senior leaders and management provides support for EBP by ensuring that necessary
resources are available to deliver EBPs within organizational capacity (e.g., purchase books
and materials, identify online resources, set up a spot on the organization server to store
electronic resources, therapy room set ups, etc.).
12. Senior leaders and supervisors find ways to recognize staff that is doing a good job with
EBP. Recognition can take a variety of forms:
a. Staff acknowledgement: call-out in a meeting; feature a successful completion of
a case; ask a provider who attended EBP training to present at a staff meeting.
CBT+ EBP Practical Guide 12
b. Frame the Certificates of Participation, Completion or Certification so that
providers can prominently post in their offices.
c. Make sure that training programs provide CEUs.
d. Recognition in an organization newsletter or monthly email update.
e. Tangible rewards: Bonuses; small gift cards (even $5 goes a long way); a
notecard or email from an administrator; small, purchased reward; relief from an
onerous activity.
f. Encourage public Rostering if available on local or national sites.
g. Criteria for promotion from within explicitly emphasizes commitment to and
proficiency with EBP.
13. Proactively connect up with other organizations that are installing EBPs to “steal
shamelessly and share relentlessly” (e.g., sharing materials like EBP progress notes or clinical materials,
splitting a trainer/consultant cost across agencies, cross–site EBP discussions/calls, senior leader networking
across agencies to develop a larger voice about EBPs) .
CBT+ EBP Practical Guide 13
II. EBP Training
Providers and supervisors have to learn EBPs in order to deliver them. Research has clearly
established that traditional training alone does not lead to change in practice or delivery of the
EBPs in practice (Beidas & Kendall, 2010; Herschell et al., 2010; Wiltsley Stirman et al., 2012).
The optimal training is skill-based (not just lecture on theory) and of sufficient dosage for
providers and supervisors to learn and practice the new skills, as well as become familiar with
the basic underlying theory and content. But the key is moving beyond initial training so that
providers have an opportunity to deliver the model with actual clients while receiving expert
consultation (on the job coaching). Training without follow-up case consultation and support
does not result in practice changes.
The Learning Collaborative (LC) model as originally developed by the Institute for Healthcare
Improvement IHI (http://www.ihi.org/Pages/default.aspx) has taken hold as an EBP training
approach. For example, the National Child Traumatic Stress Network (http://www.nctsn.org).
has created an LC model for EBPs that are widely used with children affected by trauma and
abuse. An LC consists of several components including organizational consultation or
preparation for implementation, basic learning sessions followed by an action period during
which providers and supervisors do the EBP with clients while receiving expert case
consultation, and subsequent learning or booster sessions.
Another more extensive but potentially more powerful approach to a LC is directed to a
community, not just a single organization or a group of providers. This model incorporates
learning sessions and consultation activities for the other organizations, institutions and
professionals that comprise the systems of care (e.g., child welfare, CASA, juvenile court).
Project Best is an example of such an approach with TF-CBT in South Carolina and other parts of
the country (http://academicdepartments.musc.edu/projectbest/partners/partners.htm).
The other key goal of EBP training is establishing initial provider competence following the
training. Studies of EBP initial trainings show that they result in changes in provider knowledge,
attitudes and self-reported comfort with using EBP strategies. But this does not ensure that
providers have actually acquired the skills needed to provide the EBP without supervision or
coaching following training.
Methods of evaluating providers’ skill acquisition include direct observation of delivery of the
model with clients and structured behavioral rehearsal (role plays), observed by a trained
supervisor. In behavioral rehearsal, the provider practices an EBP skill either with someone
formally playing a “standard patient” or actor, or more informally in which another provider, or
CBT+ EBP Practical Guide 14
the supervisor, who plays a child who needs the skill (Beidas, Cross, & Dorsey, in press; Beidas,
Edmunds, Marcus, & Kendall, 2012). Results from a 2011 CBT+ evaluation using pre and post
training behavioral rehearsal (8 minute role play) over the phone objectively revealed provider
competence and skill (what had the provider learned, what areas still needed improvement).
Challenges
The LC model of training plus consultation involves an explicit expectation for practice. It is
more expensive because of the follow-up consultation requirement, but has a far better return
than traditional methods in which providers attend workshops, lectures or conferences and
incorporate bits and pieces from the training (termed “train and hope” in the dissemination
literature). Commitment to the LC approach can be logistically complicated because EBP
trainings using the LC model frequently involve multiple days or even weeks, limit the number
of participants to facilitate a more intensive learning experience (or require multiple trainers),
have requirements for consultation following the learning sessions, and expect collection of
metrics to evaluate success of EBP implementation. LC trainings can engender trainee
discomfort because they include a high percentage of time spent on actual practice of skills,
with peer and trainer feedback. In addition to the costs of the training itself, in order to attend,
providers are out of the office and not seeing clients. There may also be costs such as hotel, per
diem and other expenses.
Practical Strategies
1. Shift how the organization approaches basic and continuing education. Policy is established
to favor training in EBPs and support training models that are consistent with the LC
approach. Establish the expectation that a change in practice is the goal of training.
Continuing education moneys are devoted to training or supporting EBP. Consider fewer
trainings—and prioritize those that offer consultation/supervision/coaching support to get
the biggest bang for the buck.
CBT+ EBP Practical Guide 15
2. Take advantage of any and all opportunities for accessing training that meets the criteria for
EBP training:
a. Training supported by government sources either on an ad hoc or ongoing basis
(e.g., WA DBHR CBT+).
b. Training associated with child welfare or other government contracts to provide
EBPs (e.g., contracts for TF-CBT, AF-CBT, PCIT, Triple P, Incredible Years, SafeCare).
c. Training associated with research. Researchers seek to carry out clinical research in
public mental health settings. Agree to participate as subjects or to recruit clients as
subjects in exchange for access to free expert training and consultation (OK/CDC
Funderburk PCIT supervision; UW Dorsey TF-CBT Supervision study; UW Dorsey TF-
CBT with children in foster care, UW Spieker Promoting First Relationships).
d. Training associated with federal initiatives (e.g., SAMHSA NCTSN, HHS-ACF Initiative
to Improve Access to Evidence-Based Behavioral Health Services in Child Welfare;
DOJ Safe Start and Defending Childhood). Agree to support an application, be a
participating site.
3. Seek out specific grants for EBP training, consultation, and support.
a. Grants that are time limited and have a specific purpose are often available from
local, state and federal sources. Request funds for EBP training and provide
justification for costs based on the goal of changing practice to change outcomes,
not simply having providers receive training.
4. Establish a policy that if providers do attend training that is non-EBP or is not a specific LC
endorsed by the organization, they will not be permitted to deliver the non-EBPs and/or
CBT+ EBP Practical Guide 16
they must describe how the learning is consistent with EBP and how it would be
incorporated into the organization’s existing EBPs.
5. Collaborate with other local mental health organizations to contribute toward LC type
training and reduce costs to individual organizations.
6. Make a decision to invest in organization-wide training in a broadly applicable EBP approach
to create a fully trained work force.
a. Contract to bring in an expert trainer and pair them up with a qualified internal
supervisor who can do the consultation and oversee providers meeting criteria.
Consider training supervisors first either internally or in collaboration with other
organizations to establish commitment to the model, become internal champions and
develop expertise to eventually become internal trainers.
7. Invest in and develop the training expertise of supervisors who can learn to provide ongoing
internal training.
a. Clinical supervisors are supported to learn to do EBP training and supervision (e.g.,
participate in opportunities such as being a CBT+ co-consultant, join CBT+ once
yearly supervisor training, participate in CBT+ monthly supervisor calls).
b. Clinical supervisors and experienced EBP providers deliver training on EBP skills or
deliver organization-wide training on EBP components, skills to share knowledge.
c. Clinical supervisors learn to provide an initial EBP training for new hires before they
have the opportunity to attend formal external training in specific EBPs. Such
training would emphasize the key principles of EBPs (e.g., structured and focused,
targets a specific outcome that is measured, skill building in the primary goal,
homework for between session practice is routine, etc.) as well as the key skills (e.g.,
systematic assessment with feedback, teaching skills to clients).
d. Clinical supervisors explicitly prepare and support new providers for attending
formal EBP training and follow up to create continuity and reinforcement in applying
new skills.
CBT+ EBP Practical Guide 17
a. If they’ve been to the training themselves, prepare providers for what to
expect, next steps, and how the organization will support them after the
training.
e. When possible, allow supervisors to attend EBP trainings to observe HOW the
training is conducted as much as the content. Most good trainers have learned by
watching other good trainers. These opportunities can improve supervisor ability to
offer training and support within the organization.
CBT+ EBP Practical Guide 18
III. EBP Supervision
The importance of ongoing active supervision in EBP cannot be overstated. It is widely agreed,
and proven, to be necessary for successful implementation and adoption of EBP; it is extremely
unlikely that providers will continue to deliver an EBP with fidelity in the absence of competent
supervision. The research and the experience of supervisors in the field attest to the
importance of EBP-specific supervision. By this we mean active clinical supervision on how to
deliver the EBP with children and adolescents on provider caseloads, not just administrative
and case management support. Supervisors need to be knowledgeable about the EBP and
qualified to assist providers in acquiring and/or maintaining the skills that were learned in the
training and that are necessary to deliver the model and achieve the outcomes.
There are many advantages to supervisors having direct experience delivering the EBP, either
through continuing to maintain a small practice or having prior experience. Tremendous
credibility and practical knowledge are gained from having had to transfer book or classroom
learning into the real world context of routine practice.
The key is understanding the underlying theoretical framework and knowing the model
components very well. In addition, supervisors need to be good teachers and coaches. Doing
therapy involves applying skills. Some skills fall into the category of the common or non-specific
factors such as forming and maintaining a therapeutic alliance, or engaging clients in the
change process. Other skills involve teaching new behaviors to clients such as emotion
regulation, challenging maladaptive cognitions, or positive parenting. Skills are only learned
though practice and rehearsal, not via reading about them or hearing a verbal description of
what to do. One reason why the specific skill training approach for supervision is so critical is
that evidence-based therapy always involves teaching clients skills.
Challenges
Supervision is typically used to accomplish a variety of organizational objectives in addition to
teaching clinical skills and monitoring direct clinical activities. Supervisors are responsible for
the overall management of cases including the non-clinical activities that are common in public
mental health. Typical issues that must be addressed are making child abuse reports,
addressing serious crises (e.g., suicide attempts or other self harm behaviors, school
suspension/expulsion, extreme family conflict, and running away). Supervision of cases of
children in the child welfare system (CWS), especially foster care, frequently requires attention
CBT+ EBP Practical Guide 19
to case management, attendance at staffings, writing reports, making recommendations, and
care coordination across multiple systems.
Beyond the case-specific supervision activities, clinical supervisors are typically responsible for
administrative oversight of providers including monitoring completion of required paperwork,
doing performance evaluations, managing sick and vacation time, setting and monitoring
productivity standards, checking on licensing and continuing education status, and the myriad
of other workplace requirements and expectations. As well, supervisors help providers deal
with stressors that interfere with work, scheduling conflicts, burnout, and vicarious
traumatization. When providers are not meeting organizational standards for performance, it is
supervisors who are responsible for creating and monitoring improvement plans and dealing
with HR. Yet, part of supervision, if EBP is to be supported, has to be dedicated to monitoring
provider practice and providing support for improved practice and fidelity to the model. This
can be a very difficult balancing act for supervisors.
The other major challenge is organizational. Most EBPs establish standards of supervision that
are specific to the model. For some brand names, this supervision must be purchased or is
required as part of delivering the model (e.g., MST, FFT, SafeCare, MTFC). In other cases,
although a requirement for model specific supervision is not imposed by the developers, it may
be imposed by a local contract. Regardless of whether the developers require model specific
supervision, it is a universal assumption that model specific supervision is necessary to maintain
fidelity and achieve outcomes.
This presents complications for organizations where children and families come with a broad
array of clinical problems and needs. Few EBPs reach more than a subset of all children entering
public mental health because of the focus on specific clinical conditions. Take the example of
behavior problems (e.g., ODD, CD). They account for approximately 20% of children’s primary
presenting problems, therefore PMT would reach only one fifth of all children seeking services.
Beyond the breadth of the generic model, brand name interventions can have even more
restricted criteria (e.g., PCIT is for children aged 2-7 with behavior problems). Many brand
names have modifications or versions for different developmental stages or clinical contexts
(e.g., Triple P, Incredible Years). This means that the list of EBPs and subtypes of EBPs that
theoretically might be necessary or useful to meet the needs of all referred children is long.
Research has not explicitly addressed how many EBPs would be required to cover the array of
mental health problems for which youth present to public mental health organizations, but it is
likely that the number is at minimum 10 different models that address clinical problem areas
excluding substance abuse disorders. The list would have to include: infant mental health, child
behavior problems, anxiety, depression, PTS/PTSD, ADHD, serious family conflict, extreme
emotion dysregulation, eating disorders, conduct disorder/delinquency.
CBT+ EBP Practical Guide 20
Supervision is not just about supervisory skills, but also about the time it entails, both that of
supervisors and providers. It is not realistic within current existing resources for organizations
to support routine individual or even group supervision on every single potential EBP that might
be desirable. This mean that organizations will need to factor into decisions about which EBPs
to adopt what would be necessary in terms of provision of active ongoing supervision and how
that fits with organizational capacity.
Practical Strategies
1. Establish an organizational standard that evidence-based clinical supervision is an essential
ingredient of EBP delivery. Once an EBP is adopted there will be ongoing supervision
provided that is specific to the EBP and/or the relevant principles and practices.
2. Set the standard that clinical supervision on EBP must be delivered regularly and cannot be
subsumed or overtaken by supervision on non-clinical concerns/issues.
i. Talk with other supervisors within the organization, or other
organizations, about how to ‘protect’ part of supervision for
EBP/model focus. Providers may have ideas too.
ii. Provide concrete and specific assistance to supervisors and providers
on how to ensure that more time is devoted to focus ON clinical
interventions.
3. Organize supervision around EBP principles and components:
a. Create a format for case presentation in supervision that emphasizes the EBP and/or
EBP component (e.g., therapist provides a quick summary of client demographics
and referring concerns, then states what EBP is being used and what EBP
components have already been delivered). Supervision is then oriented to
supporting the provider in delivering the EBP with the case.
b. Devote supervision sessions to specific and critical EBP skills (e.g., exposure for
anxiety, selective attention for behavior problems).
c. Have supervisors teach, model, and then practice the basic and common skills in
supervisory sessions with supervisees. True learning happens with practice.
Supervisees are more comfortable practicing if the supervisor practices first.
d. Encourage supervisors to monitor clinical outcomes (e.g., scores on standard
symptom assessments) as a routine part of case review with providers. Focus on
CBT+ EBP Practical Guide 21
outcomes can help supervisors and providers identify when more intensive or
directed supervisory support is needed.
4. Whenever possible combine supervision for EBPs based on the same theoretical foundation
or containing common elements.
a. Combine all CBT or all Parent Management Training supervisions since they are
based on a common theory and contain comparable elements. Focus on teaching
the common skills found in all versions depending on the clinical target (e.g.,
selective attention and consequences for PMTs; psychoed, coping skills for anxiety,
PTS, depression CBTs).
i. Combine non- and brand name in the same theoretical family of
interventions when possible.
b. Organize supervision around key skills that cross EBPs (e.g., assessment,
engagement, motivational enhancement, psychoeducation, assigning/reviewing
homework, modeling, role playing).
c. Create an EBP team or supervision group that exclusively focuses on clinical delivery
of EBPs (e.g., case discussions, demonstration/discussion of EBP components, reviewing
treatment manual/online training components, integrating the EBP within the organization,
promoting with brokers, etc).
5. Cultivate and support EBP supervisors within organizations.
a. Support supervisor EBP training and participation on supervision support activities
(e.g., CBT+ supervisor monthly call and annual meeting; EBP specific supervision
training or certification).
b. Specify that EBP competence is a factor in promotion, hiring and maintaining
supervisor role.
c. Align supervisor performance evaluations to specifically address EBP supervision
skills.
6. Buy outside expert supervision/consultation once a month (or some other interval).
7. Give supervisors responsibility for ensuring that providers have access to EBP resources
(intranet folders with provider and client handouts, hot boxes with measures/psychoed
handouts, therapeutic materials such work books, games, rewards; equipment such as
recorders, observation rooms).
CBT+ EBP Practical Guide 22
8. Require supervisor confirmation of provider initial competence in the EBP.
a. Require providers to enter cases and document that they have administered clinical
measures and delivered EBP components. Use the EBP Roster Clinical Toolkit
(http://ebproster.org/roster/toolkit.php). The Toolkit also shows whether clients
report clinical change following EBP intervention and can serve as indication for
increased supervisory support if client symptoms are not showing improvement.
b. Confirm completion of required activities such as TF-CBTWeb.
c. Document initial provider competence by observing or listening to sessions,
especially initially.
d. Align performance evaluations to contain standards regarding EBP competence and
delivery to the model.
CBT+ EBP Practical Guide 23
IV. Standardized Trauma Screening and Assessment of Clinical Targets
Screening for trauma. A history of trauma or abuse is associated with increased risk for many
emotional and behavioral consequences. Although not all exposed children develop persisting
mental health problems, when they have other risk factors such as multiple traumas, prior
mental health problems, or compromised psychosocial circumstances the risk increases. Most
children in public mental health have multiple traumas as well as other adversities (e.g.,
compromised socio-economic circumstances, parents with substance abuse or mental illness,
parents who are incarcerated, foster care, etc). In other words, they are at high risk for mental
health and other negative outcomes.
Screening for trauma is straightforward and highly acceptable to clients. It creates an
opportunity not just to assess for trauma specific impact, but as well to make sure that children
are safe, and to validate and normalize their experiences. Trauma screening can serve as an
engagement strategy.
Screening for a trauma history has become a standard of care and is the essential ingredient of
being a trauma-informed organization. Many settings are screening for trauma by including
items about trauma or abuse on standardized intake forms or in other standardized assessment
protocols and finding that clients will readily tick the boxes or respond to the questions.
Unfortunately, in many instances there is no specific acknowledgment, validation or
normalizing with the client, only screening. It may actually be counterproductive or even
harmful to screen for trauma and not acknowledge that the client has shared this information
and how it is helpful to have the information for treatment. Failure to acknowledge may be
interpreted by the client as disinterest or disbelief.
Screening for trauma exposure is only clinically meaningful or useful if the information is
explicitly used to provide acknowledgement, support and validation to the client and as a
gateway for assessing trauma specific impact. If an organization screens for trauma, it is
necessary to institutionalize procedures for providing specific acknowledgement and
feedback.
Standardized assessment of the clinical target. EBPs target specific clinical conditions. The way
they are shown to work is by (1) measuring the symptoms of the targeted conditions at
baseline; (2) matching clients to an appropriate EBP for the baseline symptoms; and (3)
measuring the symptoms again at one or more subsequent points to understand
response/improvement. By definition, EBPs are only evidence-based for the clinical target(s)
they have been tested on. This is a core characteristic of EBPs.
CBT+ EBP Practical Guide 24
Although many EBPs are intended for a single or specific outcome, manyhow benefits for
associated conditions. For example, TF-CBT is designed for children affected by trauma who
have trauma specific impact (PST/PTSD), but it also has benefits for depression, anxiety and
moderate behavior problems, if present. PCIT is designed for behavior problems but there is
some evidence that it also helps with child depression and anxiety, as well as parenting stress
and risk for future physical abuse in cases where physical abuse has occurred.
The key to EBP is to match the specific EBP approach to an identified clinical target(s), make
sure the client is linked with a provider trained to provide that EBP, and have some way of
measuring the level of the clinical target(s) at baseline and over the course of therapy. This is
the mechanism for determining whether the EBP is working and whether the treatment needs
to be adjusted or changed.
Mental health clinical assessments are standard practice and are intended to identify the
clinical problems and needs. They routinely produce a clinical formulation, diagnosis and make
treatment recommendations. The specific addition that is consistent with EBP is incorporating a
structured, systematic and standardized method for measuring and quantifying the clinical
target(s).
Challenges
There are two main challenges to incorporating standardized methods of trauma screening and
clinical target(s). One has to do with the organizational usual procedures. All organizations have
certain information that must be collected from clients before they can begin therapy including
determining whether they meet eligibility requirements (access to care criteria, catchment
area), completing required organizational paperwork including consent for care and release of
information forms, filling out checklists and forms that are required by funders, and carrying
out clinical assessments according to standards that are set by the organization or the
government. Organizations differ in how and when these activities are completed. Some collect
certain information during phone screening/intake; in some organizations there are assessment
units and the client is then assigned to a provider; in other organizations the assessor will
become the provider. Decisions need to be made about what point in time standardized trauma
screening or clinical target assessment measures will be administered and who will provide the
feedback on results to the client.
The second challenge is deciding on which trauma screening and clinical target assessment
measures to use. There are many different trauma screens and many standardized measures
for clinical targets. In addition, there are measures of functioning. Some standardized trauma
screens and assessment measures are proprietary which means the organization has to buy
CBT+ EBP Practical Guide 25
them. For some EBPs, proprietary standardized measures are required (e.g., PCIT requires the
ECBI and Parenting Stress Inventory).
Fortunately, no one can copyright a list of traumas, so organizations are free to make their own
list. CBT+ has developed a non-proprietary Trauma Screen. As well CBT+ makes available a
number of short, reliable, valid, and free measures on the CBT+ Notebook
(http://depts.washington.edu/hcsats/PDF/TF-%20CBT/CBT_Plus_NB.htm).
When an organization is ramping up to having an EBP trained work force, there are often
situations where only a few providers are trained in a particular intervention. This means that a
client seeking services for a condition such as PTSD may be assigned to a non-TF-CBT trained
provider despite the fact that there are some trained providers within the organization. It is
imperative that organizations hold discussions and have a strategy for how to handle this type
of situation since the service availability will be uneven.
Practical Strategies
1. Decide how and when trauma screening will be done. Choose a method and make it part of
the standard operating procedure. Require providers who screen to give some type of
acknowledging/validating response directly to the clients. CBT+ Notebook contains simple
user friendly “cheat sheets” to help the provider in how to do this with clients.
2. Incorporate standardized measures into the routine intake/assessment process.
3. Select specific clinical targets to assess during the intake/assessment process. Preferably
use a standardized measure in addition to the results of clinical interview.
4. Ensure that the treating provider has the results of trauma screening and standardized
measures if they do not conduct the initial assessment and provides clinical feedback prior
to initiation of treatment.
5. Establish a practice standard that an EBP for the identified clinical target will be used. The
provider will treat to the target until there is improvement. Adopt a systematic method of
measuring progress (preferably using a standardized measure).
6. Ensure that clients are assigned to a provider who is qualified to deliver an EBP for the
identified clinical target if there is a trained provider in the work force.
CBT+ EBP Practical Guide 26
a. Set intake procedures to require referral to an EBP trained provider when clients
meet the clinical criteria (e.g., have clinical elevations on measures, dx).
b. Facilitate access to an EBP trained provider when a referent or family explicitly
requests a certain EBP and meets the clinical criteria for the condition.
c. Establish organizational policies for how to make provider assignment decisions
when clients meet the criteria but there are insufficient trained providers.
a. Require consultation with a supervisor about case assignment
b. Assign to providers with training in comparable interventions or those who have
the basic knowledge and skills even if they have not attended a formal EBP
training.
c. Provider higher levels of supervision in those cases.
CBT+ EBP Practical Guide 27
V. Quality Assurance
QA for EBP refers to processes and procedures designed to ensure that the EBPs are being
delivered in accordance with the model and measuring outcomes to confirm that the desired
results are being achieved. There is extensive research documenting the importance of
maintaining fidelity in order to get the results that are found in clinical trials (Schoenwald et al.,
2011). The failure to achieve equivalent outcomes in real-world settings is generally attributed
to the fact that the models are not being delivered faithfully as opposed to the cases being
more difficult or complex (Weisz, Southam-Gerow, & McCarty, 2001). In some cases, an EBP can
produce worse results when delivered by providers who are judged to not be competent in the
model. For example, the recidivism rates for youth receiving Functional Family Therapy (FFT)
from therapists who were evaluated as not meeting QA standards were higher than for the
comparison (non-FFT) group, although FFT has been found to be an effective intervention for
delinquency (Barnoski, 2004). How well the model is delivered according to the guidelines,
matters.
One reason for these results may be that EBPs, when delivered by the book, are very focused
on a particular outcome, are structured and involve active efforts to help clients learn and do
new behaviors in the real world (e.g., via assigning homework). Careful research on usual public
mental health practice has found that usual care looks quite different from that (Garland et al.,
2010). Providers tend to cover many topics each session, few in much depth. Some strategies
are consistent with evidence-based strategies and others are not. In usual practice, modeling
or practicing skills in session, and assigning and reviewing homework practice are rare (Garland
et al., 2010). These are integral parts of most EBP.
It has been established that even when providers have received high-quality training and
consultation with demonstration of skill acquisition, they tend not to maintain the practice over
time (Whitaker et al., 2012). Providers tend to drift away from the model without ongoing
supervision and monitoring.
The purpose of fidelity monitoring is to go beyond supervision. Typically supervision does not
involve actually learning whether or not providers are doing EBPs faithfully across their
caseload. The only way to really assess drift or inexpert delivery of EBP treatment components
is to specifically track in-session provider behavior.
It is important to keep in mind that fidelity monitoring, as part of QA, is actually a mechanism
for achieving the intended client outcomes, not a goal in and of itself. Outcomes are the real
goal of EBP clinical practice. There are alternative methods in use. For example, another
CBT+ EBP Practical Guide 28
approach is to allow providers a certain amount of latitude in how EBP models are delivered
but to require routine assessment of the clinical target at frequent intervals and require
consultation/supervision if clients are not improving. A focus on outcomes can be effective
(Un tzer, 2012).
Challenges
Fidelity monitoring and outcome monitoring are the two strategies that ensure that EBPs are
being delivered as intended and the expected outcomes achieved. Both activities require staff
time to accomplish. The level of staff time required depends on the methods used.
There are three ways to monitor fidelity: observing or listening to sessions (live or recorded),
provider self-reported adherence as reported via checklist or progress notes, and client report
of provider fidelity to the model. Unfortunately, some studies have shown that provider self-
report may not be very reliable (in comparison to observation of actual practice). This is not
because providers intentionally misrepresent, but rather that it is challenging to objectively
report on one’s own behavior.
Given the limitations of self-report, in research studies, typically sessions are recorded (audio or
video); initially the entire session is coded or reviewed for adherence, over time random tapes
are coded or reviewed. Some interventions lend themselves to this methodology outside of a
research setting, such as PCIT, in which sessions are recorded as part of model delivery. For
most other therapies recording some or all sessions is outside of usual practice.
Buying the QA from developers or their companies is an excellent but expensive solution.
Many of the most well-established brand name interventions require organizations to purchase
the ongoing supervision and fidelity monitoring (e.g., MST, FFT, MTFC, SafeCare, KEEP, PMTO).
In other instances these services may be available from developers but not required for
advertising the service (e.g., PCIT). In some cases the QA monitoring by the company is
required on an ongoing basis, whereas in other cases the model developers create a pathway to
having independent internal organizational monitoring (e.g., SafeCare).
There will be many instances in which organizations will not be able to afford to buy model-
specific supervision at the level and intensity that typically is associated with the brand names.
As well, organizations are obligated to serve their whole client population, not only those who
meet the criteria for one of a few clinical targets that a brand name intervention may address.
Alternative, even if less rigorous methods, will be necessary in many cases.
CBT+ EBP Practical Guide 29
Practical Strategies
Routine or periodic checkups. The key to QA is feedback to providers and assistance in
improving skills, sticking with the model or tweaking strategies to address case-specific
considerations. In some settings, clinical supervisors will be able to review all progress notes,
provider or client self-report checklists, and outcomes measures. However, in most settings,
caseload size and supervisory capacity will preclude review of all possible QA measures.
Alternative strategies include creating methods for periodic or random review of tapes of
sessions (to compare against self-report), progress notes or completion of provider or client
self-reports. However, unless done consistently and reviewed with feedback provided to the
providers, the procedures will not achieve the desired goals. Therefore they must be feasible
within the extant job descriptions and the capacity to make adjustments in other duties.
1. Supervisor or consultant explicitly monitors fidelity.
When providers are initially learning an EBP it is optimal to have high levels of direct
observation to ensure they have acquired the basic skills. Organizations may require
supervisors to observe or listen to enough sessions to be confident that providers are
competent. Once initial competence is established, supervisors may occasionally but
systematically observe (or listen) to sessions or require tapes.
a. Supervisors sit in on sessions, observe through a one-way mirror, and/or listen in on
the phone during sessions.
b. Providers audio or video record sessions and review with supervisors.
2. Treatment plan templates require the clinical condition and/or diagnosis to align with a
specified EBP. The EBP is specifically identified on the treatment plan.
3. Progress note documentation explicitly addresses EBP and EBP components.
Providers complete progress notes for all sessions. Progress notes can be structured or
templated to prompt for documentation of certain evidence-based components or
activities.
a. Check boxes for EBP approach elements (e.g., primary treatment model;
homework assignment/review and compliance monitoring; modeling, role
play in session).
b. Check boxes for EBP session content (e.g., psychoed, exposure, behavioral
activation, selective attention, rewards and consequences, etc.).
c. Documentation of symptoms using standardized measure score, client rating.
CBT+ EBP Practical Guide 30
d. Narrative standards such as what the EBP model is and what component was
delivered, what homework was assigned, compliance with homework, etc.
e. End of session rating of client satisfaction or progress towards goals.
4. Providers complete fidelity checklists.
a. Providers are required to complete adherence checklists on a routine or periodic
basis. Most EBPs have model specific adherence checklists that ask providers to rate
whether and/or the degree to which they did the EBP components during a session.
5. Client Fidelity Self-Report
a. Clients complete checklists or respond to questions about what happened in session.
These checklists are brief and comparable to provider self-report checklists. They
can be administered on routine or systematic periodic basis. Some organizations
have clients complete these checklists at the front desk on the way out of the
appointment. Supervisors review the results.
6. Outcomes monitoring
a. Providers are required to administer standardized checklists or provide systematic
assessment of clinical or functional progress at set intervals. These measures are
used by the supervisor to monitor and adjust the treatment regimen. Failure to
progress after a selected number (4-5) of sessions prompts a requirement for
supervision/consultation.
b. Once the client outcomes for a specific clinical target are at non-clinical levels,
clients are discharged from active therapy or from active therapy for the specific
clinical target. The d/c is accompanied by explicit acknowledgement of the
achievement.
CBT+ EBP Practical Guide 31
VI. Technology
Technology and IT systems are critical ingredients for creating organizational QA procedures.
Technology offers many possibilities for quick, attractive and meaningful presentations of data
to give clients, providers and organizational senior leaders regarding the delivery of EBPs, EBP
fidelity monitoring and outcomes. These data are important at the individual client feedback
level, at the provider level for supervisors, and in the aggregate level for Senior Leaders and
organizational administrators. Measurement of targets, provider behaviors and outcomes is a
core principle of EBP.
IT systems in human service organizations most often focus on documenting activities that
funders or government regulators require. There has been less attention paid to focusing on
measuring the types of clinical activities and outcomes that reflect clinical change over time
that can be used to provide individual or clinic wide feedback.
The ideal would be to have mechanisms that do not add additional burden, but document
baseline and repeat measurement of clinical targets and adherence to the EBP at the individual
provider and the aggregate clinic level. Dashboards that track use of EBP components and
outcomes can be created. For example, Practicewise is a proprietary method of creating EBP
dashboards for EBP components in child psychotherapy (http://www.practicewise.com). The
Washington State Mental Health Integration Program (MHIP; http://integratedcare-nw.org)
tracks mental health outcomes within primary care clinics where standardized mental health
measures are incorporated into the electronic medical record and scores are treated like lab
values that can be tracked.
Most public mental health clinics will have limitations in the ability to use technology because
of funding constraints. However, there are lower tech strategies that might be incorporated
within existing resources. Methods of documenting fidelity to EBP components and outcomes
that require individual level review of client files or chart and that do not permit tracking are
unwieldy, inefficient and do not allow for provider or clinic-level analyses. At the same time,
systems that require double or triple entry are expensive and burdensome.
Practical Strategies
1. Electronic medical record.
a. Most organizations buy electronic medical records system off the shelf. There may
be restrictions on the amount of modifying or adding that is possible. Whenever
possible in decision-making, consideration should be given to how readily data can
CBT+ EBP Practical Guide 32
be extracted from the EMR. The most important information for which to have
access is scores on standardized measures and documentation of use of EBP
components (e.g., self-report fidelity checklists).
a. It is ideal if the EMR permits data entry/scoring of standardized measures
which would them appear in the same way that lab values do.
b. If the EMR does not come with capacity for extracting quantitative data, it is a
worthwhile investment to seek IT consultation about how to add, to modify or to
incorporate repeat assessment with standardized measures and documentation of
use of EBP components. Some organizations create proprietary web-based progress
note systems. This approach allows for tailoring and modification but requires a
substantial investment to achieve.
2. Creating dashboards.
a. Simple dashboards can be created using an ordinary excel spreadsheet. For example,
a fidelity checklist for TF-CBT is can be constructed based on the components
acronym PRACTICE. Providers document which components were delivered during a
session and how much time was expended. The dashboard representation shows
the progression through the components and the percentage of time over the
course of therapy. If this type of checklist is not incorporated into the progress note,
it does require double data entry.
a. Supervisors can create their own dashboards to monitor provider progress
with specific clients. CBT+ supervisors have created few versions that are
available upon request.
3. Random review of progress notes.
a. IT creates a list of randomly selected visits per week for supervisor review.
Supervisors read the randomly selected progress notes to review how closely
providers are sticking with and documenting the EBP that is being used. By randomly
selecting cases for review, the supervisor is able to get an overall sense of the
provider’s practice approach versus relying on provider selected cases brought for
discussion since most discussed cases are likely to be outliers or unusual.
4. Data entry procedures for standardized measures.
a. The EBP Roster Clinical Toolkit (http://ebproster.org/roster/toolkit.php) has many
different standardized measures. Providers who have participated in CBT+ or other
approved EBP training can get an account that allows them to enter cases and score
standardized measures. Providers or clients can enter the data.
CBT+ EBP Practical Guide 33
b. Kiosks or notepads can be set up in waiting areas and clients can enter their own
data (i.e., respond to computer-administered symptom questionnaires) which are
transmitted to the provider for in session review. Provider can assist client in data
entry in the office and review it immediately.
5. Scoring of standardized measures.
a. The EBP Roster Clinical Toolkit (http://ebproster.org/roster/toolkit.php) offers
automatic scoring of a variety of standardized measures. Providers who have
participated in CBT+ Learning Collaborative can get an account that allows them to
enter cases and score standardized measures.
b. Many free, reliable and valid clinical target measures are brief and easily scored.
They yield a score that classifies the level of the clinical targets. Scoring programs
can easily be constructed to provide visual representations (bar graphs, line graphs)
that can be shown to clients as a means of providing feedback and motivating client
engagement in treatment.
c. Purchased proprietary measures have scoring procedures that create profiles and
track change over time (e.g., ECBI for PCIT; CBCL, BASC).
6. Monitoring outcomes.
a. IT can use administrative data systems to identify cases that have had a certain
number of visits to create prompts for providers or supervisors to examine progress.
Providers may be required to re-administer standardized measures at selected
intervals. The key is initiating standardized methods for systematically identifying
cases and the number of visits.
CBT+ EBP Practical Guide 34
Summary and Recommendations
Public mental health organizations can undertake a variety of organizational strategies, some of
which are low or no cost, to embed EBPs within their organizational structures. There are
specific steps that can be taken in all of the key areas that have been identified by the
dissemination and implementation science as important to keep EBPs going and flourishing
within organizations. Setting, context and resources will influence the way that organizations go
about these activities.
Adoption and sustainment of EBPs in public mental health settings will improve the lives of
children and families. Many of these children and families suffer from a disproportionate share
of adversity and are struggling to manage many aspects of their lives. High quality, effective
mental health care that produces results within a relatively short period of time will not only
ease their immediate distress, but can help set them on a pathway towards success in many
areas of functioning. They deserve no less than the best that is available.
CBT+ EBP Practical Guide 35
Appendices
CBT+ EBP Practical Guide 36
References
Aarons, G. A., Fettes, D. L., Flores, L. E., & Sommerfeld, D. H. (2009). Evidence-based practice and staff
emotional exhaustion in children's services. Behaviour Research and Therapy, 47(11), 954-960.
Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2011). Advancing a conceptual model of evidence-based
practice implementation in child welfare. Administration and Policy in Mental Health & Mental
Health Services Research, 38(1), 4-23.
Aarons, G. A., & Sommerfeld, D. H. (2012). Leadership, innovation climate, and attitudes toward
evidence-based practice during a statewide implementation. Journal of the American Academy
Child & Adolescent Psychiatry, 51(4), 423-431.
Aarons, G. A., Sommerfeld, D. H., Hecht, D. B., Silovsky, J. F., & Chaffin, M. J. (2009). The impact of
evidence-based practice implementation and fidelity monitoring on staff turnover: Evidence for
a protective effect. Journal of Consulting & Clinical Psychology, 77(2), 270-280.
Barnoski, R. (2004) Outcome evaluation of Washington State’s research based programs for juvenile
offenders. (Document No. 04-01-1201)Olympia, WA: Washington State Institute for Public
Policy.
Beidas, R. S., Cross, W. F., & Dorsey, S. (in press). Show me don't tell me: Behavioral rehearsal as
a training and fidelity tool. Cognitive and Behavioral Practice.
Beidas, R. S., Edmunds, J. M., Marcus, S. C., & Kendall, P. C. (2012). Training and consultation to promote
implementation of an empirically supported treatment: A randomized trial. Psychiatric
Services, 63(7), 660-665.
Beidas, R. S., & Kendall, P. C. (2010). Training therapists in evidence-based practice: A critical review of
studies from a systems-contextual perspective. Clinical Psychology: Science & Practice, 17, 1-30.
Burley, M. (2009). Outpatient treatment differences for children served in Washington’s public mental
health system (Document No. 09-10-3401). Olympia, WA: Washington State Institute for Public
Policy.
Damschroder, L., Aron, D., Keith, R., Kirsh, S., Alexander, J., & Lowery, J. (2009). Fostering
implementation of health services research findings into practice: a consolidated framework for
advancing implementation science. Implementation Science, 4, 50-65.
Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research:
A synthesis of the literature. Tampa: University of South Florida, Louis de la Parte Florida Mental
Health Institute, the National Implementation Research Network (FMHI Publication #231).
CBT+ EBP Practical Guide 37
Garland, A. F., Brookman-Frazee, L., Hulburt, M., Accurso, E. C., Zoffness, R., Hain, R. A., & Ganger, W.
(2010). Mental health care for children with disruptive behavior problems: A view inside
therapists’ offices. Psychiatric Services, 61(8), 788-795.
Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K., Mayberg, S., & Green, P. (2008).
Assessing the Organizational Social Context (OSC) of mental health services: Implications for
research and practice. Administration & Policy in Mental Health & Mental Health Services
Research. Special Issue: Improving Mental Health Services, 35(1-2), 98-113.
Herschell, A. D., Kolko, D. J., Baumann, B. L., & Davis, A. C. (2010). The role of therapist training in the
implementation of psychosocial treatments: A review and critique with recommendations.
Clinical Psychology Review, 30, 448-466.
Kolko, D. J., Baumann, B. L., Herschell, A. D., Hart, J. A., Holden, E. A., & Wisniewski, S. R. (2012).
Implementation of AF-CBT by community practitioners serving child welfare and mental health:
A randomized trial. Child Maltreatment, 17(1), 32-46
Palinkas, L. A., Aarons, G. A., Chorpita, B. F., Hoagwood, K., Landsverk, J., & Weisz, J. R. (2009). Cultural
exchange and the implementation of evidence-based practices: Two case studies. Research on
Social Work Practice, 19(5), 602-612.
Palinkas, L. A., & Soydan, H. (2012). Translation and implementation of evidence-based practice. New
York: Oxford University Press.
Proctor, E. (2012). Implementation science and child maltreatment: Methodological advances. Child
Maltreatment, 17(1), 107-112.
Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A.
(2011). Toward the effective and efficient measurement of implementation fidelity.
Administration & Policy in Mental Health & Mental Health Services Research, 38(1), 32-43.
Stirman, S. W., Kimberly, J., Cook, N., Calloway, A., Castro, F., & Charns, M. (2012). The sustainability of
new programs and innovations: A review of the empirical literature and recommendations for
future research. Implementation Science, 7(17), 1-19.
n tzer, ., Chan, . ., afer, E., naster, ., Shields, ., Powers, D., eith, . C. (20 2). uality
improvement with pay-for-performance incentives in integrated behavioral health
care. American Journal of Public Health, 102(6), 41-5.
Whitaker, D. J., Ryan, K. A., Wild, R. C., Self-Brown, S., Lutzker, J. R., Shanley, J. R., Edwards, A. M., ...
Hodges, A. E. (2012). Initial implementation indicators from a statewide rollout of SafeCare
within a child welfare system. Child Maltreatment, 17(1), 96-101.
CBT+ EBP Practical Guide 38
Wiesz, J. R., Southam-Gerow, M. A., & McCarty, C. A. (2001). Control-related beliefs and depressive
symptoms in clinic-referred children and adolescents. Journal of Abnormal Psychology, 110, 97-
109.
Evidence Based Practice Institute & Washington State Institute for Public Policy (2012). Inventory of
evidence-based, research-based, and promising practices (Document No. E2SHB2536). Olympia:
Washington State Institute for Public Policy.
CBT+ EBP Practical Guide 39
CBT+ Team
We are your team. Please feel free to contact us for further information and consultation.
Lucy Berliner, MSW Harborview Center for
Sexual Assault & Traumatic Stress University of Washington 325 9th Ave, Box 359947
Seattle, WA 98104 206 744 1645
Shannon Dorsey, PhD Department of Psychology University of Washington Guthrie Hall, Box 351525
Seattle, WA 98195 206 543 4527
Laura Merchant, MSW Harborview Center for
Sexual Assault & Traumatic Stress University of Washington 325 9th Ave, Box 359947
Seattle, WA 98104 206 744 1637
Nathanial Jungbluth, PhD Psychiatry & Behavioral Medicine
Seattle Children’s ospital M/S W-3636, PO Box 5371
Seattle, WA 98105 303-241-2261
Georganna Sedlar, PhD Division of Public Behavioral Health
& Justice Policy University of Washington
2815 Eastlake Ave E, Ste 200 Box 358015
Seattle, WA 98102 206 685 2085 [email protected]
CBT+ EBP Practical Guide 40
CBT+ Senior Leader Meeting Participants
Stacey Alles Compass Health
PO Box 3810 Everett, WA 98213
425 349 6171 [email protected]
Ralph Fragale Atlantic Street Center
2103 S Atlantic St Seattle, WA 98144
206 329 2050 [email protected]
Joelle Blair
Community Psychiatric Clinic 10501 Meridian Ave N, Ste D
Seattle, WA 98133 206 366 3001
Dan Fox Lutheran Community Services Northwest
210 W Sprague Spokane, WA 99201
509 343 5089 [email protected]
Karen Brady Ryther
2400 NE 95th St Seattle, WA 98115
206 517 0267 [email protected]
Melissa Gorsuch-Clark Catholic Family and Child Service
5301 Tieton Dr, Ste C Yakima, WA 98908
509 965 7100 [email protected]
Darlene Darnell
Catholic Family and Child Service 5301 Tieton Dr, Ste C
Yakima, WA 98908 509 965 7100
Lynne Guhlke NEW Alliance Counseling Services
165 E Hawthorne Ave Colville, WA 99114
509 685 0637 [email protected]
Edith Elion
Atlantic Street Center 2103 S Atlantic St
Seattle, WA 98144 206 329 2050
Carole Kosturn Compass Health
4526 Federal Ave Everett, WA 98203
425 349 8363 [email protected]
CBT+ EBP Practical Guide 41
Sandy Ellingboe Multicare Good Samaritan Behavioral Health
325 Pioneer Ave E Puyallup, WA 98372
253 697 8417 [email protected]
Heike Lake Lutheran Community Services Northwest
210 W Sprague Spokane, WA 93854
509 343 5008 [email protected]
Jack Maris
Central WA Comprehensive Mental Health 402 S 4th Ave
Yakima, WA 98901 509 575 4900
Susie Winston Sound Mental Health
1600 E Olive St Seattle, WA 98122
206 302 2340 [email protected]
Robin McIlvaine
Child Study and Treatment Center 8805 Steilacoom Blvd SW
Lakewood, WA 98498 253 761 3353
DeAnn Yamamoto KCSARC
PO Box 300 Renton, WA 98034
425 282 0349 [email protected]
Gena Palm
Navos 2600 SW Holden St Seattle, WA 98126
206 933 7164 [email protected]
Unable to Attend-Phone Consultation
Beth Friedman-Darner Kitsap Mental Health Services
900 Sheridan Rd, Ste 106 Bremerton, WA 98310
360 415 5866 [email protected]
Don Fields Moriarty Kitsap Mental Health Services
900 Sheridan Rd, Ste 106 Bremerton, WA 98310
360 415 5866 [email protected]
CBT+ EBP Practical Guide 42
DBHR Contract Manager
Jeannie D'Amato DSHS/DBHR 4500 10th Ave SE Lacey, WA 98503 360 725 2313 [email protected]
CBT+ EBP Practical Guide 43
What is CBT+?
CBT+ describes a training and consultation approach that encompasses Trauma-Focused CBT (TF-CBT),
CBT for anxiety, CBT for depression, and parent management training (PMT or BPT). Starting in 2009,
the Washington State Division of Behavioral Health and Recovery (DBHR) supported Harborview and the
University of Washington Evidence-Based Practice Institute (UW EBPI) to bring TF-CBT to public mental
health organizations across the state.
TF- CBT is a well-supported and highly acceptable intervention for children affected by trauma. Public
mental health settings were especially interested to have an effective therapy for the impact of trauma
because most of the children they serve have complex trauma histories.
Harborview and UW EBPI are committed to the evidence-based goal that children in need have access to
effective treatments for all conditions. We expanded the approach to include teaching evidence-based
treatments that cover the most common clinical conditions of children in public mental health:
depression, anxiety, and behavior problems as well as trauma-specific impact such as posttraumatic
stress.
The program includes organization consultation, active learning sessions, and case consultation. Clinical
supervisors are supported to take leadership internally to sustain the EBPs through monthly calls and an
annual meeting. There are listservs, web-based resources and a web-based Toolkit that scores
standardized measures and collects metrics. Providers can roster on a public web-site.
We have very high regard for our colleagues in public mental health who have embraced the evidence-
based ideal and taken steps to bring effective interventions to the children and families who have the
highest need. We have learned a lot about how complicated it is to successfully take research to the real
world practice context. It is inspiring to see what can be accomplished with leadership, creativity and
commitment. None of this would have happened if it had not been for the WA State DBHR commitment
to invest in EBP and support for the Initiative. We specifically thank Robin McIlvaine and Jeannie
D’ mato.
We are very grateful to TF-CBT developers Judy Cohen, MD and Tony Mannarino, PhD, Allegheny
General Hospital/Drexel University College of Medicine, and Esther Deblinger, PhD, UMDNJ-School of
Osteopathic Medicine for inventing an intervention that really resonates with public mental health and
for being so supportive of our local variation on the theme. We appreciate so much that they have come
out to train. And we thank our colleagues Michael de Arellano, PhD and Carla Danielson, PhD, Medical
University (MUSC) of South Carolina; David Kolko, PhD, University of Pittsburg; and Laura Murray, PhD,
Johns Hopkins University for participation in advanced training on their cultural adaptations and CBTs
for violent families and high risk adolescents. Special thanks to Jason Lang, PhD Connecticut Center for
Effective Practice Child Health and Development Institute (CHDI) and Rochelle Hanson PhD, MUSC for
additions to the Guide.