evidence based practice what is it? why do it? points to consider
DESCRIPTION
Evidence Based Practice – An Overview Webinar for Reclaiming Futures October 23, 2008 Randolph Muck, M.Ed. CSAT/SAMHSA Contact Info: [email protected] 240-276-1576. Evidence Based Practice What is it? Why do it? Points to consider. - PowerPoint PPT PresentationTRANSCRIPT
Evidence Based Practice – An Overview
Webinar for Reclaiming Futures October 23, 2008
Randolph Muck, M.Ed.CSAT/SAMHSAContact Info: [email protected]
Evidence Based Practice
What is it?
Why do it?
Points to consider
Evidence Based Practice
The term evidence-based practice (EBP) refers to preferential use of mental and behavioral health interventions for which systematic empirical research has provided evidence of statistically significant effectiveness as treatments for specific problems. Alternate terms with the same meaning are evidence-based treatment (EBT) and empirically-supported treatment (EST).
Evidence Based Practice
Tested with good outcomes
Manual exists so it can be replicated/trained
A training program exists
Supervision leading to certification
Ongoing monitoring
Outcomes measurement
Ways of Viewing EBP
EBP is a process. EBP is a way of doing practice that integrates the best evidence with clinical expertise and consumer values. (EBP as a verb.) (Sackett et al., 2000)
PractitionerExpertise
BestEvidence
Client Values & Preferences
EBP
Ways of Viewing EBP
EBP is a product. An evidence-based practice is any practice that has been established as effective through scientific research according to some set of explicit criteria. (EBP as a noun.) (Drake, 2001)
– EB Interventions. (A-CRA, MET/CBT5)– EB Skill sets. (CBT, Behavioral Parent Training)
Definition of Implementation
“…Specified set of activities designed to put into practice an activity or program of known dimensions…such that independent observers can detect its presence and strength.”
(Fixsen et al, 2004, p. 5)
Definition of Fidelity
Strategies used to monitor the faithful delivery of a manual-guided behavioral intervention
Important dimensions include – adherence (i.e., extent to which intervention
procedures were delivered as prescribed in the treatment manual)
– competence (i.e., qualitative measure of the skillfulness in which intervention procedures are delivered)
Different Types of Manuals
Session Driven
Procedure Driven
Principle Driven
Randomized Clinical Trials (RCT) are to Evidence Based Practice (EBP) like Self-reports are to Diagnosis
They are only as good as the questions asked (and then only if done in a reliable/valid way)
They are an efficient and logical place to start But they can be limited or biased and need to be
combined with other information Just because the person does not know something
(or the RCT has not be done), does not mean it is not so
Synthesizing them with other information usually makes them better
The field is increasingly facing demands from payers, policymakers, and the public at large for “evidence-based practices (EBP)” which can reliably produce practical and cost-effective interventions, therapies and medications that will
– reduce risks for initiating drug use among those not yet using, – reduce substance use and its negative consequences among those who are
abusing or dependent, and– reduce the likelihood of relapse for those who are recovering
NIDA Blue Ribbon Panel on Health Services Research (see www.nida.nih.gov )
Context
So what does it mean to move the field towards Evidence Based Practice (EBP)?
Introducing reliable and valid assessment that can be used – At the individual level to immediately guide clinical judgments
about diagnosis/severity, placement, treatment planning, and the response to treatment
– At the program level to drive program evaluation, needs assessment, and long term program planning
Introducing explicit intervention protocols that are– Targeted at specific problems/subgroups and outcomes– Having explicit quality assurance procedures to cause adherence
at the individual level and implementation at the program level
Having the ability to evaluate performance and outcomes – For the same program over time, – Relative to other interventions
The Current Renaissance of Adolescent Treatment Research
Feature 1930-1997 1997-2005
Tx Studies* 16 Over 200
Random/Quasi 9 44
Tx Manuals* 0 30+
QA/Adherence Rare Common
Std Assessment* Rare Common
Participation Rates Under 50% Over 80%
Follow-up Rates 40-50% 85-95%
Methods Descriptive/Simple More Advanced
Economic Some Cost Cost, CEA, BCA
* Published and publicly available
Juvenile Justice involved youth increasing presence in the treatment system
Support for funding relies on ability to demonstrate effectiveness Treatment needs of the youth that we see and the need to
incorporate appropriate and effective interventions for these needs
Continuing Care is as or more important than the treatment delivered
Issues to Consider
-
10,000
20,000
30,000
40,000
50,000
60,000
70,000
80,000
90,000
Juve
nile
Jus
tice
Sch
ool
Sel
f/F
amil
y
Oth
erC
omm
unit
y
Oth
er S
A T
xA
genc
y
Oth
er H
ealt
hC
are
Em
ploy
ee/E
AP
0%
20%
40%
60%
80%
100%
120%
140%
1993
2003
Change
Change in Referral Sources: 1993-2003
Source: Treatment Episode Data Set (TEDS) 1993-2003.
JJ referrals have doubled, are 53% of 2003 admissions and
driving growth
Other sources of Referral have grown, but less than expected
41%
37%
12%
37%
114%
115%
5%
61% growth
53% Have Unfavorable Discharges
Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX, UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration. Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .
0% 20% 40% 60% 80% 100%
Outpatient(37,048 discharges)
IOP(10,292 discharges)
Detox(3,185 discharges)
STR(5,152 discharges)
LTR(5,476 discharges)
Total(61,153 discharges)
Completed Transferred ASA/ Drop out AD/Terminated
Despite being widely recommended, only 10% step down after intensive treatment
Median Length of Stay is only 50 days
Source: Data received through August 4, 2004 from 23 States (CA, CO, GA, HI, IA, IL, KS, MA, MD, ME, MI, MN, MO, MT, NE, NJ, OH, OK, RI, SC, TX, UT, WY) as reported in Office of Applied Studies (OAS; 2005). Treatment Episode Data Set (TEDS): 2002. Discharges from Substance Abuse Treatment Services, DASIS Series: S-25, DHHS Publication No. (SMA) 04-3967, Rockville, MD: Substance Abuse and Mental Health Services Administration. Retrieved from http://wwwdasis.samhsa.gov/teds02/2002_teds_rpt_d.pdf .
0 30 60 90
Outpatient(37,048 discharges)
IOP(10,292 discharges)
Detox(3,185 discharges)
STR(5,152 discharges)
LTR(5,476 discharges)
Total(61,153 discharges)
Lev
el o
f C
are
Median Length of Stay
50 days
49 days
46 days
59 days
21 days
3 days
Less than 25% stay the
90 days or longer time
recommended by NIDA
Researchers
Past 90 day HIV Risk Behaviors
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
84%
38%
32%
26%
21%
3%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Sexually active
Sex Under the Influence of AOD
Multiple Sex partners
Any Unprotected Sex
Victimized Physically, Sexually, orEmotionally
Any Needle use
Recovery Environment
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
57%
49%
28%
74%
65%
14%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Social Peers Getting Drunk Weekly+
School/Work Peers Getting Drunk Weekly+
Others at Home Getting Drunk Weekly+
Social Peers Using Drugs
School/Work Peers Using Drugs
Others at Home Using Drugs
Co-Occurring Psychiatric Problems
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
79%
54%
45%
37%
26%
17%
59%
47%
31%
25%
16%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Any Co-occurring Psychiatric
Conduct Disorder
Attention Deficit/Hyperactivity Disorder
Major Depressive Disorder
Traumatic Stress Disorder
General Anxiety Disorder
Ever Physical, Sexual or Emotional Victimization
High severity victimization (GVS>3)
Ever Homeless or Runaway
Any homicidal/suicidal thoughts past year
Any Self Mutilation
Past Year Violence & Crime
*Dealing, manufacturing, prostitution, gambling (does not include simple possession or use)
Source: CSAT AT Outcome Data Set (n=9,276 adolescents)
82%
69%
66%
51%
49%
45%
84%
68%
39%
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Any violence or illegal activity
Physical Violence
Any Illegal Activity
Any Property Crimes
Other Drug Related Crimes*
Any Interpersonal/ Violent Crime
Lifetime Juvenile Justice Involvement
Current Juvenile Justice involvement
1+/90 days In Controlled Environment
Multiple Problems* are the Norm
Source: CSAT AT Common GAIN Data set
NoneOne
Two
Three
Four
Five to Twelve
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Most acknowledge 1+ problems
Few present with just one problem
(the focus of traditional research)
In fact, over half present
acknowledging 5+ major problems
* (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD, CD, victimization, violence/ illegal activity)
No. of Problems* by Severity of Victimization
Source: CSAT AT Common GAIN Data set (odds for High over odds for Low)
* (Alcohol, cannabis, or other drug disorder, depression, anxiety, trauma, suicide, ADHD,
CD, victimization, violence/ illegal activity)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Low (31%) Moderate (17%) High (51%)
Five or More
Four
Three
Two
One
None
Those with high lifetime levels of
victimization have 117 times higher
odds of having 5+ major problems*
GAIN General Victimization Scale Score (Row %)
Most Lack of Standardized Assessment for…
Substance use disorders (e.g., abuse, dependence, withdrawal), readiness for change, relapse potential and recovery environment
Common mental health disorders (e.g., conduct, attention deficit-hyperactivity, depression, anxiety, trauma, self-mutilation and suicidality)
Crime and violence (e.g., inter-personal violence, drug related crime, property crime, violent crime)
HIV risk behaviors (needle use, sexual risk, victimization)
Child maltreatment (physical, sexual, emotional)
Summary of Problems in the Treatment System
The public systems is changing size, referral source, and focus – often in different directions by state
Major problems are not reliably assessed (if at all) Less than 50% stay 50 days (~7 weeks) Less the 25% stay the 3 months recommended by
NIDA researchers Less than half have positive discharges After intensive treatment, less than 10% step down
to outpatient care While JJ involvement is common, little is known
about the rate of initiation after detention
EBPs and Treatment for Youth in the Juvenile Justice
System
Some Programs Have Negative or No Effects on recidivism
“Scared Straight” and similar shock incarceration program
Boot camps mixed – had bad to no effect
Routine practice – had no or little (d=.07 or 6% reduction in recidivism)
Similar effects for minority and white (not enough data to comment on males vs. females)
The common belief that treating anti-social juveniles in groups would lead to more “iatrogenic” effects appears to be false on average (i.e., relapse, violence, recidivism for groups is no worse then individual or family therapy)
Source: Adapted from Lipsey, 1997, 2005
Meta Analysis of the Effectiveness of Programs for Juvenile Offenders
N of
Offender Sample Studies
Preadjudication (prevention) 178
Probation 216
Institutionalized 90
Aftercare 25
Total 509
Source: Adapted from Lipsey, 1997, 2005
Most Programs are actually a mix of components
Average of 5.6 components distinguishable in program descriptions from research reports
Intensive supervisionPrison visitRestitutionCommunity serviceWilderness/Boot campTutoringIndividual counselingGroup counselingFamily counselingParent counselingRecreation/sportsInterpersonal skills
Anger managementMentoringCognitive behavioralBehavior modificationEmployment trainingVocational counselingLife skillsProvider trainingCaseworkDrug/alcohol therapyMultimodal/individualMediation
Source: Adapted from Lipsey, 1997, 2005
Most programs have small effectsbut those effects are not negligible
The median effect size (.09) represents a reduction of the recidivism rate from .50 to .46
Above that median, most of the programs reduce recidivism by 10% or more
One-fourth of the studies show recidivism reductions of 30% or more
The “nothing works” claim that rehabilitative programs for juvenile offenders are ineffective is false
Source: Adapted from Lipsey, 1997, 2005
Major Predictors of Bigger Effects
1. Chose a strong intervention protocol based on prior evidence
2. Used quality assurance to ensure protocol adherence and project implementation
3. Used proactive case supervision of individual
4. Used triage to focus on the highest severity subgroup
Impact of the numbers of Favorable features on Recidivism (509 JJ studies)
Source: Adapted from Lipsey, 1997, 2005
Usual Practice has little
or no effect
Program types with average or better effects on recidivism
AVERAGE OR BETTER BETTER/BEST
Preadjudication
Drug/alcohol therapy Interpersonal skills training
Parent training Employment/job training
Tutoring Group counseling
Probation
Drug/alcohol therapy Cognitive-behavioral therapy
Family counseling Interpersonal skills training
Mentoring Parent training
Tutoring
Institutionalized
Family counseling Behavior management
Cognitive-behavioral therapy Group counseling
Employment/job training Individual counseling
Interpersonal skills trainingSource: Adapted from Lipsey, 1997, 2005
Cognitive Behavioral Therapy (CBT) Interventions that Typically do Better than Practice in Reducing Recidivism (29% vs. 40%)
Aggression Replacement Training Reasoning & Rehabilitation Moral Reconation Therapy Thinking for a Change Interpersonal Social Problem Solving Multisystemic Therapy Functional Family Therapy Multidimensional Family Therapy Adolescent Community Reinforcement Approach MET/CBT combinations and Other manualized CBT
Source: Adapted from Lipsey et al 2001, Waldron et al, 2001, Dennis et al, 2004
NOTE: There is generally little or no differences in mean effect size between these brand names
Implementation is Essential (Reduction in Recidivism from .50 Control Group Rate)
The effect of a well implemented weak program is
as big as a strong program implemented poorly
The best is to have a strong
program implemented
well
Thus one should optimally pick the strongest intervention that one can
implement wellSource: Adapted from Lipsey, 1997, 2005
Moderate to large differences in Cost-Effectiveness by Condition
Source: Dennis et al., 2004
$0
$4
$8
$12
$16
$20
Cos
t per
day
of
abst
inen
ce o
ver
12 m
onth
s
$0
$4,000
$8,000
$12,000
$16,000
$20,000
Cos
t per
per
son
in r
ecov
ery
at m
onth
12
CPDA* $4.91 $6.15 $15.13 $9.00 $6.62 $10.38
CPPR** $3,958 $7,377 $15,116 $6,611 $4,460 $11,775
MET/ CBT5MET/
CBT12FSN MET/ CBT5 ACRA MDFT
* p<.05 effect size f=0.48** p<.05, effect size f=0.72
Trial 1 Trial 2
* p<.05 effect size f=0.22 ** p<.05, effect size f=0.78
MET/CBT5 and 12 did better
than FSN
ACRA did better than MET/CBT5, and both did better than MDFT
Choosing an EBP
Best evidencePractitioner experienceYouth/Family values and preferencesReadiness for change (buy-in at all levels
of agency, but particularly the top)Cost/ResourcesAbility to implement well
What are the pitfalls of EBP?
EBP generally causes some staff turnover EBP often shines a light on staff or work place problems
that would otherwise be ignored EBP often impact a wide range of existing procedures and
policies – requiring modification and provoking resistance EBP (and most organizational changes) will fail without
good senior staff leadership EBP typically require going for more funds from grant or
other funders On-going needs assessment will create demand for more
change and more EBP
A Fearless Appraisal… We are entering a renaissance of new knowledge in this area, but are only
reaching 1 of 10 in need
Several interventions work, but 2/3 of the adolescents are still having problems 12 months later
Effectiveness is related to severity, intervention strength, implementation/adherence, and how assertive we are in providing treatment
As other therapies have caught up technologically, there is no longer the clear advantage of family therapy found in early literature reviews
While there have been concerns about the potential iatrogenic effects of group therapy, the rates do not appear to be appreciably different from individual or family therapy if it is done well (important since group tx typically costs less)
Effectiveness was not consistently associated with the amount of therapy over a short period of time (6-12 weeks) but was related to longer term continuing care
Common Strategies you can do NOW Standardize assessment and identify most common problems Pool knowledge about what staff have done in the past, whether it
worked, and what the barriers were Identify system barriers (e.g., criteria to local access case management,
mental health) that could be avoided if thought of in advance Identify existing materials that could help and make sure they are
readily available on site Identify promising strategies for working with the adolescent, parents,
or other providers Develop a 1-2 page checklist of things to do when this problem comes up Identify a more detailed protocol and trainer to address the problem,
then go for a grant to support implementation
Evidenced Based Practice - Summary
Achieving reliable outcomes requires reliable measurement, protocol delivery and on-going performance monitoring.
The GAIN is one measure that is being widely used by CSAT grantees and others trying to address gaps in current knowledge and move the field towards evidenced based practice.
Standardized and more specific assessment helps to draw out treatment planning implications of readiness for change, recovery environment, relapse potential, psychopathology, crime/violence, and HIV risks.
Adolescents entering more intensive levels of care typically have higher severity.
Multiple problems and child maltreatment are the norm and are closely related to each other.
There is a growing number of standardized assessment tools, treatment protocols and other resources available to support evidenced based practices.