chris borgmeier portland state university evaluating research on challenging behavior
TRANSCRIPT
www.pbis.org
Horner, R., & Sugai, G. (2008). Is school-wide positive behavior support an evidence-based practice? OSEP Technical Assistance Center on Positive Behavioral Interventions and Support. http://www.pbis.org/files/101007evidencebase4pbs.pdf.
Why evidence-based?
Maximize outcomesMinimize harmIncreased accountabilityIncrease efficiencyImprove decision makingImprove resource use
Basic Approach
Start w/ what has greatest likelihood of addressing (evidence-based) confirmed problem/question Explained/supported
conceptually/empiricallyAdapt to local context/culture/needMonitor regularly & adjust based
on dataAdapt for efficient & durable
implementation
4 Evaluation Criteria
Effectiveness Has/will practice produced desired
outcome?Efficiency
What are costs (time, resources, $) to implement practice?
Relevance Is practice & outcomes appropriate for
situation?Conceptually soundness
Is practice based on theory?
Identify practicethat addressesneed/problem?
Is practiceresearchbased?
Specify features ofneed/problem
Is evidence ofeffectiveness
available?
Can practicebe adapted?
Implement &monitor effects
Consider anotherpractice
No No
Yes
Yes
No
Is adequateprogress
observed?
No
Yes
Improve efficiency& sustainability of
practiceimplementation
Review questions& data on regular
basis
Does problemexist?
Yes
No
Yes
Start
Basic Practices Evaluation
Identify practicethat addressesneed/problem?
Specify features ofneed/problem
Review questions& data on regular
basis
Does problemexist?
Yes
No
Start
Identify practicethat addressesneed/problem?
Is practiceresearchbased?
Is evidence ofeffectiveness
available?
Can practicebe adapted?
Consider anotherpractice
No No
Yes
Yes
No
Yes
Can practicebe adapted?
Implement &monitor effects
Yes
Yes
Is adequateprogress
observed?
No
Yes
Improve efficiency& sustainability of
practiceimplementation
Design Questions
Has functional or cause-effect relationship been demonstrated & replicated?
Have alternative explanations been accounted & controlled for?
Have threats or weaknesses of methodology been controlled for?
Was study implemented w/ fidelity/accuracy?
Research Designs
Experimental - RCT & SSREvaluation - Descriptive w/ baselineCase Study - Descriptive w/o
baselineTestimonial - No/Limited data
Results QuestionsWho were subjects?
How much like my participants?Where was study conducted?
How much like where I work?What measures were used?
Do I have similar data?What outcomes were achieved?
Are expected outcomes similar
Effectiveness Logic
Significance (“believe”) Likelihood of same effect by chance
Effect Size (“strength”) Size of effect relative to business as usual
Consequential Validity (“meaning”) Contextually meaningful
School-wide PBIS – Let’s compare!
www.pbis.org Click on ‘Resource Catalog’ Then ‘Literature List’
Evidence Base for SW-PBISRandomized Control Trials
Randomized Control Trials of SW-PBIS
Tier 1/ Universal SW-PBIS
Bradshaw, C. P., Mitchell, M. M., & Leaf, P. J. (2010).Examining the effects of school-wide positive behavioral interventions and supports on student outcomes: Results from a randomized controlled effectiveness trial in elementary schools. Journal of Positive Behavior Interventions, 12(3), 133-148.
Bradshaw, C.,Koth, C., Bevans, K., Ialongo, N., & Leaf, P. (2008). The impact of school-wide positive behavioral interventions and supports (PBIS) on the organizational health of elementary schools.School Psychology Quarterly.
Bradshaw, C., Reinke, W., Brown, L., Bevans, K., & Leaf, P. (2008).Implementation of school-wide positive behavioral interventions and supports (PBIS) in elementary schools: Observations from a randomized trial. Education and Treatment of Children, 31, 1-26.
Horner, R. H., Sugai, G., Smolkowski, K., Eber, L., Nakasato, J., Todd, A. W., & Esperanza, J., (2009). A randomized, wait-list controlled effectiveness trial assessing school-wide positive behavior support in elementary schools. Journal of Positive Behavior Interventions, 11(3), 133-144.
Sprague, J., & Biglan, A., et al (in progress).A Randomized Control Trial of SWPBS with Middle Schools.
90-School RCT StudyHorner et al., in press
• Schools that receive technical assistance from typical support personnel implement SWPBS with fidelity
• Fidelity SWPBS is associated with▫ Low levels of ODR
▫ .29/100/day v. national mean .34▫ Improved perception of safety of the school
▫ reduced risk factor▫ Increased proportion of 3rd graders who meet state
reading standard.
RCT Project TargetBradshaw & Leaf, in press
PBIS (21 v. 16) schools reached & sustained high fidelity
PBIS increased all aspects of organizational health
Positive effects/trends for student outcomes Fewer ODRs (majors + minors) Fewer ODRs for truancy Fewer suspensions Increasing trend in % of students scoring in
advanced & proficient range of state achievement test
Collaborative Problem Solving
Visit the website http://www.livesinthebalance.org/
What do I notice? A canoe? Advertising products for purchase Lots of testimonials Little bit of research (10 citations listed under
research) let’s take a closer, evaluative look at the research
Evaluating a Research Study
AbstractIntroduction & Literature ReviewResearch QuestionsMethod & Design
Subjects & Settings / Measures/ ProceduresResultsDiscussion & ConclusionsReferences
Results QuestionsWho were subjects?
How much like my participants?Where was study conducted?
How much like where I work?What measures were used?
Do I have similar data?What outcomes were achieved?
Are expected outcomes similar
Collaborative Problem Solvingdata from CPS website on 6/18/12
Johnson, M., Ostlund, S., Fransson, G., Landgren, M., Nasic, S., Kadesjo, B., Gillberg, C., and Fernell, E. (2012). Attention-Deficit/Hyperactivity Disorder (ADHD) with Oppositional Defiant Disorder (ODD) in Swedish Children: An Open Study of Collaborative Problem Solving. Acta Paediactrica, in press.
Ollendick, T. H. (2011). Invited Address: Effective Psychosocial Treatments for Emotional and Behavioral Disorders in Youth. University of Stockholm, Sweden..
Fraire, M., McWhinney, E., & Ollendick, T. (2011). The effect of comorbidity on treatment outcome in an ODD sample. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies, Reykjavik, Iceland.
Halldorsdottir, T., Austin, K. & Ollendick, T. (2011). Comorbid ADHD in children with ODD or specific phobia: Implications for evidence-based treatments. In T. Ollendick (Chair), Comorbidities in children and adolescents: Implications for evidence-based treatment. Symposia presented at the 41st European Association for Behavioral and Cognitive Therapies, Reykjavik, Iceland.
Epstein, T., & Saltzman-Benaiah, J. (2010). Parenting children with disruptive behaviors: Evaluation of a Collaborative Problem Solving pilot program. Journal of Clinical Psychology Practice, 27-40.
Martin, A., Krieg, H., Esposito, F., Stubbe, D., & Cardona, L. (2008). Reduction of restraint and seclusion through Collaborative Problem Solving: A five-year, prospective inpatient study. Psychiatric Services, 59(12), 1406-1412.
Greene, R.W., Ablon, S.A., & Martin, A. (2006). Innovations: Child Psychiatry: Use of Collaborative Problem Solving to reduce seclusion and restraint in child and adolescent inpatient units. Psychiatric Services, 57(5), 610-616.
Greene, R.W., Ablon, J.S., Monuteaux, M., Goring, J., Henin, A., Raezer, L., Edwards, G., & Markey, J., & Rabbitt, S. (2004). Effectiveness of Collaborative Problem Solving in affectively dysregulated youth with oppositional defiant disorder: Initial findings. Journal of Consulting and Clinical Psychology, 72, 1157-1164.
Greene, R.W., Biederman, J., Zerwas, S., Monuteaux, M., Goring, J., Faraone, S.V. (2002). Psychiatric comorbidity, family dysfunction, and social impairment in referred youth with oppositional defiant disorder. American Journal of Psychiatry, 159, 1214-1224.
Greene, R. W., Beszterczey, S. K., Katzenstein T., Park, K., & Goring, J. (2002). Are students with ADHD more stressful to teach? Patterns of teacher stress in an elementary school sample. Journal of Emotional and Behavioral Disorders, 10, 27-37.
These studies to not evaluate
effectiveness of CPS
NO studies have been conducted in school settings; all research is either
with parents or in-patient clinical settings
Evaluating the Research studies
2004; Greene et al
2006; Greene , Ablon, Martin
2008; Martin et al
2010 – Epstein & Saltzmann
SubjectsAge & #
47 kids w ODD4-12 yrs. old 3-14 yrs. old School-age
12 Kids w ODD
Under 12 yrs.
Settings Outpatient MH clinic @ hospital
Inpatient Psyc unit (13 beds)
Inpatient Psyc unit (15 beds)
Outpatient clinic
Procedure
Compare CPS w parent training (PT) group
Trained unit staff (pre/post)
Trained unit staff (pre/post)
Group CPS parent training (pre/post)
Outcome measure
ODDRS (unpublished rating scale created by Greene; Improvement ratings (maternal & therapist)
Restraints & seclusion
Restraints & Seclusions
Eyberg Child Beh. Inv. Parent Stress Index
Outcome Improved slightly more than PT
Reduced Reduced Improvement pre to post
What does the research tell us?
So what do we know?
Based on 4 evaluation studies All include children ages 12 or less (2008 study does not
specify an age range; simply ‘school age’) 2 are in inpatient psychiatric hospitals 1 is an outpatient mental health clinic 1 is a parent training program # in school settings = 0
The research tells us nothing about the efficacy of CPS in school settings
What does the research tell us?
Outcome measures ODD Rating Scale (unpublished assessment created by the
author) & improvement ratings from parent & therapist Similar scores to parent training
Reductions in restraint & seclusion (Pre/Post) Is this due to student behavior change or adult behavior
change?
Eyberg CBI & Parent Stress Index (Pre/Post)
No studies directly measure changes in student behavior
Concerns
Only 4 research studies evaluating CPS in 4 years 2 on parent training (1 individual training & 1 group training) 2 in inpatient psyc facilities
Make sure research you are looking at takes place in settings that match your application E.g. school settings v. treatment centers
2 of 4 studies have been conducted by the author of the program Concern if authors are benefiting financially from sale of the
program
Research on CPS in Schools!
Schaubman, A., Stetson, E., & Plog, A. (2011). Reducing Teacher Stress by Implementing Collaborative Problem Solving in a School Setting. School Social Work Journal, 35(2), 72-93.
Abstract Student behavior affects teacher stress levels and the student-teacher
relationship. In this pilot study, teachers were trained in Collaborative Problem Solving (CPS), a cognitive-behavioral model that explains challenging behavior as the result of underlying deficits in the areas of flexibility/adaptability, frustration tolerance, and problem solving. It was hypothesized that teacher stress would be reduced when teachers' understanding of the underlying causes of student behavior shifted to a framework of skills development, and they began using a proactive, positive approach to misbehavior (CPS), with the support of mental health consultation. Results showed a significant decrease in teacher stress, as measured by self-report. Further, discipline referrals were significantly reduced. Limitations of the study and implications for school mental health consultation are also discussed. (Contains 4 figures.)