toward a technology of treatment integrity ronnie detrich wing institute apbs, denver, colorado 2011
TRANSCRIPT
![Page 1: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/1.jpg)
Toward a Technology of Treatment Integrity
Ronnie DetrichWing Institute
APBS, Denver, Colorado2011
![Page 2: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/2.jpg)
Goals for Today
• Review the link between data-based decision making and treatment integrity.
• Discuss the dimensions of treatment integrity.
• Review methods for assuring treatment integrity.
![Page 3: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/3.jpg)
Data-based Decision Making and Treatment Integrity
• Data-based decision making at heart of PBS and RtI.• The impact of PBS and RtI depends on the
effectiveness of specific interventions.• The effectiveness of interventions is a function of the
integrity with which they are implemented.• The quality of decisions about effects of an
intervention is directly linked to the quality of implementation.
![Page 4: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/4.jpg)
Data-based Decision Making and Treatment Integrity
• Student data provides feedback about progress.• If we know about adequacy of treatment integrity
then can make decisions: Adequacy of intervention Adequacy of implementation
If implementation is inadequate then focus should be on improving implementation.
If implementation is adequate then focus should be on changing intervention so student can succeed.
![Page 5: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/5.jpg)
Grade Level Standard
Aim Line
Trend Line
Data-based Decision Making and Treatment Integrity
![Page 6: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/6.jpg)
Positive NegativeH
igh
Low
Continue Intervention Change Intervention
Unknown reason Unknown reason• Intervention problem?
• Implementation problem?
• Other life changes?
• Unknown intervention?
• Intervention is effective?
OutcomeIn
tegr
ityPositive Negative
Hig
hLo
w
![Page 7: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/7.jpg)
Assumptions about Treatment Integrity
• Students cannot benefit from interventions they do not experience.
• Unless educators know what they are supposed to be doing they cannot do more of it.
![Page 8: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/8.jpg)
Dimensions of Treatment Integrity(Dane & Schneider,
1998)
• Exposure (Dosage): the extent to which participants are exposed to the intervention as prescribed. Curricula usually prescribe frequency and duration of
exposure that is necessary for benefit. Ex: 3/week for 30 minutes/session.
Failing to satisfy either can impact student benefit.Ex: 1/week for 30 minutes.
![Page 9: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/9.jpg)
Dimensions of Treatment Integrity(Dane & Schneider,
1998)
• Adherence: the extent to which the components of an intervention are delivered as prescribed. Most commonly measured dimension. It is necessary but not sufficient to produce benefits.
Adherence with low dosage not likely to produce positive outcomes.
![Page 10: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/10.jpg)
Dimensions of Treatment Integrity(Dane & Schneider,
1998)
• Quality of delivery: qualitative measure of how well the intervention is implemented. Importance has been acknowledged for years.
Have not developed good measures or how to influence it. Possible measures through social validity methods:
Enthusiasm Sincerity Variations in inflection and content of speech.
![Page 11: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/11.jpg)
Variables that Influence Treatment Integrity
• Organizational Systems that are perceived by teachers to be supportive
and provide strong leadership have higher levels of integrity.
• Personal characteristics Burnout: Inverse relation between measures of teacher
burnout and treatment integrity. Personal efficacy: Positive correlation between measures
of personal efficacy and treatment integrity.
![Page 12: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/12.jpg)
Variables that Influence Treatment Integrity
• Characteristics of Intervention Complexity
As complexity increases integrity decreases.
![Page 13: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/13.jpg)
Relationship between Complexity, Precision, and Treatment Integrity
• Be as precise as necessary but no more.co
mpl
exity
precision
Catch’em being good
Check inCheck out
Individualized support plan
Integrity
![Page 14: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/14.jpg)
Variables that Influence Treatment Integrity
• Characteristics of Intervention Perceived effectiveness. Acceptability. Contextual fit. Necessary resources available.
![Page 15: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/15.jpg)
Variables that Influence Treatment Integrity
• Training Training is necessary but not sufficient. Not all training is equal.
![Page 16: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/16.jpg)
OUTCOMES
(% of Participants who demonstrate knowledge, demonstrate new skills in a training setting,
and use new skills in the classroom)
TRAININGCOMPONENTS
Knowledge SkillDemonstration
Use in the Classroom
Theory and Discussion
10%
5% 0%
..+Demonstration in Training 30% 20% 0%
…+ Practice & Feedback in Training
60% 60% 5%
…+ Coaching in Classroom 95% 95% 95%
Joyce and Showers, 2002
Not All Training is Equal
![Page 17: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/17.jpg)
What We Know About Treatment Integrity
• It is estimated that drug prevention programs are implemented with integrity only 19% of the time. (Hallfors & Godette, 2002) Programs have extensive research base as being effective. This may be a generous estimate.
• No reason to assume that other programs are immune to poor implementation.
![Page 18: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/18.jpg)
What We Know About Treatment Integrity
• Most of what we know has been developed at the level of individual student support plans.
• SET is a measure of treatment integrity at the school level. Does not address what is happening at the level of the
individual classrooms or individual students.
![Page 19: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/19.jpg)
What Do We Know About Treatment Integrity and Student Behavior?
• Different levels of integrity result in different levels of student behavior.
(Wilder, Atwell, & Wine, 2006) • High integrity followed by declines in integrity has
limited disruptive effect on student behavior. (Northup, Fisher, Kahng, Harrel, & Kurtz, 1997)
• Low levels of integrity followed by increases in integrity do not produce the same level of student response as when integrity high from the beginning.
(Groskreutz,, Groskreutz, & Higbee, 2011)
![Page 20: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/20.jpg)
What Do We Know About Treatment Integrity and Student Behavior?
• Implications Make sure that integrity is high at the beginning of
intervention. It is better to start with high levels of integrity and let it decline
than to start with low integrity and try to increase it. Maximizes impact of intervention.
![Page 21: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/21.jpg)
What Do We Know About Integrity of Interventions at the Universal Level?
• Kovaleski, Gickling, Morrow, & Swank (1999)Evaluated high vs low implementation of Instructional Support Teams
(IST). School-wide organizational change. Students benefited from IST processes only when implemented with high
fidelity. Implementing with low fidelity resulted in no better outcomes for
students than control group not exposed to IST processes. Having structures in place was not sufficient to assure high fidelity. Fidelity assessed one time per year.
![Page 22: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/22.jpg)
What Do We Know About Integrity of Interventions at the Universal Level?
• Horner (2005) Effect of high fidelity vs low fidelity on office discipline
referrals. Schools that implemented with high fidelity had 25% fewer
office referrals for major rule violations than schools that did not meet fidelity criterion.
Fidelity measures taken 2 times per year.
![Page 23: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/23.jpg)
How Do We Assure High Levels of Integrity?
• Technology for assuring treatment integrity is emerging.
• Most of the research is at level of individual support plans.
• Feedback is most common approach.
![Page 24: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/24.jpg)
Mortenson & Witt, 1998
![Page 25: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/25.jpg)
Effective Performance Feedback
• More frequent the feedback better effects (Jones, Wickstrom, & Friman, 1997Mortensen & Witt, 1998).
• Daily better than weekly.• Immediate better than delayed.• Immediate more preferred than delayed.
![Page 26: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/26.jpg)
Limitations of Performance Feedback
• Requires direct observation.• May be too resource intensive to implement at large
scale. Would require significant restructuring to implement
effectively.
![Page 27: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/27.jpg)
Alternative Approaches: Teacher Self Report
• Teacher rate their own implementation. Teacher ratings tend to overestimate accuracy of
implementation ( Wickstrom, Jones, Lafleaur, & Witt). Teachers rated integrity at 54% accurate. Direct observation= 4%.
Integrity may be increased by rating immediately following intervention session.
Integrity may be improved by having teacher score video tape.
It may be possible to teach teachers to rate more accurately (see self-evaluation literature).
![Page 28: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/28.jpg)
Alternative Approaches: Quizzes
• Quizzes (Detrich et al., 2001)
Staff quizzed weekly on elements of multi-component individualized behavior support plans.Given feedback on quiz but no feedback on actual implementation
of support plan.4 versions of the quiz. One question per element of the plan
(student preferences, antecedent interventions, teaching replacement behavior, responding to misbehavior).
![Page 29: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/29.jpg)
![Page 30: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/30.jpg)
Alternative Approaches: Quizzes
• Advantages: Easier to implement than direct observation. May be easy to implement at scale when standard
protocols are utilized.
• Limitations Does not produce maintenance effects in the absence of
quizzes.
![Page 31: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/31.jpg)
Alternative Approaches: Video Modeling
• Has been utilized to train a standard protocol as well as individualized interventions. Problem solving curriculum. Discrete trial instruction. Functional assessment.
![Page 32: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/32.jpg)
Alternative Approaches: Video Modeling
• Advantages: For standard protocols can be used to train many
implementers. Demonstration of implementation can be standardized. Flexible scheduling of training.
• Limitations Has not been evaluated at large scale. Coaching resources required to assure modeling was
effective.
![Page 33: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/33.jpg)
Alternative Approaches: Checklists
![Page 34: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/34.jpg)
Alternative Approaches: Checklists
• Advantages: Standardize implementation. Increase integrity especially adherence.
• Limitations: Upfront effort to develop. Often low acceptability.
![Page 35: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/35.jpg)
Alternative Approaches: Intensive Professional Development
• University faculty/coach assigned to school for full year (Klingner, Vaughn, Hughes, & Arguelles, 1999). Provided in-depth training on three reading programs. Classroom based coaching. Problem solving implementation. Produced moderate levels of treatment integrity at three
years follow-up.
![Page 36: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/36.jpg)
Alternative Approaches: Intensive Professional Development
• Advantages: Highly trained resource on site.
• Limitations Difficult to replicate at large scale. Expensive.
![Page 37: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/37.jpg)
Unresolved Issues
• Adoption or adaptation?Must we implement exactly as prescribed or can we
adjust to fit local circumstances?
• Research suggests that programs are almost always adapted.
• Presumably adapted to improve outcomes. Some adaptation for other reasons:
Better fit teaching style. Do not like some elements of program.
![Page 38: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/38.jpg)
Unresolved Issues
• What does this mean for treatment integrity? Are adaptations systematic?
If so, then we can assess integrity. What parts of program can we adapt without doing
harm to effectiveness? Core elements?
If adapted is program still research-based? If teachers allowed to adapt then program more
acceptable. Teacher’s made better adaptation of reading programs if they
were well grounded in principles. (Klingner, Vaughn, Hughes, & Arguellas, 1999).
![Page 39: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/39.jpg)
Treatment Integrity and PBS
• Regardless of the level of the intervention, it is necessary to know that it was implemented with integrity. High integrity is necessary in a data based decision
making approach.
• Integrity should be assessed at the same level that the intervention is being evaluated.
![Page 40: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/40.jpg)
Treatment Integrity and PBS
• A program or intervention is a set of protocols that guides behavior of the adult. If protocols are not followed then by definition the
program has not been implemented or sustained.
• PBS is an excellent model for making decisions about when, where, and how to intervene. Intervention without process for assuring integrity is likely
to result in wasted effort.
![Page 41: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/41.jpg)
Taking Treatment Integrity Measurement to Scale: Data-based Decision Making
• Insufficient resources to frequently observe all educators working with students to determine adequacy of implementation.
• One solution: Measure student behavior and analyze at different units of analysis:SchoolClassroomIndividual student
• Any level there is gap between obtained and expected performance assess levels of treatment integrity.
![Page 42: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/42.jpg)
Taking Treatment Integrity Measurement to Scale: Data-based Decision Making
• Requires that data are routinely collected across all levels. RtI:
CBM on all students 3/year. PBS
Continuous measurement of Office Discipline Referrals.
• Systems have to be in place to assure integrity data-based decision making process.
![Page 43: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/43.jpg)
Where are We?
• Implementation is where good interventions go to die unless there is active plan to assess and influence integrity.
• Research-based approaches to influencing treatment integrity are emerging.
• We have ethical obligation to allocate resources to influencing treatment integrity.
![Page 44: Toward a Technology of Treatment Integrity Ronnie Detrich Wing Institute APBS, Denver, Colorado 2011](https://reader030.vdocuments.site/reader030/viewer/2022032708/56649e855503460f94b87741/html5/thumbnails/44.jpg)
Thank you
Copies may be downloaded atwinginstitute.org