treatment integrity dadd cec

Treatment Integrity: What it is and How to do it

Presented by:Karen Umstead-Yosmanovich, M.Ed, B.C.B.A, LBS

Beautiful Minds of Princeton“Teach, Reach, & Expand Potential”

www.beautifulmindsofprinceton.com [email protected]

1

http://www.beautifulmindsofprinceton.com/

Who am I?Who am I ?

– Teacher (Special/General Education) in private and public schools

– Board Certified Behavior Analyst (BCBA)– Executive Director of BMOP– Adjunct Professor

• Populations worked with– Schools– Agencies– Community Providers– Families

2

karen


For more information:Call: 1-800-675-2709

Email: kyosmanovich@ beautifulmindsofprinceton.com

or Visit us: www.beautifulmindsofprinceton.com

3

Who are you• BCBA• University/Researcher• Teacher• Child Study• Related Services• Administrator• Parent

4

Warm Up ActivityWhat do you know: (thumbs up, in middle, down)Definition of treatment integrityMulti-dimensional approach to treatment integrity Barriers to treatment integrityVariables or interventions that increase treatment integrityWhat do you want to know (why are you here)More info on defining integrityDealing with barriersSpecific variables or interventions to increase integrity

5

Our Objectives

• Participants will be able to: Label and define multidimensional components of

treatment integrity, including related variables Identify at least four methods of assessing

treatment integrity Label at least 3 solutions to common challenges to

high treatment integrity in school and home settings

6

Proposed definition

• Multidimensional construct (from several fields)• Not as simple as “Did they do it?”• Not just percentage of components implemented as planned

• Content related (quantity)– “How much of the intervention was implemented?”• Process related (quality)– “How well was the intervention implemented?”

7

Process-related

8

Creating a Treatment Integrity Tool

• Define treatment integrity • Pick an appropriate assessment method • Delineate the intervention into operational

steps • Decide how intervention steps will be rated

and the measure will be summed • Determine how often treatment integrity will

be assessed

9

(Collier-Meek, Fallon, Sanetti, & Maggin, 2013; Gresham, 1989; Gresham, 2013; Sanetti, Fallon, & Collier-Meek, 2011; Sheridan, Swanger-Gagne, Welch, Kwon, & Garbacz, 2009)

From PRIME: Planning Realistic Implementation and Maintenance by Educators

Content-Related

• Adherence– This is the most common one we are used to– Requires operational definitions of the intervention

steps• Exposure

– Is it being delivered as often as it supposed to?– For the length of the time it is supposed to happen?

• Another view or key term in research literature– Errors of Omission (not doing something)

10

Content-Related• Program Differentiation

– Just like we consider it when doing research– Especially important when comparing two approaches or

programs or when doing a component analysis of a multi-component plan

– Often times implementers start to incorporate other elements into existing treatment

– Need to understand the “MUST do” versus “Would LIKE to do”

• Another view or key term in research literature– Errors of Commission (doing something wrong)

11

12From PRIME: Planning Realistic Implementation and Maintenance by Educators

ExampleNET (natural environment teaching)- Procedures for Early, Intermediate, & Advanced Students

Staff:____________________ Student:____________________ Observer:____________________ Date:____________________ Section 1: Setting Section 4: Behaviors ____Area is sanitized ____Staff uses appropriate antecedent strategies ____Materials needed are organized and ready ____Staff takes accurate behavior data ____Variety of reinforcers are available ____Staff implements behavior interventions correctly ____NET lesson plan/targets are updated and visible ____Staff pairs social reinforcement with tangible reinforcers Section 2: NET Teaching Procedures Section 5: Data Collection ____Staff follows student’s MO ____Begins session with manding ____Any necessary mand data sheets are updated & ready ____Appropriate level of enthusiasm ____Mand data is taken ____Staff’s voice is natural ____ Behavior data sheets are updated & ready ____Staff provides adequate number of manding trials ____Behavior data is taken ____Staff fades prompts as quickly as possible ____NET lesson plans/target data sheets are updated & ____Staff follows appropriate variable ratio of reinforcement (VR) schedule ready ____Number of demands is faded in ____Data on targets is taken ____Difficulty of demands is faded in ____Staff generalizes skills already taught at IT across: ____Environments ____People ____Stimuli ____Staff mixes verbal operants ____Staff uses errorless teaching with appropriate time delay prompts: ____0 second time delay for new targets ____2-3 second time delay for acquisition or maintenance targets ____Staff uses transfer trials (re-present Sd after error and prompts) ____Staff differentially reinforces novel behaviors/appropriate play skills ____Skills are taught to fluency ____Extinction is used for problem behavior Rating: 1= Yes/Consistently 2= Sometimes/Inconsistent 3= No/Not Occurring 4= N/A Not Applicable 13

From PATTAN Resource Guide for Autism Initiative

14

Natural Environment Teaching Evaluation form Teacher: ______________________________ Date: ___________________________ Observer: _____________________________ Activity: _________________________

Area 1: Organization Area 4: Reinforcement ____ Instructional area is neat and clean ____ Sr+ Reinforcer competes with Sr-/SrA+ ____ All materials needed are organized and ready ____ Pairs social reinforcement w/ tangible items ____ Begins promptly/avoids wasted time ____ Lesson plan current and visible Area 5: Behavior Management ____ Correctly implements extinction procedures Area 2: Instructional Delivery ____ Maintains composure during procedures ____ Follows EO of student ____ Accurately records behavior data ____ Begins NET session with manding ____ Implements effective antecedent interventions ____ Positioned at child’s eye level ____ Appropriate level of enthusiasm Area 6: Error Correction ____ Mixes verbal operants ____ Re-presents after error w/ Sd and 0 sec time delay ____ Uses errorless teaching with appropriate time delay ____ Returns to target several trials later ____Averages 4-5 responses per minute ____ Requires correct response ____Spends recommended time per day in NET ____ Uses non-verbal stimulus to evoke responses Area 3: Data Collection Additional Comments: __________________________ ____ NET data sheet is available and data is taken ____ Data is graphed by verbal operant ______________________________________________ ______________________________________________ Rating: 1 = consistently 2 = Sometimes/inconsistently 3 = Not occurring N/A = Not applicable Responses Across Verbal Operants: 3 Minute Sample Mands Tacts Receptive Intraverbals Motor Imitation Echoics

Responses Per Minute: 1 Minute Timing 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 18 20

From PATTAN Resource Guide for Autism Initiative

Process-Related

• Harder to operationally define (should still attempt to though)

• Qualitative • Quality of delivery

– How well was the treatment delivered– Can tie into some of the content-related elements– Often times a rating scale

• Participants responsiveness15

16From PRIME: Planning Realistic Implementation and Maintenance by Educators

Participant Responsiveness

• Implementer and intended receipt of intervention• Enlisting help to ensure integrity

– Fruit helper of the day (study on increasing fruit)• Implementers wanted to quit, were able to identify solution by

incorporating kids

• Treatment acceptability research has not shown a functional relationship between acceptability of intervention and integrity of implementation

• Responsiveness relates engagement in the program/intervention vs. acceptability

17

Example: Usage Rating Profile

18Chafouleas, S.M., Briesch, A.M., Neugebauer, S. R., & Riley-Tillman, T. C. (2011). Usage Rating Profile – Intervention (Revised). Storrs, CT: University of Connecticut.

Example: Usage Rating Profile

• Factor I: ACCEPTABILITY Items - 1, 7, 9*, 11, 12, 18, 21, 22, 23

• Factor II: UNDERSTANDING Items – 4, 6, 25 • Factor III: HOME SCHOOL COLLABORATION Items – 5,

15, 28 • Factor IV: FEASIBILITY Items – 3, 8, 13, 17, 19*, 27 • Factor V: SYSTEM CLIMATE Items – 10, 14, 16, 20, 26 • Factor VI: SYSTEM SUPPORT Items – 2, 24, 29

19Chafouleas, S.M., Briesch, A.M., Neugebauer, S. R., & Riley-Tillman, T. C. (2011). Usage Rating Profile – Intervention (Revised). Storrs, CT: University of Connecticut.

Intervention characteristicsConsider for implementation, include… •Perception of intervention benefit, as compared to current practice •Intervention complexity •Time or duration required •Materials and resources required •Number of interventionists required •Compatibility •Rate of behavior change •Extent to which the intervention can be adapted

20

Sanetti & Kratochwill (2009)

From PRIME: Planning Realistic Implementation and Maintenance by Educators

Please do the following

• Draw a 2 dimensional picture that represents a plane with a parabola and include a line indicating the vertex

• In a vociferous manner, avow \hu̇-ˈrā\ • Extend your adductor pollicis while

simultaneously lifting your flexor digitorum superficialis above your ossicles

21

Interventionist Characteristics

• Comprehension– does the implementer actually know what the plan is saying?

• Feasibility – can the implementer actually do the components of the plan?

• Acceptability – will the plan itself and related student behavior be considered appropriate to the implementer and others in the setting?

23

Readability and Comprehension

• Ease of understanding or comprehension • National Average is 8th grade reading level

– Know your audience and it’s background• SMOG formula (McLaughlin 1969)• Readability Assessment Instrument (RAIN)

(Singh, 2003)

24Singh et al 2009

RAIN Suggestions

• Text must have titles and subtitles with highlighting

• Introductory or summary statement;• Pronoun references and connectives (e.g.

conjunctions) must be clear• New words highlighted and accompanied by a

definition or synonym• Use 12 – 14 pt print size

25Singh et al 2009

Suggestions• Can check readability with Microsoft Word• Under Options for Spelling/Grammar

– Check readability– Run spelling and grammar check– At the end it will give you a reading level and ease of reading

• Get feedback from people who will implement– Change wordings if need be or clarify

• If you must use technical jargon, pair it with easy to understand terms or definitions

26

Don’t forget to consider culture

• Your OWN – Ex: Your priorities versus others (the case of eating

with forks and knives versus hands)• Organizational

– Ex: We can’t allow that here• Implementers

– Ex: What does agreement looking (e.g. nodding head does not indicate agreement in some cultures, but means I heard you)

27

Collecting Data

• Several ways to do this:– Direct observation– Permanent Product– Self-monitoring, self-reporting, behavioral

interviews

There are Pros and Cons to each approach!

28

Direct observation

Pros Cons

•See exactly what is being done•Most accurate representation•Can account for program differentiation issues or external variables

•Reactivity•Time consuming•May be cost prohibitive (funding source)

29

Permanent ProductPros Cons

•Less chance of reactivity•More efficient•Potentially more accurate than other assessments

•You don’t see what is going on•Not all interventions result in a permanent product

30Wilkinson 2007

Self-monitoring, self-reporting, and behavioral interviews

Pros Cons•Not as time consuming for the evaluator•Treatment implementers take ownership for their actions

•Honesty?•Don’t actually see what is happening

31

Self Monitoring Example

32

Self Monitoring Example

33

Treatment Integrity Tools

• Treatment Monitoring Interview– Semi-structured interview meant to increase the

strength of a plan by providing direct support and performance feedback.

• Treatment Integrity Planning Protocol– Standardized process for planning and creating a

treatment integrity assessment

34

True or False?

• Better treatment integrity results in better outcomes.

35

True or False?

• Better treatment integrity results in better outcomes.– It depends!– There are other factors involved in determining

outcomes other than treatment integrity.– How much treatment integrity is needed depends

on the intervention itself – some interventions require more stringent treatment integrity than others.

36

True or False?

• Treatment integrity must remain high at all times for an intervention to be successful.

37

True or False?

• Treatment integrity must remain high at all times for an intervention to be successful.– This would be nice, BUT not necessarily true.

• Treatment drift happens.– Don’t think of treatment integrity as an all or

nothing.• We can view it as a percentage (25%, 50%, 100%)

38

Plan from the start • When you design the plan/intervention,

consider how you can easily adapt materials to check for treatment integrity

• Determine what aspects of integrity you want to check

• Determine a schedule of assessment

39

Determine barriers to treatment

• What are the barriers to treatment• What can be done to address the barrier ahead of time

– Staff says “I don’t have time to do prize box at end of day”» Can an assistant do it? Is there another time/way for it to be

done? Can the student go to another person for prize box (think of level 2/secondary level interventions in PBS)?

• Consider best practices and work from there– This is what best practice would say…now how can we make it

feasible– Start from a positon of strength and then negotiate – CONSIDER THE FUNCTION!

40

Training

• Know your audience• Direct methods

– Modeling, role –playing, rehearsal and feedback• Indirect methods

– Didactic instruction– Written instruction

41

Video clips for training OR feedback

• Online• Youtube• Made by you

– Example- easily add “captions” to video and share with staff on secure drive

• Used a combination of short-video clips (3-5 min) and feedback– Incorporated video-self monitoring, Teacher Performance Rate and Accuracy Scale or

Form (TPRA), and mentoring/feedback from “expert” – Slim, Lina, "Exploring A Staff Training Model For Enhancing Post-Training Procedural Integrity And

Staff Performance Outcomes When Working With Children Diagnosed With ASD" (2015). Seton Hall University Dissertations and Theses (ETDs). Paper 2058.

42

Staff Training • Important to distinguish staff’s ability to perform skill versus

verbally describing skill or enhancing knowledge• EBP consists of performance-based and competency based

strategies– Performance refers to trainer and trainee actions during training– Competency refers to continuing training until mastery criterion is met

• Training is data-based• Essentially being referred to in literature as behavioral skills

training (we can’t cover more specifics, but online handouts include more slides at end of presentation). BST should also be covered in supervision training for BCBAs

43Parson et al 2012

Initial training is just the start

• We cannot train and hope– Several studies showed that post initial training-

high treatment integrity lasted 0-10 days. • How often has a plan worked great initially

and then you hear about problems 2 months later?

• Treatment drift

44

Performance Feedback

• Consider:– What?

• Specify exactly what feedback will be about.– Where?– When?

• Immediately? Daily? Weekly?– How?

• Verbal, written, graphic

• Try an incorporate preferences when possible45

School Challenges • Using negative reinforcement to increase treatment

integrity (teachers avoided consultant meeting by achieving integrity)

• Meeting summaries– Write up any decisions/changes that were agreed upon at

the meeting.– Include action steps ((who will do what by when (and if

need be how))– Delineate roles and responsibilities– Role/Job analysis

46

School Challenges

• Buy-in and accountability– Engage student, various staff members, and parents

as much as possible• REINFORCE!!!! • By engaging others, individuals more hesitant may get

“swept” along • Have the student help with elements of intervention (e.g.

graphing own data = math practice)

• Start with one routine or teacher then expand– Prevent-Teach Reinforce example

47

School Challenges

• Permanent product methods to assist/augment direct observation.

• Write treatment integrity checks into plan. Prepare and include sample forms with your assessment/BIP.

• Using Google Docs to assess other elements

48

School Challenges

• Connect with other resources that can support intervention– Autism Internet Modules– Evidence Based Practice Briefs from National

Professional Development Center on Autism Spectrum Disorders

– PBIS • Internal coaches

– Pyramidal approach to training• Train the Trainer model

49

Scheduling

• Mini-training series (start of day or end of day required time by contract)

• Augment with video-taping and sharing securely– Could even be used by implementer for self-

assessment – Could be used by consultant to model intervention,

including important elements (captioning)

50

Scheduling

• Secure conferencing• Utilize auxiliary resources

– Mentor Teacher– School Psychologist– Case Manager– Administration

51

Home Challenges

• Why do we face treatment integrity issues in the home?– Parents may not agree on what is the best approach

OR – Who has more energy to tough it out?– Disconnect between parents and grandparents

52Gulick (2014)

From Improving Treatment Integrity in Home-based Settings

How do we get better treatment integrity in the home:

– We have to show that our approach will ultimately be more EFFICIENT

– We address the various factors of parental resistance

• Faulty rules – self-generated or found in media– Consider teaching parent behavior that is

necessary to implementing a treatment plan BEFORE having the parents do it.

• Shaping

53Gulick (2014)From Improving Treatment Integrity in the Home-based Setting

Specifically…

• We can collaborate with school to see how they are doing things– Videos– Observation sessions– If there is a home program aspect, have them come

in and demonstrate skills to parents/families

54


For more information:Call: 1-800-675-2709

Email: kyosmanovich@ beautifulmindsofprinceton.com

or Visit us: www.beautifulmindsofprinceton.com

55© BMOP 2010 www.beautifulmindsofprinceton.com

ReferencesAllen, K.D., & Warzak, W.J.(2000). The problem of parental nonadherence in clinical behavior analysis: Effective treatment is not enough. Journal of Applied Behavior Analysis, 33(3), 373-391.Collier-Meek, M. (2013). Increasing parents’ treatment integrity to a homework intervention through conjoint behavioral consultation and performance feedback (Doctoral Dissertation). Retrieved from http://digitalcommons.ucon.edu/dissertationsCourtemanche, A., Sheldon, J., Sherman, J., Schroeder, S., Bell, A., & House, R. (2014). Assessing the effects of a staff training package on the treatment integrity of an intervention for self-injurious behavior. Journal of Developmental and Physical Disabilities, 26(4), 371-389. doi: 10.1007/s10882-014-9372-6.Dart, E. H., Cook, C.R., Collins, T.A., Gresham, F.M., & Chenier, J.S. (2012). Test driving interventions to increase treatment integrity and student outcomes. School Psychology Review, 41(4), 467-481.De Fazio, C.M., Fain, A.C., & Duchaine, E.L. (2011). Using treatment integrity in the classroom to bring research and practice together. Beyond Behavior, 20(1), 45-49.DiGennaro-Reed, F.D., Reed, D.D., Baez, C.N., & Maguire, H. (2011). A parametric analysis of errors of commission during discrete-trial training. Journal of Applied Behavior Analysis, 44(3), 611-615.Fiske, K.E. (2008). Treatment integrity of school-based behavior analytic interventions: A review of the research. Behavior Analysis in Practice, 1(2), 19-25.Fryling, M.J., Wallace, M.D., & Yassine, J.N. (2012). Impact of treatment integrity on intervention effectiveness. Journal of Applied Behavior Analysis, 45(2), 449-453.Gresham, F.M., Gansle, K.A., & Noell, G.H. (1993). Treatment integrity in applied behavior analysis with children. Journal of Applied Behavior Analysis, 26(2), 257-263.

56

ReferencesGulick, R.F (2014). Improving treatment integrity in home-based services [PowerPoint Slides]. Retrieved from http://www.achievemntctr.org/files/abai-presentation-2014-gulick.pdfHagermoser Sanetti, L.M., Fallon, L.M., & Collier-Meek, M.A. (2013). Increasing teacher treatment integrity through performance feedback provided by school personnel. Psychology in the Schools, 50(2), 134-150. doi: 10.1002/pits.21664.Hagermoser Sanettin, L.M., & Kratochwill, T. R. (2008). Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes. International Journal of Behavioral Consultation and Therapy, 4(1), 95-114.Hagermoser Sanetti, L.M., Kratochwill, T. R. (2009). Toward developing a science of treatment integrity: Introduction to the special series. School Psychology Review, 38(4), 445-459.Hagermoser Sanetti, L. M., & Kratochwill, T. R. (2011). An evaluation of the Treatment Integrity Planning Protocol and two schedules of treatment integrity: Impact on implementation and report accuracy. Journal of Educational and Psychological Consultation, 21(4), 284-308. doi: 10.1080/10474412.2011.620927.Kuhn, S.A.C., Lerman, D.C., & Vorndan, C.M. (2003). Pyramidal training fro families of children with problem behavior. Journal of Applied Behavior Analysis, 36(1), 77-88.McIntyre, L.L., Gresham, F.M., DiGennaro, F.D., & Derek, D.R. (2007). Treatment integrity of school-based interventions with children in the Journal of Applied Behavior Analysis. Journal of Applied Behavior Analysis, 40(4). 659-672.

57

ReferencesPage, T.J., Iwata, B.A., & Reid, D.H. (1982). Pyramidal training: A large-scale application with instructional staff. Journal of Applied Behavior Analysis, 15(3), 335-351.Plavnick, J.B., Ferreri, S.J., & Maupin, A.N. (2010). The effects of self- monitoring on the procedural integrity of a behavioral intervention for young children with developmental disabilities. Journal of Applied Behavior Analysis, 43(2), 315-320.Sanetti, L.M.H. (2012). Assessing and promoting high levels of treatment integrity of school-based Interventions [PowerPoint Slides]. Retrieved from: http://www.cber.uconn.edu/up-content/uploads/2012/02/NE-PBIS-2011-10-03-11.pdfSanetti, L.M.H., Collier-Meek, M.A., Byrne, D.C., Byron. J.R., Wisniewsk, A., & Dobey, L. (2014). Research-based strategies to assess and promote educators’ intervention implementation [PowerPoint Slides]. Retrieved from http://www.primeimplementation.com/wp-content/uploads/Promoting-TI-Miniskills_NASP2014.pdfSanetti, L., Williamson, K., Byron, J., Long, A., Kim, J., & Kratochwill, T. (2014). Defining and quantifying treatment integrity: Conceptual and methodological barriers to assessment and interpretation [PowerPoint Slides]. Retrieved from http://www.primeimplementation.com/wp-content/uploads/Measuring-TI_NASP2014.pdfSchulte, A.C., Easton, J.E., & Parker, J. (2009). Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity. School Psychology Review, 38(4), 460-475.

58

ReferencesSolomon, B.G., Klein, S.A., & Politylo, B.C. (2012). The effect of performance feedback on teachers' treatment integrity: A meta-analysis of the single-case literature. School Psychology Review, 41(2), 160-175.St. Peter Pipkin, C., Vollmer, T. R., & Sloman, K. N. (2010). Effects of treatment integrity failures during differential reinforcement of alternative behavior: A translational model. Journal of Applied Behavior Analysis, 43(1), 47-70.Vollmer, T.R., Sloman, K.N., St. Peter Pipkin, C. (2008). Practical implications of data reliability and treatment integrity monitoring. Behavior Analysis in Practice, 1(2), 4-11.

59

Staff Training • Step 1: Describe target skill (instruction)

– Include rationale as to why important– Behaviorally define target skill

• Include performance checklist of necessary staff actions

• Step 2: Succinct Written Description– This is not the same as the formal behavior plan– This is essentially just the basics and a cheat sheet– Can refer back to background documents like FBA and

full BIP– Keep it simple and watch the language used

60Parson et al 2012

Staff Training • Step 3 Demonstrate the skill (model)

– Can use role-play or video model– One person is staff, One is targeted individual– Make sure well-scripted/planned

• Can use cue cards to prompt people to perform certain actions (e.g. make an error, no response, touch only the left side, etc)

– Stop and freeze at certain points to describe what is being done and why

61Parson et al 2012

Staff Training • Step 4 Trainee practices skill (rehearsal)

– Role-play– Practice is critical

• Step 5 Provide performance feedback during practice– Supportive

• Describe EXACTLY what was done correctly – Corrective

• Describe EXACTLY what was incorrect• Providing instruction on how to correct error

62Parson et al 2012

Staff Training • Consider giving feedback at the end rather than in

the middle. Can be a matter of preference though• Step 6 Repeat steps 4 and 5 until mastery• On the job training

– Remember initial acquisition does not mean that skills will be maintained or generalized

– Some “drift” has been noted in as little as 10 days• Consider effectiveness, efficiency, and

acceptability of staff training

63Parson et al 2012