1

• A Description of Performance Management & Data Based Decision Makingwithin a Center Based Program for

• Children with AutismPresented by:

Vincent J. Carbone, Ed.D., BCBAMargaret M. Hagerty, BCABA

Emily J. Sweeney-Kerwin, BCABA

Carbone ClinicValley Cottage, NY

www.carboneclinic.com

NYS Association for Behavior Analysis 17th Annual Conference

Verona, NY

2

Introduction • The purpose of this workshop is to provide a description

of the application of evidence based procedures to manage staff performance and learner instruction within a center based educational program for children with autism.

SETTING

• The setting is a clinic for children with autism in Rockland County, NY.

• The clinic serves children with autism between the ages

of 2 years and 14 years.

3

• The mission of the clinic is to assess learner needs, provide instructional services and then train parents and community based educational providers to implement effective procedures so that the child may transition back to the home school environment.

• The methodology is exclusively behavior analytic with an emphasis upon teaching learner cooperation, reducing problem behavior, and teaching language using B.F. Skinner’s analysis of verbal behavior as a conceptual guide.

• Ten (10) instructors provide 1-1 instructional services. Most of the instructors are board certified behavior analysts.

4

• About 30 children are served on a weekly basis in the center with no child attending for more than 15 hours per week.

• All of the children receive services in other educational settings or their homes during the other instructional hours of the day.

• The setting is divided into two classrooms with five instructors and five children in each classroom.

• A full time board certified behavior analyst supervisor is assigned to each classroom to manage the instructional environment.

5

DirectorEvaluation Services

OutreachServices

AssistantOffice Manager

Supervisor of Education

Assistant to the Director

LeadInstructor

LeadInstructor

Instructor

Carbone Clinic Organizational Chart

Instructor Instructor Instructor Instructor Instructor Instructor Instructor InstructorInstructor

6

PERFORMANCE MANAGEMENT

• The center staff have adopted a radical behavioral approach.

• As a result, performance of staff behavior is analyzed according to the same principles that are implicated in the control of the learner’s behavior.

• Since learner outcomes are directly related to the management of contingencies of reinforcement by the staff in an educational setting, precise management of staff performance may result in the best outcome for learners.

7

• Performance management has been defined by Aubrey and James Daniels (2004) as “ …a technology for creating a workplace that brings out the best in people while generating the highest value for the organization.”

• In an educational setting the “highest value” is the learner’s outcomes.

• In the past 40 years the principles of behavior analysis have been applied to the management of the behavior of persons within business and industry.

8

• This extension of the principles of behavior analysis to the behavior of persons within organizations has been called Organizational Behavior Management (OBM).

• The research published in this area has led us to an understanding of the empirically verified methods and procedures of performance management.

• It is this body of literature that we have relied upon to develop a performance management system to improve the quality of instruction within a center based program for children with autism.

9

EMPIRICAL SUPPPORT FOR PROCEDURES

• The management of staff performance in settings designed to provide education and treatment to persons with developmental disabilities has been the focus of many research studies in the Organizational Behavior Management literature. (see Babcock, Fleming & Oliver, 1998 and Sturmey, 1998, for a review of these studies)

• Reid, Parsons, Green, & Shepis (1991) state “The literature in organizational behavior management (OBM) offers a number of proven procedures for improving the quality of services for persons with developmental disabilities” (p.33)

10

• Organizational Behavior Management has the strongest scientific support for methods that are designed to influence staff performance (Sturmey, 1998).

• Moreover, this field of study offers “a wide range of methods for changing staff behaviors” (Sturmey, 1998).

• Reid and Parsons (1995) suggest that there are seven steps to effective training and management of the quality of staff performance within a human services delivery system.

11

• Sturmey (1998) refers to these steps as “prototypical OBM” methods.

• The seven steps to effective performance management (Reid & Parsons, 1995) are :

• 1. Specify work skills• 2. Provide staff with a checklist description of work skills• 3. Describe the work skills• 4. Model the work skill behaviors.• 5. Observe staff practice the work skills• 6. Provide feedback• 7. Continue process until staff person is competent.

12

• This is the model of performance management that has been implemented in our center based program.

• In addition monthly monetary incentives are provided for exemplary performance as measured by supervisory staff.

13

• The specific steps of our performance management approach include the manipulation of antecedent and consequent variables:

1. Ten (10) days of staff training that explicitly describes the behaviors of instructing children with autism using the methods of applied behavior analysis (ABA). See an outline of the 10 days of training in Appendix A of this document.

STEPS OF THE PERFORMANCE MANAGEMENT SYSTEM

14

2. Measurable competency checklists explicitly describing most of the relevant instructional skills to include, Discrete Trial Instruction, Natural Environment Teaching (NET), teaching self-care skills, etc.

3. Demonstration of the skills

4. Required practice of the skills and measurement of skill competency on the checklists.

5. Unannounced monthly competency checklist skill assessment for the term of employment.

6. Contingency management of staff behavior through verbal feedback immediately following assessment, public posting of performance, monetary incentives for exemplary performance and remedial checks for all skills targeted for improvement.

15

IMPLEMENTATION OF THE PERFORMANCE MANAGEMENT SYSTEM

• Following training and assignment to several learners, monthly assessments of performance are conducted by a supervisor.

• The supervisor watches an instructional session with three to four relevant competency checklists on a clipboard and measures the quality of instruction across a range of instructional settings by recording the occurrence or non-occurrence of the list skills on the checklist.

16

• The competency checklists were developed to measure the accuracy of instructor implementation of various instructional practices and are derived from behavior analytic literature related to the effective treatment of children with autism and other developmental disabilities.

• Competencies were developed to measure instructor performance related to:– Natural environment teaching– Discrete trial teaching– Teaching adaptive living skills– Implementation of behavior reduction protocols

17

• The following slides provide examples of competency checklists developed for Intensive Teaching, Natural Environment Teaching, and Teaching Vocal Manding. Copies of these competency checklists are provided in the handouts.

18

• When competency checklist sessions are complete the supervisor provides verbal feedback to the instructor regarding the quality of instruction.

• While feedback alone can be effective, its effectiveness can be increased by adding other methods such as programmed consequences for performance (Balcazar, Hopkins, & Suarez, 1985; Alvero, Bucklin & Austin, 2001).

• A score of 90 percent on each checklist is required to receive a “passing” score”.

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

• Any areas scored as non-occurrence or of poor quality are listed as targets to be re-checked and remediated within a week.

• In addition, on each competency a number of skills have been identified as “critical” to effective instruction. Therefore failure to emit these “critical” responses or incorrect responses are considered “critical errors” and result in a failing score regardless of performance across other skill areas.

• When the instructor receives a score of 90 percent or greater on all the competencies scored for that session their name is publicly posted in the staff dinning room indicating their exemplary performance (See Norstrom, Lorenzi, & Hall, 1990, for a review of the research on this topic).

35

• For those who do not meet the 90 percent criterion their names are omitted from the list.

• Instructors whose names appear on the list also receive a check for $150.00.

• This check is delivered by the instructor’s supervisor at the next Friday morning staff meeting.

• Congratulatory applause by the entire staff occurs each time a check is distributed.

36

MONETARY INCENTIVE SYSTEMThe monetary incentive system at the center conforms to the OBM research findings (See Bucklin & Dickinson, 2001) that suggests the most effective delivery of money as a consequence includes the following:

• Incentives are based only on the employee’s performance.

• Incentives are based upon clearly specified behaviors.

• Incentives are certain; if the target behaviors occur they will receive the money.

• Incentives are paid as soon as possible following the performance rating.

37

• In addition, the monthly performance scores for each instructor is referred to in the employee’s annual review and contributes largely in determining the size of the annual raise and future promotions.

• Finally, a similar competency checklist system is used monthly to evaluate each child’s program/data book for accuracy. These competencies are referred to as book audits.

• An additional $150.00 amount is awarded for exemplary performance across this measure.

• A copy of the book audit checklist is provided in the supplementary packet.

38

Case Study• As an illustration of our system of performance

management, a case study of one instructor’s performance on competency checklists during her initial 10 day training and on all subsequent monthly competencies through September 2007 will be presented.

• Kristin has Masters in Education Degree from The Pennsylvania State University. She has completed all the coursework necessary for the national board certification in behavior analysis and is currently completing the supervision requirements.

39

• Prior to her employment at the at our clinic, Kristin worked in the field of applied behavior analysis for a number of years but decided to apply for and accept a position at the clinic specifically for the purposes of receiving extensive training and supervision.

• Kristin began her training at the clinic in June 2007. During her initial 10 day training, a variety of competency checklists were completed by a supervisor across instructional areas for the purposes of training and as a measure of her baseline performance.

• Samples of the following completed competency checklists across these areas are presented below.

– Effective Teaching Procedures During Intensive Teaching– Natural Environment Teaching– Teaching Vocal Manding

40

• The examples presented below are representative of the range of possible outcomes on the competency checklist including:

• Passing score of 90% correct or greater without critical errors.• Failing score of 90% correct or greater with critical error.• Failing score below 90% correct.

• One alternative outcome not presented here is a score 100% correct.

• In addition, the samples below provide examples of target remediation.

41

42

43

44

45

46

47

• On the Effective Teaching Procedures competency checklist Kristin’s initially scored 91% correct and one critical error was identified.

• Based on the competency checklist three skill areas were identified as in need of improvement and specific targets for remediation were developed.

• Targets included: – Teacher uses prompts the reliably evoke the response– Varies elements of teaching procedures based on unique

teaching situation– Delivers reinforcer immediately (critical error).

• • Remediation competencies were then conducted approximately

one week later. Kristin remediated all previously identified targets and received a score of 100% correct.

48

• On the Natural Environment Teaching competency Kristin’s initial score was 94% correct and no critical errors were identified.

• Although she received a passing score, one target was identified, which was varies elements of teaching procedures based on unique teaching situation.

• A remediation competency was therefore conducted approximately one week later on which Kristin received a score of 100% correct.

49

• On the Vocal Manding competency Kristin received an initial score of 87% correct.

• Based on the competency checklist two skill areas were identified as not occurring or in need of improvement. Specific targets for remediation were therefore developed.

• Skills targeted for improvement included: – Stimulus control transfer is done properly– Varies elements of teaching procedures based on unique

teaching situation

• Remediation competencies were then conducted approximately one week later. All previously identified targets were remediated and Kristin received a score of 100% correct.

50

• Kristin’s competency scores during the 10 day training period and during subsequent monthly competency sessions are presented in the figure below as a demonstration of her performance over time.

• Notice for the Effective Teaching Competency two symbols are presented. The open circles indicate that at least one critical error occurred during the competency. The closed circles indicate that there were no critical errors on this competency.

51

● = % Correct on Effective Teaching Procedures Competency without critical error

○ = % Correct Effective Teaching Competency Procedures Checklist with critical error

▲ = % Correct Manding Competency Checklist without critical error

■ = % Correct Natural Environment Teaching Competency Checklist without critical error

Figure 1. The percentage of correct responses on Effective Teaching, Manding, and Natural Environment Teaching Competency Checklists during training and unannounced monthly competency sessions for Kristin.

52

• As shown in Figure 1, during the 10 day training period in June Kristin’s score on the Effective Teaching Procedures competency checklist was 87% correct with one of critical error. Her score on the Natural Environment Teaching competency checklist was 87% correct and her score on the Manding Competency checklist was 80% correct.

• In July Kristin’s score on the Effective Teaching Procedures competency improved to 97% correct without any critical errors. Her score on the Natural Environment Teaching also increased slightly to 89% correct and her score on the Manding competency increased to 87% correct. However, since the criterion for passing is a score of 90% correct across all competency checklist conducted she was not eligible for the monthly bonus during this month.

53

• In August she again scored 97% correct on the Effective Teaching Procedures competency but one critical error was identified and her score on the Manding competency checklist decreased slightly to 86% correct. She was again not eligible to receive the monthly bonus. Her score on the Natural Environment Teaching competency checklist did, however, improved to 100% correct and no critical errors were identified.

• In the month of September Kristin received a passing score of 97% correct on the Effective Teaching Procedures competency checklist and no critical errors were identified. She also received passing scores of 94% correct on the Natural Environment Teaching checklist and 93% correct on the Manding checklist. Since Kristin received scores greater than 90% across all competency checklist conducted during the month of September she received a $150 bonus for exemplary performance.

54

• A check for that amount was delivered by her supervisor at a Friday morning staff meeting and her name was added to the publicly posted list of all “passing” instructors.

• In addition, Kristin received passing scores on book audits conducted during the months of August and September.

• During these months she therefore received an additional $150 bonus delivered publicly during a weekly staff meeting.

55

• Although not an experimental investigation, this case study is offered as an example of both the implementation of our performance management system and the effectiveness of this system in improving the instructional practices of staff.

• Since its formal introduction in January 2005 similar results as those presented in the case study have been found across a number of new and existing instructors with whom the performance management system has been used.

• Since January 2005 competency sessions have been conducted 308 times with a total of over 924 individual competency checklists completed. During that time a total of 308 book audits have also been conducted.

• The cumulative number of competency sessions and the cumulative number of book audits conducted from January 2005 to September 2007 are presented in Figure 2 and Figure 3 respectively.

56

Figure 2. The cumulative number of competency sessions conducted per year from January 2005 through September 2007.

Year

Competency Checklists

57

Figure 3. The cumulative number of book audits per year from January 2005 through September 2007.

Year

Book Audits

58

• During the first year of the performance management systems implementation, 2005, 120 competency checklist sessions and book audits were conducted.

• In 2006, 119 competency checklist sessions and book audits were conducted.

• As of September in 2007, 69 competency checklist sessions and books audits have been conducted.

59

• Data collected during competency sessions and books audits have also been summarized across all staff to allow for an overall analysis of staff performance.

• The percentage of monthly competencies and book audits passed across all staff during the years of 2005, 2006, and through September 2007 are presented in Figure 4, 5, and 6 respectively.

60

Figure 4. The percentage of monthly competencies and book audits passed across all instructors during 2005.

Month

2005

61

Figure 5. The percentage of monthly competencies and book audits passed across all instructors during 2006.

Month

2006

62

Figure 6. The percentage of monthly competencies and book audits passed across all instructors from January 2007 to September 2007.

Month

2007

63

• As shown in the figures, a significant percentage of instructors pass both their competencies and book audits each month.

• Analysis of these data in comparison to employment records suggests that months during which the percentages of passing scores are the lowest correspond with the promotion of previous instructors to other positions and the hiring of new staff.

• For example during May, June, and July 2005, four new instructors were hired to replace two lead instructors and two additional instructor who were being promoted to other positions within the organization.

64

• During 2006, however, few new instructors were hired which may account for the higher percentages of passing competencies and book audits. The lowest percentage of passing rates during this year was in March. It was during this month that one new staff member was hired which may account for the slight drop in percentage of passing scores.

• In 2007 the percentage of competencies passed decreased during the months of June and July. It was during these months three new instructors were again hired to replace previous instructors who had received promotions.

• Kristin, from the case study presented above is an example of one of the new instructors hired during this time.

65

• Due to scheduling changes learners are also occasionally assigned to new instructors. Since a number of highly individualized competency checklists have been developed based on the specific needs of some learners, the assignment of new learners to instructors may also account for lower percentages of passing scores during some months.

• In addition, it is the goal of the clinic to provide extensive training and supervision to our staff and to adequately prepare them for a career in behavior analysis. To this end the competency checklists are completed diligently by the supervisor to ensure continued improvement by each instructor for the term of their employment. Only exemplary performances are reinforced.

66

• For those instructors who receive passing scores on either or both the monthly competency checklists and book audits, the performance management system provides a sizable increase in pay from their base salaries.

• In total instructors have the opportunity to receive up to $300 a month or $3600 a year in bonuses.

• As of September 2007 a total of $54,900 has been paid to staff in monthly bonuses since the introduction of the performance management system.

• The cumulative dollar amount paid to staff per month for the years 2005, 2006 and through September 2007 are presented in Figures 7, 8, and 9 respectively.

67

Figure 7. Cumulative dollar amount paid to staff per month for performance management during 2005.

Month

2005

68

Figure 8. Cumulative dollar amount paid to staff per month for performance management during 2006.

Month

2006

69

Figure 9. Cumulative dollar amount paid to staff per month for performance management from January 2007 to September 2007.

Month

2007

70

• During 2005 a total of $19,500 was paid to the staff through the performance management system. This total includes all money awarded to instructors based on exemplary performance on competency checklists and book audits.

• In 2006 a total of $23,250 was paid to staff through the performance management system.

• As of September in 2007 an additional $12,000 has been paid in performance management bonuses.

71

Limitations• Although we have collected some data in support of the effectiveness of

our performance management system, without an experimental investigation of its effect this conclusion is tentative.

• We take the replication of the results presented in the case study above across a large number of additional instructors as at least minimal evidence of the effectiveness of this program.

• In addition the procedures included in our performance management system are supported by a large body of empirical research within the organizational behavior management literature.

72

• An additional limitation is that we do not have outcome data related to changes in learner behavior as a result of improved instructor performance.

• To address this limitation we are currently developing a system that will directly determine the relationship between staff performance and learner outcomes.

• We have, however, developed a data based decision making system to evaluate learner progress and to ensure that our interventions are producing the desired changes in the behavior of the children we serve.

73

DATA BASED DECISION MAKING

• The recognition that behavior is the legitimate subject matter of scientific investigation is a hallmark of the experimental analysis of behavior and applied behavior analysis (Skinner, 1938, Baer, Wolf, & Risley, 1968).

• As a result of this basic assumption, direct and

repeated measures of behavior serve as the data for analyzing the relationship between independent and dependent variables (Skinner, 1938,1953).

74

• The field of behavior analysis has settled on visual display of direct measures of the target response on a line graph in a time series format to derive functional relationships and analyze the effectiveness of various treatments (Baer, et al., 1968; Parsonson & Baer, 1992; Baer, 1977; Michael, 1974).

• Educators and behavior analysts, particularly in the fields of precision teaching (Lindsley, 1971) and curriculum based measurement and decision making (Fuchs, Deno, & Mirkin 1982) have adopted this type of data display as a visual aid to analyzing the effectiveness of instructional practices.

75

• The general practices of educators who have adopted visual display as an aid to measuring the effectiveness of instructional methods have followed these guidelines: (White 1986)

– Identify in an observable measurable form the response to be measured (dependent variable)

– Record the occurrence of the response frequently and repeatedly.

– Display the response measure on a line graph for visual inspection of the relationship between the educational tactic and response of the student.

– Evaluate the results regularly and change educational practices by applying decision rules as a form of data analysis.

76

• Moreover, instructors who measure performance frequently and directly and analyze the trends, variability and levels of responding to inform changes in instructional methods are more effective than those who do not (Fuchs, et al., 1982).

• However, teachers who merely rely on clinical intuition to decide when and how to change instructional practices are not effective.

• • Teachers whose decisions were guided by decision

rules, (e.g. 3 days of limited progress) which led to a change in instructional practices were most effective. (Ysseldyke, Thurlow, Grade, Wesson, Deno, & Algozinne, 1982).

77

Data Based Decision Making for Student Progress

Within our center based program, rules for making data based decisions ensure that instructors analyze the relationship between instructional practices (independent variable) and acquisition of targeted skill areas (dependent variable).

Our data based decision making system includes the following:

• Daily measurements of the targeted response• Graphic displays of data on line graphs• Daily analysis of learner progress based on visual inspection of

graphed data• Standardized rules for changing teaching practices

78

• Based upon the evidence regarding data based decision making, the staff within our clinic for children with autism have developed a program for evaluating student progress frequently and changing instructional practices based upon the data analyzed.

• What follows is a description of that process and some case study data to illustrate the activities.

79

Data Collection Systems for Acquisition of Skills

We use mainly daily probes to measure progress within instructional programs. – Daily Yes/No probes to measure accuracy of responding

and acquisition and mastery of skills. • Recording data on the targeted response on a daily basis

allows instructors to monitor progress of the student frequently.

• Data regarding learner progress indicate the effectiveness of the performance management system in improving instructional practices of staff.

Research Support for Probe Data

80

81

The following describes procedures for data collection and display that are specific to the Carbone Clinic.

• First Trial data collection, as opposed to trial-by-trial data collection, has been selected for the following reasons (Cummings & Carr, 2009; Dollins & Carbone, 2003; Najdowski et al., 2009):– First Trial probe data collection is less time consuming,

which results in more time being spent teaching.– First Trial data collection is equally as efficient (amount of

time to mastery).– First Trial data may be equally as effective at leading to

maintenance of skills over time, although some results suggest slightly lower maintenance over time for skills mastered using first trial probe data.

82

83

84

85Behavior Analysis in Practice, Spring (4), 2011

86

87

Mand Probe Data Recording

• Mand data are recorded on a daily basis using a first trial probe recording system.

• In the next couple of slides contained examples of this process.

• We do not use rules to make data based decisions related to manding.

• Instead we analyze the acquisition of this skill continuously through daily observations and change teaching procedures when acquisition is slow.

88

89

90

Data Recording During Discrete Trial Instruction

91

• Daily Data Recording and Display• Probe trials are conducted at the beginning of the

learner’s session before any teaching of the skill to measure of accuracy learner responding.

• The probe trials are conducted one time per day.• The data from the probe trial are recorded on a probe

data sheet daily. • The data from the probe trials are used to track leaner

progress and are displayed on a line graph weekly.

92

• Retention Probes• A skill will go into retention after a learner has met

the criteria for either the timings or cold probes. The instructor will not teach that particular skill for seven calendar days.

• On the seventh day, the teacher will probe the skill once more and if the child meets the criteria with both rate and accuracy (timings) or accuracy (cold probe) then the target skill is considered “retained” and it is then placed within the child’s maintenance program.

93

Yes/No Probes• Baseline: Conduct three probes for each target.

– Write the target name along the far left of the yes/no probe data sheet.

– Write date across the first three spaces on the probe sheet.– In the space above the date line label this section as “Baseline.”– Below the date line record Y/N as to whether or not target was

answered correctly on each of the three probes.• Write target name and date introduced on skills tracking

sheet.• Two things can happen after baseline:

1. Child meets criteria in baseline.• Baseline Mastery Criterion 3 consecutive Y’s

2. Child does NOT meet criteria.

94

Yes/No Probe: 1) Child meets criteria in baseline

• On Y/N probe sheet draw a RED phase change line after baseline data and write “mastered in baseline.”

• Highlight baseline data in pink.• Write date on skills tracking sheet in both the

acquired and retained columns. • Fill in target on the retention side (back) of

skills tracking sheet and write “mastered in baseline” (MIB) across boxes.

95

96

Yes/No Probe:2) Child does NOT meet criteria

• Target placed into teaching using prompt/transfer/probe procedure; teach 2 times per session.

• Draw a black phase change line on probe sheet after baseline data.• After the black phase change line label the criterion for acquisition

(e.g., 3 consecutive Yes); each day fill in the date and whether the learner scored a Yes or No. Connect data points to show graphic display.

• Once the criterion is met, skill is acquired.– Draw black phase change line on probe sheet and highlight (yellow)

the days on which the acquisition criterion was met.– After the phase change line fill in the date on which the retention

probe will be conducted; in the space above the date line label this RP for retention probe.

– Skill goes into retention.– On the skills tracking sheet fill in the date the skill was acquired.– Fill out back of the skills tracking sheet with retention information

(i.e., target name and date of retention probe).

97

• 1 week later probe target (i.e., 7-day retention probe).– If learner gets “Y” the skill is considered acquired and retained.

• On the probe sheet draw a RED phase change line and write “acquired and retained,” “retained,” or “mastered”; highlight (pink) the date on which the target was retained.

• Fill in the date the target was retained on skills tracking sheet.• On the back of the skills tracking sheet record a Y under the

column for the date on which the retention probe was conducted.• Target is placed into mixed and vary.

– If learner does NOT get “Y” return to teaching and make a program change according to the frame analysis and data based decision making clinic wide protocol.

• On the probe sheet draw a black phase change line.• After the phase change label with the new acquisition criterion

(i.e., 4 consecutive Yes).• Repeat the process described above until the target is retained.

98

99

100

101

Yes/No Probe: Graphing

• Weekly basis• Graph total number of skills retained per week

within each skill/goal.• Graph total number of skills retained per week

for whole operant.

102

103

104

105

References

Cummings, A.R., & Carr, J.E. (2009). Evaluating progress in behavioral programs for children with autism spectrum disorders via continuous and discontinuous measurement. Journal of Applied Behavior Analysis, 42, 57 – 71.

Dollins, P., & Carbone, V.J. (2003, May). Using probe data recording methods to assess learner acquisition of skills. In V.J. Carbone (Chair), Research related to Skinner’s analysis of verbal behavior with children with autism. Symposium conducted at the 29th annual convention of the Association for Behavior Analysis, San Francisco.

Najdowski, A.C., Chilingaryan, V., Bergstrom, R., Granpeesheh, D., Balasanyan, S., Aguilar, B., & Tarbox, J. (2009). Comparison of data collection methods in a behavioral intervention program for children with pervasive developmental disorders: A replication. Journal of Applied Behavior Analysis, 42, 827 – 832.

106

Benefits of Data Based Decision Making

• A large body of research supports the use of data based instructional practices and data based decision making.

• Within our clinical setting, data are collected daily and visually displayed on a line graph.

• The data are used to identify trends within a student's progress and in turn analyze the relationship between instruction and student performance.

• Rules for when to apply a program change are standardized and in turn supervisors can monitor progress of both the students and instructors so that students improve their performance and instructors improve their instructional decision making.

107

Benefits of Data Based Decision Making

• A data based decision making system which includes: daily measurements of the targeted response, visual displays of the data on line graphs, evaluation of the targeted response on a daily basis and standardized rules for changing teaching practices are more effective than those programs that do not.

• Educators and practitioners who are guided by decision rules are more likely to enhance the performance of students even those with the most challenging educational needs.

108

Data-Based Decision Making:Evaluation of the Data

• Daily cold probes, allow the instructor to evaluate and monitor a learner's progress on a frequent basis.

• All data are displayed on hand written graphs which allow instructors to closely examine the data and identify trends of a student's progress and in turn analyze the functional relationship between instruction and student performance.

• Trends in learner performance may show:– Acceleration of skills (increasing)– Deceleration of skills (decreasing)– Stability (no change)– Variability (inconsistent)

• The instructor uses the trend line to make a data based decision.

109

• However, continuous measurement of the data alone is simply not enough to ensure sound data based decision making. Standardized guidelines must be established to dictate when changes in instructional practices are necessary.

• It is only when guidelines to implement a program change are established, combined with graphic display and data evaluation, are improvements in student performance achieved.

110

Applying Decision-Making Rules

• Decision rules have been established within our clinical setting so that instructors will examine the data to identify when changes in instructional practices are necessary.

• The decision rules apply to both the instructors and the supervisors, but the decision making process first begins with the instructor.

111

Frame Analysis• After baseline, the learner has up to 6 consecutive sessions to meet the

acquisition criterion (3 consecutive days of “yes” on the cold probe). These 6 sessions, or data points, are broken up into 2 frames, each frame consisting of 3 data points. (***The frame size changes based upon the number of days required for acquisition.)

• Following baseline, an instructor must make a program change according to the following guidelines: Frame 1 – If the learner receives 3 consecutive “no’s,” immediately make a

program change. However, if the learner receives at least 1 yes, then maintain the current procedures.

Frame 2 – If a program change was not made in frame 1, the learner must receive 3 consecutive “yes’s” across frames 1 and 2 to acquire the target. However, recording of the first “no” in frame 2 prompts an immediate program change.

Once an instructor change is made, the frame analysis is re-started (i.e., at frame 1).

• If a second change is needed according to this frame analysis, it is made by a lead instructor.

• If a third change is needed according to this frame analysis, it is made by a supervisor.

112

113

114

General Guidelines to Implement an Instructor Program Change

• The instructor fills out a program change form and writes a brief description as to why an instructional change is necessary. The best description of the reason for the program change usually includes information such as: the student does not attend to the stimulus, scan the field, or makes frequent discrimination errors.

• The instructor then chooses a change in instruction that is listed on the form or changes some element of the teaching through an analysis of the unique learning needs of the student.

• The change in the teaching (independent variable) is indicated on the graph and the probe data sheet.

Antecedent:Stimulus Control/Motivation

CHECK THOSE APPROPRIATE Increase pairing Reduce the number of demands (Decrease VR) Provide higher rate of interspersing mastered

skills with target skills Decrease the response effortFurther reduce errors: Modify prompt procedure

and prompt fade procedureChange the pace of instruction (ITI)Decrease/increase session time Conduct a Sr+ assessment Increase the saliency of Sr+ Change the field of the stimuliIncrease the number of teaching trials Change the physical environment Change the aim Teach pre-requites skillsDecrease the number of goals/objectives Build MO by deprivation of specific reinforcersChange the teaching procedureOther:

Consequence: Reinforcement/Extinction/Punishment

CHECK THOSE APPROPRIATE Provide more valuable reinforcer Provide higher rate of reinforcement (lower

VR) Reinforce immediately Provide greater magnitude of reinforcement Reinforce on transfer trials Better use of extinction Improved implementation of Differential

Reinforcement Consider punishment contingency: Check with supervisor Other:

Program Change Procedures

115

116

General Guidelines to Implement an Instructor Program Change

• The instructor fills out a program change form and writes a brief description as to why an instructional change is necessary. The best description of the reason for the program change usually includes information such as: the student does not attend to the stimulus, scan the field, or makes frequent discrimination errors.

• The instructor then chooses a change in instruction that is listed on the form or changes some element of the teaching through an analysis of the unique learning needs of the student.

• The change in the teaching (independent variable) is indicated on the graph and the probe data sheet.

Instructor Program Change Analysis Checklist

117

118

Example of an Instructor Program Change

• The following slides illustrate an instructor program change for yes/no cold probes.

• Following baseline, the learner received three consecutive “no” responses on the cold probe for the tact “grey.”

• The instructor identified that the learner did not discriminate between “white” and “grey.”

• The instructor made a program change to teach “grey” in discrimination with “white” and increase the number of teaching trials to four times.

• The instructor program change was successful in increasing the accuracy of the response which led to acquisition and retention of this skill.

119

1065

121Lead instructor Analysis Checklist

122

123

Measurement of Instructor Performance on Data Based Decision Making

• In addition to data collected on learner performance following instructional changes, the supervisor also collects and graphically displays data regarding the effectiveness of individual instructor program changes.

• These data are used to track the performance of the instructor on their ability to make effective data based decisions per individual learner as well as their overall decisions across a variety of learners and instructional programs.

124

• These data are used by the administration to evaluate how effective the instructors are at making data based decisions.

• This information can suggest something about their ability to change instructional methods based upon the data to improve learner outcomes.

• These data become important for two reasons:– First, it allows the supervisor to identify which instructors

require training on making effective data based decisions. In other words, instructors who have a solid understanding of evidence based literature in teaching children with autism will make effective instructional decisions.

– Second, it becomes useful in annual reviews as a way of evaluating teacher performance related to implementation of instructional practices.

125

Measurement of Instructor Performance

• As an illustration of our data based decision making system, a case study of one instructor’s performance on making instructional changes from January, 2010 through April, 2010 will be presented.

• Brian has Masters in Education Degree and has completed all the coursework and supervision requirements necessary for the national board certification in behavior analysis.

126

• Figure 1 illustrates Brian’s data on the number of instructor program changes made and the percent of instructor program changes that led to retention of skills for three learners (i.e., were successful).

• In January, 2010, Brian made 13 instructor program changes where 69% of those changes led to the Learners’ retention of skills.

• In February, 2010, Brian made 4 instructor program changes of which 100% led to the Learners’ retention of target skills.

• In March, 2010, he made 8 instructor program changes of which 50% led to the Learners’ retention of skills.

• In April, 2010, he made 7 instructor program changes of which 43% led to the Learner’s retention of skills.

Brian Data

127

• Figure 2 illustrates the cumulative number of instructor program changes made and the percentage of those program changes that led to retention of target skills per month for all nine instructors within our center based program.

• In January, 2010, a total of 63 instructor program changes were implemented of which 54% led to retention of skills for the individual learners.

• In February, 2010, a total of 50 instructor program changes were implemented of which 50% led to retention of skills for the individual learners.

• In March, 2010, a total of 72 instructor program changes were implemented of which 42% led to retention of skills for individual learners.

• In April, 2010, a total of 85 instructor program changes were implemented of which 46% lead to the retention skills for individual learners.

Figure 2. The number of instructor program changes made and the percentage of those program changes that were successful (i.e., led to retention of skill) per month for all instructors.128

129

• As shown in figures 2, the percent of successful program changes for all instructor changes decreased from January to March, and then showed a slight increase in April.

• The reduction in percentages of data based decisions from January to March suggest to the Supervisor that individual instructors may need additional supervision within instructional decisions.

• Novel or unfamiliar teaching situations, individual learner skill level, etc. can all be taken into account by the supervisor.

• Information gathered based upon the data, observation of the learner during instructional sessions and interview with the instructor will help the supervisor identify the areas in need of improvement.

• Analysis of these data help the supervisory staff directly determine the effectiveness of the instructor’s data based decisions.

• Although we have collected some data in support of the effectiveness of our data based decision making system, an experimental investigation of its effectiveness on learner outcomes is still needed.

130

ReferencesAlvero, A. M., Bucklin, B. R., & Austin, J. (2001). An objective review of the effectiveness and essential characteristics of

performance feedback in organizational settings (1985-1998). Journal of Organizational Behavior Management, 21, 3-29.

Babcock, R. A., Fleming, R. K., & Oliver, J. R. (1998). OBM and quality improvement systems. Journal of Organizational Behavior Management, 18(2/3), 33-59.

Balcazar, F., Hopkins, B. L., & Suarez, Y. (1985). A critical, objective review of performance feedback. Journal of Organizational Behavior Management, 7 65-89.

Baer, D., Wolf, M., & Risley, T. (1968). Some current dimensions of applied behavioral analysis. Journal of Applied Behavioral Analysis, 1, 91-97.

Baer, D. (1977). Perhaps it would be better not to know every. Journal of Applied Behavior Analysis, 10, 167-172.

Bucklin, B. R. & Dickinson, A. M. (2001). Individual monetary incentives: A review of different types of arrangements between performance and pay. Journal of Organizational Behavior Management, 21, 45-137.

Daniels, A. C., & Daniels, J. E. (2004). Performance management: Changing behavior that drives organizational effectiveness (Rev. 4th ed.). Atlanta, GA: Aubrey Daniels International, Inc.

Fuchs, L.S., Deno, S. L., & Mirkin, P. K. (1982). Effects of frequent curriculm-based measurement and evaluation on student acheivement and knowledge of performance: An experimental study. (Research Report No. 96) November 1982.

Lindsley, O.R. (1971). From Skinner to precision teaching: The child knows best. In J.B. Jordan & L.S. Robbins (Eds.), Let’s try doing something else kind of thing. Arlington, VA: Council for Exceptional Children.

131

• Michael, J. (1974). Statistical inference for individual organism research: Mixed blessing or curse? Journal of Applied Behavior Analysis, 7, 647-653.

• Nordstrom, R. R., Lorenzi, P., Vance, H. R. (1990). A review of public posting of performance feedback in work settings. Journal of Organizational Behavior Management, 11, 101.123.

• Parsonson, B. S. & Baer, D. M. (1992). The visual analysis of data, and current research into the stimuli controlling it. In T. R. Kratochwill & J. R. Levin (Ed.) Single-case research design and analysis: New directions for psychology and education. Hillsdale, N.J. : Lawrence Erlbaum Associates

• Reid, D.H., Parsons, M.B., Green, C.W., & Shepis, M.M. (1991). Evaluation of components of residential treatment by Medicaid ICF-MR survey: A validity assessment. Journal of Applied Behavior Analysis, 24, 293-304.

• Reid, D.H. & Parsons, M.B. (1995). Motivating Human Services Staff. Supervisory Strategies for Maximizing Work Effort and Work Enjoyment. Morganton, NC: Habilitiative Management Consultants, Inc.

• Skinner, B. F. (1938). The behavior of organisms. New York: Appleton-Century.

• Skinner, B.F. (1953). Science and human behavior: New York: Macmillan.

• Sturmey, P. (1998). History and contribution of organizational behavior management to services for persons with developmental disabilities. Journal of Organizational Behavior Management, 18(2/3), 7-32.

• White, O. R. (1986). Precision teaching – Precision learning. Exceptional Children, 52, 522-534.

• Yssledyke, J. E., Thurlow, M. L., Graden, J. L., Wesson, C., Deno, S. L., & Algozzine, B. (1982). Generalizations from five years of research on assessment and decision making. Research Report No. 100) November 1982.