+Evidence Based Practice University of Utah

Presented by Will BacknerDecember 2009

Training School Psychologists to be Experts in Evidence Based Practices for Tertiary Students with Serious Emotional Disturbance/Behavior Disorders

US Office of Education 84.325KH325K080308

+The Use of Single-Subject Research to Identify Evidence Based Practice in Special Education

Council for Exceptional Children

Horner, R., Carr, E., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The Use of Single-Subject Research to Identify Evidence Based Practice in Special Education. Exceptional Children, 71(2), 165-179.

+Council for Exceptional Children

(CEC)

CEC was first established in 1922 by school administrators and faculty at the Teachers College of Colombia University

Its purpose is to improve the success of children with disabilities and gifted/talented children

It is the largest professional organization in the country devoted to special education (SPED)

Focus is on teachers and administrators (applied research)

+Single Subject Research

Methodology first operationalized over 40 years ago

Has been relevant for defining educational practices at the level of the individual learner

Provides a level of experimental rigor beyond that found in traditional case studies.

Over 45 professional journals report single subject research

Operant principles of behavior have benefitted most from single-subject methodology and been empirically demonstrated with single-subject experiments for over 70 years

Single-subject methodology can be used to evaluate diverse approaches to human behavior such as social learning theory, social psychology, and social work

+Goals of the Article

Present the defining features of single-subject research methodology

Clarify the relevance of single-subject research methods for special education

Offer objective criteria for determining when single-subject research results are sufficient for documenting evidence-based practices

Goal is to clarify how single-subject research is used to establish knowledge within special education and define the empirical support needed to document evidence-based practices

+Single Subject Research Methodology

Experimental, rather than correlational or descriptive

Documents casual relationships between independent and dependent variables

Uses between- and within-subject comparisons to control for threats to internal validity

Requires systematic replication to enhance external validity

+Individual Participant is the Unit of Analysis

May only involve one participant but typically includes multiple participants in a single study

Each participant serves as his own control Performance prior to intervention is compared to

performance during and/or after intervention

Usually the participant is an individual, but it can be a group that generates a single score during a measurement period (e.g. a classroom)

+Participant and Setting Description

Operational descriptions of the participants, setting, and the selection of participants is required

Another researcher should be able to use the descriptions to recruit similar participants and use similar settings

Global descriptions are usually insufficient

+Dependent Variable

In educational research is usually some sort of observable behavior

Operationally defined- Need valid and consistent assessment and allow for replication of the assessment process

Measured repeatedly across controlled conditions to allow identification of performance patterns prior to intervention and comparison of performance across conditions

Recording is assessed for consistency by frequent monitoring of interobserver agreement

Variables are selected for their social significance. Chosen because it is important for the individual participant, those

who are in contact with the individual or society

+Independent Variable

Typically the practice, intervention, or behavioral mechanism under investigation

Operationally defined to allow valid interpretation of results and accurate replication of procedures Materials and actions are often documented

Variable is actively rather than passively manipulated to demonstrate experimental control

Fidelity of variable implementation fidelity is documented Can be done by continuous direct measurement or of the IV

or an equivalent

+Baseline Comparison Condition

Performance during this condition is compared to the effects of the intervention condition

Similar to a treatment as usual condition in group design research

Requires measurement during this condition and a detailed description of the condition which is precise enough to allow for replication

Measurement should continue during baseline until the observed pattern of responding is consistent enough to allow prediction of future responding Requires multiple data points without a substantial trend or a trend in

the opposite direction predicted by the intervention. A trend in the direction predicted by the intervention compromises the

ability to document an effect.

+Experimental Control

Demonstration of a functional relationship between manipulation of the independent variable and change in the dependent variable. Demonstrated when the design documents three demonstrations of

the experimental effect at three different points in time with a single participant or across different participants

Experimental effect is demonstrated when predicted change in the dependent variable covaries with manipulation of the independent variable.

Documentation of experimental control is demonstrated through introduction and withdrawal of the independent variable, the staggered introduction of the independent variable at different points in time, or the iterative manipulation of the independent variable across observation periods.

+Analysis of Single-Subject Research Single subject research can be interpreted with statistical

analyses, but the traditional approach involves systematic visual comparison of responding within and across conditions of a study.

Visual analysis involves interpretation of the level, trend, and variability of performance occurring during baseline and intervention conditions

Also involves judging: Immediacy of effects following the onset and/or withdrawal of the

intervention The proportion of data points in adjacent phases that overlap in level The magnitude of changes in the dependent variable The consistency of data patterns across multiple presentations of

intervention and nonintervention conditions

+Analysis Continued…

Integration of information from these assessments and comparisons is used to determine whether a functional relationship exists between the independent and dependent variable

A functional relationship is a documentation of an effect. It is compromised when: There is a long latency between manipulation of the independent

variable and change in the dependent variable Mean changes across conditions are small and/or similar to changes

within conditions Trends do not conform to those predicted following introduction or

manipulation of the independent variable

Meta-analysis can also be used to document comparative trends in a field

+External Validity

Concern whether an effect documented by one study has relevance for participants, locations, materials, and behaviors beyond those defined in the study.

External validity of single-subject research is enhanced through replication of the effects across participant, conditions, or different measures of the dependent variable Also improved if study includes multiple participants, settings,

materials, or behaviors Enhanced by using operational descriptions of parameters

Generality of an intervention is established through systematic replication of effects across multiple studies

External validity narrowed when selection and attrition bias limit the range of examples available for analysis

+Social Validity

In education single subject research has been used to document interventions that are functionally related to change in socially important outcomes.

Social validity of single subject findings is enhanced by: Selection of dependent variables that have high social importance Demonstration that independent variables can be implemented with

fidelity by teachers and parents in typical contexts across time Demonstration that teachers and parents report the procedures to be

acceptable, effective, feasible with available resources, and continue to use interventions after formal support is removed

Demonstration that the intervention produced an effect that met the clinical need

Single subject research can also be used to distinguish between efficacy research and effectiveness research

+Research Questions Appropriate for Single Subject Research Methods No research approach is appropriate for all research

questions

Research questions appropriately addressed with single-subject methods include: Examining casual or functional relations by examining the

effects of the introduction or manipulation of an independent variable on change in one or more dependent variables

Focus on the effects that altering a component of a multicomponent independent variable has on one or more dependent variables

Focus on the relative effects of two or more independent variable manipulations on one or more dependent variables

+Quality Indicators for Single-Subject Research Description of Participants and Setting

Participants are described with sufficient detail to allow others to select individuals with similar characteristics.

The process for selecting participants is described with replicable precision.

Critical features of the physical setting are described with sufficient precision to allow replication.

Dependent Variable Dependent variables are described with operational precision. Each dependent variable is measured with a procedure that

generates a quantifiable index. Measurement of the dependent variable is valid and describes

with replicable precision. Dependent variables are measured repeatedly over time. Data are collect on the reliability or interobserver agreement

associated with each dependent variable, and IOA levels meet minimum standards (e.g., IOA=80%; Kappa=60%).

+Quality Indicators Continued…

Independent Variable Independent variable is described with replicable precision. Independent variable is systematically manipulated and

under the control of the experimenter. Overt measurement of the fidelity of implementation for

the independent variable is highly desirable.

Baseline The majority of single-subject research studies will include

a baseline phase that provides provides repeated measurement of a dependent variable and establishes a pattern of responding that can be used to predict the pattern of future performance, if introduction or manipulation of the independent variable did not occur.

Baseline conditions are described with replicable precision.

+Quality Indicators Continued…

Experimental Control/Internal Validity The design provides at least three demonstrations of experimental

effect at three different points in time. The design controls for common threats to internal validity. The results document a pattern that demonstrates experimental

control.

External Validity Experimental effects are replicated across participants, settings, or

materials to establish external validity.

Social Validity The dependent variable is socially important. The magnitude of change in the dependent variable resulting from

the intervention is socially important. Implementation of the independent variable is practical and cost

effective. Social validity is enhanced by implementation of the independent

variable over extended time periods, by typical intervention agents, in typical physical and social contexts.

+ Importance of Single Subject Research Methods for Research in Special Education Single-subject research focuses on the individual

Single-subject research allows detailed analysis of “nonresponders” as well as “responders.”

Single-subject research provides a practical methodology for testing educational and behavioral interventions.

Single subject research provides a practical research methodology for assessing experimental effects under typical educational conditions.

Single-subject research designs allow testing of conceptual theory.

Single-subject research methods are a cost-effective approach to identifying educational and behavioral interventions that are appropriate for large scale analysis.

+Identification of EBP using Single-Subject Research

The practice is operationally defined. Described with sufficient proficiency so others can replicate it with

fidelity.

The context in which the practice to be used is defined. Conditions where the practice should be used, individuals qualified

to apply the practice, the population for whom the practice is expected to be effective, and the outcomes affected by the practice are defined.

The practice is implemented with fidelity. Studies should provide adequate documentation that the practice

was implemented with fidelity.

+Identification of EBP Continued… Results from single-subject research document the practice

to be functionally related to change in dependent measures. Studies should document a casual, or functional, relationship

between use of the practice and change in a socially important dependent variable by controlling for the effects of extraneous variables.

The experimental effects are replicated across a sufficient number of studies, researchers, and participants to allow confidence in the findings. Considered Evidence-based if: A minimum of 5 single-subject studies that meet minimally

acceptable methodological criteria and document experimental control have been published in peer-reviewed journals.

The studies are conducted by at least 3 different researchers across at least 3 different geographical locations.

The 5 or more studies include a total of at least 20 participants.

+Conclusion

Single-subject research offers powerful and useful methods for improving the practices that benefit individuals with disabilities.

Any systematic policy for promoting the development or dissemination of evidence-based practices in education should include single-subject research as an encouraged methodology.