Applied Behavioral Research

Session 5: Single Subject Research Methodology

◦ Presentation template posted on wiki

◦ Please look at the APA Style website presented by Purdue’s Online Writing Lab (OWL)

◦ Find and review an experimental, quasi-experimental or single subject design study on an intervention for people with disabilities! Article review due next class August 3rd!

◦ Research proposal status- should be working at the point of defining your dependent and independent variables….thinking about a research design, but we haven’t discussed all of the designs yet.

Updates/Questions

Review for Quiz

Steps in the Research/Scientific Process

1. Identify socially important issue

2. Review current literature

3. Define conceptual model

4. Define specific hypothesis(es) and research question(s)

5. Define dependent variable(s)/measure

6. Identify independent variable(s)/measures

7. Select appropriate research design

8. Obtain consents 9. Collect data 10. Analyze data 11. Communicate

results Written presentation Oral presentation

Investigators attempt to determine the cause of differences that already exist between or among groups of individuals.

Describes conditions that already exist (a.k.a. ex post facto).

The group difference variable is either a variable that cannot be manipulated or one that might have been manipulated but for one reason or another, has not been.

Studies in medicine and sociology are causal-comparative in nature, as are studies of differences between men and women.

What is Causal Comparative Research?

Similarities and Differences Between Causal-Comparative and Experimental Research Similarities

Require at least one categorical variable Both compare group performances to determine relationships Both compare separate groups of subjects

Differences In experimental research, the independent variable is manipulated Causal studies are likely to provide much weaker evidence for

causation In experimental studies, researchers can assign subjects to

treatment groups The researcher has greater flexibility in formulating the structure of

the design in experimental research

Similarities Ex Post Facto research Attempt to explain

phenomena of interest Seek to identify variables

that are worthy of later exploration through experimental research

Neither permits the manipulation of variables

Attempt to explore causation

Differences Causal studies compare two

or more groups of subjects Causal studies involve at

least one categorical variable

Similarities and Differences Between Causal-Comparative and Correlational Research

Examples of the Basic Causal-Comparative Design

Step 1- Determine Purpose

Step 2- Identify a Sampling Plan & Mode

Step 3- Design survey instrument

Step 4- Test survey instrument

Step 5- Send out a letter of transmittal

Step 6- Deliver the survey

Step 7- Analyze data from survey

Steps to designing, delivering, and analyzing surveys

Quiz

Correct Quiz

Discussion

Please get into your research groups for the lecture portion.

You will be completing the in-class activity together with your group.

Lecture

Systematic analysis using individual subjects as their own experimental control.

Main message:◦ Single subject research is an approach to

rigorous experimentation that involves small numbers of subjects, repeated observations of subjects over time, and employs research designs that allow each subject to provide his/her own experimental control. Within-subject analysis Fine-grained analysis across time and conditions

Single Subject Research

Defining Features of Single Subject Research

An experimental research method focused on defining causal (e.g., functional) relations between independent and dependent variables.

Focus is on individuals as unit of analysis◦ can treat groups as participants with focus on the

group as a single unit Repeated measures of participants’ behavior

(DV) over time Within-subject comparison to analyze effect

◦ Observed change in individual’s behavior from “Baseline” to “Intervention”

Focus on an individual rather than group means◦ Interest is in the behavior of a single individual

or on within-subject variability A “group” may be treated as an “individual”

◦ Group descriptive statistics may not "describe" any actual individual

◦ Generalizations from a group to an individual are problematic in many instances Predicting the behavior of a specific individual is

different from predicting that of a “typical” individual

Reasons for using single subject methodology

Many populations of interest are low incidence populations◦ Practically, large numbers of subjects may not

be available◦ Assumptions of normal distribution and

homogeneity of variance may not be valid Can be used in clinical practice contexts

◦ Single subject research studies may develop out of and be conducted on a specific problem or need of an individual(s) in a practical context Scientist-practitioner model

Reasons for Using Single Subject Methodology (continued)

Using Single Subject Research to Establish “Evidence-based Practices”

A “practice” may be considered “evidence-based” when:◦ The practice is operationally defined, and

implemented with fidelity.◦ The outcomes associated with the practice are

operationally defined.◦ The context in which the practice in use is

operationally defined◦ Results from the single subject studies used to

assess the practice demonstrate experimental control.

◦ The effects are replicated across 5 single subject studies conducted in at least 3 locations, and with at least 20 different participants.

Dependent variable (DV) – the behavior (measure) that you are analyzing◦ You want to produce change (variability) in the

dependent variable◦ Studies may have multiple DVs

Independent variable (IV) – the variable (event, intervention, condition) that is of experimental interest and that the researcher manipulates in an experimental research design◦ May be discrete or continuous◦ May be a single element or multi-component

compound◦ Studies may have multiple IVs

Dependent and independent variables

The degree to which observed differences/changes in the dependent variable are a direct result of manipulation of the independent variable, and not some other extraneous variable

Extent to which a functional relation can be documented. Control of extraneous variables that provide alternative explanations for results.◦ It is okay to try to maximize internal validity,

especially in initial documentation of a functional relationship Doing this may come with a cost, however

Internal Validity

History – everything happening outside of the research study

Maturation Testing - repeated measurement Instrumentation

◦ with human observers, observer bias and drift Attrition - loss of participants Multiple treatment interference Diffusion of treatment - intervention is

inadvertently provided when not intended

Threats to Internal Validity

Loss of baseline through generalization or spread of effects (across settings, behaviors, or participants)

Instability and/or high variability of behavior◦ cyclical variability

Statistical regression toward mean Selection biases with participants Inconsistent or inaccurate implementation

of the IV (Treatment Drift/Treatment Integrity)

Threats to Internal Validity (continued)

Defined: The extent to which results can be applied to settings, activities, people, etc. other than those involved in the study.◦ Given that you have found an effect for this

intervention with this participant under one set of conditions, will it work with other participants, in other settings, when implemented by other interventionists, and when implemented with minor variations in the basic procedures?

◦ What can we generalize from this single study?◦ Importance of systematic and direct replication.

External Validity

Reactive experimental arrangements - Hawthorne effect

Reactive assessment - reactivity to observers

Pretest sensitization Experimenter bias Interaction between selection bias and

treatment effects - i.e., intervention only works if the "right" participants are selected◦ Specificity of effects

Threats to External Validity

In single subject designs the research question typically examines a causal, or “functional” relation, between the independent and dependent variable. As such the research question should have three features

Identify the dependent variable(s) Identify the independent variable(s) Proclaim intention to determine if change in the IV

is functionally related to change in the DV.

The Research Question

Dependent variable is socially important Independent variable(s) can be controlled

(e.g. manipulated) across time. Both the dependent and independent

variable(s) can be operationally described and measured.

For “experimental” research, the question must ask if change in the DV is caused by (or functionally related to) change in the IV.

Research Question Features

Is there a functional relation between development of reading fluency and scores on comprehensive reading assessments?

Will walking in water facilitate development of appropriate gait by individuals with “gait imbalance hypertension”?

Is there a functional relation between use of escape-extinction and reduction of escape-motivated food refusal?

Does Jason act out because he has ADHD?

Research Question Examples

Dependent Variable (Outcome):

Independent Variable (Intervention):

Research question: “Is there a functional relationship between …… and …… ?”

For your research study define your DV, IV, & SSD research question

Level Trend

VariabilityImmediacy of Effect

Overlap

Phase A Phase B

Phase A Phase B

Research Question???

Level Trend

VariabilityImmediacy of Effect

Overlap

Phase A Phase B

Phase A Phase B

Research Question???

In SSD, a Functional Relationship/Experimental Control has occurred when

There are 3 demonstrations of an effect at 3 points in time.◦ Effect could be: change in trend or level◦ Also want to see immediacy of effect

Good research has at least 5 data points in each phase to establish a consistent pattern in the data.

Establishing a Baseline

Baseline - phase in a design that serves as the reference point or comparator for analysis of change in behavior (effect of IV)◦ Used in withdrawal/reversal and multiple baseline

designs; may be included in alternating treatments design (but not needed)

◦ Generally, the first phase, but not always Returned to periodically in withdrawal/reversal designs

◦ Provides (should provide) a representative picture of behavior under pre-intervention (typical, status quo) conditions Baseline is the “control condition” in within subject

analysis May involve some alternative intervention/treatment

Guidelines for Establishing a Baseline

Collect repeated measures of a DV under “baseline” conditions◦ Goal is to establish the stability of behavior

Look at level, trend, and variability of data◦ At minimum, Horner et al. (2005) propose 5

data points in baseline phase (at least for initial phase) Fewer points can be defended in some situations -

e.g., participant cannot perform the behavior (has not learned) or ethical considerations

◦ Variability in DV requires more data points◦ Can go forward with variability, particularly if

intervention effect can be documented despite baseline variability

High Variability in Baseline?

Use baseline phase to do close observation to reveal potential sources of variability◦ Control variability through elimination or

holding constant extraneous variable(s)◦ Consider whether sources of variability should

be studied as IVs◦ Be alert to dramatic changes within the phase

and identify potential causes Balance logistical and clinical needs with

research goal of stability◦ Recognize potential limitations and threats to

internal validity if you have high variability

Trends in baseline data? Trends (increasing or decreasing slope)

can be accepted, if the trend is in the opposite direction of the anticipated effect of the IV◦ Visual analysis does consider changes in

trend across/between phases Trend in the “expected” change

direction is problematic◦ Collect more data points◦ Consider whether intervention is warranted◦ If substantial change in slope is expected,

you may go forward with intervention Statistical analysis may be used to supplement

visual analysis

When to move from BL to Intervention? When a pattern of BL responding is

established.◦ Can you predict the next data points?◦ Current BL pattern will allow you to document

anticipated intervention effects? Note:

◦ High BL variability requires extending BL◦ Trend in direction of expected effect requires

extended BL.◦ If BL level matches expected IV level, then

extend baseline.

Assessing Baselines Define research question and dependent

variable.◦ Does BL document a predictable pattern of

behavior?◦ Does BL document a pattern that will allow

comparison with expected effect when Intervention (IV) is implemented?

Implementing the Independent Variable The “traditional” rule - implement one

variable at a time◦ Allows for clearest demonstration of a functional

relationship Package interventions create issues

◦ May be able to establish relationship between the package and DV, but not know about effects of specific components Component analysis designs address this issue

Demonstrating interaction effects also is a challenge

Guidelines for Implementing IVs Implement based on data collected in

baseline (or previous phases), rather than on a predetermined schedule that is independent of the data

Establish effects of IV on one baseline (data path) before implementing IV in another baseline (data path) in a multiple baseline

Collect and report measures of IV implementation fidelity

Length of Phases

Phases should be long enough to establish representativeness of data within the phase◦ Reach stability within the phase (at least 5 points)◦ Some have argued that for power, the number of data

points in SS design is comparable to number of subjects in group design

Researchers often want to use relatively short phases◦ Because of logistical issues, ethical issues,

impatience, costs◦ Be aware of limitations and threats to validity

Phases of very different lengths within a design (particularly ABAB) can create issues for visual analysis and interpretation of effects

Timing of Phases Data may be collected in sessions that are

daily, multiple within a day, or longer spaced (e.g., weekly, etc)

Consider timing between sessions and phases◦ Avoid carryover effects by spacing sessions or

phases ◦ Timing between phases can raise potential

threats to internal validity e.g., running all sessions for a phase within a day, and

then all sessions for the next phase on the next day

A multiple baseline design involves three or more AB interventions (series) with phase changes staggered across at least three points in time.

Key Features◦ Series are independent of each other

People, places, materials, behaviors/skills◦ The same IV is applied in each series◦ Staggered implementation of IV

Defining Features of Multiple Baseline Designs

◦ Identify Research Question(s)◦ Assess Baselines for each series

Do the Baselines document a predictable pattern? Do Baselines allow opportunity to document IV

effect? Are Baselines similar?

◦ Horizontal Analysis of Effect (per series) Level, trend, variability, overlap, immediacy of effect

◦ Vertical Analysis DV change in one series is associated with NO

change in other series? Similar effect (consistent effect) across series?

◦ Functional Relationship? At least three demonstrations of effect at three

points in time

Interpreting MBL Designs

Vivian

0

20

40

60

80

100

Tammy0

20

40

60

80

100

Dr. Cathy20

40

60

80

100

0 10 20 30 40 50 60 70

Per

cent

age

of C

orre

ct R

espo

ndin

gBL

Sessions

Treatment Lollipop for R+

Lollipop for R+

Lollipop for R+

6

Sequential phases of data collection involving the implementation and withdrawal of an independent variable(s)◦ within each phase, multiple data points are collected

to establish a representative pattern of behavior◦ phase change should occur only after stability of

behavior within the phase is established◦ traditionally, the first phase is Baseline, followed by

implementation of the IV (Intervention) this is not required, however, as you may begin a study

with an intervention phase

Defining features of withdrawal and reversal designs

Behavior measured as DV is “reversible”◦ Learning will not occur

Limited carryover effects between phases Ethical concerns

◦ Can do a reversal DV is not a dangerous behavior, or you can protect participant Staff cooperation

Can compare multiple conditions◦ Comparison of too many conditions makes design

cumbersome

When are reversal and withdrawal designs appropriate?

0

1

2

3

4

5

6

1 5 10 15 20 25 30 35Sessions

Tota

l SIB

per

min

ute

FCTBaseline Baseline FCT4B

Within subject analysis Independent variable needs to have at least four

levels (e.g. criteria) Document baseline performance with one level of

the IV Change the level of the IV and monitor change in

DV◦ Immediacy of change important◦ Absence of trend and variability important

Repeat level (criterion) change in IV two more times.

Defining Features of Changing Criterion Designs

Examine the graphs below◦ 1. What is the research question?◦ 2. Is there a functional relationship?◦ 3. Does the design document three

demonstrations of an “effect” at three different points in time? Where?

Example

Changing Criterion Design

0

5

10

15

20

25

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Days

Occ

urr

ence

s o

f P

rob

lem

Beh

avio

r

BL: No Reinf Reinf < 17 Reinf < 12 Reinf < 5

Changing Criterion Design

0

5

10

15

20

25

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29

Days

Occ

urr

ence

s o

f P

rob

lem

Beh

avio

r

BL: No Reinf Reinf < 17 Reinf < 12 Reinf < 5

Alternating Treatment (Multi-Element) Designs employ rapid phase reversals across 2 or more conditions to assess sensitivity of change in the dependent variable to change in condition.

ATD/ MED Defined

Student 1Hypothesis: Escape Math Work

1 2 3 4 50%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Control Condition

Escape Condition

Attention Condition

IOA

Sessions

Perc

ent I

nter

vals

with

Occ

urre

nce

of P

robl

em B

ehav

ior

1. Is Esc different than Control?

2. Is Esc different than Attn?

State the research design you would use for your study and why?

54

In-class Activity #4

The selection of measures is PART of building a single subject design.

All single subject designs require measures that allow documentations of:◦ A stable pre-intervention pattern of

performance, and◦ A rapid and dramatic change in performance

following intervention. Measures must be reliable/consistent

enough to document pre-intervention stability, and sensitive enough to document rapid, dramatic change.

Measurement in Single Subject Designs

Fundamental Dimensions of Behavior

• Frequency: – The number of occurrences of a response within an observation period.

• Duration:– The total time taken to perform a response (typically indexed as the mean

duration)

• Latency:– The time between the presentation of the Sd, and the initiation of a

response.

• Perseveration:– The proportion of the observation period/interval in which responding was

occurring. (Total time for all occurrences)

• Rate:– The frequency of a response divided by the total time for an interval

(typically occurrences per minute…or occurrences per second).

Measurement Procedures• Event recording:

– Observe number of occurrences within an observation period

• Duration recording:– Observe the mean time of responding per occurrence (tempo)

• Interval recording:– Observe the proportion of intervals in which the behavior occurs.

» Whole interval versus partial interval recording.

• Time sampling:– Proportion of time sampled moments in which behavior is

occurring.

• Permanent product:– Count of products from behavior. Note: No direct observation

• Narrative:– Continuous description of behavior in real time

Define a research question For the Dependent Variable

◦ Select a measure◦ Select a measurement process

For the Independent Variable◦ Select a measure◦ Select a measurement process

In-Class Activity #5 & #6 select measures for your variables.

Building Data Collection Forms• Paper/Pencil or Computer Entry/PDA• Key Features

– Logistical Information– Date, observer, observed,

– Ease of recording (eyes on context)– Key strokes or checks instead of writing words.– Number of variables recorded simultaneously (3 is plenty)

– Operational definitions– Fit the context and range of observed behavior

– Instructions on setting up a data session– System for summarizing session results.

Nifty Observation Form Date: ________________________ Observer: _____________________ Context: ______________________ Request: Statement from teacher requesting response by target studentCompliance: Initiation of requested response within 5 s of requestNoncompliance: Absence of initiation of requested response within 5 s of request.Problem behavior: Talking out, aggression, property destruction, disruption.

10 s Interval

Request Compliance (+)/Noncompliance (0)

ProblemBehavior

 Comments/Issues

1        

2      

3      

4      

5      

6      

7      

8      

9      

10      

In-class Activity #7

• Build a data collection form based on how you plan to measure the data.

Inter-observer Agreement• Proxy for reliability but not really a measure

of reliability.• Poor IOA means poor reliability, but good IOA does

not prove good reliability.

• Two practical measures• Percent agreement (Total, Occurrence Only)• Kappa

Percent Agreement• Defined: The extent to which two, independent

observers agree they observed the same events at the same time.– Operationalized. Given a group of observation intervals,

to what extent do the frequencies or interval recordings co-vary across two, independent observers. What percent of the intervals index agreement?

• Calculation. – (Frequency of observations with agreement/ total

number of observations) * 100%– Frequency observed by Observer 1/Frequency observed

by Observer 2 (correlation)

Percent Agreement

• Advantages• Easy to compute• Easy to understand• Failure to obtain criterion level is informative.

• Disadvantages• Is not a measure of reliability• Provides an over-estimate of agreement (especially

when <10% or >90% of intervals include occurrence.

Percent Agreement

• Professional Standards– 85% agreement is expected for good IOA

• Occurrence Only vs Total Percent Agreement– Occurrence/Nonoccurrence Only is used to assess

agreement when <10% or > 90% of intervals include occurrence.

– Calculate (use in denominator) only using those intervals in which either of the observers recorded a response (Occurrence Only) or only those intervals with either of the observers did not record a response (non-occurrence only).

– Controls for one source of bias.

Cohen’s Kappa• Purpose of Kappa is to provide an index of

observer agreement that controls for chance agreements. – Kappa can range from –1.00 to +1.00

• .40-.60 = fair agreement• .60-.75 = “good” agreement• .75+ = generally needed for publication in

Tier 1 journals

Kappa

• Calculation– Kappa = (Po- Pc) / (1 – Pc)

• Where Po = the proportion of observed agreements• Where Pc = the proportion of agreements expected by

chance.

• Recommendation:– Report both percent agreement and Kappa.– Use Occurrence/Non-occurrence Only when

appropriate

68

Ethics in Single Subject Research

Issues related to single subject research design features Withdrawal/Reversal Designs

– Implementing withdrawal/reversal phases & length of phases when DV is problematic

– End study with participants in the "optimal" phase– Adequate baseline length

Multiple Baseline Designs– Extended baselines & treatment phases– No treatment/intervention "control" baselines

Reaction to measurement or other research procedures– Set research session termination guidelines & criteria to

protect everyone – terminate sessions when criteria are met– Have a plan to protect participants and others, and to bring

situations under control if crisis occurs

Issues related to applied research in natural settings

• Minimize negative images and stigma• Use unobtrusive measurement (as possible)• Appropriate selection of DV measures

• For example, use latency to problem behavior rather than rate in community settings

• Dignified procedures• Responding to "citizen" questions or comments• Ensuring cooperation and support of others in natural

settings• Open communication before and during study• Obtain appropriate permissions & consents• Be courteous & respectful

• Allow people in the setting (teachers, families, staff) some voice• Include community "others" as research partners/collaborators

Exiting research projects gracefully Plan for exit Leave participants in "optimal" phase or state of

performance Provide training and support (i.e., plan, materials, etc)

for natural community members to assume and maintain implementation of intervention

Provide information on results and their implications for natural setting

Provide follow-up if necessary– Agree on researcher responsibilities on the front end

(before study)

Activity #8. Draw your research design and proposed data

0

1

2

3

4

5

6

1 5 10 15 20 25 30 35Sessions

Tota

l SIB

per

min

ute

FCTBaseline Baseline FCT4B