Clinical Science

Steven A. Del Chiaro, PsyD.San José State University 

[PSYC 160]

2 of 83

Review of the homework Trull: Chapter 4 Kalal: Critical thinking in clinical practice

(article) Questions? Callaghan: Demonstrating effectiveness

(article) Richards et al.: Single subject research

(chapter) Questions? Comments? How do these fit in?

3 of 83

Scientists, Clinicians, and Clinical Science Questions posed by clinical researchers

and practicing clinicians What are the predictors for suicide

attempters and completers? Who benefits most from therapy? How does therapy work? Is one treatment approach (theory) better

than another? How do you assess for clinical change? How do you assess for psychopathology?

4 of 83

Epistemological assumptions of science Assumption of Science #1:

Determinism There is cause related to effect Need to be able to show: If I manipulate

this variable (independent variable) then I should be able to show a change in that variable (dependent measure)

We are eliminating the assumption of a variable having free will

5 of 83

Epistemological assumptions of science Assumption of Science #2: Empiricism

Attempt to restrict attention to observables or measurable variables

Can also have non-observables or hypothetical constructs

There must be theoretically available observables, though, that are related to actual observable variables

Examples: black holes, freedom, and self-esteem (how about cognitive schema)

6 of 83

Epistemological assumptions of science Assumption of Science #3: Order

Need lawful patterns Goal of science is prediction and control Can we do this with a science of

behavior? Some have backed off into prediction

and influence Argue that prediction is implausible

7 of 83

Clinical Psychology Research Methodology The Case Study

Records in detail and in narrative form what happens during therapy

Useful for documenting rare cases Used to generate hypotheses we

might want to test Can generate research questions

8 of 83

Clinical Psychology Research Methodology Case Study

Not convincing data for causal modeling because it does not control for outside factors (can't even know all of the factors)

Does not generalize Information is subject to interviewer's bias

for soliciting selective info and to patient's bias for recollecting selective information

9 of 83

Clinical Psychology Research Methodology Epidemiological research

Incidence New case rate per unit of time

Prevalence Total number of cases in certain time

frame Retrospective studies Prospective studies

Longitudinal study

10 of 83

Prevalence Rates Example of Kessler et al. (1994)

National Comorbidity Study NCS 8000 people interviewed Lifetime (and 12-month) prevalence rates

for all disorders Have any disorder at any point in life 48% life time prevalence for all disorders

Depression 12-20% lifetime prevalence

Anxiety disorders 17-25% lifetime prevalence

11 of 83

Lifetime and 12-mo Prevalence RatesNCS (1994) data

12 of 83

Critical Thinking Moment Making sense of prevalence data

What do these data say to us? What do these data mean? Are these numbers changing?

Consider the numbers on the next slide using 1985 ECA data and 1994 NCS data for 12-month prevalence rates

What does this mean? Why else might this be occurring?

13 of 83

1985 ECA Prevalence

(% )

1994 NCS Prevalence

(% )Any Anxiety Disorder 13.1 18.7

Simple Phobia 8.3 8.6Social Phobia 2 7.4Agoraphobia 4.9 3.7

GAD (1.5)* 3.4Panic Disorder 1.6 2.2

OCD 2.4 (0.9)*PTSD (1.9)* 3.6

Any Mood Disorder 7.1 11.1MD Episode 6.5 10.1Unipolar MD 5.3 8.9

Dysthymia 1.6 2.5Bipolar I 1.1 1.3

Bipolar II 0.6 0.2Schizophrenia 1.3 —Somatization 0.2 —

ASP 2.1 —Anorexia Nervosa 0.1 —

Severe Cog. Impairment 1.2 —Any Disorder 19.5 23.4

14 of 83

Clinical Psychology Research Methodology Correlational Research

Example of abuse history and psychological disorders or later abuse

Causation and direction of causality Height and weight Education and IQ number SES and prevalence of schizophrenia Serotonin levels and depressive

symptoms

15 of 83

Group level experimental designs Psychotherapy outcome research

Do therapy and evaluate people pre-therapy, during therapy, and post-therapy to see if the therapy has an effect

Also do follow-up research to see whether there is a lasting effect

16 of 83

How do you know if therapy works? Depends on how you define “works” Depends also on your variables

(assessment and measurement are key here)

Scientifically Reliability Internal validity External validity

17 of 83

Efficacy and Effectiveness Clinical Efficacy Trails

Concerned primarily with Internal Validity

why and how does the treatment work?

Clinical Effectiveness Trials Concerned more with External Validity

Where and with whom does the treatment work, given that it seems to work

See as stage 1 and stage 2 process

18 of 83

Clinical Psychology Research Methodology Group level experimental designs Group methodology

Larger design with manipulated IV and DV

Compare with group not receiving treat’t.

Statistical significance testing Results (differences between groups)

cannot be explained given that chance was operating

19 of 83

Depression S

core

40

30

20

10

Pre-treatment Post-treatment

Does CBT make people less depressed?

CBT treatment

16 weeks of therapy

Control condition

No therapy

20 of 83

Depression S

core

40

30

20

10

Pre-treatment Post-treatment

Difference between two averages

F(1,98) = 3.68, p <.05

A statistical result showing a significant difference

Post = 38

Pre = 40

Post = 22

Does CBT make people less depressed?

21 of 83

Depression S

core

40

30

20

10

Pre-treatment Post-treatment

Post = 38

Pre = 40

Post = 22

THIS IS NON-DEPRESSED

THIS is where we are aiming our treatments

22 of 83

Depression S

core

40

30

20

10

Pre-treatment Post-treatment

Post = 38

Pre = 40

Post = 22

THIS IS NON-DEPRESSED

This is clinically significant change

Post = 8

Does CBT make people less depressed?

23 of 83

Clinical Psychology Research Methodology Group methodology

Clinical significance testing Move from one population to another in a

meaningful way

24 of 83

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0

Effect Size

freq

uen

cy

Dysfunctional Functional

25 of 83

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0

Effect Size

freq

uen

cy

Dysfunctional Functional

26 of 83

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0 6.0

Effect Size

freq

uen

cy

Dysfunctional

Best Available Treatment

Functional

27 of 83

The Placebo Effect in psychotherapy Benefit from just coming in to

therapy Been an issue since the advent of

psychotherapy research What is the placebo for therapy?

To whom do we compare our group who receives treatment?

Many solutions Waiting list Innocuous "treatment"

28 of 83

The Placebo Effect Still unresolved Appears that therapy is more effective

than just a placebo effect There is an active mechanism in therapy

that makes a difference across therapies Types of research designs to address

the placebo problem Double blind procedure Problems with this approach

29 of 83

Meta-analysis Studying Therapy Research that has

Already Been Done: Meta-analysis Therapy and research

Early Example: Eysenck - examined 24 studies Most were psychodynamically oriented Results - you couldn't tell that therapy was effective Thoughtful analysis BUT it has been criticized

Study occurred at a primitive stage in the development of research

30 of 83

Meta-analysis Meta-analysis

technique of analyzing multiple studies already existing in the literature

collapse several studies to come up with one finding

Need empirical research for this to work there must be a measurable test of whether

and how much the therapy worked Go to literature and find every article that

relates to the question you are interested in

31 of 83

Meta-analysis You must have set criteria for

whether you will use the articles Conduct statistical analysis to

collapse findings and get an Effect Size Tells you how useful or effective the

treatment was

32 of 83

Meta-analytic findings Research now indicates that the

average person who is treated is better off than 80% of the untreated sample and effects of therapy seem to last for a while Smith, Glass, & Miller (1980)

Hundreds of clinical evaluation studies compared meta-analytically

33 of 83

34 of 83

Meta-analytic findings What is not found, however, are

important differences between different kinds of therapy One study reports that all therapies are the

same in effectiveness if we control for experimenter or therapist bias

Caveat: with meta-analytic studies, there is an issue with sensitivity of actually finding a difference which is not yet resolved

35 of 83

Typical outcomes How much do therapies typically

effect client outcome? Most therapies report an effect size of

0.5 Considered a moderate effect size Change did occur statistically

Remember clinical significance?

36 of 83

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Effect size (d = .5)

freq

uenc

yControl

Comparison

37 of 83

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

-5.0 -4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0 5.0

Effect size (d = .5)

freq

uenc

yControl

Comparison

Is this meaningfulchange?

38 of 83

Goal is informative science for clinical practice Reality is science does not get

consumed by practitioners Why don’t clinicians consume

Science? Morrow-Bradley and Elliot, 1986 Cohen, Sargent, and Sechrest, 1986

Research conducted to answer these questions

Still same reasons today Here are the top six reasons…

39 of 83

Why don’t clinicians consume Science?

Questions addressed are not clinically relevant

Variables contained in reported studies are not representative of actual practice

Methods and populations used are not adequately described or selected

Group statistics and statistical significance are overemphasized in data analyses

Researchers do not attempt to convey results in ways clinicians can use them

Limitations are imposed on clinicians themselves which prevent them from utilizing research, such as limited time and inertia

40 of 83

Critical Thinking Moment So how do we make the scientist

practitioner model really work How do we solve this dilemma Do we go after scientists? Do we go after clinicians? What do we do, specifically???

Rethinking Science

At least a little bit…

42 of 83

Alternative approaches to science There are better ways to answer

some of these questions There are other ways to conduct

science that allow us to answer these questions

These other methods may be much better for clinical psychology

43 of 83

Alternative approaches to clinical science Observational coding

methodologies Access the mechanism of change

Single case experimental designs Research at the level of primary

interest

44 of 83

Observational Research We Are All Observational Researchers To some extent we are all interested in

observational research You have to observe something to engage

in research Not all of us are using observation to

our advantage in the social sciences

45 of 83

Chaos and Psychology Much of what we are interested in

occurs in an open system It is uncontrollable It is unpredictable at the level of specific

instance This drives us crazy as “scientists”

We try to create situations that will help us exert control and make better predictions

46 of 83

Advantages to Observation Naturalistic (cf. analog experiments or

post hoc analysis) Not tied to verbal self-report or other

report (cf. questionnaires) Good when we cannot exert control

over an open system e.g., weather (NASA research) e.g., human interactions (therapy, crisis

situations, etc.)

47 of 83

Differences in Science Observational research is not the

same as classical experimentation Philosophical issues with

understanding of what “science” is e.g. ethology and not exerting

experimental influence (i.e. prediction and control)

48 of 83

What is “Science?” Is this what science looks like?

49 of 83

What is “Science?” Need to rethink our version of

science Consider the compromise between

internal validity and external validity

Many are questioning the constraints of classic “science” Example: clinical psychology

50 of 83

Limitations of Experimentation May not be able to answer our questions

of interest “Why does this happen?” is not as easily

answered as “Whether this happens” The “Why” questions get us closer to

hypothesized mechanisms of action in psychology research Hypothesized mechanisms help us answer

our deeper research questions

51 of 83

Consider a Clinical Question Does Therapy X reduce levels of

depression? Answered with a straightforward research

design with high internal validity AKA clinical efficacy trial

Can anyone else do Therapy X after or outside of that study? Look at external validity AKA clinical effectiveness trial

52 of 83

What do We Want to Know? Why does Therapy X make clients

better? This can’t be as easily answered

Does the therapist (doing Therapy X) have anything to do with making clients better?

53 of 83

Mechanisms of Change These questions help address our

underlying hypothesis of change in settings we cannot easily change most of research on human behavior

occurs in an open system need to be careful about what we are

we are controlling and limiting get at this with process research

54 of 83

Process Vs Outcome Research Most of us are interested in outcome

variables What changes? How much does it change?

Process Research Why is change occurring? Why is change not occurring? What is going on in this situation or

interaction?

55 of 83

Process Research Process refers to what occurs during a

situation or interaction Examining process can get us closer to what

is really going on Look at how process relates to outcome

This requires us to rethink how we answer our research questions

Can you think “outside the box” about your research?

What do you really want to know?

56 of 83

Observational coding No longer tied to what the

experiment can tell us Now we can answer questions that

get us closer to our research questions of “why” not “whether”

57 of 83

Types of Observation Qualitative

Ethnography (e.g., field notes) Case studies (narratives)

Quantitative Observational coding methodologies

e.g., marital interactions, families, parent-child interactions, psychotherapy research

58 of 83

Observational Coding Research Want to be able to approach the task of

observation in empirical manner

Can answer important research questions with data This is quantitative research Almost any qualitative research can be

made quantitative with coding methodology

59 of 83

Issues with Observational Coding Attempting to become more

“scientific” or consistent with observations

Coding refers to consistent documentation of observed variables using a formal system Need to specify what you are trying to

code

60 of 83

Types of Coding Approaches Time based or Time series

Look at occurrence of codes at specified time intervals

Violence on a playground

Event based coding Look at occurrence of events that are

not dependent on time intervals “Floor changes” or “turns” in psychotherapy

61 of 83

Coding Research Coding requires us to answer our

questions in stages What is our question?

What is occurring at the level of process? What is occurring at the level of outcome? How do we tie these together?

62 of 83

Problems with Observational Coding Time consuming

Creating coding systems Training coding of observations

Observer Effects Observer Bias

63 of 83

Advantages to Observational Coding Gets us closer to some of our real

hypotheses Does not constrain our research

models Do not have to create analogs Do not have to restrict external

validity

64 of 83

Advantages to Observational Coding You can relate process to outcome

You can’t do this without either the process or the outcome component

You can’t do the process component without observational coding You can test these hypothesized

mechanisms of action Notice: You must be very explicit about your variables Notice: These variables must be specifiable and

observable

65 of 83

Advantages to Observational Coding Innovation!

Asking hard questions and coming up with unique solutions to help answer them

66 of 83

Single subject designs In depth, scientific study of one person

at a time Tend to emphasize measurable,

observable, meaningful change Not a case study! Key is repeated measurement and

evaluation of variability, level, and trend Can show true experimental control OR

true accountability of treatment effect By the clinician!

67 of 83

Single subject designs ABAB design

A = baseline measure Record behavior without intervening No treatment time Want to observe regular pattern

B = Treatment Record changes in behavior

2nd A = remove treatment See if you have a return to baseline Naturally done with termination of treatment in

psychotherapy 2nd B = reinstate treatment

See if behavior returns to level of 1st B or maintains!

68 of 83

Single subject designs Some terminology

Baseline Need sufficient data to determine: Variability Level (2?) Trend / stability (3?) Some data will be easier to hold at baseline

69 of 83

70 of 83

71 of 83

72 of 83

Single subject designs Also use Multiple Baseline Approach look at baseline across

Situations Behaviors Participants and implement treatments staggeringly and look at

the effects Shows experimental control at n = 1 level of

analysis

73 of 83

MB across situations

74 of 83

Example of AB design for a treatment for social skills

75 of 83

Strengths of MB design Can’t always to do a baseline

reversal Want to be able to show effects

across situations or even subjects This helps convey generalization

Actual generalization is across variables that really shown change

76 of 83

Using Single Subject Research in Clinical Practice Callaghan (2001) article

This is the science that clinicians can do in clinical practice

This shows accountability Can combine these studies across

variables to demonstrate clinical effectiveness

Across cases Across clinicians

Issues Affecting All of Science

78 of 83

Problems to look for in experimental research Experimental Confounds

another variable can influence results, that when not identified and controlled will lead to inaccurate statements of causality

Direction of causality does Variable X cause Y or does Y cause X?

(Depression and loneliness) Post Hoc explanation

not prediction, just after the fact explanation

79 of 83

Problems The Quine-Duhem Problem and letting

go of bad theories Auxiliary Hypotheses and testing Core

Hypotheses Measurement issues are central Tend to measure illness, not

psychological health – this may impose a limit to how we analyze how therapy works

80 of 83

Problems Kalal’s position on baloney-

detection tools Parsimony Falsifiability Multiple hypotheses Explaining mechanisms What happens when we let fraudulent

therapies exist?

81 of 83

Human Subjects Committee Ruling board which analyzes the

components of an experiment and decides whether the experiment can be conducted ethically and legally with the minimum of risk to the subject

Very formal protocol involved

82 of 83

Human Subjects Committee One of main features

Participant gives informed consent to participate and is debriefed as to the purpose of the study after it is over

Deception is frowned upon and is not used in experiments very often

Want to ensure confidentiality of participant data

83 of 83

Fraudulent data The contingencies of fraudulent

data This is not frequently done, but Research and the bell curve

The ethics of fraudulent data