Experimental Design

Reminders

Graduate symposium deadline Friday CSBS student conference April 25

(deadline April 10) Thesis defenses Outline due on Friday

Validity

What is internal validity? How can it be increased?

What are the relationships between internal, external, and construct validity?

What is needed to establish causality?

Single-group threats to int val

History Maturation Testing Instrumentation Mortality Regression to the mean What are each of these (examples)

and how can they be decreased/controlled?

Regression to the mean

Why does it happen? When will it happen more? How can you estimate it?

Multiple group threats

Selection Selection-history Selection-maturation Selection-testing Selection-instrumentation Selection-mortality Selection-regression What are each of these (examples) and

how can they be decreased/controlled?

Social interaction threats

Diffusion of treatment Compensatory rivalry Resentful demoralization Compensatory equalization of

treatment What are each of these (examples)

and how can they be decreased/controlled?

Design notation

X O

R O X O R O O

N O X ON O ON O

Importance of random assignment

vs. random selection What is the strongest design in

terms of internal validity threats? Why would you want to add a

pretest? Or not? Are there times that you wouldn’t

want to randomly assign?

Basic design terms

Independent vs. dependent variables Within vs. between-participants

designs: advantages? What are control variables? What can you do to reduce

confounds in research?

Factorial designs

What are they and why do we care? What are levels vs. variables? What kind of FD is this? How many main

effects are possible? How many interactions? How many groups? How many people at n=10? IVs: Presence of an audience, difficulty of task

(easy, medium, or hard). DV = nervousness IVs: Gender, Drug vs. CBT vs. control,

previous experience with treatment (yes or no). DV=reduction in symptoms

Main effects and interactions

People who were alone helped more often than those who were with a confederate. Seminary students also helped more often than business students.

Men and women helped equally overall, but men were more likely to help women than to help men.

Examples on sheets

Terms

Null results Main effect Interaction Block design Covariate Solomon 4-group Switching design ANOVA vs. ANCOVA vs. regression

Issues related to design

If a factorial design is good, is a bigger one better?

When would you want to include a variable as an IV vs. a covariate vs. something you match on vs. a control variable?

How do you decide which variables to include?

Placebos

Moderators

What is a moderator? How is a moderator different from a/an

IV? DV? Confound? Mediator? Covariate?

How can you design a study to test a moderator (various ways)?

Mediators

What is a mediator? How is a mediator different from a/an

IV? DV? Confound? Moderator? Covariate?

How can you design a study to test a mediator (various ways)?

Examples A study finds that convicted criminals are more likely than noncriminals to score

low (negatively) on the Attitudes toward Women Scale. Further research shows, however, that this is only true for violent criminals. There is no relationship between nonviolent criminal activity and negative attitudes toward women. Identify the mediator or moderator. 

2. A researcher finds that by increasing self-focus in children, she can decrease their likelihood of cheating on a test. By decreasing cheating, in turn, she finds that academic self-efficacy increases. Identify the mediator or moderator.

3. Dr. Laylor finds a relationship between physical attractiveness and self-confidence. He later determines that the primary cause of this relationship is the positive feedback physically attractive people receive from others. Identify the mediator or moderator.

4. Boys who are popular with the same-sex tend to also be popular with the opposite-sex, whereas girls who are popular with the same-sex tend to be less popular with the opposite-sex. Identify the mediator or moderator.

5. Identify and explain at least two potential mediators and two potential moderators for the following relationship: School size and academic achievement.

MacKinnon et al.

Table 1—why do some areas of psychology use more mediators than others?

Analyzing moderators

Case 1: Cat IV, Cat Mod: 2 x 2 ANOVA

Case 2: Cont IV, Cat Mod: Could do correlations sep and compare

BUT Better to do regression and compare

unstandardized Betas Or SEM software

More moderators

Case 3: Cat IV, Cont Mod: Figure out how level of moderator affects IV-

DV relationship If linear, do hierarchical regression, showing

that XZ adds to the effects of X and Z on Y Case 4: Both Cont:

Use Case 2 if step function or Use Case 3 if linear, quadratic

How do you know which one is the moderator vs. IV?

Analyzing mediators

Causal steps (Baron & Kenny) IV related to M M = i + aX + e IV related to DV Y = i + c1X + e M and IV related to DV Y = i + c2X + bM + e C1 greater than C2 (look at size and sig)

Limitations: Not good for multilevel, probit, logistic, survival Need to test for whether C1 > C2 (Sobel test) Low power esp when IV and DV aren’t related Overestimates effect of IV on DV if error in M

Sobel test

Used to test whether C1 > C2 Good for sample sizes of 50+ with 1

M Or 100-200 for >1 M

Alternatives to B & K

Use multiple measures of M and use SEM

Distribution of the product PRODCLIN Better Type I error rates, higher power

Computer-intensive methods Aka resampling Fewer assumptions

Assumptions of mediational tests

Residuals are independent in equations 2 and 3

No XM interaction in equation 3 Direction is correct (DV doesn’t cause M) Measurement is perfect, esp. in M No unmeasured variables that cause X, Y,

or M IV related to M M = i + aX + e IV related to DV Y = i + c1X + e M and IV related to DV Y = i + c2X + bM + e

Other mediation issues

Complete vs. partial mediation Use hierarchical regression Test C2 significance

Inconsistent mediation Multilevel mediation

Can increase T1 error if you ignore Categorical DV mediation

Use logistic or probit regression Multiple mediators

Longitudinal mediation Moderated mediation Mediated moderation Mediated baseline by treatment

moderation

Meds and Mods

How do you know which variable is the mediator?

How can a moderator lead you to a mediator? How can a mediator lead you to a moderator? Can a variable be both a mediator and a

moderator at the same time? Are there variables that are always going to

be one or the other? Does a mediator have to be correlational? Remember that one study does not a

mediator/moderator make

Outlines

Think of sections as headings in your thesis

Keep headings parallel Always have at least 2 in each group Think about the best logical order for

things, and keep the order consistent throughout the paper

Next week

Quasi-experiments 2 book chapters plus other chapter