dealing with missing data - learning research and ... · in the mlm context simulations by quene...

Dealing With Missing Data

Possible Future Topics

● Novice user topics:● Using R● Contrast coding &

hypothesis testing● Random slopes &

model comparison● Logit & probit

models

● Advanced topics:● Growth curve

modeling of eye fixations

● Overfitting● Path analysis● Principal

components analysis

Dealing With Missing Data

Outline

● Ways Data May Be Missing● Solutions

● Deletion● Imputation

● In MLM framework● Means & Centering

● Special Case: Incomplete Designs

Big Issue: WHY data is missing

● The fact that data is missing may itself be data!

● Missingness of data may not be arbitrary● Affects what conclusions we can draw from the

data we do have

Big Issue: WHY data is missing

● Missing Completely at Random: Missingness unrelated to variables in experiment● Computer crashes, snow day, etc...

● Missing at Random: May be related to predictor● Production experiments: More unusable responses

in some conditions

● Missing Not at Random: Missingnessrelated to outcome measure evenafter controlling for predictors● RT w/ a cutoff● People w/ low memory don't return for test

Outline





Solutions: Deletion Methods

● Listwise deletion● Drop cases with any missing variables● Default in R … and in most software

● Properties● Only OK if missingness completely random –

otherwise, looking at selective group● Potentially, losing a lot of data!

Solutions: Deletion Methods

● Listwise deletion● Pairwise deletion

● Drop cases separately for computing each effect

● Properties● Less data loss● Results not completely consistent / comparable● Again, missingness needs to be completely random

Solutions: Imputation Methods

● Mean Imputation● Replace missing values with the variable's mean● Underestimates variance

– Thus, increases chance of detecting spurious effects

5, 8, 3, ?, ?

M = 5.33σ2 = 12.5

5, 8, 3, 5.33, 5.33

M = 5.33σ2 = 3.17


● Mean Imputation● Conditional Imputation

● If Y missing, impute value predicted by regression based on other cases

● Possibly with some amount of error (to preserve variance)

● OK in a lot of missing-at-random cases

WM VocabRT = + ? RT


● Mean Imputation● Conditional Imputation● Multiple Imputation

● Impute multiple possible values & fit model to each● Final result averages over these● Can see how much that one value affects results● Solution preferred by Schafer & Graham (2002)● Software available for this, at least for standard

regression

What if Missing NOT at Random?

● Cases where the DV determines missingness● Solutions

● In many cases, “only a minor impact” on results (Schafer & Graham, 2002, p. 152)

● Can also try:– Model the missingness in some way

● e.g. missingness and observed DV are both indicators of a latent variable

– Grouping participants by missingness– See Schafer and Graham (2002) for more details

Outline





In the MLM Context

● Simulations by Quene & van den Bergh (2004) of casewise deletion● Robust even with lots of data missing (25%)

● But this would require missingness to be completely at random● “This robustness is only if data are missing in a random

fashion. If observations were predominantly missing for certain participants and/or under certain treatments, then the full and reduced data sets would not have yielded similar estimates” (p 116).

Outline





Means & Unbalanced Data

● When unbalanced, mean of all observations may not be the same as mean of means

Primed

Unprimed

600, 600, 700, 700, 700

900, 900, 1000

660

933

796.67MEAN 762

Centering

● If missingness is completely at random, assumption is that “mean of means” is what you're interested in● “Controlling for the missingness”

Centering

● Mean centering / reweighting of fixed effects

● Suppose I code Primed as 1 as Unprimed as -1

● Reweight: less numerous level gets stronger weight

1 1 1 1 1

-1 -1 -1

Primed

UnprimedOverall mean (intercept) more influenced by Primed condition

.375 .375 .375 .375 .375

-.625 -.625 -.625

Primed

Unprimed

MEAN: 0.40

MEAN: 0.00Overall mean (intercept)

equally influenced by each condition

Centering

● If missingness is completely at random, assumption is that “mean of means” is what you're interested in● “Controlling for the missingness”

● Can get this by centering fixed effects● lmer does this automatically with random

effects (subjects & items)

Centering

● What does centering affect?● Value & interpretation of intercept● Main effect estimates & tests if an interaction

● What is unaffected?● Interactions● Main effects if no interaction

Outline





Incomplete Designs

● Designs where an entire cell is missingWORDS FACES

FASTPRESENTATION

SLOWPRESENTATION

WORDS FACES

FASTPRESENTATION

SLOWPRESENTATION

Young Adults

Older Adults

Incomplete Designs

● Designs where an entire cell is missing● Not possible to include all

interactions in the model● We don't know the 2-way

interaction effect for olderadults … so can't look at the3-way interaction involving age

● Can still look at some lower-order effects (e.g. Age x Speed) if youassume no 3-way interaction– Would be inappropriate if there

is an interaction since we'remissing part of the picture!

FAST,WORDS

FAST,FACES

SLOW,WORDS

SLOW,FACES

FAST,FACES

SLOW,WORDS

SLOW,FACES

Incomplete Designs

● Designs where an entire cell is missing

● lmer error message:● Error in mer_finalize(ans) : Downdated X'X is not positive definite.

● Dependencies in data → matrix is not full rank → not invertible

● The good news: If cell missing bydesign, clearly predicted only byIVs and unrelated to the DV● Thus, missing at random

FAST,FACES

SLOW,WORDS

SLOW,FACES

Outline





● Encyclopedia Brown confronted local troublemaker “Bugs” Meany about the missing data. Bugs says he distinctly remembers storing the missing sheet of data between pages 151 and 152 of his lab notebook. Bugs says that the sheet must have just fallen out when Bugs's gang, the Tigers, were cleaning their clubhouse.

● How did Encyclopedia know Bugs was lying?

Pages 151 and 152 are the front and back of the same sheet.

dealing with missing data - learning research and ... · in the mlm context simulations by quene...

Documents