list experiment talk no pauses

BackgroundFormalizing the List ExperimentSummary of Additional Results

Empirical ExampleDiscussion

Statistical Theory of the List Experiment/ItemCount Technique

Jay VerkuilenThe City University of New York Graduate Center

[email protected] at Fordham University, Department of Psychology

Psychometrics Colloquium, January 20, 2010

January 19, 2010

Jay Verkuilen The City University of New York Graduate Center [email protected] Presented at Fordham University, Department of Psychology Psychometrics Colloquium, January 20, 2010List Experiment



Acknowledgements

I Alexander Janus, UC-Berkeley

I Ida Jeltova, CUNY Graduate Center

I James Kuklinski, UIUC

I David van Dantzig’s very convenient notation (Hemelrijk,1966)




Social Desirability Bias

I Goal: Want to measure prevalence of an attitude or behaviorsubject to social desirability bias of some sort. Examples:

I Have you had sex with someone you don’t know in the lastmonth? (Sex.)

I Have you used marijuana in the last month? (Drugs.)I Have you listened to Britney Spears’ new album? (Rock ’n’

roll—of a sort)

I Not self-deception. We need to assume that respondentswould respond truthfully if they felt anonymous but won’t dueto sanction implied by affirmative response.




Alternative Strategies

There are two main alternative strategies for handling socialdesirability bias. I can’t really summarize these due to timelimitations.

I Use a covariate such as a lying scale. Main problems:Obtrusive, long and low validity (I. Jeltova, personalcommunication).

I Randomized response: Provide a “probabilistic fig leaf.”Admirably summarized in a previous talk. Main problems:Obtrusive and somewhat complex from the respondent’sperspective.




The List Experiment

Noting the limitations of other methods—particularly as regardstelephone or paper form surveys—Kuklinski et. al. and Droitcoeuret. al. (see references) devised a relatively simple method.

I Randomize the sample into two experimental groups of equalsize.

I Administer distractor items plus the hot item to the treatmentgroup and only control items to the control group, asking onlyfor how many (not which) items the respondents agree with.

I Use contrasts among the experimental groups to remove theeffect of the distractors and provide an estimate of theprevalence of the hot item.




LE Example: Janus’ (2008) Immigration Question

“Now I am going to read you three/four things that sometimespeople oppose or are against. After I read all three/four, just tellme HOW MANY of them you oppose. I don’t want to know whichones, just HOW MANY.” Afterwards, subjects in the treatmentgroup were presented with the following four items:

I Cutting off immigration to the United States

I The federal government increasing assistance to the poor

I Professional athletes making millions of dollars per year

I Large corporations polluting the environment




Example: Janus (2008) Immigration Question (cont.)

I Janus used a design we term simple deletion:I 407 respondents in the treatment group, 505 in the control.I Order of presentation of items also randomized.

I Estimate of proportion favoring the hot item is

π1 = yT− y

C(1)

I Results:

Education Direct List Experiment

HS or less 44% 68%Some college 48% 66%

Bachelor’s 29% 66%Grad./Prof. 28% 31%

Average 40% 62%




List Experiment: Observations

The LE is

I Easy to administer,

I Relatively unobtrusive,

I Straightforward to analyze (just regression).

I What’s not to like . . .?

Unfortunately in practice the list experiment often gives bafflingresults, e.g., negative proportion estimates and is perceived(erroneously) not to allow use of multiple regression, whichsuggests a closer look is necessary.




List Experiment: Questions (and Answers)

Some questions (answers in parentheses):

I What is the underlying model? (Provided.)

I Can we guarantee a consistent estimator of the desiredproportion? (Yes.)

I How much efficiency do we lose? (Often a lot.)

I Are there designs that also lead to consistent estimates of πbut are more efficient than simple deletion? (Yes, givenoptimization of group sizes.)

I What are the best ways to write the items? (The distractorsare of key importance.)

I Are there ways to remove the effect of the distractors, ineffect pull back the probabilistic fig leaf? (Yes, given goodregressors for the distractors.)




Goals

I Develop the underlying mathematical model that the creatorsof the list experiment had in mind (verbally) when the originalwork was done.

I Focus is on the simple deletion case (henceforth SD). Ourresults are more general and can accommodate other designs,but the math becomes tedious and will be summarized afterexamining SD in depth.




List Experiment as a Missing Data Problem

Assume there are J items (w.l.o.g. j = 1 is the “hot item”) andfurthermore there is a latent checklist, represented by the binaryvector ui for respondent i . Using the multivariate binomial (MVB)reparameterization of the multinomial distribution (Teugels, 1990),let

ui ∼ MVB(1,π,Ω,θ), (2)

whereE(u) = π, (3)

var(u) = Ω, (4)

and θ represents all higher-order marginals (which will be ignoredhenceforth).




An Obvious Lemma

LemmaFor g indexing group (treatment or control), let

πg =

∑ngi=1 ui

ng. (5)

Then, by the multivariate central limit theorem,

πgD→ NJ

(π,

1

ngΩ

). (6)

Remark: Convergence is quite good so long as πj ∈ [.25, .75], justas for univariate sample proportions.




What Is Observed?

In a list experiment we do not get to observe ui . Instead weobserve

yi

= q′iui , (7)

where

qi =

1, item j is given to respondent i

0, item j is not given to respondent i .(8)

I A key assumption: Not being exposed to an item hasnegligible impact on other item responses!

I For SD, there are only two q: qT = (1, 1, . . . , 1)′

andqC = (0, 1, . . . , 1)

′.




Asymptotic Distribution of π1

Based on the lemma from before and the fact that linearcombinations of normals are normal,

yg

D→ N

(q

′gπ,

1

ngq

′gΩqg

). (9)

From this, we have

π1 = yT− y

C

D→ N

(π1,

1

nTq

′TΩqT +

1

nCq

′CΩqC

). (10)




var (π1) Again

Insight into the structure of the problem is gained by looking atvar (π1) more deeply. Partition

Ω =

(ω11 ω

′1D

ω1D ΩDD

). (11)

For the SD design,

var (π1) =1

nT

(ω11 + 21

′Jω1D + 1

′J−1ΩDD1J−1

)+

1

nC1

′J−1ΩDD1J−1.

(12)Notice that ΩDD enters twice. It is therefore desirable for this tobe as sparse as possible.




What Do We Know About Ω?

Given how important the unknown covariance matrix of the itemsis, it would be nice to know what is known about it. In short: Notmuch. We can directly compute s2

T and s2C but, in addition,

s2T = ω11 + 21

′Jω1D + 1

′J−1ΩDD1J−1 (13)

and

s2C =

1

nC1

′J−1ΩDD1J−1. (14)

Also, because the underlying items are binary, we have ofω11 = π1(1− π1). Thus we have (generally poor) estimates ofsums of the blocks of the covariance matrix and no informationabout the higher order margins. That’s it!




Results I Don’t Have Time to Discuss in Detail

There is not enough time to discuss these results in detail, but Iwill mention something about:

I Regressing on distractors.

I Consistency of designs.

I Alternative designs.




Regressing on Distractor Covariates

Assume in addition to yi

we also obtain zi of covariates from therespondents designed to predict the responses to the distractors.

I We cannot assume perfect consistency of response betweenformats but instead regress on zi .

I This leads to an unbiased estimator π1|Z for whichvar(π1|Z) ≤ var(π1).

I The process works primarily by annihilating ΩDD .

I Correlation between actual distractors and the covariatesneeds to be relatively high (≥ 0.5). These could simply be thedistractors given as single item responses elsewhere in thesurvey.




Consistency Condition

TheoremIf e1 ∈ col(Q), i.e., ∃c : Qc = e1, then the design is consistent, i.e.,

π1P→ π1 (15)

I c is the contrast vector necessary to find π1.

I It is found easily using the singular value decomposition of Q.




Alternative Designs: Circulant Deletion

The circulant family is built from binary circulant matrices. Themost notable is balanced deletion (BD), which has matrices of theform

Q =

0 1 11 0 11 1 0

. (16)

Because col(Q) = I, BD has the unusual property of being able toestimate the entire π. This may be useful if separate items areadministered to test assumptions about the latent checklist.




Alternative Designs: Simple Substitution

The simple substitution design has two groups with matrices of theform

Q =

1 00 11 1

. (17)

This design is inconsistent because e1 6∈ col(Q)! Alas it has beenused in practice and partly explains the baffling results sometimesobtained.




Examining Simple Deletion with Equal and Optimal GroupSizes

3 4 5 6 7

05

1015

J

RE

SD Equal Groups Cov WorstSD Opt. Groups Cov WorstSD Equal Groups Cov BestSD Opt. Groups Cov BestRR WarnerRR InnocuousSingle Item

Figure: This graph compares different scenarios for the SD design.ΩBest = I/4 and ΩWorst = 1J1

′

J/4. Optimal group sizes are determinedby solving a quadratic optimization problem to minimize var(π1). Notethat optimization does very little in this case. The relative efficiencies forthe classic Warner design, the innocuous question design and single itemare presented. All assume that π1 = 0.5.




Examining Balanced Deletion with Equal and OptimalGroup Sizes

3 4 5 6 7

05

1015

J

RE

BD Equal Groups Cov WorstBD Opt. Groups Cov WorstBD Equal Groups Cov BestBD Opt. Groups Cov BestRR WarnerRR InnocuousSingle Item

Figure: This graph compares different scenarios for the BD design.ΩBest = I/4 and ΩWorst = 1J1

′

J/4. Optimal group sizes are determinedby solving a quadratic optimization problem to minimize var(π1). Notethat optimization is very helpful in this case. The relative efficiencies forthe classic Warner design, the innocuous question design and single itemare presented. All assume that π1 = 0.5.




Comparing Simple and Balanced Deletion with OptimalGroup Sizes

3 4 5 6 7

05

1015

J

RE

SD Opt. Groups Cov WorstSD Opt. Groups Cov BestBD Opt. Groups Cov WorstBD Opt. Groups Cov BestRR WarnerRR InnocuousSingle Item

Figure: This graph compares the SD and BD designs, assuming optimalgroup sizes over the previously specified assumed Ω values.




Experiment Outline

I 237 participants from a large Midwestern universitydepartmental subject pool. 53% female with median age 19(IQR = 1).

I Experiment done using MediaLab software, which imposedcertain limitations on what could be done.

I Two questions, immigration and cheating, administered usingSD4 and SD5, respectively:

I Randomized into treatment and control groups by experiment.These were roughly equal in size.

I Hotelling’s T 2 on standard demographic characteristicssuggest no important differences between treatment andcontrol groups, i.e, randomization worked.

I All items within an experiment were randomized.I Covariates for the distractors (and hot item for immigration)

were administered much later in the experiment as directquestions.Jay Verkuilen The City University of New York Graduate Center [email protected] Presented at Fordham University, Department of Psychology Psychometrics Colloquium, January 20, 2010List Experiment



Immigration Question

I Experiment exactly as shown before.I Results (coded in direction of opposition):

I Direct questioning has π1 = 0.27, SE(π1) = 0.029.I Indirect questioning has π1 = 0.53, SE(π1) = 0.082.I Social desirability gap consistent with those found previously

by Janus.I Regression on distractors didn’t really change anything here.

However, examining the covariance matrix of the directquestions show that the items making up the Janusimmigration question are reasonably uncorrelated.




Cheating Question

This is a new item constructed based on a pilot study. Wepreviously surveyed the subject pool for topics they and collegestudents in general would find uncomfortable. The distractors werechosen to have low social desirability whereas the hot item waschosen to be likely to have high social desirability.

1. Cheated on an examination or paper (hot item)

2. Sorted garbage in order to recycle plastics, paper, and glass

3. Consumed more than four alcoholic drinks in an evening

4. Eat four or more meals per week at a “fast food” chain likeMcDonald’s

5. Routinely wash your hands after using the bathroom




Cheating (cont.)

I Note: Due to IRB concerns we were not able to ask the hotitem directly.

I Results:I Indirect questioning has π1 = 0.21, SE(π1) = 0.12.I Upon regression on distractors, π1 = 0.27, SE(π1) = 0.12. All

distractors appear to help here. The efficiency gain appears tobe about 6% (it is masked in the rounding). Not a lot, but it’sessentially free. . . .

I Regressing on other covariates seems to be even morebeneficial but due to some missing data problems, we don’tshow them here.




Why Bias Matters

The Janus immigration question seems to suggest that socialdesirability bias is around 0.25 (or so) and we know estimatorvariances for direct and indirect questioning. As usual, quantify theestimator’s performance as mean squared error:

MSE(π) = var(π) + bias2(π). (18)

I Direct: MSE = 0.0292 + 0.252 = 0.063. Note that estimatorvariance makes up a negligible component of the MSE.

I Indirect: MSE = 0.0822 + β2 = 0.0067 + β2.

I Depending on what you assume about β, which you choosewill depend. However, for a bias this large, ICT almost alwaysbeats direct questioning. At minimum, you shouldn’t just holdyour nose in the presence of unknown bias.




Summary

I The LE can—given some partly testable assumptions—providea means to study attitudes, behaviors, preferences, etc.,subject to social desirability effects in a less obtrusive mannerthan for RR.

I Analysis is fairly straightforward and requires nothing morethan ordinary regression.

I Regression main effects help tighten the standard error for thehot item.

I To examine group differences for the hot item (say by gender),one needs interactions.

I Regression on distractor proxies is attractive for “peekingbehind the probabilistic fig leaf.”

I Key is to get good distractors, namely ones that are:I Likely to have univariate marginals near 0.5.I Likely to be independent of each other.




Limitations

I Efficiency loss can be substantial, but optimizing group sizesand regression on distractors helps (sometimes).

I Least squares estimation can be problematic for relativelysmall proportions and in small samples.

I Bayesian estimation using an informative prior (or constrainedestimation) restricting π1 ∈ [0, 1] would prevent inadmissibility(if in a Procrustean manner).

I Some kind of ML model would be nice. Unfortunately thedistribution of y is far from tractable, except under strongassumptions. Maybe an overdispersed binomial?

I Tsushiya, Hirai and Ono (2007) discuss some additionalestimators.

I We don’t understand the psychological aspects very well.Some psychological experimentation by an intrepid socialcognition researcher would definitely be in order to test them.




References

1. Droitcour, J., Caspar, R.A., Hubbard, M.L., Parsley, T.A., Visscher, W., & Ezzati, T.M. (1991). The itemcount technique as a method of indirect questioning: A review of its development and a case studyapplication. In ed. P.P. Biemer et al., (eds.) Measurement Errors in Surveys, pp. 185-210. New York:John Wiley.

2. Gilens, M., Sniderman, P.M., & Kuklinski, J.H. (1998). Affirmative action and the politics of realignment.British Journal of Political Science, 28, 159-183.

3. Hemelrijk, J. (1966). Underlining random variables. Statistica Neerlandica, 20, 1-7.

4. Janus, A.L. (2008). The list experiment as an unobtrusive measure of attitudes toward immigration.Department of Sociology, University of California-Berkeley.

5. Kuklinski, J.H., Cobb, M.D., & Gilens, M. (1997). Racial Attitudes and the New South. The Journal ofPolitics, 59, 323-349.

6. Kuklinski, J.H., Sniderman, P.M., Knight, K., Piazza, T., Tetlock, P.E., Lawrence, G. R., & Mellers, B.(1997). Racial prejudice and attitudes toward affirmative action. American Journal of Political Science, 41,402-419.

7. Lensvelt-Mulders, G.J.L.M., Hox, J.J., van der Heijden, P.G.M. & Maas, C.J.M. (2005). Meta-analysis ofrandomized response research: 35 years of validation studies. Sociological Methods and Research, 33,319-348.

8. Tsuchiya, T., Hirai, Y., & Ono, S. (2007). A study of the properties of the item count technique. PublicOpinion Quarterly, 71, 253-272.

9. Teugels, J.L. (1990). Some representations of the multivariate Bernoulli and binomial distributions.Journal of Multivariate Analysis, 32, 256-268.

10. Warner, S.L. (1965). Randomized response: A survey technique for eliminating evasive answer bias.Journal of the American Statistical Association, 60, 63-69.


list experiment talk no pauses

Education

fordham university

environmentjay verkuilen

control items

britney spears new album

distractor items

presentation of items

main problems

control group