multidirectional survey measurement errors: the latent class mtmm model

Multidirectional errors Latent class MTMM Conclusions

Multidirectional survey measurement errors:the latent class MTMM model

Daniel Oberski / doberski@uvt.nl

Department of methodology & statisticsAAPOR 2015

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

..1 Multidirectional survey measurement errors

..2 Latent class multitrait-multimethod model

..3 Conclusions

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errorsTwo respondents who:

• Went to the doctor the same number oftimes

• Have the same opinion about the role ofwomen in society

give different answers to these questions.

This will bias estimates of relationshipsbetween the variables.

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered

• But latent variable models can recover theamount of influence they exert

• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert

• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors• Latent variables can never be recovered• But latent variable models can recover the

amount of influence they exert• This is useful to remove bias

• Multitrait-multimethod(MTMM)approach to estimating this influence(Andrews 1984; Saris & Andrews 1991; Saris &Gallhofer 2007);

• Quasi-simplex approach (Wiley & Wiley 1970;Alwin 2007).

↑ Linearmodels

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Stochastic errors in the literature

• Random mistakes• Acquiescence• Answering in the socially desirable direction• Tending to choose the first/last of several categories• Extreme response (outer categories)• Avoiding some particular category for whatever reason• Preferring the midpoint• Heaping (rounding)• ...

Problem: Not all of these are linearStochastic errors are multidirectional

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Example of the effect of a nonlinear error on estimate ofrelationships

True relationship1 2 3 4

1 136 112 91 752 112 75 50 343 91 50 28 154 75 34 15 7

Polychoric correlation = -0.25

Relationship with ERS1 2 3 4

1 374 41 34 2052 41 4 3 123 34 3 1 64 205 12 6 19

Polychoric correlation = -0.43

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Problem: Stochastic errors can strongly bias relationshipestimates;

Problem: Linear latent variable models assume errors are allone-way, so bias is not appropriately removed.

Solution: Latent class models to allow for multidirectional errors.Here MTMM but could also be quasi-simplex

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

The latent class MTMM modelOberski, Hagenaars & Saris, to appear in Psychological Methods.

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

M1 M2 M3

T1 T2 T3

y11 y21 y31 y12 y22 y32 y13 y23 y33

• Latent variables are discrete (categorical)• Observed may be continuous or discrete• Relationships (can be) nonparametric

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Experimental design: split-ballot MTMM

Method 1 Method 2 Method 3Random group 1 . .Random group 2 . .

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Opinion about the role of women experiment: Mainquestionnaire (first method)

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Opinion about the role of women experiment:Supplementary group 1 (second method)

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Opinion about the role of women experiment:Supplementary group 2 (third method)

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Effect of trait on item distribution, Greece

Men more right, positive agree-disgree

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Effect of trait on item distribution, Slovenia

Men more right, negative agree-disgree

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Method factors have a nonlinear influence on the items

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Cut down, method 1

Trait

Me

tho

d

1

234

5

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Cut down, method 2

Trait

Me

tho

d

1

2 34

5

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Cut down, method 3

Trait

Me

tho

d

1

23

4

5

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Prevalence of method behavior

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Conclusion

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Conclusion

• Stochastic errors are important when you are interested inrelationships;

• Need latent variable models to estimate their extent so theireffects can be removed statistically;

• In traditional MTMM and quasi-simplex models, these effectsare all one-way;

• Latent class models allow for multidirectional errors.• Example: the latentclassMTMM model

(Oberski et al., to appear - see http://daob.nl/publications)

Latent class MTMM Daniel Oberski / doberski@uvt.nl

Multidirectional errors Latent class MTMM Conclusions

Thank you for your attention!

doberski@uvt.nl@DanielOberski

See http://daob.nl/publications for preprints

Supported by Veni grant number 451-14-017 from the NetherlandsOrganization for Scientific Research (NWO).

Latent class MTMM Daniel Oberski / doberski@uvt.nl