estimating measurement effects of survey modes from
TRANSCRIPT
Estimating Measurement Effects ofSurvey Modes from Between- andWithin-Subject Designs
Thomas KlauschJoop HoxBarry Schouten
Presentation at the 68th annual conference of the AAPORMay 16-19 2013, Boston, MA
Department of Methodology and Statistics
7 augustus 2013
1ViaInvoegen|KoptekstenVoettekstinvoegenSubafdeling<2spaties>|<2spaties>Titelvandepresentatie
Common Mixed-Mode Redesign Options
7 augustus 2013 2Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Mode A
1. Single-to-Single Mode
Mode B
Mode A
2. Single-to-Mixed Mode
B A
3. Single-to-Mixed Mode
Mode A
A B
Redesigns require the same expected answer under modes A and B
1. For respondents under mode B2. For respondents under mode B3. For respondents under mode A
Estimate average difference in answers (measurement effects)
How to do this in the presence of nonresponse in both modes?
Between-Subject Mode Experiments
7 augustus 2013 3Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Marginal Measurement Effect:= ( ) − ( )PotentialOutcomes
Between-Subject Mode Experiments
7 augustus 2013 4Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Marginal Measurement Effect:= ( ) − ( )Response
Nonresponse
Response
Nonresponse
Between-Subject Mode Experiments
7 augustus 2013 5Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Marginal Measurement Effect:= ( ) − ( )Response(S=1)
Nonresponse(S=0) Conditional Measurement Effects
for respondents in mode a or b:
= | = 1 − | = 1= | = 1 − | = 1
Use of Cond. MEs in Mixed-Mode Redesign
7 augustus 2013 6Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Mode A
1. Single-to-Single Mode
Mode B
Mode A
2. Single-to-Mixed Mode
B A
3. Single-to-Mixed Mode
Mode A
A B
It can be shown that, if:
Redesigns 1 and 2 are possible
Redesign 3 is possible
Because then there is no relative measurement bias between modes(respondents give in expectation the same answers)
Marginal MEs not needed for these decisions
0bRME
0aRME
Estimation in Between-Subject Designs
7 augustus 2013 7Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Exploit information on ‘X’, available for allunits (frame information)
Matching, regression or weightingtechniques can be used
Available X normally weakly associatedwith Ys or response mechanisms (S), e.g.socio-demographics
Thus: implausible MAR assumptions
Estimates then underlie selection bias(aka ‘selection effects’)
We suggest a new approach allowing forweaker assumptions when estimatingcond. MEs
Within-Subject Designs
Within-Subject Mode Experiments
7 augustus 2013 8Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Time
Observed ‘Potential’ Outcome Strong relation of to Strong auxiliary information
Occasion twoSwitch to mode a
Forward Estimation of MEs for respondents in B
7 augustus 2013 9Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Time
The forward method toestimate
MAR of at occasion 2 onearlier Assumes is time-stable ⊥ | , , = 1
Backward Estimation of MEs for respondents in A
7 augustus 2013 10Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Time
The backward method toestimate
MAR of at occasion 1 onrepeated Assumes and aretime-stable ⊥ | , , = 1
Within-Subject Design with Control Group
7 augustus 2013 11Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Time
Time
No mode switch Adjust time related
change of Y and S in‘treatment’ condition
Assumes that change isequivalent in mode A andB (plausible)
Control
‘Treatment’
Example: Quality of Social Life (Index)
Data collection Within-subject mode experiment with control group
Telephone, Mail, Web (1st occasion) followed up by F2F (2nd) Additionally F2F at 1st occasion (control)
Statistics Netherlands in 2011
Crime Victimization Survey, QSL index central variable in reporting
Estimation Regression Estimation (aka GREG)
Other methods possible, e.g. propensity score methods
Standard errors using 1000 bootstrap samples
7 augustus 2013 12Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Missing Data Pattern for F2F (a) and Mail (b)
7 augustus 2013 13Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Control
‘Treatment’
ME estimates with bootstrapped CI’s
7 augustus 2013 14Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Within-Subject Within-Subject-Control-Group
Est. 95% CI p (Adj.)Est.
95% CI p
-.766 [-.925,-.608] <.001 -.729 [-.928,-.531] <.001
-.434 [-.556,-.314] <.001 -.489 [-.680,-.302] <.001
-.444 [-.554,-.336] <.001 -.389 [-.525,-.255] <.001( ) -.332 [-.185,-.479] <.001 -.240 [-.068,-.409] <.01( ) -.323 [-.178,-.469] <.001 -.340 [-.134,-.547] <.01∆ - - - .055 [-.025,.135] .179∆ - - - .093 [.008,.180] .033
Use of Cond. MEs in Mixed-Mode Redesign
7 augustus 2013 15Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Mode A
1. Single-to-Single Mode
Mode B
Mode A
2. Single-to-Mixed Mode
B A
3. Single-to-Mixed Mode
Mode A
A B
Example: Cond. Measurement Effects identified For this variable: redesigns are not possible (incomparable answers)
Advisable to redesign questions to unified mode design, if possible(Dillman et al., 2009)
Otherwise a correction method is needed (will be possible!)
Conclusion
New technique to estimate measurement effects (aka mode effects)using within-subject designs
Weaker assumptions than estimation using socio-demographicauxiliary information in between designs
Better estimates, useful for effective mixed-mode design
Could be particularly useful in (mixed mode) panel settings
We can adjust for time-related change using control groups
Within subject design: no extra cost (compared to between design)
Control group extension: extra cost
Based on our model, adjustment of measurement effects will becomepossible
7 augustus 2013 16Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
ViaInvoegen|KoptekstenVoettekstinvoegenSubafdeling<2spaties>|<2spaties>Titelvandepresentatie
17
Appendix
7 augustus 2013
MAR Assumptions of Between-Designs
7 augustus 2013 18Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
Marginal Measurement Effect:
Conditional Measurement Effects:
⊥ |⊥ |⊥ | , = 1⊥ | , = 1:
:
Available X are seldom closelyrelated to Y or S
Illustration of Interplay of all Effects
7 augustus 2013 19Klausch, Hox, & Schouten | Estimating Measurement Effects in Within-Subject Designs
| = 1
| = 1