mixing modes

An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB

Issues of Coverage, Sampling and Participation in Mixed Mode Surveys

Peter Lynn, University of Essex

6th ESRC Research Methods FestivalOxford, 08-07-2014

Mixing Modes

“mixing modes gives an opportunity to compensate for the weaknesses of each individual mode at affordable cost”

- de Leeuw (2005)

Involves an explicit trade-off between costs and (multiple sources of) survey errors

Recent interest in mixed modes is particularly stimulated by the (possibly false) notion that the marginal cost of data collection by web is close to zero

Lynn | ESRC RMF, 8 July 2014

A Distinction

• Multi-mode (or multiple mode) data collection: Different modes used for different survey items, e.g.- CASI component within a CAPI survey;- CATI follow-up to a mail questionnaire;- etc

• Mixed mode data collection: The same survey items can be collected by different modes for different sample members:- sequential;- concurrent selective;- concurrent with respondent choice; etc


Motivation for Mixing Modes

Motivation for mixed modes rather than a single mode:• Cost reduction; or• Coverage/ participation enhancement.(Designs to achieve both simultaneously are proving elusive.)Different motivations tend to suggest rather different designs:• Combinations of modes;• Sequence of modes (or non-sequential).The coverage, sampling and participation issues may differ

between these types of design


Sequential and Concurrent Mixed Mode DesignsSequential design:• Use a number of modes in sequence;• Get as much response as possible in one mode, before trying remaining

non-respondents in the next mode• Cheaper modes first if motivation is to reduce costs• Higher response rate modes first if motivation is to maximise participation

Concurrent selective design:• Offer a different mode to each of 2 or more subsets of sample members

Concurrent elective design:• Offer each sample member a choice of mode

Combination: e.g. Concurrent selective sequential design


Sequential Design: Example

Phase 1:• Mail an invite to a web survey to all sample members;• Mail a reminder to those who have not responded after a week or two

Phase 2:• Mail a paper self-completion q’re to those who have still not responded

after a further week (or include this with the reminder above)

Phase 3:• Approach for a face-to-face interview those who have still not responded

after a further periodKey design choices: Which modes? Which order? Criterion for switching to next phase?


Coverage Issues with Mixed Mode Surveys IMixed modes (of approach) can help address frame quality

problems. Examples:• Inconsistent contact details on sampling frame: only address for

some people, only email for others, etc• Dual-frame approaches: good coverage is provided only by the

union of multiple frames, which have different contact details

Note: Mixed modes of approach need not necessarily imply mixed mode data collection


Inconsistent Contact Details: ExampleDutch GPSs (e.g. Labour Force Survey, ESS experiment)• Select addresses from Postal Address Register• Match phone numbers to addresses (via names)• 70% match successfully: Can approach by phone• 30% do not match: First approach face-to-face• Some of the addresses with phone numbers also require face-to-face

follow-up


Dual-Frame: Example

• RDD or list-assisted sampling to generate sample of phone numbers:- These numbers are screened to identify households (with phones)

• Supplementary sample of addresses:- These addresses are screened (face-to-face) to identify households with no (landline) phone

• The two samples combined give good coverage


Coverage Issues with Mixed Mode Surveys IICoverage issues can introduce constraints on mixed mode data

collection. Example:• Desire is a sequential web → face-to-face design• Not all sample members are web users• Non-web users must skip the web phase, either explicitly or

implicitly


Coverage Issues with Web as a Primary ModeA. Restrict survey to web users:• Obvious cost advantages;• Non-random under-coverage: requires evaluation and adjustment.Or B. Include non-web users in web mode:• Requires provision of hardware, software and training;• Various models, e.g. LISS, KnowledgePanel, GIP, ELIPSS.Or C. Include non-web users in a different mode…


Coverage Issues with Web as a Primary Mode, ctd.Option C, including non-web users in a different mode:• May have cost advantages compared to providing equipment –

depends on frequency/nature of data collection, etc;• May have measurement disadvantages (see next two presentations!)Various designs possible:• Web + mail, based on mail-only approach;• Interviewer-administered recruitment, followed by web+mail,

web+phone, or other mixes;• More options are feasible in longitudinal context.Examples: GESIS panel and Gallup panel are both web+mail


Sampling Issues with Single Mode WebNo general population frames with email addresses, therefore

first approach must be by a different mode.For frames without names, mail approach requires self-

administered respondent selection:• This is error-prone if paper-based and may cause dropout if

web-based.Interviewer-administration preferred but costly.Alternatively, use non-probability recruitment methods (opt-in

panels) and model-based inferential paradigm.


Sampling Issues with Mixed Mode including WebFrames with partial information can be used (e.g. email

addresses for a subset) – but none yet in UK.Instead, single-mode initial approach needed:• Mail approach, with web+mail data collection, may offer low-

cost solution of reasonable quality in some situations;• In UK, this design may work with named-person frames such

as admin records, but is hampered by the need for respondent selection for general population surveys;

• Interviewer approach and respondent selection may be preferable, but only cost-effective for longitudinal surveys.


Respondent Selection

Next / last birthday method:• European Social Survey (UK) Experiment (Villar, 2013)• Selection compared to birth date info from questionnaire;• Approx. 50% correct, 20% incorrect , 30% uncertain• (amongst households with 2+ adults)

Household roster / grid method:• Community Life Survey (Williams 2013)• Approx. 25% incorrect selections

→ Difficult / impossible to control who completes a self-completion questionnaire;

→ Chance of incorrect respondent may be greater with web


Participation in Mixed Mode SurveysResponse rates in single-mode surveys:

Face-to-face > telephone > self-completion – typically

Amongst self-completion surveys:Mail > web – often, but not always

Composition of response:Broadly similar between modes – typicallyBut some differences


Participation in Mixed Mode Surveys

Population

FTF Response

Web Response


Participation in Mixed Mode Surveys


Population

Mixed Modes Response?

Increasing Response Rates

• Success requires:- All (or most) people who would have responded in mode 1 continue to respond;- Additional people respond too (in mode 2)

• But it is generally the case that:- (Average) response propensity declines FTF → Phone → Mail → Web- Data collection costs decline in same order- Refusal in one mode reduces propensity to respond in next mode


Examples of Increasing Response Rates• Examples are (all?) from surveys that use a subset of the

sequence FTF – Phone – Mail – Web

• And even then, it is also necessary to “exhaust” each mode

• E.g. 1 British Household Panel Survey: FTF → phone

- Approx. 93% response FTF alone; 96% FTF + phone (amongst previous wave respondents)


Examples of Increasing Response Rates• Examples are (all?) from surveys that use a subset of the

sequence FTF – Phone – Mail – Web

• And even then, it is also necessary to “exhaust” each mode

• E.g. 1 British Household Panel Survey: FTF → phone

- Approx. 93% response FTF alone; 96% FTF + phone (amongst previous wave respondents)

• E.g. 2 British Crime Survey 2011 Follow-Up: Mail → web

- 60% response mail alone; 63% mail + web (amongst BCS respondents)


Examples of Failing to Increase Response Rates• All (?) surveys that use a sequence which is not a subset of

FTF – Phone – Mail – Web

• E.g. 1 UKHLS-IP 2009 wave 2: Phone → FTF- 76% response FTF alone; 67% Phone + FTF (w1 resps)





• E.g. 2 UKHLS-IP 2012 wave 5: Web → FTF - 84% response FTF alone; 79% Web + FTF (w4 resps)







• E.g. 4 NL-ESS 2009: Web → Phone/FTF (new sample)- 52% response FTF alone; 46% Web + Phone/FTF


Non-Response Bias

• Most (of the few) studies to date either found no effect or a modest (assumed) positive effect

• Hope is that web may disproportionately add young, full-time employed, busy people – who are generally under-represented in surveys

• But very little evidence either way on this point:- NL-ESS 2009 found almost identical sample composition in the FTF-only and Web + Phone/FTF samples (age, employment, education, etc)- UKHLS 2012 too found no significant differences between FTF-only and Web+FTF (age, gender, household type, etc)


Mixed mode including Web: UK ExamplesTwo cross-sectional surveys:

Community Life Survey (Williams 2013):• Random sample of addresses from PAF• Advance letter → Mail invitation → Mail reminder → Mail questionnaire• 16% responded online + 11% mail = 27% response (no incentive)• 19% responded online + 12% mail = 31% response (£5 conditional)• 22% responded online + 13% mail = 35% response (£10 conditional)• 25% responded online + 14% mail = 39% response (£5 unconditional)

European Social Survey (UK) Experiment (Villar, 2013):• Random sample of addresses from PAF• Advance letter → Mail invitation → Mail reminder → Face-to-face fieldwork• 21% responded online + 18% face-to-face = 39% response


Mixed mode including Web: Longitudinal ExamplesBritish Crime Survey re-contact study (Fong & Williams 2011):• Issued sample = 30% of BCS respondents who gave an email address;• Email invitation → Email reminder → Postal q’re → Postal reminder • 35% responded online + 27% by mail = 62% response


Web+Interviews: Examples1958 Birth Cohort (Brown et al 2014):• 9th wave (age 55) – 5 years after previous wave• Sequential web → telephone• 62% responded online + 21% phone = 83% responseUnderstanding Society (Jäckle et al, 2013):• Issued sample = wave 5/wave 2 of a household panel• Mail(+email) invitation → (email reminders) → Mail reminder → Face-to-face• 21% responded online + 55% face-to-face = 76% response (w5 sample)• 30% responded online + 50% face-to-face = 80% response (w2 sample;

higher incentives)• 23% responded online + 55% face-to-face = 78% response (w5 sample, £10)• 23% responded online + 51% face-to-face = 74% response (w2 sample, £10)


Conclusions

Potential cost savings from mixed mode may erode if we:- Aim for full population coverage- Aim for response at least as high as could be achieved with

single-mode

Participation advantages of mixed mode likely to be costly

Most promising cost-quality trade-off in the UK currently may be:- Web → Mail → Face-to-face

But … measurement concerns (next presentations!)


An initiative by the Economic and Social Research Council, with scientific leadership by the Institute for Social and Economic Research, University of Essex, and survey delivery by NatCen Social Research and TNS BMRB

Issues of Coverage, Sampling and Participation in Mixed Mode Surveys

Peter Lynn, University of Essex

6th ESRC Research Methods FestivalOxford, 08-07-2014

mixing modes

Documents

different modes

modecheaper modes

number of modes

single mode

individual mode

mixed mode surveys peter

multiple mode data collection

types of design lynn