overview: online surveys vasja vehovar university of ljubljana, slovenia handbook of online...

Overview: Online Surveys

Vasja VehovarUniversity of Ljubljana, Slovenia

http://WebSM.org

Handbook of Online Research Methods Colloquium 28-29 March 2007

• Computer-assisted survey information collection

• Key methodological issues

• Related issues

• Applications

Structure

Introduction

1930s: applications of probability sampling

1960s: expansion of telephone surveys

1970s: computer technology appears in surveys

1980s: computer-assisted surveying

1990s: Internet mediated surveys

Technology and surveys

Computer-assisted survey information collection

• CATI – Computer-assisted telephone interviewing

• CAPI – Computer-assisted personal interviewing

• CASI – Computer-assisted self-interviewing

Early CASIC interviewer assisted modes

Computer-assisted survey information collection

• Reduced time and costs for data input

• Elimination of errors during data transcription

• Implementation of advanced features:

automatic skips and branching

randomization of questions and response options

control of answers

inclusion of multimedia elements…

CASIC benefits

Computer-assisted survey information collection

Different modes of CSAQ:

disk-by-mail

touch-tone data entry (TDE)

interactive voice response (IVR)

e-mail surveys

web surveys

Computerized self-administered questionnaires (CSAQ)

Computer-assisted survey information collection

Computer-assisted survey information collection

Web surveys

Internet surveys

Online, Internet, Web, CASIC, CSAQ, CADAC

Computer-assisted survey information collection

Web surveys

Internet surveys

CSAQ

Online, Internet, Web, CASIC, CSAQ, CADAC

Computer-assisted survey information collection

Web surveys

Internet surveys

CSAQ

CADAC

Online, Internet, Web, CASIC, CSAQ, CADAC

Interviewer involvement

Survey mode

Paper and pencil CASIC

Interviewer presence

Paper and pencil (face-to-face) interviewing (PAPI)

CAPI, CASI, Audio/Video CASI

Remote interviewer

Paper assisted telephone interviewing (PATI)

CATI, CAVI (computer assisted video interviewing)

No interviewerSelf-administered paper

questionnaires (mail questionnaires)

CSAQ telesurveys (web CSAQ, TDE, IVR, Virtual interviewer,...)

Interviewer-less and paper-less surveys

Computer-assisted survey information collection

Interviewer involvement

Survey mode

Paper and pencil CASIC

Interviewer presence

Paper and pencil (face-to-face) interviewing (PAPI)

CAPI, CASI, Audio/Video CASI

Remote interviewer

Paper assisted telephone interviewing (PATI)

CATI, CAVI (computer assisted video interviewing)

No interviewerSelf-administered paper

questionnaires (mail questionnaires)

CSAQ telesurveys (web CSAQ, TDE, IVR, Virtual interviewer,...)

Interviewer-less and paper-less data collection

Computer-assisted survey information collection

Asking  question(INPUT)

Recording of responses (OUTPUT)

Manual recording Automatic voice recognition

Written questions

Standard CSAQ (PC, TV, PDA,

Mobile,...)Visual CSAQ with AVR

Audio questionsVideo CSAQ, Audio

CSAQ, TDE

IVR, Video IVR,TTS with AVR, Virtual

interview

CSAQ input-output technology

Computer-assisted survey information collection

Asking  question(INPUT)

Recording of responses (OUTPUT)

Manual recording Automatic voice recognition

Written questions

Standard CSAQ (PC, TV, PDA,

Mobile,...)Visual CSAQ with AVR

Audio questionsVideo CSAQ, Audio

CSAQ, TDE

IVR, Video IVR,TTS with AVR, Virtual

interview

CSAQ input-output technology

Computer-assisted survey information collection

• Developments of the Internet and related technologies

• Importance of broadband Internet access

• Possibilities of distribution across various platforms and devices

• Fast and easy implementation using specialized software tools

Technological aspects

Computer-assisted survey information collection

• Two major problems of probability samples in Internet surveys:

non-coverage

sampling frame problems

• Image of Internet surveys as inherently non-probability ones

1. Probability vs. non-probability sampling

Key methodological issues

Issue of a statistical inference:probability vs. non-probability samples

Internet survey

Information-communication technologies

Mode of survey data collection

Management of the survey process

The spurious link

Key methodological issues

Samplingtype

SolicitationType of invitation

Personal invitation General invitation

Probability samples

Samples from closed population with email,

Pre-recruited with mail, telephone, f2f

Web intercept surveys

Non-probability samples

Self-recruited Internet panels,

Opt-in lists of e-mail adresses

General banner-ad invitation

2. Sampling and invitation

Key methodological issues

• Generally low and variable response rates to web surveys

• Complexity of the response process

• Measures to increase response rates:

incentives (e.g. using PayPal)

multiple follow-up contacts

other motivations

3. Non-response

Key methodological issues

Target population

Operational population

Frame population

Sampled units

Absorbed units

Complete respondents

Partial respondents

Starting units

Clickers

Aware units

Solicitation

Informed units

Involvement process

Key methodological issues

Respondent’s characteristics

Social environment

Researcher-respondent interaction

Survey design

Technological environment

No researcher’s control

Partially under researcher’s control

Participation in web surveys

Key methodological issues

• Question types and visual elements

• Advanced features of computerized questionnaires

• Inclusion of multimedia

• Problems and considerations:

standardization

impact on respondent’s answers

technical problems

measurement error

4. Questionnaire design

Key methodological issues

• Commonly utilized for correcting:

deviations from probability selections

non-coverage and non-response problems

• Importance for non-probability Internet surveys

• Promising approaches:

calibration methods

propensity score weighting

modelling and causal analysis

5. Post-survey adjustments

Key methodological issues

• The managerial process of survey implementation:

costs optimization

data quality

relations between data quality and costs

• Cost effectiveness of web surveys:

costs for increasing response rates

evaluations of costs in relation to errors

1. Costs, errors and management

Related issues

• Web surveys vs. other modes:

lower response rates (Lozar Manfreda et. al., 2007)

comparable or higher level of validity and reliability (e.g. Fricker et al., 2005; Chang & Krosnick, 2002)

2. Survey mode

Related issues

Contact with the respondent

personal, telephone, e-mail, mail,…

Interview

PAPI, CAPI, CATI, CASI,…

Centralized data

management

Solicitation

Mixing survey modes

Related issues

Response rates and behaviour

Modelling mixed modes

Data quality requirements

Willingness to participate

Structure of the data (MAR)

Cost models and impact of environment

Optimal combination

Selecting the right mixture

Related issues

Programing of the questionaire

Related issues

User-friendly interface

Standardized questionnaire

description in QML

Paper questionnaire

CAPICATICASI

Internet survey

• Quantitative and qualitative research methods

• Flexible combinations of approaches (Morgan, 1998):

qualitative preliminary

quantitative preliminary

qualitative follow-up

quantitative follow-up

• New possibilities with the Internet

3. Mixing research methods

Related issues

Qualitative method

Quantitative method

Analysis

Beginning

Conclusion

Flexible on-line mixed methods (FOMM)

Related issues

• New ethical dilemmas in survey research:

solicitation process, spam and privacy

data protection

children and minors

combining data…

• Development of new standards (e.g. ESOMAR, AoIR, MRA…)

4. Ethics, guides and standards

Related issues

• Different applications according to:

type (internet access panels, on-line voting, intercept…)

organizations (official, academic, commercial)

topics (marketing research, polling opinions, health…)

complexity (simple one-shot surveys vs. complex integrated data collection)

Applications

Current and future applications

psychological research

online experiments

paradata and online measurement

specific research fields (usability research, HCI...)

emerging areas (e-learning, e-government…)

Related areas

Current and future applications

• Expansion of CASIC modes, options and mixtures

• Integration of devices (TV, mobile,..)

• Speech recognition and touch-screens

• Multimedia and interactivity

• Integration with other CADAC (GPS, paradata, observations,...)

• Specialisation and professionalisation

Trends

Current and future applications