1

Advantages and Disadvantages of Internet Survey Methods for Official

Statistics

Mick P. Couper, PhDSurvey Research Center, University of Michigan

andJoint Program in Survey Methodology

4th International Workshop on Internet Survey Methods

September 2012

2

Outline

Background: Innovation in survey research and the rise of Internet surveys

The use of Internet data collection in national statistical offices• Internet for surveys of households or people• Internet for surveys of establishments

Possible futures for the role of the Internet

Challenges and research opportunities

2

3

Background: Research Sectors

There are four main research sectors with interest in online survey methods:• Market/consumer research – includes advertising, product

research, customer satisfaction, etc.• Public opinion and political polling research• Academic research• Government research

Each sector has a different focus and use of online surveys• Adopting Internet surveys for different reasons • But there is a lot of overlap between sectors• Some types of surveys (e.g., customer satisfaction) are done

by all four sectorsMuch of the innovation and push for online surveys has come from the market research world; the other sectors have been more cautious

4

Research Sectors and Key Concerns

Market and consumer research

Public opinion and political

polling

Government

Speed, cost Speed, cost, accuracy

Accuracy, broad representation

Academic

Cost, complexity

3

5

Internet Survey Adoption in Different Sectors

Market researchers have fully embraced online survey methods, and were early adopters• See next slide

Public opinion research is using Internet methods alongside RDD telephone surveys

Academic researchers doing theory testing are using Internet surveys for cost reasons

Governments have been relatively slow to adopt• Greater focus on mixed-mode surveys than stand-

alone Internet surveys

6

Growth in Online Market Research in U.S. and Europe

0200400600800

10001200140016001800200022002400

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

Spen

ding

U.S. (millions $)Europe (millions €)

Source: Inside Research, February 2011

4

7

Competing Pressures

National Statistical Offices (NSOs) face a number of pressures:• Declining response rates and increasing costs of

traditional methods• General budgetary pressures – do more with less• Pressures to be modern and adapt to user

preferences (“digital government”)• Responsibility to collect data on all segments of

society (both the “haves” and “have not’s”)• Need for accuracy, reliability, and trustworthiness –

data serve as benchmarks and have vital policy implications

8

Competing Pressures (continued)

Tension between the need to adapt and innovate and the need to preserve comparability of data across time

Innovation tends to be slow and cautious, relative to other sectors• Innovation is rewarded in some sectors (competitive

edge, publications)• Risk of failure much higher in the government

sector, given relative scale of operations (e.g., censuses)

• Cost pressures not as great as other sectors

5

9

Reasons for Slow Adoption of Internet Surveys

Coverage concerns• Not everyone has Internet access• Those with access are different from those without (the digital

divide), and the differences are not disappearing• NSOs are particularly concerned about broad representation

Sampling concerns• There is no sampling frame of Internet users• There is no RDD-like method to sample e-mail addresses

Nonresponse concerns• Concern that Internet surveys may decrease response rates

and increase nonresponse bias

10

World Internet Penetration Over Time

6

11

Internet Penetration in Korea

44.7

56.659.4

65.5

72.7 73.578.1 78.8 81 81.6 83.7 83.8

0

10

20

30

40

50

60

70

80

90

100

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Pece

nt

Source: ITU World Telecommunication / ICT Indicators

12

Internet Survey Response Rates

Evidence from two meta-analysis that Internet response rates generally lower than comparable modes

Lozar Manfreda et al. (2008)• Meta-analysis of 45 experimental comparisons of Web to

other modes (mail, phone, IVR, etc.) in 24 papers• Web has an average 11% lower response rate than the

alternativeShih and Fan (2008)• Meta-analysis of 39 experimental comparisons of Web to mail• Web has an average 11% lower response rate than mail• No significant effect of publication year (i.e., the gap is not

closing over time)

Response rate charts

7

13

So What Are NSOs Doing?

Because of these concerns, greater focus on mixed-mode surveys involving Internet than on stand-alone Internet surveys

Different approaches for household surveys and establishment surveys• I focus primarily on household surveys, then offer

some observations on establishment surveys

14

Household Surveys

Different types of surveys • Censuses• One-time or repeated cross-sectional surveys• Panel surveys

Internet as replacement/supplement for different modes• Mail• Interviewer-administered (face-to-face or telephone)

8

15

Internet Use in Household/Population Censuses

Typically used as an option in a sequential mixed-mode design• Mail-out census form includes option for Internet

completion• Non-experimental design• Varying rates of success across countries

Zewoldi (2011) noted that more than 30 countries are providing or experimenting with an Internet option in the 2010-2011 round of censusesSelected examples follow

16

Census Experiences with Mail+Web

Switzerland• In 2000, about 4.2% of forms returned via Internet• Register-based census in 2010

Norway• In 2001, about 9.9% of responses via Internet• Register-based census in 2011

United Kingdom• In 2011, about 27% of households responded online

9

17

Czech Republic and Hungary 2011

25.1 19

62.2

15

12.7

66

0102030405060708090

100

Czech Republic Hungary

EnumeratorMailInternet

18

Census Experiences with Mail+Web (cont.)

Canada• In 2006, 18.3% of households completed the census

online• In 2011, 54.4% of households did so

United States 2000 and 2010• In 2000, about 67,000 households (out of 105.5

million) completed the census form online, despite no advertising of the availability of a Web option

• Tested Internet in 2005 National Census Test• The Census Bureau did not have a Web option in

2010 – I return to this later

10

19

Census Experiences with Mail+Web (cont.)

New Zealand • In 2006, about 7.0% of responses via Web

Australia• In 2006, 10% of occupied households completed the

census online• Aimed for 30% to 40% uptake in 2010• Preliminary estimate of 27% uptake of Internet

option for August 2011 census

20

Census Experiences with Mail+Web (cont.)

Thailand• In 2010, planned for 10% online response• Only 1.4% responded via Internet or telephone call-

in South Korea• In 2005, 0.9% of responses via Internet• In 2010, planned for 30% of forms online, achieved

47.9% (47.7% for short form and 49.3% for long form)

• Estimated reduced expenditure of ±$13.6m USD in personnel costs and ±$4.1m USD in data processing (on base of $157.2m USD)

11

21

Singapore Census 2000 and 2010

Both 2000 and 2010 censuses were register-based, supplemented with sample of 200,000 households

Telephone involved both inbound and outbound calling

15

38

62

46

22 16

0102030405060708090

100

2000 2010

EnumeratorTelephoneInternet

22

Japan Census 2010

Tested Internet response option in Tokyo area only

Population of about 13 million, or 10% of total population of Japan

Expected 5% Internet return rate

Response mode (in percent) of households in Tokyo area

8.4

59.2

32.4

0

10

20

30

40

50

60

70

80

90

100

EnumeratorMailInternet

12

23

Comments on Census Internet Options

Success rates vary across countries and regions

Internet penetration rates do not explain all of this variation

Other factors may play a role:• Level of security requirements and ease of access

to online instrument• Length and complexity of instruments• Promotion of online version and reasons given• Trust in government, concerns about online privacy,

etc.

24

Internet Use in Personal/Households Surveys

Internet-only surveys are rare• Internet status (access or use) is not universal or

known• Except for some specialized populations (e.g.,

college graduates)Most NSOs are exploring mixed-mode designsTwo types of mixed-mode designs• Concurrent mixed-mode surveys, e.g., mail survey

with an Internet option• Sequential mixed-mode surveys, e.g., start with

Internet, then switch to mail, telephone, and/or face-to-face

13

25

Response Rates to Mail Survey With Internet Option

Recent meta-analysis (Medway and Fulton, 2012) of 19 experimental comparisons found that providing a Web option in a mail survey significantly decreases response rates OR=.87)• Meta-analysis results• Detailed examples• Response rate charts

The first study (Griffin et al., 2001) on the ACS led to concerns about implementing an Internet option for the U.S. census and delayed implementation of the Internet on ACS for many years

26

What Do We Make of These Findings?

Several hypothesized reasons have been offered for surprising effects

Cautions• Not clear if this holds for censuses too – two

exceptions were census examples• Censuses are generally short and mandatory• Length and complexity of surveys may be a factor• Cost and/or speed of return is also important

These findings have led many to argue that sequential mixed-mode designs are better

14

27

Sequential Mixed-Mode Designs

In general, the design is to start with a cheaper mode (e.g., Internet) and then follow up with increasingly more costly modes (e.g., telephone and face-to-face)In practice, because e-mail addresses are usually not available, this starts with a mailed invitation to complete an Internet survey• When accompanied by a paper questionnaire, this

is a concurrent or choice mixed-mode design• When followed by a mail survey, this is a sequential

designMany NSOs are exploring sequential designs

28

Research on Sequential Mixed-Mode Designs

There are various mode combinations involving the Internet• E.g., Web followed by mail or Web followed by telephone

and/or face-to-faceThere are different sequences• E.g., Web as the initial mode or as a follow-up mode

The sequential designs are compared to different designs• E.g., single mode designs (e.g., mail only) or choice designs

All of this makes drawing conclusions from a limited number of studies difficulty

Selected examples follow

15

29

Sequential Mail + Web Designs

Studies comparing a sequential design to a mail-only design:• Cantor et al. (2010)• Israel (2009)• ACS Internet tests (see later)

Studies varying the order of the modes in a sequential design:• Holmberg et al. (2010)• Börkan (2010)• Stevenson et al. (2011)• Converse et al. (2008)• Smyth et al. (2010)• Lebrasseur et al. (2010)

Studies comparing choice versus sequential approaches:• Bensky, Link, and Shuttles (2010)• Lesser et al. (2010)• Friese et al. (2010) • Lagerstrøm (2011)

30

Summary of Overall Response Rates for Sequential Mail + Web Designs

Mail only versus Web+mail sequential designs:• 3 studies (Cantor et al; Israel; Elliott et al.) find mail only >

sequential• 1 study (ACS Internet) finds sequential > mail only

Order of mail+Web sequential design:• 4 studies (Stevenson et al.; Converse; Smyth et al.; Friese et

al.) find mail→Web > Web→mail• 3 studies (Holmberg et al.; Börkan; Lebrasseur et al.) find

mail→Web ≈ Web→mailChoice versus sequential design:• 3 studies (Bensky et al.; Lesser et al.; Friese et al.) find choice

> sequential• 1 study (Lagerstrøm) finds choice ≈ sequential

16

31

Comments on Sequential Mail + Web Designs

The results are still quite mixed• The belief that a sequential design starting with the

Internet is best has mixed supportThe benefit of sequential mixed-mode design may lie in cost, speed of response, and data quality – rather than response rates• Even if overall response rates aren’t increased,

substantial cost savings and data quality improvements are possible at large volumes

• Thieme (2011) estimated that each paper form in the U.S. 2010 census cost $12-$14 to print, mail, and process

32

The American Community Survey (ACS)

The ACS is a rolling cross-sectional survey to replace the U.S. census long form

Early development and testing around the 2000 census; now fully implemented

Employs a sequential mixed-mode design with mail and Internet choice• Mail/Internet → telephone (if available) → face-to-

face (for subsample)Recent research on sequential mixed-mode designs has shown more promise

17

33

ACS Internet Tests

Several developments since early tests• Improved interface design• Made login process easier• Made Web choice more prominent• Tested sequential (push) designs

Two recent field tests show more promising results• April 2011 Test• November 2011 Test

Based on this, the Census Bureau is now implementing a sequential mixed-mode design

34

Sequential Designs Involving Interviewers

Many of the sequential designs employed by NGOs include follow-up by telephone and/or face-to-face• This includes the ACS and censuses discussed

earlierStatistics Netherlands is one NSO aggressively exploring use of the Internet in mixed-mode designs

18

35

Web and Interviewer-Administered Sequential Designs: Statistics Netherlands

Dutch Health Interview Survey (Van Nunspeet, Cuppen, and van der Laan, 2011)• Cross-sectional survey, sample drawn from

population register• Response rates slightly higher (64%) for the

sequential mixed-mode design than for old CAPI design (61%)

• About half of the respondents in the mixed-mode design reported via the Web

Dutch Labor Force Survey

Dutch Security Monitor

36

Web and Interviewer-Administered Sequential Designs: Other Examples

Netherlands Kinship Panel Study

Netherlands ESS Round 4 Experiment

UKHLS or Understanding Society innovation panel experiment

UK LFS and OPN tests

Danish LFS test

19

37

What About Web as Follow-up Mode?

Several organizations have explored following up nonrespondentswith a Web optionVoorpostel and Ryser (2011): Swiss Household Panel• Telephone refusals contacted and offered a Web survey• Only 87 of the 1,962 panel members opted for Web and were sent

login information; of these, only 43 completed the Web surveyBanning and Schouten (2009): Dutch Labor Force Survey • One person per household recontacted• One group offered a short instrument by phone, mail, or Web; 45%

did so (mode not reported)• Other group followed up using CATI and CAPI; 77% answered basic

questionsDykstra et al. (2007): W2 of NKPS• Web survey offered to those who refused participation in main data

collection• Only 1.6% of all completes via Web

Findings suggest this strategy is not particularly effective

38

Challenges of Mixed-Mode Surveys Using the Internet

Within-household selection• Some surveys require the selection of one person at random

within the household, but the frame is one of addresses• Several attempts at self-administered within-household

selection have yielded mixed resultsHave everyone complete questionnaire

Use a method such as last birthday to select

Accept a household respondent

Measurement error• Assumption of no measurement differences between modes is

being questioned• Development of statistical methods to adjust for measurement

differences is underway

20

39

Comments on Measurement Error Differences

Different reasons for measurement error differences, e.g.,• Self- versus interviewer-administration: more (i.e., better)

reporting of socially undesirable information in self-administration

• Visual versus aural presentation: question order and response order effects

• Cognitive demands of modes, including distributed cognition• Motivational differences

In general differences between visual modes (mail and Web) and between aural modes (face-to-face and telephone) are smaller than those crossing mode types

Growing evidence that the design of the instruments can have an impact on mode differences

40

Internet Use for Establishment Surveys

Much of the research on mode alternatives is conducted in the context of household surveys

While many NSOs use the Internet as part of a mixed-mode strategy, much less research evidence is available

For this reason, I offer a set of observations, rather than present detailed empirical evidence

21

41

Internet Surveys of Establishments Versus Households/Persons

Establishments include businesses, educational facilities, government agencies, farms, institutions, etc.

Varying levels of Internet use across sectors, but probably greater penetration than for households

42

Features of Establishment Surveys That Promote Internet Survey Use

Surveys are often mandatory

Surveys are usually repeated• Often at regular intervals (e.g., monthly, annually)

Surveys are often short• Small number of items asked repeatedly

Sampling frames usually more complete• No need for listing, screening, selection, etc.

Questions are mostly factual• May be less susceptible to measurement error

22

43

Features of Establishment Surveys That Promote Internet Survey Use (cont.)

Administrative data often available• Permits verification/validation of responses

Extensive editing required• Could be reduced through built-in edits

Speed of reporting is important• Increase reliability of preliminary estimates

44

Characteristics of Establishment Surveys That Impede Internet Survey Use

Information is often distributed or available in records• Requires questionnaire to be answered by several

different people• Requires looking up information in administrative

recordsNeed for documentation• Because of mandatory nature and repeated

measures, businesses often need copy of survey for records

Need access to prior survey’s data• Additional security requirements

23

45

How Widespread is Use of Internet in Establishment Surveys?

Data on this issue is much less easy to obtain than for household surveys• Two exceptions: Bremner (2011); Kovac et al. (2009)

Some estimates from Haraldsen et al. (2011):• Statistics Norway: more than 150 business surveys available

in Internet portal; in 2009 an average of 63% of businesses responded via Internet or electronic data interchange (EDI)

• Statistics Netherlands: about 50% of all business responses in electronic form (Web or EDI)

• Statistics Denmark: more than 45 Web questionnaires on portal; percent of respondents choosing Internet ranged from 2% to 80%

46

Research Opportunities in Establishment Surveys

Unlike household surveys, there is not a tradition of methodological research in establishment surveys• Other than in editing and estimation process• Or in qualitative design work

Yet, the potential benefits of improved design and implementation using the Internet may be greater in establishment surveys

The ongoing nature of establishment surveys lend themselves to incremental improvements

24

47

Research Opportunities in Establishment Surveys (cont.)

Data quality improvements could be achieved with improved design of online business forms• Very little empirical work in this area

Postsurvey processing could identify key areas of concernParadata (e.g., keystroke files, time stamps) could identify inefficiencies• E.g., items that trigger edit checks• Time taken to complete items

These could lead to design improvements that could be tested in the next round of the survey

48

What About “Organic Data”?

Groves (2011) introduced the notion of organic data• Designed data: data created and collected for a specific

purpose (as in surveys)• Organic data: data generated by digital transactions

Examples of organic data• Digital transactions: scanners recording purchases, RFID

tracking goods, vehicle tracking• Social media: blogs, tweets, Internet searches, FaceBook

updates, etc.Examples:• Google trends• Nielsen BlogPulse• Billion prices project• Mappiness

Potential for supplementing rather than replacing traditional survey methods

25

49

Organic Data

Advantages• Cheap, widespread, timely

Disadvantages• Usually only one variable at a time; no covariates

E.g., we know gas prices are rising, but we don’t know how households are changing other spending to accommodate the increase

• Miss the very poor, those offline, etc.E.g., can’t look at how those in poverty, or the old, are behaving relative to others

• Stability of estimates: availability and meaning may change over time

Organic data tend to be wide but thin, while survey data tend to be narrow but deep

50

Conclusions: Opportunities

Internet-only surveys are unlikely in the foreseeable future for NSOs• Except for customer satisfaction surveys, surveys of

special populations, or supplemental tracking studies

The Internet is likely be used in combination with other modes• Sequential mixed mode designs show the most

promiseIncreasing the proportion of surveys completed online is likely to reduce costs and improve data quality

26

51

Conclusions: Challenges or Research Opportunities

We know that increasing Internet data collection benefits NSOs, but we need to identify benefits for respondents (households or establishments) to increase take-up ratesWe need to deal with measurement error differences between modes• Identify under what circumstances they occur• Test instrument designs to minimize such differences• Develop statistical models to adjust for differences

We need more research on Internet surveys for establishment surveys• How to get businesses responding online and keep them there• How to improve the design of survey instruments to leverage

the advantages of the Internet, minimize respondent burden and improve data quality

Thank You