advantages and disadvantages of internet survey methods ... ?· advantages and disadvantages of...

Download Advantages and Disadvantages of Internet Survey Methods ... ?· Advantages and Disadvantages of Internet…

Post on 01-Sep-2018

222 views

Category:

Documents

5 download

Embed Size (px)

TRANSCRIPT

  • 1

    Advantages and Disadvantages of Internet Survey Methods for Official

    Statistics

    Mick P. Couper, PhDSurvey Research Center, University of Michigan

    andJoint Program in Survey Methodology

    4th International Workshop on Internet Survey Methods

    September 2012

    2

    Outline

    Background: Innovation in survey research and the rise of Internet surveys

    The use of Internet data collection in national statistical offices Internet for surveys of households or people Internet for surveys of establishments

    Possible futures for the role of the Internet

    Challenges and research opportunities

  • 2

    3

    Background: Research Sectors

    There are four main research sectors with interest in online survey methods: Market/consumer research includes advertising, product

    research, customer satisfaction, etc. Public opinion and political polling research Academic research Government research

    Each sector has a different focus and use of online surveys Adopting Internet surveys for different reasons But there is a lot of overlap between sectors Some types of surveys (e.g., customer satisfaction) are done

    by all four sectorsMuch of the innovation and push for online surveys has come from the market research world; the other sectors have been more cautious

    4

    Research Sectors and Key Concerns

    Market and consumer research

    Public opinion and political

    polling

    Government

    Speed, cost Speed, cost, accuracy

    Accuracy, broad representation

    Academic

    Cost, complexity

  • 3

    5

    Internet Survey Adoption in Different Sectors

    Market researchers have fully embraced online survey methods, and were early adopters See next slide

    Public opinion research is using Internet methods alongside RDD telephone surveys

    Academic researchers doing theory testing are using Internet surveys for cost reasons

    Governments have been relatively slow to adopt Greater focus on mixed-mode surveys than stand-

    alone Internet surveys

    6

    Growth in Online Market Research in U.S. and Europe

    0200400600800

    10001200140016001800200022002400

    1996

    1997

    1998

    1999

    2000

    2001

    2002

    2003

    2004

    2005

    2006

    2007

    2008

    2009

    2010

    2011

    Spen

    ding

    U.S. (millions $)Europe (millions )

    Source: Inside Research, February 2011

  • 4

    7

    Competing Pressures

    National Statistical Offices (NSOs) face a number of pressures: Declining response rates and increasing costs of

    traditional methods General budgetary pressures do more with less Pressures to be modern and adapt to user

    preferences (digital government) Responsibility to collect data on all segments of

    society (both the haves and have nots) Need for accuracy, reliability, and trustworthiness

    data serve as benchmarks and have vital policy implications

    8

    Competing Pressures (continued)

    Tension between the need to adapt and innovate and the need to preserve comparability of data across time

    Innovation tends to be slow and cautious, relative to other sectors Innovation is rewarded in some sectors (competitive

    edge, publications) Risk of failure much higher in the government

    sector, given relative scale of operations (e.g., censuses)

    Cost pressures not as great as other sectors

  • 5

    9

    Reasons for Slow Adoption of Internet Surveys

    Coverage concerns Not everyone has Internet access Those with access are different from those without (the digital

    divide), and the differences are not disappearing NSOs are particularly concerned about broad representation

    Sampling concerns There is no sampling frame of Internet users There is no RDD-like method to sample e-mail addresses

    Nonresponse concerns Concern that Internet surveys may decrease response rates

    and increase nonresponse bias

    10

    World Internet Penetration Over Time

  • 6

    11

    Internet Penetration in Korea

    44.7

    56.659.4

    65.5

    72.7 73.578.1 78.8

    81 81.6 83.7 83.8

    0

    10

    20

    30

    40

    50

    60

    70

    80

    90

    100

    2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

    Pece

    nt

    Source: ITU World Telecommunication / ICT Indicators

    12

    Internet Survey Response Rates

    Evidence from two meta-analysis that Internet response rates generally lower than comparable modes

    Lozar Manfreda et al. (2008) Meta-analysis of 45 experimental comparisons of Web to

    other modes (mail, phone, IVR, etc.) in 24 papers Web has an average 11% lower response rate than the

    alternativeShih and Fan (2008) Meta-analysis of 39 experimental comparisons of Web to mail Web has an average 11% lower response rate than mail No significant effect of publication year (i.e., the gap is not

    closing over time)

    Response rate charts

  • 7

    13

    So What Are NSOs Doing?

    Because of these concerns, greater focus on mixed-mode surveys involving Internet than on stand-alone Internet surveys

    Different approaches for household surveys and establishment surveys I focus primarily on household surveys, then offer

    some observations on establishment surveys

    14

    Household Surveys

    Different types of surveys Censuses One-time or repeated cross-sectional surveys Panel surveys

    Internet as replacement/supplement for different modes Mail Interviewer-administered (face-to-face or telephone)

  • 8

    15

    Internet Use in Household/Population Censuses

    Typically used as an option in a sequential mixed-mode design Mail-out census form includes option for Internet

    completion Non-experimental design Varying rates of success across countries

    Zewoldi (2011) noted that more than 30 countries are providing or experimenting with an Internet option in the 2010-2011 round of censusesSelected examples follow

    16

    Census Experiences with Mail+Web

    Switzerland In 2000, about 4.2% of forms returned via Internet Register-based census in 2010

    Norway In 2001, about 9.9% of responses via Internet Register-based census in 2011

    United Kingdom In 2011, about 27% of households responded online

  • 9

    17

    Czech Republic and Hungary 2011

    25.1 19

    62.2

    15

    12.7

    66

    0102030405060708090

    100

    Czech Republic Hungary

    EnumeratorMailInternet

    18

    Census Experiences with Mail+Web (cont.)

    Canada In 2006, 18.3% of households completed the census

    online In 2011, 54.4% of households did so

    United States 2000 and 2010 In 2000, about 67,000 households (out of 105.5

    million) completed the census form online, despite no advertising of the availability of a Web option

    Tested Internet in 2005 National Census Test The Census Bureau did not have a Web option in

    2010 I return to this later

  • 10

    19

    Census Experiences with Mail+Web (cont.)

    New Zealand In 2006, about 7.0% of responses via Web

    Australia In 2006, 10% of occupied households completed the

    census online Aimed for 30% to 40% uptake in 2010 Preliminary estimate of 27% uptake of Internet

    option for August 2011 census

    20

    Census Experiences with Mail+Web (cont.)

    Thailand In 2010, planned for 10% online response Only 1.4% responded via Internet or telephone call-

    in South Korea In 2005, 0.9% of responses via Internet In 2010, planned for 30% of forms online, achieved

    47.9% (47.7% for short form and 49.3% for long form)

    Estimated reduced expenditure of $13.6m USD in personnel costs and $4.1m USD in data processing (on base of $157.2m USD)

  • 11

    21

    Singapore Census 2000 and 2010

    Both 2000 and 2010 censuses were register-based, supplemented with sample of 200,000 households

    Telephone involved both inbound and outbound calling

    15

    38

    62

    46

    22 16

    0102030405060708090

    100

    2000 2010

    EnumeratorTelephoneInternet

    22

    Japan Census 2010

    Tested Internet response option in Tokyo area only

    Population of about 13 million, or 10% of total population of Japan

    Expected 5% Internet return rate

    Response mode (in percent) of households in Tokyo area

    8.4

    59.2

    32.4

    0

    10

    20

    30

    40

    50

    60

    70

    80

    90

    100

    EnumeratorMailInternet

  • 12

    23

    Comments on Census Internet Options

    Success rates vary across countries and regions

    Internet penetration rates do not explain all of this variation

    Other factors may play a role: Level of security requirements and ease of access

    to online instrument Length and complexity of instruments Promotion of online version and reasons given Trust in government, concerns about online privacy,

    etc.

    24

    Internet Use in Personal/Households Surveys

    Internet-only surveys are rare Internet status (access or use) is not universal or

    known Except for some specialized populations (e.g.,

    college graduates)Most NSOs are exploring mixed-mode designsTwo types of mixed-mode designs Concurrent mixed-mode surveys, e.g., mail survey

    with an Internet option Sequential mixed-mode surveys, e.g., start with

    Internet, then switch to mail, telephone, and/or face-to-face

  • 13

    25

    Response Rates to Mail Survey With Internet Option

    Recent meta-analysis (Medway and Fulton, 2012) of 19 experimental comparisons found tha

Recommended

View more >