turbulent times for survey methodology

TURBULENT TIMES FOR TURBULENT TIMES FOR SURVEY METHODOLOGYSURVEY METHODOLOGY

Damon BurtonDamon Burton

University of IdahoUniversity of Idaho

WHAT DO YOU KNOW WHAT DO YOU KNOW ABOUT SURVEY ABOUT SURVEY

RESEARCH? RESEARCH?

Jot down at least 3 types of surveys Jot down at least 3 types of surveys you’ve taken in the past few years.you’ve taken in the past few years.

What value or benefit do surveys What value or benefit do surveys have in your life? How do surveys have in your life? How do surveys help make your life better?help make your life better?

How much does survey data How much does survey data influence your life?influence your life? provide 1-2 examplesprovide 1-2 examples

CHANGES IN SURVEY CHANGES IN SURVEY RESEARCH OVER 50 RESEARCH OVER 50

YEARSYEARS

SURVEYING THROUGHSURVEYING THROUGH THE 1960’S THE 1960’S

Until the late 1960’s, most survey research Until the late 1960’s, most survey research was conducted via in-person interviews.was conducted via in-person interviews.

Random sampling was developed in the Random sampling was developed in the 1940’s & 1950’s. 1940’s & 1950’s.

Efficient surveying required sampling in Efficient surveying required sampling in stages, and clustering locations to reduce stages, and clustering locations to reduce costs.costs.

Planning and conducting studies took Planning and conducting studies took months and sometimes years, with data months and sometimes years, with data collection alone normally taking months.collection alone normally taking months.

Sample surveying was limited to only the Sample surveying was limited to only the most important topics.most important topics.


Polite, well-dressed women interviewers Polite, well-dressed women interviewers came to a person’s home to conduct the came to a person’s home to conduct the interview.interview.

Icebreaker questions (e.g., How is your Icebreaker questions (e.g., How is your family?) were used to develop report.family?) were used to develop report.

Responses were recorded via paper and Responses were recorded via paper and pencil in great detail. pencil in great detail.

Interviewers attempted to get people to Interviewers attempted to get people to respond thoroughly to each question.respond thoroughly to each question.

Every effort was made to show that they Every effort was made to show that they were genuinely interested in the were genuinely interested in the respondent as a person.respondent as a person.


Survey research took a major blow when Survey research took a major blow when the Literary Digest’s predicted the wrong the Literary Digest’s predicted the wrong winner in the 1936 Presidential Election.winner in the 1936 Presidential Election.

Based on a 20% response rate to over 10 Based on a 20% response rate to over 10 million straw ballots mailed to 36% of U.S. million straw ballots mailed to 36% of U.S. households, Alf Landon was predicted to households, Alf Landon was predicted to beat Franklin Roosevelt 55% to 41%.beat Franklin Roosevelt 55% to 41%.

Roosevelt actually beat Alf Landon 61% to Roosevelt actually beat Alf Landon 61% to 37% for the most famous botched 37% for the most famous botched prediction in the history of American prediction in the history of American politics, thus discrediting survey research.politics, thus discrediting survey research.


Phone surveys were not practical because Phone surveys were not practical because only one-third of households had phones in only one-third of households had phones in 1936. 1936.

Area codes didn’t exist so operators had to Area codes didn’t exist so operators had to place calls, making them extremely costly.place calls, making them extremely costly.

Call quality was low, and many households Call quality was low, and many households shared party lines with other households. shared party lines with other households.

Long distance was used mostly for Long distance was used mostly for emergencies. emergencies.

Sharing personal information over the phone Sharing personal information over the phone to someone you didn’t know was to someone you didn’t know was unappealing. unappealing.


Mail surveys were used extensively in the Mail surveys were used extensively in the 1940’s for specialized populations. 1940’s for specialized populations.

Adequate household lists did not exist for Adequate household lists did not exist for doing general public surveys, and response doing general public surveys, and response rates were low.rates were low.

Surveys were constructed with typewriters, Surveys were constructed with typewriters, limiting design and composition limiting design and composition possibilities. possibilities.

Copies were duplicated on mimeo machines, Copies were duplicated on mimeo machines, with time demands high and quality low. with time demands high and quality low.

The process was time-consuming, costly and The process was time-consuming, costly and often ineffective. often ineffective.

SURVEY RESEARCH IN THE SURVEY RESEARCH IN THE 1970’S AND 1980’S1970’S AND 1980’S

Development of modern telephone and mail Development of modern telephone and mail survey methods began in the early 1970’s. survey methods began in the early 1970’s.

Area codes were developed, making direct Area codes were developed, making direct dialing possible, and development of Wide dialing possible, and development of Wide Area Telecommunication/Telephone Service Area Telecommunication/Telephone Service (WATTS) lowered long distance costs.(WATTS) lowered long distance costs.

Phone quality improve greatly, allowing Phone quality improve greatly, allowing conversations in a normal tone of voice. conversations in a normal tone of voice.

By 1970, 87% of U.S. households had phones. By 1970, 87% of U.S. households had phones. Random-digit-dialing (RDD) was also Random-digit-dialing (RDD) was also

developed to systematically sample developed to systematically sample households. households.


Research showed phone & in-person interviews Research showed phone & in-person interviews yielded similar response rates and results. yielded similar response rates and results.

Phone calls came without prior notice, normally Phone calls came without prior notice, normally in the early evening hours. in the early evening hours.

Interviewers read a script in a conversational Interviewers read a script in a conversational manner, and persons had only a few seconds to manner, and persons had only a few seconds to decide whether to respond. decide whether to respond.

Scripted questions were read and respondents Scripted questions were read and respondents had to answer in predesigned ways.had to answer in predesigned ways.

Open-ended questions were eliminated and Open-ended questions were eliminated and elaboration discouraged to reduce call length. elaboration discouraged to reduce call length.


Breakthroughs such as photo copy machines Breakthroughs such as photo copy machines made mail surveys more efficient. made mail surveys more efficient.

Personal computers and better printing Personal computers and better printing methods streamline construction of surveys. methods streamline construction of surveys.

Survey researchers personalized mailings Survey researchers personalized mailings and used multiple followups to raise and used multiple followups to raise response rates. response rates.

Costs came down for conducting surveys, Costs came down for conducting surveys, allowing smaller organizations in any area of allowing smaller organizations in any area of the country to collect survey data.the country to collect survey data.

Most government surveys were done by mail, Most government surveys were done by mail, and public surveys were conducted by phone.and public surveys were conducted by phone.


Gated communities and locked apartment Gated communities and locked apartment buildings reduced access to people’s homes. buildings reduced access to people’s homes.

Unlisted phone numbers and removal of Unlisted phone numbers and removal of addresses from phone books reduced sample addresses from phone books reduced sample inclusiveness. inclusiveness.

Telemarketers used survey methods to market Telemarketers used survey methods to market products over the phone, increasing the number products over the phone, increasing the number of unsolicited calls and raising public mistrust. of unsolicited calls and raising public mistrust.

The public responded with unlisted numbers, The public responded with unlisted numbers, answering machines, caller ID and call blocking.answering machines, caller ID and call blocking.

Respondents said Respondents said “no” “no” to almost all requests.to almost all requests.


Cell phones made phones personal not Cell phones made phones personal not household devices. household devices.

Scientific sampling became more difficult Scientific sampling became more difficult because not all persons had an equal or because not all persons had an equal or known chance of selection. known chance of selection.

Getting a list of cell numbers is also a Getting a list of cell numbers is also a problem. problem.

Conversations while people were driving Conversations while people were driving and dropped calls are other cell problems. and dropped calls are other cell problems.

The result is a much lower proportion of The result is a much lower proportion of people who can be reached by phone and people who can be reached by phone and who will agree to be surveyed.who will agree to be surveyed.


Computer technology has changed survey Computer technology has changed survey methodology significantly. methodology significantly.

Computer-Assisted Telephone Interviewing Computer-Assisted Telephone Interviewing (CATI) allowed scripted responses to be (CATI) allowed scripted responses to be entered directly into the computer. entered directly into the computer.

Touchtone Data Entry (TDE) had respondents Touchtone Data Entry (TDE) had respondents providing responses to the computer by providing responses to the computer by pressing keys on their touchtone phone. pressing keys on their touchtone phone.

Interactive Voice Response (IVR) used the Interactive Voice Response (IVR) used the computer to conduct an interview and computer to conduct an interview and register vocal as well as keyed respondent register vocal as well as keyed respondent answers.answers.


Internet surveying changed survey research. Internet surveying changed survey research. Cost savings were huge because interviewer Cost savings were huge because interviewer

wages, long-distance charges, postage, wages, long-distance charges, postage, printing and data entry costs were eliminated. printing and data entry costs were eliminated.

Web surveys also made finding a sample Web surveys also made finding a sample easier and quicker, and increased sampling easier and quicker, and increased sampling locales. locales.

Programs such as Survey Monkey made Programs such as Survey Monkey made constructing surveys simpler and better constructing surveys simpler and better because the survey process was engineered.because the survey process was engineered.

Conducting survey research became easier Conducting survey research became easier and quicker on-line. and quicker on-line.


Internet surveying also presents a host of Internet surveying also presents a host of challenges and problems. challenges and problems.

Major gaps exist in computer and internet Major gaps exist in computer and internet access and computer skills for certain access and computer skills for certain samples. samples.

Lack of standards for creating email addresses Lack of standards for creating email addresses that would allow development of a sampling that would allow development of a sampling algorithm has been problematic. algorithm has been problematic.

Internet surveys are limited to populations Internet surveys are limited to populations with high internet access and skill levels such with high internet access and skill levels such as certain professional organizations and as certain professional organizations and purchasers of computer-related products.purchasers of computer-related products.


The Internet has changed our culture. The Internet has changed our culture. Over time, distrust of the Internet has grown Over time, distrust of the Internet has grown

because of cyber crimes (e.g., identity theft) because of cyber crimes (e.g., identity theft) and viruses. and viruses.

Many people now avoid responding to Many people now avoid responding to unsolicited emails, particularly ones that unsolicited emails, particularly ones that involve clicking on a link. involve clicking on a link.

Email and texting have become preferred Email and texting have become preferred methods for communicating at work and methods for communicating at work and home.home.

Survey researchers must promote the value Survey researchers must promote the value of their surveys from other communicationsof their surveys from other communications. .


The 18 to 30-year-old population segment is The 18 to 30-year-old population segment is the most difficult to get to respond to surveys.the most difficult to get to respond to surveys.

Mail surveys can now achieve higher response Mail surveys can now achieve higher response rates than telephone surveys, although rates than telephone surveys, although identifying a sample has become harder. identifying a sample has become harder.

Many survey researchers use multiple survey Many survey researchers use multiple survey modes (e.g., phone, mail, internet). modes (e.g., phone, mail, internet).

Respondents now can be given a choice of Respondents now can be given a choice of survey mode.survey mode.

Refusal to respond via one mode often prompts Refusal to respond via one mode often prompts requests to respond using another mode.requests to respond using another mode.

GETTING BEYOND THE GETTING BEYOND THE TURBULENCETURBULENCE

Certain survey modes are better for specific Certain survey modes are better for specific populations, topics and sponsors than others. populations, topics and sponsors than others.

Survey research has shifted from one to Survey research has shifted from one to several survey modes for the same data several survey modes for the same data collection. collection.

In the mixed-mode era, questions must work In the mixed-mode era, questions must work across modes, and modes must be across modes, and modes must be complementary to maximize response rates. complementary to maximize response rates.

Goal is to design scientifically-sound data Goal is to design scientifically-sound data collection systems that allow precise collection systems that allow precise estimates of behaviors and attitudes of a estimates of behaviors and attitudes of a population by sampling only a small population by sampling only a small proportion of them. proportion of them.

FROM TOTAL DESIGN TO FROM TOTAL DESIGN TO TAILORED DESIGNTAILORED DESIGN

Dillman’s (1978) 1Dillman’s (1978) 1stst edition focused on the edition focused on the Total Deign Method (TDM-1) that personalized Total Deign Method (TDM-1) that personalized all aspects of surveys. all aspects of surveys.

TDM-1 used the same procedures for all TDM-1 used the same procedures for all populations and situations (i.e., one size fits populations and situations (i.e., one size fits all). all).

The Tailored Design Method (TDM-2) tailors The Tailored Design Method (TDM-2) tailors surveys to fit specific populations & surveys to fit specific populations & situations. situations.

In TDM-2, criteria for successful surveying In TDM-2, criteria for successful surveying were expanded to include coverage, sampling, were expanded to include coverage, sampling, measurement and nonresponse error.measurement and nonresponse error.

Layout design and mixed modes were added. Layout design and mixed modes were added.

33RDRD EDITION OF THE EDITION OF THE TAILORED DESIGN METHODTAILORED DESIGN METHOD

Huge changes have occurred in the past 9 Huge changes have occurred in the past 9 years as technology has confronted culture.years as technology has confronted culture.

The Internet has reached most businesses but The Internet has reached most businesses but is still unavailable in many homes. is still unavailable in many homes.

Phones have become personal rather than Phones have become personal rather than household tools and become harder to access. household tools and become harder to access.

Survey responses are more dependent on Survey responses are more dependent on respondent choice.respondent choice.

Survey design, mode and choice require Survey design, mode and choice require greater up-front preparation and greater up-front preparation and understanding of how to best reach the understanding of how to best reach the “target population.” “target population.”

33RDRD EDITION OF THE EDITION OF THE TAILORED DESIGN METHODTAILORED DESIGN METHOD

Consideration of internet applications influences Consideration of internet applications influences coverage, sampling, question construction and coverage, sampling, question construction and choice of modes to achieve high response rates.choice of modes to achieve high response rates.

Many surveys demand mixed modes to best Many surveys demand mixed modes to best target a specific population and/or topic. target a specific population and/or topic.

Visual design concepts can enhance effectiveness Visual design concepts can enhance effectiveness to create a more “personal feel.” to create a more “personal feel.”

Longitudinal and internet panel surveys are Longitudinal and internet panel surveys are particularly sensitive to visual design elements.particularly sensitive to visual design elements.

Sponsorship of surveys may also dramatically Sponsorship of surveys may also dramatically impact their design and implementation needs. impact their design and implementation needs.

The EndThe End

turbulent times for survey methodology

Documents

discrediting survey

years surveying

sample surveying

ssurvey research

efficient surveying

sphone surveys

smail surveys

sample households