Surveys: What Are They?

• A survey is a collection of questions asked repetitively to a sample of a population to mathematically derive characteristics of the total population.

Surveys: What Are They Good For?

Why is This Cycle Important?

• It’s a framework• It provides guidelines and reminders as

you work with clients and stake-holders• You’re likely doing parts of it already• Those are likely the parts of your process

that work!

Great Survey Design Cycle

The Trifecta: Need, Design & Act

Unit 1: Need

Set Your Goals and Objectives

Needs: We All Have Them

• Questions to ask:– What are we trying to figure out?– What kinds of reports or data do we want or

expect?– What will we do with this data when we’re

done?– Who is our intended audience or population? – How are you going to access the target

audience?

Examples of Need

– How well known is my brand?– Will customers buy this product?– If we offer X benefit, will our employee

happiness go up?– Why are my customers not converting?– Will my product do well in a new market?

Set a Survey Goal

• Setting goals and objectives for a survey– Define your goal. A goal is not a single learning

point – a goal is what you are going to do with this data, and why.

• Good goal: grow your company into new markets. (“A survey will determine which markets are good for our existing products.”)• Bad goal: make more money for your

business.

Learning Objectives

• Determine your learning objectives– These should all support your overall need and

goal– A good amount of learning objectives: three– You should have no more than five!

Brainstorm Your Questions

Selection and Refinement

Eye on the Prize: ROI

• If you are going to spend more time and money on running the research for this need than the overall completion of goals would generate, it’s a waste of time and money

• If there is no ROI measurement, there is no encouragement to take action

• Without communication from the start about possible actions to take, survey results may have no obvious meaning

Unit 2: Design

Organize Brainstorm

Refine Brainstorm Ideas into Questions

Guide: Writing Questions

• Multiple choice versus open-text questions– Quantitative versus qualitative

• Phrasing and language use – unclear language, grammar, ambiguity can all be issues– Remember that language can differ

between demographic groups• Keep your questions:– Brief - Simple– Relevant - Specific and direct

Qualitative Versus Quantitative

• You may introduce bias into your survey with every qualitative answer you ask, unless the resulting answers are discrete.

• This data should never be added to quantitative data without the information being entirely clear in all reporting.

Guide: Exploratory Studies

• Open-text questions• You should never have a required question

that does not have an opt-out option (this creates bad data)

The Four Horsemen of the Surveypocalypse

Emotional Bias

• Asking loaded questions• Asking neutral-seeming questions on a

loaded topic

Identity Bias

• “How much do you love SurveyGizmo?”• Asking “Do you like SurveyGizmo?” with a

SurveyGizmo logo in the corner of the survey

Option Bias

• Required, non-applicable questions• Leading or restrictive options• Different types of scales• Option lists of death

Conversational Bias

• Surveys as a conversation• Respondents giving the answer they think

you want to hear

Lack of Focus

• Covering too many diverse topics• Additional questions that do not meet the

survey goal• Questions that are not inline with the

learning objectives• Questions that do not derive actionable

results

Miscommunication

• Know your audience and the language that they use and understand– Avoid technical terms unless it is

appropriate– Define terms if necessary

• Remember to speak in your company’s voice

• Have a peer review for clarity

Survey fatigue as a cultural trend

• Cultural survey fatigue– The average

respondent is fatigued already, just by nature of: • Receiving emails

from organizations• Suggestions on

receipts and from cashiers

• Try to avoid…– Leading questions– Loaded or suggestive questions (like our

star rankings)– Fatiguing question types – large tables,

lots of open-text or essay questions– Sensitive questions– Highly technical language

The Wrap-Up: Question Mistakes to Avoid

Unit 3: Build

• Design: Involves thinking about psychology, emotions and words. It is the more abstract phase.

• Build: Involves taking into account security walls, logic, combatting fatigue, bias, and poor data collection; It is the more active phase.

How are Design and Build different?

Stages of Build

The Usual Suspects

The Radio Button

• Quantitative – Scale (should be horizontal)– Categorical (should be vertical)• “All of the above” is a no-no!

The Scale: In its Natural Environment

Neutral or not?

Scale questions: The controversy

NPS: Net Promoter Score

Neutral or not?

The Checkbox: Choose All That Apply

The Checkbox: Beware!

Choosing more than one option changes statistical reporting a lot!

Multi-Text Questions

• Qualitative• Explorative or un-

aided response; used for lists

Please list the names of phone providers that you have seen or heard advertised.

Essay Questions

• Qualitative and explorative

• This is a way to gather unaided responses for your survey

3. What is your favorite thing about SurveyGizmo?

Essay Questions: Your Friend? Maybe.

Table Questions

Do NOT use as a space-saver – these are fatiguing!

Table Questions: What’s Totally Okay

Build your survey.Test It.

Get buy-in from your stakeholders.

Why Validation Matters:

Test Reports• Are your questions reporting the way

you expect?• Are you able to create the reports you

need using the data you are collecting?• Is the data in the format you need?

Apply and Test Logic

Different Types of Logic

• Fatigue-fighting:• Page jumping• Show-when logic• Percent branching• Piping (repeating)

• Bias-fighting:• Randomization• Disqualifiers• Survey timing/combatting straight-

lining• Vote protection

Great Survey Design Cycle

Unit 4: Collect

Survey Mode

Mode introduces different forms of bias – so how the data is collected is important!

Bias of Sample

• The mode of survey you choose – Example: Percentage of households

with internet capability in the US versus households with no internet

• Choosing to email (versus telephone) this survey will create a highly biased sample

Sample

• Your options are: survey everyone, or survey a percentage

• Why?– Cost– Survey fatigue– You will miss certain sections of the

population– Using a statistically valid sample is just as

effective (or more effective) than trying to survey your entire population

What is Sample? Why is it Important?

More on Sample

A sample is statistically valid when every single person in that population has a equal chance or probability to be in a sample that you select.

What is Sample Size?

How many responses do you need for your survey to be statistically accurate?– It depends.• How accurate do you want the data

to be? (margin of error or confidence interval)• How repeatable do you want the

results to be?• How large is your total population?

How to Determine Sample Size

• Estimate 400 responses• Use a sample calculator!

Sample Calculators: Magic?

Caveats

#1: If you are segmenting your sample data for comparison, you need to make sure that the segments that you are using for comparison are the same as the segments in the represented population– Example: When comparing men and

women in the United States, you would need to make sure that the ratio within your survey was the same as the ratio within the larger US population

Caveats, continued

#2: If you are using cross-tabbing for your data, you need to ensure that the data you have (per question that you are cross-tabbing) is statistically valid for representing the larger population

Where Do You Get Sample?

• Pull a population from your own customer list.–Warning: Do NOT use your entire

customer base.– If everyone has the same chance of being

randomly selected, you are not biasing your results in any way.

• Panel Companies: A panel company is an organization that exists to sell anonymous survey responses to marketers and market researchers.

Panel Companies: The Issues

Drawbacks:– Using incentives– Cannot access market researchers– Some panel companies will buy from

each other when they cannot provide the sample needed

– Hard to determine level of bias in sample• If the panel companies award “points”

for websites like Amazon – helps reduce sample bias based on incentive

Incentives

• Biases your sample (ex. Toys R Us gift card as incentive)

• Incentives can jeopardize your data (because respondents just want to get to the end)

• Safeguards:– Survey page timer with

disqualification– Shorter surveys– Red herring questions– Clean data (eliminating straight

liners, Christmas trees, etc)

Unit 5: Report

Clean your data

How to clean data, Step 1

• Look out for:– Straight liners or Christmas tree behavior– Unusually quick responses (when using a

timer)

Warning signs

• Look out for:– Choosing all checkbox options

Warning signs

Look out for:– Red herring fails or logically inconsistent

answers

Warning signs

• Look out for:– Nonsense or missing open-ended

answers

How to clean data, Step 2

• Prepare your data for analysis– Beware of:• Inconsistent numeric values (How old

are you? Etc.)• Breaks in validation

• Do not introduce new bias! – Changing question text

Finding outliers

Analyzing text responses

• How will you deal with your qualitative data?– Keyword frequency–Word clouds– Positive/negative

Analyzing text responses

Bucketing– You can use the SurveyGizmo Open Text

Analysis tool.

Run Preliminary Reports

• Your preliminary reports should be focused on your original learning objectives.– Did you get your questions answered? – Is the data in the format you expected?– Are you seeing the trends that you

anticipated?• Run individual reports for each learning

objective.– Use this process to determine the “highlights”

of data collected as they relate to ultimate actions so that you can truly understand the most significant findings of your research.

Running reports: Key factors

• Make sure your data makes sense• For any overt trends you are finding in the

data, make note of them and ensure that they are important towards the objectives that you had set for your survey

Segmenting data for analysis

• Often, your survey will contain demographic and firmographic questions to create segments in your survey.

• These segments should remain the same from start to finish of the survey process.

Good indicators of a trend:

• When you have data that isn’t statistically sound but is still interesting, you can call it “directional data”.– This data gives you an idea of what your

population is saying, thinking or feeling, but you cannot use statistics to back it up.

Analyze Your Data

Weight your data during design

Some studies will over-sample certain populations – Example: Over-sampling females in an

election poll

Report on your Findings

Suggestions for effective reports

Stage 1: Write a summary–What was the ultimate goal of this survey?–Who was surveyed?–Who was the population?–Who responded?– Include basic highlights of the survey

audience and your data to introduce the findings

Stage 2: Write a mini-report for each individual learning objective (ex: 401K changes).•The last section for every learning objective report will include the recommended actions to take based on the results of the survey (these should not be a surprise!)

Suggestions for effective reports

Stage 3 (optional): Interesting and unexpected trends found– Good to know, not need-to-know– Ex. Perhaps you found a new, unintended

segment of your population that could help you to make good business decisions moving forward

•This is going the extra mile for your clients!

Suggestions for effective reports

Stage 4: Conclusion– Recap what actions are going to be taken (if

any) based on your findings.– Get all of stakeholders to agree to those

actions.– Create a survey to be sent to stakeholders in

order to gain feedback for the project and put actions in motion.

– Important for the next stage, Act: ask stakeholders to provide metrics that can be used to measure the success of the actions that will be taken.

Suggestions for effective reports

Tips for communicating data

• Try to anticipate questions about the report

• Know the details• Be honest

Unit 6: Act

Actions: The key to success!

• In order to ensure that your study has a purpose, it is important to reiterate and motivate the stakeholders to take action based on the data collected.

• It can be helpful to establish a reasonable timeframe in which actionable results (positive or negative) can be expected.

Monitor Actions

How to get feedback

• This should be in the form of a short survey that goes out to all stakeholders in order to gain feedback on the study.

• Ask for any suggestions they may have so that you can work better together in the next study and improve the process.

Publish and Share Study Results

Great Survey Design Cycle