surveys and usability - uc berkeley school of...

Surveys and Usability

I214 28 Oct 2008

What Info Can Be Collected via Surveys?   Facts

  Characteristics of respondents   Demographics, experience, employment...

  Self-reported behavior   This instance   Generally/usually   Past   Anticipated (who will you vote for?)

  Opinions and attitudes:   Preferences, opinions, satisfaction, concerns, perceptions   Their expectations of their future behavior

  Knowledge   What do they they know about x, y, z

Some sources of error   Sample, respondents   Question choice   Question wording   Question order   Method of administration

  Surveyor, if administered person to person

  Inferences from the data   Users’ interests in influencing results

  “vote and view the results”

Response Rates

  % of sample who actually participate   low rates may indicate bias in

responses   Whom did you miss? Why?   Who chose to cooperate? Why?

  How much is enough?   For statistically valid sample

  Babbie: 50% is adequate; 70% is very good

  Web surveys tend to be 3%, 5%....

Increasing response rates

  Good design   Short   Easy to answer immediately   NOT an extra step   Harder to say ‘no’ to a person   Explanation/request

  Explain purpose of study   Don’t underestimate altruism

  Why you need them   Incentives

  Reporting back to respondents as a way of getting response   Money; entry in a sweepstakes

  Follow up on non-respondents (if you can)

Ways of Administering Surveys (roughly in order of expected response rate)

  In person   Phone   Paper, in person   Email (usually with a link)   Web   Mail

  .

Active vs passive   Active: solicit respondents

  Send out email (with questions, or link), letters, phone   Use sampling frame to develop a sample, I.e. list

  Ideal:   Keep track of who responds   Follow up on non-respondents if possible   Compare respondents/non-respondents looking for biases

  Passive – “poll”   Popup box: “would you take a few minutes to help us…”   Link on a site

Passive: problems may include   Response rate probably unmeasurable   May be difficult to compare respondents to

population as a whole   Likely to be biased (systematic error)

  Frequent users probably over-represented   Busy people probably under-represented   Disgruntled and/or happy users probably over-

represented

What to look for in web survey software/site

  # of questions you can ask

  # of respondents   How long the survey will be available   Variety of question designs/formats, or at least the designs you need   Kind of data analysis they support   Reports they produce for you   Whether you can download data into Excel etc   How long the data will be available to you   Most have free/lowcost versions and premium

Web survey problems   Loss of context – what exactly are you asking about,

what are they responding to?   Are you reaching them at the appropriate point in their

interaction with site etc?   Incomplete responses   Multiple submissions

  NOW (PBS) using cookies to prevent repeated voting   Response rate problems

  Low rate   Hard to calculate

  Is your sampling unit the user or the user, the interaction?

Questionnaire construction   Content

  Goals of study: What do you need to know?   What can respondents tell you?

  Conceptualization   Operationalization – e.g., how exactly do you define

“household with access to internet”?   Question design   Question ordering   Short, short, short!

Respondent characteristics include:   Demographics:

  age, sex, race…   Income   Family/living circumstances

  Living conditions   Where   Urban/suburban/rural   House, apartment…etc etc.

  Experience, expertise   Occupation

Know how you’ll use the data. If you don’t need to know, don’t ask

Respondent characteristics

Pew Typology Quiz Burning Man Survey

What is your gender? M, F




What is your age? Fill in

A. 

  N/A





Where do you live? Drop down - Outside US,

and list of states





Where do you live? Drop down

What is your educational background?

drop down – from currently in middle school to grad degree


Burning Man Survey

Continuous, variable translated into intervals

Pew: Race   RACE What is your race?   1 White   2 Black or African-American   3 Asian or Pacific Islander   4 Mixed race   5 Native American/American Indian   6 Other (SPECIFY)   9 Don't know/Refused

Burning Man: Race

Income: Pew Last year, that is in 2006, what was your total family income

from all sources, before taxes. 1 Less than $10,000 2 $10,000 to under $20,000 3 $20,000 to under $30,000 4 $30,000 to under $40,000 5 $40,000 to under $50,000 6 $50,000 to under $75,000 7 $75,000 to under $100,000 8 $100,000 or more 9 Don't know/Refused





Where do you live? Drop down

What is your educational background?

Respondent characteristics: specific

Opinions, Behavior, Activity

Opinions, preferences, concerns

  About the site: Content, organization, architecture, interface

  Ease of use   Perceived needs   Preferences   Concerns

  E.g., security

  Success, satisfaction   Subdivided by part of site, task, purpose…

  Other requirements   Suggestions

Behavior   Tasks (e.g., what did you do today?)   Site usage, activity

  Frequency; common functions – hard to answer accurately   Self-reports vs observations

  Time:   This event   Today   The last time you...   The last week   The last month   Generally...

Pew: Activity (open-ended) We'd be interested in hearing about the most amazing thing that has ever happened while you were online.

It could be a story about how something on the Web changed your life, something interesting you discovered on the Web, or something really important or even really amusing that happened during an exchange of email.

Internet Experience: Gates Foundation

Internet experience 2

Pew – networked workers

PEW: Opinions 1. Some people say they feel overloaded with information these days, considering all the TV news shows, magazines, newspapers, and computer information services. Others say they like having so much information to choose from. How about you... do you feel overloaded, or do you like having so much information available?

Feel overloaded Like having so much information available

2. Overall, do you think that computers and technology give... people MORE control over their lives, LESS control over their lives, or don't you think it makes any difference?

More Less No difference

PEW: behavior/activity 3. About how often do you go online from home?

Several times a day About once a day 3-5 days a week 1-2 days a week Every few weeks

Less often Never

Activities: Matrix question

Matrix question

Asking for follow-up

Writing Questions: Guidelines

Writing questions involves:   Deciding on topics to be addressed   Ordering the topics   Writing a specific SET of questions   Writing the ANSWERS

  Comprehensive list   Unambiguous   Mutually exclusive (unless you ask “check all that apply”)   GOAL:

  consistency x respondents   Unambiguous findings (insofar as possible)

  Remember   People skim   People answer quickly

Variations on a theme   What type of website do you visit?   What type(s) of websites do you visit?   What type of website do you visit most often? (Check

one)   What types of websites do you visit most often? (Check

no more than three)   What types of websites do you visit most often? (Check

all that apply)   What type(s) of websites have you visited today?   What type(s) of did you visit the last time you used the

internet?

Variations on a theme (cont): Types of websites

  Search (e.g., Google, Yahoo)   News (e.g. Google News, newyorktimes.com)   Entertainment (???)   Social networking (e.g., Facebook, YouTube)   Video or photo sharing (e.g., YouTube, Flickr)   Corporate or organizational sites (www.berkeley.edu;

www.microsoft.com; www.apple.com)   Email ??

Types of questions

Start a survey > questions > examples

Writing questions – some guidelines   Clear instructions

  “Choose one only” or “choose as many as apply”

Bad survey questions   Barreling:

  Bad: Is this site clear and useful?   Good: Do you find the information on this site clearly presented? Is it useful to

you?

  Loose Bundling vs. Anchoring: Questions are too general and not grounded in specific behaviors.   Bad: How often do you use the internet?   Good: For the following internet uses, please indicate how often…LIST: email,

online shopping, online banking, reading blogs…etc etc   Assumptions

  Bad: How has blogging changed your life?   Good: Do you blog? If yes, how well does this statement describe you? “Blogging

has changed my life.”   Answer: scale from “not at all” to “very well.”

Writing questions – some guidelines

  Unambiguous questions   Caution re “what would you do”

Not too many items for ranking

Not too many ratings (with neutral!)   No more than 7; preferably 5 or fewer

Writing: Parallel questions

White space, attractive design

Design: attractive; white space

Avoid convoluted instructions, questions, answers

  People SKIM

How interesting would each of the following displays be to you?

* Historic displays with old-fashioned and famous police artifacts from the beginning of America, through the Wild West, to the mobster era and more recent times

* Modern displays where visitors go behind the badge and experience what it's like to walk in an officers shoes today, featuring the latest crime-fighting equipment and tactics

(Another ex) Avoid convoluted instructions, questions, answers

Questions should be broadly applicable (or use a skip pattern)

Recognition rather than recall   P

Please evaluate the topics covered in this workshop:

Topic 1: ____________ How useful was this topic? How would you rate the presenter? What did we NOT cover that you were hoping to learn about?

Topic 2: _______________

More guidelines   Edit! Use good English syntax.   Focused questions people can answer briefly.

Reporting Survey Data

Some principles   Reader must know how many total respondents are

represented, and must be able to calculate from data provided any numbers they may be interested in

  Reader must know which responses/respondents are included and which not

  Reader needs exact wording, answer choices offered, since these affect precise answers

  Don’t know/refused: with large #s, there are always some. Are they included in %ages?

  Any table or graphic should still make sense separated from the report.   Readers don’t always read text; some just look at the “pictures”   Tables and graphics get copied and re-used

Types of Variables and Measures of Central Tendency

Type Examples Measures of Central Tendency

Nominal gender, race, college major

mode

Ordinal preferences, satisfaction, opinion; age or income in intervals

mode, median

Interval (arbitrary zero point )

Year (calendar date) temperature Celsius or Fahrenheit

mode, median, or mean

Ratio age, income, time mode, median, arithmetic mean

Ratio or Nominal?

Ratio or Nominal?

1 2 3 4

Pew – Nominal variable

Burning Man - nominal

Ordinal Variable

Tables

Questionnaire with the data filled in

Questionnaire with the data filled in

% % % %

Tables

Pew – cross tabs

Independent variable Dependent variable

Table Summarizing Key Findings

surveys and usability - uc berkeley school of...

Documents