ux-research 2c methods questionnaire - hs …...ux research methods questionnaire ux research...

UX ResearchQuestionnaires / Surveys

KP Ludwig John

UX Researchmethods Questionnaire


KP Ludwig John

Research Methods

Qualitative Quantitative•describes the qualities or characteristics of something

•provide details about human behavior, emotion, and personality characteristics

•helps to understand why people do the things they do

•any research that can be measured

•provide data that can be expressed in numbers

•answers questions like “How many people clicked here”

Reference: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php

Questionnaire?

Approaches

http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php


KP Ludwig John

Reference: http://www.fao.org/docrep/w3241e/w3241e05.htm

QuestionnaireApproaches

Example: Interviewing the head of the household to find out how decisions are made within the family when purchasing breakfast foodstuffs

A formal questionnaire may restrict the discussion and prevent a full exploration of the processes.

Prepare a brief guide, listing perhaps ten major open-ended questions, with appropriate probes/prompts listed under each.

Exploratory questionnaire: • collection of qualitative data • no statistically evaluation • probably no formal questionnaire needed

http://www.fao.org/docrep/w3241e/w3241e05.htm


KP Ludwig John

Reference: https://chaione.com/blog/ux-research-standardizing-usability-questionnaires/

QuestionnaireApproaches

Benefits: Quantification: Standardized measurements

Generalization: generalizing a finding from a sample to the greater population.

Communication: easier to communicate findings by referring to standardized metrics.

Quick Comparisons: easy to compare different iterations throughout the development process.

Formal standardized questionnaire: • to test and quantify hypothesis • to be analyzed statistically

https://chaione.com/blog/ux-research-standardizing-usability-questionnaires/


KP Ludwig John

StandardizedQuestionnaires


KP Ludwig John

System Usability Scale (SUS)

https://en.wikipedia.org/wiki/System_usability_scale

Developed by John Brook in 1986 at Digital Equipment Corporation (UK) as a tool in usability engineering of electronic office systems

simple, ten-item attitude testgiving a global view of subjective assessments of usability

Likert scale

Summed up in form of a

https://en.wikipedia.org/wiki/United_Kingdom

https://en.wikipedia.org/wiki/Usability_engineering

https://en.wikipedia.org/wiki/Electronic_office

https://en.wikipedia.org/wiki/Usability


KP Ludwig John


https://en.wikipedia.org/wiki/Likert_scale

Respondents specify their level of agreement or disagreement on a symmetric agree-disagree scale for a series of statements.

bipolar scaling method, measuring either positive or negative response to a statement

Likert scalenamed after its inventor, psychologist Rensis Likert (1903 - 1981)

Five response options for respondents; from Strongly agree to Strongly disagree.

SUS: Question 1

I think I would like to use this system frequently

http://www.usabilitest.com/sus


KP Ludwig John

System Usability Scale (SUS)The 10 template questions which you can adapt:

1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person

to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use

this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things

before I could get going with this system.

https://usabilitygeek.com/how-to-use-the-system-usability-scale-sus-to-evaluate-the-usability-of-your-website/


KP Ludwig John

System Usability Scale (SUS)Calculate the ScoreUsers have ranked each of the 10 templates questions from 1 to 5

• For each of the odd numbered questions, subtract 1 from the score.

• For each of the even numbered questions, subtract their value from 5.

• Take these new values which you have found, and add up the total score. Then multiply this by 2.5.

The result is your score out of 100. (This is NOT a percentage!)


What does the score tell?


KP Ludwig John



Interpret the ScoreTranslated into notes:

https://measuringu.com/sus/

• 80.3 or higher is an A. People love your site andwill recommend it to their friends

More details at:

• 51 or under gets you a big fat F. Make usability your priority now and fix this fast.

• 68 or thereabouts gets you a C. You’re doing OK but could improve

SUS scores from about 500 surveys, conducted by Jeff SauroAverage SUS score = 68


KP Ludwig John

User Experience Questionnaire (UEQ)

http://www.cheval-lab.ch/was-ist-usability/usabilitymethoden/frageboegen/user-experience-questionnaire/

Originally developed at SAPfreely accessible since 2006

Explores the following six factors:

Used to measure the user experience of software products Takes into account classic aspects of Usability (Efficiency) as well as of User Experience (Attractiveness)


KP Ludwig John


http://www.ueq-online.org/

(Attraktivität)Overall Impression of the product.

Do users like or dislike this?


(Verständlichkeit)Is it easy to get familiar

with the product?

(Effektivität)Can users solve their tasks

with the product without unnecessary effort?

(Verlässlichkeit)Does the user feel in control

of the interaction?

(Stimulation)Is it exciting and motivating

to use the product?

(Originalität)Is the product

innovative and creative?


KP Ludwig John

User Experience Questionnaire (UEQ) http://www.ueq-online.org/


Attractivenessis a pure valence (Wertigkeit) dimension

Perspicuity, Efficiency, Dependability pragmatic quality aspects

(goal-directed)

Stimulation, Novelty hedonic (Sinnengenuss) quality aspects

(not goal-directed)

Grouped in categories


KP Ludwig John


26 pairs of contrasting attributes that may apply to the product


Questionnaire


KP Ludwig John


Please decide spontaneously. Don’t think too long about your decision to make sure that you convey your original impression

Tick a circle in every line. It is your personal opinion that counts.


Procedure


KP Ludwig John

User Experience Questionnaire (UEQ)Typical applications


Compare the UX of two products

However, if you want to draw conclusions on this (especially if your sample is small) you have to check if the differences are significant.

Version A shows for all scales, with the exception of Novelty better values than version B.


KP Ludwig John


General UX Evaluation of a product


Haptic Interface

Research Topics:Do Users understand how to interact / what to do?

Interactive Mirror

Research Topics:Dimension, Observer DistanceObserver Position

Source: IA6 2015, Team Salz, IA Installation


KP Ludwig John


General UX Evaluation of a product


Does the product fulfill the general expectations concerning UX? Such expectations of users are formed by products they frequently use.



KP Ludwig John



General UX Evaluation of a productDoes the product fulfill the general expectations concerning UX? Such expectations of users are formed by products they frequently use.



KP Ludwig John

What should be changed?Cannot be answered directly by a quantitative measurement of user experience.A connection of product features to the measurement is required.

UEQ shows a pattern of 6 measured user experience qualities. Serves as Basis to make some assumptions where to look for improvements.

User Experience Questionnaire (UEQ)Benchmark graph


Shows UX quality of evaluated product


KP Ludwig John

CustumizedQuestionnaire

(Survey)


KP Ludwig John

Source: http://www.surveysystem.com/sdesign.htm

Survey

Basic types:

Questions and Answers


KP Ludwig John

Source: https://www.nngroup.com/articles/open-ended-questions/

Survey in User Research

• allow to give a free-form answer • users can to talk in depth about a question • possible to derive valid information from small number of users

• can be answered with “Yes” or “No“ • have a limited set of possible answers • good for surveys

Closed-ended questions

Open-ended questions

• get higher response rates (easier to answer) • useful with larger number of participants • responses can easily be analyzed statistically


KP Ludwig John


Survey

Closed / Open - How to ask?

closed open

Are you satisfied? How satisfied or dissatisfied are you with this process?

Did it act as you expected? What would (did) you expect to happen when you ... ?

Do you think you would use this? How would this fit into your work?

Does that work for you? What do you think about that?

Do you normally … ? How do you normally ... ?

Do you like this?What would you most want to change about … ? Which things did you like the best about … ?

in User Research


KP Ludwig John


Survey


closed open







in User Research

2 minutes


KP Ludwig John


Survey


closed open







in User Research


KP Ludwig John


Survey

people may

• share motivations that you didn’t expect • mention behaviors and concerns that you knew nothing about. • reveal surprising mental models, problem-solving strategies, hopes, fears, and much more.

Open-ended questionsallow you to find more than you anticipate

in User Research


KP Ludwig John


Survey

Closed-ended questions stop the conversation and eliminate surprises

Choose your favorite ice cream: vanilla, strawberry, or chocolate.

Answers that you suggest can reveal what you are looking for. People may be influenced by the questions.

When you ask closed-ended questions, ask an open-ended question at the end:“What else would you like to say about that?”

in User Research

Ad Other __________

to a set of multiple-choice answers … to get open-ended feedback.


KP Ludwig John

Survey

When to Ask Open-Ended Questions

in User Research

While conducting conceptual research:

• Which problems to solve • What kind of solution to provide • Who to design for

For exploratory studies, such as

• Qualitative usability testing • Interviews and Diary studies • Persona research • Task analysis



KP Ludwig John

Survey

When to Ask Closed-Ended Questions

in User Research

• In quantitative usability studies, measuring time on task and error rates, to compare results among users

• In surveys which large number of participants

• When the set of answers is strictly limited for some reason



KP Ludwig John


treated as

Rating Scales / Agreement Scales

multiple choice

multiple choice

numeric

Survey Questions and Answers


KP Ludwig John


Structure / OrderEarly questions in a survey should be easy and pleasant to answer. These kinds of questions encourage people to continue the survey.Leave difficult or sensitive questions until near the end of your survey

Survey Questions and Answers

Whenever there is a logical or natural order to answer choices, use it. Always present agree-disagree choices in that order. Presenting them in disagree-agree order will seem odd. Same for positive to negative and excellent to poor

When using numeric rating scales higher numbers should mean a more positive or more agreeing answer


KP Ludwig Johnhttps://youtu.be/bi2JMRpL9Ik

Survey


KP Ludwig John

Source: https://youtu.be/bi2JMRpL9Ik by Chris Gray

How to Create?

Step 1What are your objectives for the survey?Formulated research topics / field

Step 2Find questions to cover them

Step 3Group similar questionsOrder them logically

Step 4Pre-Test your survey with a small sub-group of your audience

Survey

https://youtu.be/bi2JMRpL9Ik


KP Ludwig John

Questions - How to phrase?• easy to understand

• appropriate for your audience

Source: http://uxmastery.com/better-user-research-through-surveys/ by Chris Gray

!This doesn’t mean that they are qualified to answer them, or are able to provide insightful feedback.

People can and will answer just about any question put in front of them

Survey


KP Ludwig John

Questions - How to phrase?

• One Issue (concept) per questionDon’t combine two in one

(Does climate change exist + politics do not care)


• avoid double negatives or confusing questions

(Don’t you think, this isn’t true?)

Survey


KP Ludwig John

• use balanced rating scales (positive - negative)


Survey Questions - How to phrase?


KP Ludwig John

• use open questions

Not: Do you think, that … yes/noBut: To what extend do you agree with the following …

• allow „Don’t Know“ as multiple choice option

• combine multiple choice with Text field for explanations (why, comments)


Survey Questions - How to phrase?

keep the survey short!


KP Ludwig John

CustumizedQuestionnaire

(Interview)


KP Ludwig John

Source: https://www.nngroup.com/articles/leading-questions/

Interviews Questions - How to phrase?

Avoid Leading Questions

Which phrasing of a follow-up question will result in more accurate results? ?

Discuss and collect1 minute

• “What was easy or difficult about getting to the content you wanted?”

• “Why did you have difficulty with the navigation?”

• “I saw you were having difficulty with the navigation. What happened?”


KP Ludwig John







The first two questions are leading,meaning that it includes or implies the desired answer to the question in the phrasing of the question itself.

Which phrasing of a follow-up question will result in more accurate results?

Interviews


KP Ludwig John





The interviewer rephrases what was observed, which may not be an accurate representation of the user’s experience. The question also names a user interface element – the navigation – which is a term that users may or may not fully understand, relate to, or normally use.

Question 1

Interviews


KP Ludwig John




Again, this question implies the answer and assumes that navigation was the problem.

It also puts the blame on the user, rather than on the site. The question focusses on the user’s actions as opposed to the elements in the site that may have contributed to the user’s actions.

Question 2


!

Interviews


KP Ludwig John




Steers the user to the topic of interest without suggesting terms or feelings to the user.

The user can say it was simple to move around or difficult, without disagreeing with the interviewer. Here the interviewer offers a general frame for the topic of the question, rather than suggesting a response.

Question 3


!

Interviews


KP Ludwig John




Leading questions

The interviewer is the “authority” in the room and many participants will not want to disagree

… make it difficult for the participant to express another opinion.

… interject the answer we want to hear in the question itself.

… may result in biased or false answers

… rob us of the opportunity to hear an insight we weren’t expecting

Interviews


KP Ludwig John



Avoid Leading QuestionsTherefore

Interviews

• Do not rephrase in our own words. ◦ Participant: “I notice this picture here… “ ◦ Researcher: “You mentioned that the picture was helpful. What about it did you like? “ ◦ Improvement: “You mentioned the picture…?”

• Do not assume you know what the user is feeling. ◦ “When you were struggling with this task, what was happening?” ◦ Improvement: “What was easy or difficult about completing that task?”

• Do not name an interface element. ◦ “The related links on the side of the page here – where would those lead?” ◦ Improvement: “This area on the side of the page… [point to area]. What is that?”

• Do not suggest an answer. ◦ “How well would this save time for you during your workday?” ◦ Improvement: “How might this affect your efficiency, if at all?”


KP Ludwig John

Source: https://www.interaction-design.org/literature/article/how-to-prepare-for-a-user-interview-and-ask-the-right-questions

Questions - OrderInterviews

Ann Blandford - professor of Human Computer Interaction at University College London

Which questions at the beginning of an Interview


KP Ludwig John


Questions - OrderWhich questions at the beginning of an Interview

Interviews

Start with brief and easy to understand questions in a vocabulary that is familiar to your participant

Turn your overall research question into more concrete questions that will then answer your overall question


KP Ludwig John


Questions - DetailInterviews

Ann Blandford - professor of Human Computer Interaction at University College London

Concrete Examples of critical incidents


KP Ludwig John


Interviews

Use questions like “Can you tell me about the last time you used video streaming?”

Ask the concrete “how” and “what” questions before you ask the more abstract “why” questions

Ask for concrete examples.

Concrete Examples of critical incidents

Questions - Detail


KP Ludwig John


When to use Surveys?

• Who your users are• What your users want• What they purchase• Where they shop• What they own• What they think of your brand or product

Effective method for identifying:

Generally: to understand the user

Survey


KP Ludwig John



• Gathering feedback on a live product or during a pilot

• Exploring the reasons people visit a website and assessing their experience of that visit

• Quantifying results from qualitative research activities

• Evaluating usability

In UX context

Survey


KP Ludwig John



But be

Cautious asking about behavior!

For assessing the effectiveness of a public transport ticketing system,observing people buying tickets and traveling throughout the network would yield more accurate and useful feedback than asking people how they had used the system over the last week.

Because

Survey


KP Ludwig John

https://www.nngroup.com/articles/interviewing-users/

Because

The critical failing: You're asking people to either remember past use or speculate on future use of a system.

Human memory is fallible. People can't remember the details of how they used a website.They tend to make up stories …

What users say and what they do are different- Jacob Nielsen


KP Ludwig John

https://www.nngroup.com/articles/interviewing-users/

What users say and what they do are different- Jacob Nielsen

Because

This is why Agile development and paper prototyping methods are valuable.

When users have something concrete to interact with, it's usually obvious when you're solving their problems in a way that's easy and pleasant to work with — and equally obvious when you're not.

Users are pragmatic and concrete.They typically have no idea how they might use a new technology based on a description alone.


KP Ludwig John

Survey ToolsMany online tools available Explore and try them out!

survio.com/de


KP Ludwig John

Survey Tools

survio.com/de


KP Ludwig John

Practice Finalize and plan

• Research Question and measurementsWhat is your research topic?What do you want to find out?Which measurements will be used?

• Two versions of the survey• one biased

indicate the biases usedand the dedicated results / tendency

• one contra biased


KP Ludwig John

Practice Criteria

• Precise definition ofResearch Question and measurements

• definition of biases and striven tendency

• Methodically consistent implementation and execution

• Critical Evaluation of test procedure

• Clear summery of findings


KP Ludwig John

PracticeNext week

• Topic of your research • dedicated results of surveys • How to achieve these results (Biases) • How to conduct your survey practically

hs-augsburg.de/homes/john

Next appointments and script:

Discussion and Feedback

ux-research 2c methods questionnaire - hs …...ux research methods questionnaire ux research...

Documents