ux-research 6b results-evaluation questionnairejohn/ims-ux/material/2019/ux... · ux research...

UX ResearchQuestionnaires / Surveys

Analysis

KP Ludwig John

UX ResearchQuestionnaire Analysis


Analysis

KP Ludwig John

Qualitative Quantitative

•describes the qualities or characteristics of something

•provide details about human behavior, emotion, and personality characteristics

•helps to understand why people do the things they do

•any research that can be measured

•provide data that can be expressed in numbers

•answers questions like “How many people clicked here”

Reference: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php

ApproachesQuestionnaire

What you getLarge amount of Text

What you getNumbers

http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php


Analysis

KP Ludwig John

QualitativeQuestionnaire

HSA UX-Lab; Questionnaire 2015; Text Answers


Analysis

KP Ludwig John

QualitativeQuestionnaire

IA6-2018; Team „Grabpflege“; UX-Observation Test and Interview


Analysis

KP Ludwig John

Qualitative

Reference: https://www.nsf.gov/pubs/1997/nsf97153/chap_4.htm

Questionnaire

Goal

Interpreting meaningful patterns or themes

Meaningfulness is determined by goals and objectives of the project depends on the particular research or evaluation questions

Good qualitative analysis is both: systematic and disciplined

How to analyze


Analysis

KP Ludwig John

Qualitative

Reference: https://www.wikihow.com/Analyze-Qualitative-Data

QuestionnaireHow to analyzeIt's easy to be overwhelmed with the amount of content you have created.

Thematic / content analysis Generating ‘codes’ that describe themes in the text, such as ‘Anxiety’ or ‘Eating habits’.

One recommended method for Qualitative analysis to structure it:


Analysis

KP Ludwig John

Qualitative


QuestionnaireHow to analyze

• What patterns and common themes emerge in responses dealing with specific items?

• Are there any deviations (Abweichungen) from these patterns?

• What interesting stories emerge from the responses? How can these stories help to illuminate the broader study question(s)?

• Do any of these patterns or findings suggest that additional data may need to be collected? Do any of the study questions need to be revised?

Ask the following questions:


Analysis

KP Ludwig John

Qualitative


Questionnaire

Data Reduction

General steps

Data DisplayConclusion Drawing / Verification

Identifying Patterns and Themes

Procedure

Record and Process Data


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedure Record and Process Data


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedure

• Things that stuck out to you • Time/date details • Other observations • Highlights from the interaction

Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-qualitative-data/

Record and Process Data As soon as data is collectedrecord detailed notes:

Do this while the interaction is still fresh in your mind so that you can record your thoughts and reactions as accurately as possible.

Make a reflection sheet template that you carry with you and complete after each interaction so that it is standardized across all data collection points.


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedureUndergo a data reduction process

Determine what is significant and transform the data into a simplified format

To determine what is meaningful data always refer back to your research questions.


Data Reduction


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedure Data Reduction

UX-Test IAM 2015 Virtuelle Architektur

„ArchiVision“

•chronological sequence of events observed •complemented by remarks of the observers • in Tabular form!

Written Log (Beobachtungsprotokoll)


Analysis

KP Ludwig John

Qualitative


QuestionnaireProcedure

1.Coding the data for certain words or content 2. Identifying their patterns 3. Interpreting their meanings.

Content analysis:

Go through all of the text and label words, phrases, and sections of text that relate to your research questions. Now sort and examine the data to look for patterns.

Identifying Patterns and ThemesGroup data into patterns and themes by:


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedure Identifying Patterns and Themes

Themes may be:

Thematic analysis


Group data into patterns and themes by:

Group the data into themes that will help answer the research question(s).

• Directly evolved from the research questions and were pre-set before data collection even began, or

• Naturally emerged from the data as the study was conducted.


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedure Identifying Patterns and Themes

Label and assign Themes (Indexing)


Analysis

KP Ludwig John


Provide "an organized, compressed assembly of information that permits conclusion drawing

Data Display

It should help to think about the data in new ways and assist you in identifying systematic patterns.


The display can be a graphic, table/matrix, or textual display.


Analysis

KP Ludwig John


Index Tables

Data DisplayThe display can be a graphic, table/matrix, or textual display.

UX-Test IAM 2015 Virtuelle Architektur „ArchiVision“


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedureConclusion Drawing / Verification

To draw reasonable conclusions:

• Step back and interpret what your findings mean • Determine how your findings help answer the research question(s) • Draw implications from your findings


To verify these conclusions, revisit the data (multiple times) to confirm the conclusions that you have drawn.


Analysis

KP Ludwig John

QualitativeQuestionnaireProcedure Conclusion Drawing / Verification

Where did it appear?

Description

Possible Causes

UX-Test IMS.mobile 2013Digitaler AusstellungsbegleiterBritta Diehm, Xiaomeng Jiang, Yue Ma, Kerstin Vierthaler

Issue tableUX - Observation


Analysis

KP Ludwig John


Involve more than one person into the analytic process to serve as a cross-check and source of new ideas

Recommendations

Leave enough time for analysis and writing:Analyzing qualitative data almost always takes more time, thought, and effort than anticipated.



Analysis

KP Ludwig John

Questionnaire

Quantitative•any research that can be measured

•provide data that can be expressed in numbers

•answers questions like “How many people clicked here”

What you getNumbers

Reference: http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php

http://www.uxmatters.com/mt/archives/2012/09/strengths-and-weaknesses-of-quantitative-and-qualitative-research.php


Analysis

KP Ludwig John

Questionnaire Quantitative

HSA UX-Lab 2013, Measured Results


Analysis

KP Ludwig John


Reference: http://toolkit.pellinstitute.org/evaluation-guide/analyze/analyze-quantitative-data/

Provides quantifiable and easy to understand results.

Data may have one of the following formats

Nominal (Nennwert)Ordinal (Ordnungszahl)Interval (standardisierter Abstand)Ratio (Scale) (Verhältnis)

Levels of measurement


Analysis

KP Ludwig John

Questionnaire

Reference: https://www.youtube.com/watch?v=klgFMJppfcY


J David Eisenberg


Analysis

KP Ludwig John



Levels of measurementNominal (Nennwert)

Example:

Male or Female There is no order associated with male nor female Each category is assigned an arbitrary value male = 0, female = 1

no logical orderbasic classification data


Analysis

KP Ludwig John



Levels of measurementOrdinal (Ordnungszahl)

Example:

T-shirt size (small, medium, large) Military rank (from Private (Gefreiter) to General)

Data has a logical orderDifferences between values are not constant


Analysis

KP Ludwig John



Levels of measurementInterval (standardisierter Abstand)Data is continuous and has a logical orderhas standardized differences between values, but no natural zero

Example: Items measured on a Likert scalerank your satisfaction on scale of 1-5.

1 = Very Dissatisfied 2 = Dissatisfied 3 = Neutral 4 = Satisfied 5 = Very satisfied

Example: Fahrenheit or Celsius degrees Remember that ratios (Relationen) are meaningless for interval data. You cannot say, for example, that one day is twice as hot as another day.


Analysis

KP Ludwig John



Levels of measurementRatio (Scale) (Verhältnis)

Data is • continuous • ordered • has standardized differences between values • and a natural zero

Example:height, weight, age, length Having an absolute zero enables you to meaningful say that one measure is twice as long as another.10 cm is twice as long as 5 cm


Analysis

KP Ludwig John

Questionnaire

Reference: https://www.youtube.com/watch?v=eghn__C7JLQ


365 Data Science


Analysis

KP Ludwig John


• Data tabulation • Descriptive analysis • Data disaggregation • Analysis of Variance

Analysis procedures


Example Study (fictional)Summer program student surveyconducted with

Some standard methods

How to analyze

22 participants in two cities (NYC, Boston)


Analysis

KP Ludwig John

Questionnaire QuantitativeAnalysis procedures


Organize your collected data in a logical format Use a spreadsheet software (Libre Office Calc)

22 participants in two cities (NYC, Boston)

Some of the responses are entered as numerical values.

Each participant is assigned to a unique participant ID responses are organized by survey item/question.


Analysis

KP Ludwig John

QuantitativeQuestionnaireAnalysis procedures

Some responses transformed intonumerical values

• variable/question name • the answer options • the numerical code

assigned to answer option

Create and Keepa Code book that lists:


Analysis

KP Ludwig John

Questionnaire QuantitativeAnalysis procedures Data tabulation (Tabellierung)

Tabulate your results in your data set Construct frequency and percent distributions



Analysis

KP Ludwig John?

Questionnaire QuantitativeAnalysis procedures Descriptive analysis


•Mean the numerical average for a particular variable

•Minimum and maximumthe highest and lowest value for a particular variable

•Mode most common number score or value for a variable

Your Results

For a first interpretationdetermine the


Analysis

KP Ludwig John

Questionnaire QuantitativeAnalysis proceduresMean (Arithmetisches Mittel, Durchschnitt)

Numerical average for a particular variable

Obtained by summing all answers or scores and dividing by the total number of entries

Reference: http://learningstore.uwex.edu/assets/pdfs/G3658-6.pdf

Descriptive analysis


Analysis

KP Ludwig John


Numerical average for a particular variable

Obtained by summing all answers or scores and dividing by the total number of entries



?Example

The wait times (in minutes) of five customers in a bank: 3, 2, 4, 1, and 2.

(3 + 2 + 4 + 1 + 2) : 5 = 12 : 5 = 2,4 min.

The mean waiting time:


Analysis

KP Ludwig John



https://support.minitab.com/en-us/minitab/18/help-and-how-to/statistics/basic-statistics/how-to/store-descriptive-statistics/interpret-the-statistics/interpret-the-statistics/#mean

Use the mean to describe the sample with a single value that represents the center of the data.

Many statistical analyses use the mean as a standard measure of the center of the distribution of the data.

Example

The wait times (in minutes) of five customers in a bank are: 3, 2, 4, 1, and 2.

(3 + 2 + 4 + 1 + 2) : 5 = 12 : 5 = 2,4 min.

The mean waiting time:


Analysis

KP Ludwig John



Mode (Modalwert, Häufigster aller Werte)

Most commonly occurring answer or valueUsually what people refer as: „the typical“

Only important when a large number of values is availableNot effected by extrem values.


ExampleThe wait times (in minutes) of five customers in a bank are: 3, 2, 4, 1, and 2.

2 min.The mode waiting time: ?


Analysis

KP Ludwig John



Median (mittlerer aller Werte)

Middle Valuehalf of the cases fall below, half above the value

Arrange data from one extreme to the otherCount halfway through the list and find the median value

When two numbers tie for the halfway point,take the two middle numbers, add them and divide by 2 to get the median

Not effected by extremes or a range of data (like mode)


Analysis

KP Ludwig John



Analysis proceduresMean, Mode, Median

Depends on your dataand the purpose of analysis

Which makes more sense?Durchschnitt, Häufigster, Mittlerer


Analysis

KP Ludwig John


One value drastically changed


$460,000

Recalculate!

Which makes more sense?Durchschnitt, Häufigster, Mittlerer


Analysis

KP Ludwig John


One value drastically changed


$460,000

Recalculate!

Which makes more sense?

$780,000

$78,000


Analysis

KP Ludwig John


Disaggregating (Aufschlüsselung)Analysis procedures

Crosstabs (Kreuztabellen)

Disaggregate the data across multiple categories Allow to explore findings in depth Useful to show differences among subgroups



Analysis

KP Ludwig John


• Females overrepresented in New York,males overrepresented in Boston


Analysis procedures

• Over 70% of the White sample is in the Boston program

• Altogether 6 of the Boston respondents are students of color

Disaggregating (Aufschlüsselung)Crosstabs (Kreuztabellen)


Analysis

KP Ludwig John



All of the students who were dissatisfied with the program were students of color.

Analysis procedures

From these results it may be concluded that the Boston program is not meeting the needs of its students of color.

5 of the 20 students were dissatisfied with the program

All but one of the students of color in the Boston program were dissatisfied (since there were 6 students of color in the Boston program.)

Disaggregating (Aufschlüsselung)


Analysis

KP Ludwig John


Correlation (Wechselwirkung)

Relationship between two variables (strong and negative, weak and positive, statistically significant)

• Correlation does not explain causation • Indicates that a relationship or pattern exists,

but it does not mean that one variable is the cause of the other.

You might see a strong positive correlation between participation in the summer program and students’ grades the following school year. However, the correlation will not tell you if the summer program is the reason why students’ grades were higher.

Analysis procedures

Example



Analysis

KP Ludwig John


Variability (Schwankungsbreite)Analysis procedures


One extreme (high or low) may have strong influence on your over all mean (Durchschnittswert)

Looking at variability provides a better understanding of the results

Analysis of variance (ANOVA)determines whether the difference in means (averages) for two groups is statistically significant.


Analysis

KP Ludwig John

Reference: https://www.youtube.com/watch?v=ITf4vHhyGpc Analysis of variance (ANOVA)J David Eisenberg


Analysis

KP Ludwig John



Analysis proceduresMeasures of Variability

Range

Compares the highest and the lowest valueto indicate the spread of responses or scores

Example

Soil testing for phosphorus saved producers anaverage $15 / acreranging from $12 to $20 / acre

Only considers the highest and the lowest score.Helps to better understand the date but not a full measure of variation


Analysis

KP Ludwig John

Questionnaire

Reference: https://www.youtube.com/watch?v=ipYaHqutMds

Measures of Variability

Lydia Flinn


Analysis

KP Ludwig John

Questionnaire QuantitativeAnalysis proceduresMeasures of Variability

Standard deviation (Abweichung)

Measures the degree to which individual values vary from the mean

High standard deviation means responses vary greatly from the mean.

https://www.mathsisfun.com/data/standard-deviation.html https://www.easycalculation.com/statistics/standard-deviation.php

How to calculate?Detailed but easy to understand explanation and examples at:

Why is it useful in UX?


Analysis

KP Ludwig John

Questionnaire QuantitativeAnalysis proceduresMeasures of Variability

Standard deviation (Abweichung)

Compare the UX of two products To evaluate differences between two versionscheck if the differences are significant


Analysis

KP Ludwig John


Reference: IMS.UX WS 2017-18, HSA, Prof. Dr. Hariet Köstner

Presentation of Results

Who do you address?


Analysis

KP Ludwig John




suitable for

• questions with many answers • group comparisons, frequencies, percentages, means can be shown

Bar chart


Analysis

KP Ludwig John




• answers that sum up to 100 • not to many answers

suitable for

Pie chart


Analysis

KP Ludwig John



Presentation of ResultsStacking Diagram

• answers that sum up to 100 • not to many answers

suitable for


Analysis

KP Ludwig John



Presentation of ResultsLine chart

suitable for • questions with many answers • group comparisons, frequencies, percentages, means can be shown • compact graphical representation of a lot of information


Analysis

KP Ludwig John

PracticeAnalyse your dataand communicate your findings

based on your research question:

How do changes in design and implementation of the survey

influence its outcome?

Present (12 Min.) and deliver (PDF) your results next week


Analysis

KP Ludwig John

Practice Criteria

• Precise definition ofResearch Question and measurements

• definition of biases and striven tendency

• Methodically consistent implementation and execution

• Critical Evaluation of test procedure

• Clear summery of findings

Example: IMS.UX 2018 - Team Wulfpack


Analysis

KP Ludwig John

Practice Criteria

hs-augsburg.de/homes/john

Next appointments and script:

ux-research 6b results-evaluation questionnairejohn/ims-ux/material/2019/ux... · ux research...

Documents