understanding user satisfaction with intelligent assistants

44
Understanding User Satisfaction with Intelligent Assistants Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C. Crook, Imed Zitouni, Tasos Anastasakos Eindhoven University of Technology Pennsylvania State University University of Massachusetts Amherst Microsoft

Upload: julia-kiseleva

Post on 11-Jan-2017

743 views

Category:

Internet


1 download

TRANSCRIPT

Page 1: Understanding User Satisfaction with Intelligent Assistants

Understanding User Satisfaction with Intelligent Assistants

Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah,

Aidan C. Crook, Imed Zitouni, Tasos Anastasakos

Eindhoven University of Technology Pennsylvania State University

University of Massachusetts Amherst Microsoft

CHIIR’16, Chapel Hill, NC, USA

Page 2: Understanding User Satisfaction with Intelligent Assistants

Q1: how is the weather in ChicagoQ2: how is it this weekendQ3: find me hotelsQ4: which one of these is the cheapestQ5: which one of these has at least 4 starsQ6: find me directions from the Chicago airport to number one

User’s dialogue with

Cortana:Task is

“Finding a hotel in

Chicago”

Page 3: Understanding User Satisfaction with Intelligent Assistants

Q1: find me a pharmacy nearbyQ2: which of these is highly ratedQ3: show more information about number 2Q4: how long will it take me to get thereQ5: Thanks

User’s dialogue with

Cortana:Task is

“Finding a pharmacy”

Page 4: Understanding User Satisfaction with Intelligent Assistants

Research Questions• RQ1: What are characteristic types of scenarios of

use?

Page 5: Understanding User Satisfaction with Intelligent Assistants

Controlling Device• Call a person

• Send a text message

• Check on-device calendar

• Open an application

• Turn on/off wi-fi

• Play music

Page 6: Understanding User Satisfaction with Intelligent Assistants
Page 7: Understanding User Satisfaction with Intelligent Assistants

Knowledge Pane

Image Answer

Page 8: Understanding User Satisfaction with Intelligent Assistants

Knowledge Pane

Image Answer Image Answer

Organic Results

Page 9: Understanding User Satisfaction with Intelligent Assistants

Knowledge Pane

Image Answer Image Answer

Location Answer

Organic Results

Page 10: Understanding User Satisfaction with Intelligent Assistants

User:“Do I

need to have a jacket

tomorrow?”

Search Dialogue

Page 11: Understanding User Satisfaction with Intelligent Assistants

User:“Do I

need to have a jacket

tomorrow?”

Cortana: “You could

probably go without one. The forecast

shows …”

Search Dialogue

Page 12: Understanding User Satisfaction with Intelligent Assistants

Cortana: “Here are

ten restaurants near you”

User:“show

restaurants near

me”

Search Dialogue

Page 13: Understanding User Satisfaction with Intelligent Assistants

Cortana: “Here are

ten restaurants near you”

Cortana:“Here are ten restaurants

near you that have good reviews”

User:“show

restaurants near

me”

User:“show

the best restaurants near

me ”

Search Dialogue

Page 14: Understanding User Satisfaction with Intelligent Assistants

Cortana: “Here are

ten restaurants near you”

Cortana:“Here are ten restaurants

near you that have good reviews”

Cortana:“Getting you direction to the Mayuri

Indian Cuisine”

User:“show

restaurants near

me”

User:“show

the best restaurants near

me ”

User:“show

directions to the second one”

Search Dialogue

Page 15: Understanding User Satisfaction with Intelligent Assistants

Research Questions• RQ1: What are characteristic types of scenarios of use?

• RQ2: How can we measure different aspects of user satisfaction?

• RQ3: What are key factors determining user satisfaction for the different scenarios?

• RQ4: How to characterize abandonment in the web search scenario?

• RQ5: How does query-level satisfaction relate to overall user satisfaction for the search dialogue scenario?

Page 16: Understanding User Satisfaction with Intelligent Assistants

Research Questions• RQ1: What are characteristic types of scenarios of use?

• RQ2: How can we measure different aspects of user satisfaction?

• RQ3: What are key factors determining user satisfaction for the different scenarios?

• RQ4: How to characterize abandonment in the web search scenario?

• RQ5: How does query-level satisfaction relate to overall user satisfaction for the search dialogue scenario?

USE

R

STU

DY

Page 17: Understanding User Satisfaction with Intelligent Assistants

User Study Participants

55%45%

LANGUAGEEnglish Other

• 60 Participants• 25.53 +/- 5.42 years

Page 18: Understanding User Satisfaction with Intelligent Assistants

User Study Participants

75%

25%

GENDER

Male Female

55%45%

LANGUAGEEnglish Other

• 60 Participants• 25.53 +/- 5.42 years

Page 19: Understanding User Satisfaction with Intelligent Assistants

User Study Participants

75%

25%

GENDER

Male Female

55%45%

LANGUAGEEnglish Other

82%

8%2% 8%

Education

Computer ScienceElectrical EngineeringMathematicsOther

• 60 Participants• 25.53 +/- 5.42 years

Page 20: Understanding User Satisfaction with Intelligent Assistants

User Study Design• Video Instructions (same for all participants)

• Tasks are realistic – mined from Cortana logs:

o Control type of taskso Queries where users don’t clicko Search dialogue tasks – mostly localization type of

queries

Page 21: Understanding User Satisfaction with Intelligent Assistants

Find out what is the hair color

of your favorite

celebrity.

Page 22: Understanding User Satisfaction with Intelligent Assistants

You are planning a vacation. Pick a

place. Check if the weather is good enough for the period you are planning the

vacation. Find a hotel that suits you.

Find the driving directions to this

place.

Page 23: Understanding User Satisfaction with Intelligent Assistants

You are planning a vacation. Pick a

place. Check if the weather is good enough for the period you are planning the

vacation. Find a hotel that suits you.

Find the driving directions to this

place.

Page 24: Understanding User Satisfaction with Intelligent Assistants

Questionnaire: Controlling Device

• Were you able to complete the task?o Yes/No

• How satisfied are you with your experience in this task?o 5-point Likert scale

• How well did Cortana recognize what you said?o 5-point Likert scale

• Did you put in a lot of effort to complete the task?o 5-point Likert scale

Page 25: Understanding User Satisfaction with Intelligent Assistants

Questionnaire: Controlling Device

• Were you able to complete the task?o Yes/No

• How satisfied are you with your experience in this task?o 5-point Likert scale

• How well did Cortana recognize what you said?o 5-point Likert scale

• Did you put in a lot of effort to complete the task?o 5-point Likert scale

5 Tasks20 Minutes

Page 26: Understanding User Satisfaction with Intelligent Assistants

Questionnaire: Good Abandonment

• Were you able to complete the task?o Yes/No

• Where did you find the answer?o Answer Box, Image, SERP, Visited Website

• Which query led you to finding the answer?o First, Second, Third, >= Fourth

• How satisfied are you with your experience in this task?o 5-point Likert scale

• Did you put in a lot of effort to complete the task?o 5-point Likert scale

Page 27: Understanding User Satisfaction with Intelligent Assistants

Questionnaire: Good Abandonment

• Were you able to complete the task?o Yes/No

• Where did you find the answer?o Answer Box, Image, SERP, Visited Website

• Which query led you to finding the answer?o First, Second, Third, >= Fourth

• How satisfied are you with your experience in this task?o 5-point Likert scale

• Did you put in a lot of effort to complete the task?o 5-point Likert scale

5 Tasks20 Minutes

Page 28: Understanding User Satisfaction with Intelligent Assistants

Questionnaire: Search Dialogue

• Were you able to complete the task?o Yes/No

• How satisfied are you with your experience in this task?o If the task has sub-tasks participants indicate their graded

satisfaction e.g. o a. How satisfied are you with your experience in finding a hotel? o b. How satisfied are you with your experience in finding directions?

• How well did Cortana recognize what you said?o 5-point Likert scale

• Did you put in a lot of effort to complete the task?o 5-point Likert scale

Page 29: Understanding User Satisfaction with Intelligent Assistants

Questionnaire: Search Dialogue

• Were you able to complete the task?o Yes/No

• How satisfied are you with your experience in this task?o If the task has sub-tasks participants indicate their graded

satisfaction e.g. o a. How satisfied are you with your experience in finding a hotel? o b. How satisfied are you with your experience in finding directions?

• How well did Cortana recognize what you said?o 5-point Likert scale

• Did you put in a lot of effort to complete the task?o 5-point Likert scale

8 Tasks: 1 simple, 4 with 2 subtasks, 3 with 3 subtasks

30 Minutes

Page 30: Understanding User Satisfaction with Intelligent Assistants

Search Dialog Dataset• 540 tasks that incorporated

• 2, 040 queries, of which 1, 969 were unique

• the average query-length is 7.07

• The simple task generated 130 queries in total

• Tasks with 2 context switches generated 685 queries

• Tasks with 3 context switches generated 1, 355 queries

Page 31: Understanding User Satisfaction with Intelligent Assistants

Factors Determining Satisfaction

RQ3: What are key factors determining user satisfaction for the different scenarios?

Page 32: Understanding User Satisfaction with Intelligent Assistants

Across Scenar-

ious

Device Control

Web Search

Structured Dialog

50

1

2

3

4

5

6

Across Scenar-

ious

Device Control

Web Search

Structured Dialog

50

1

2

3

4

5

6

Satis

fact

ion

Leve

l

Effor

ts

Results Over ScenariosMean of Satisfaction

Page 33: Understanding User Satisfaction with Intelligent Assistants

Results `Good Abandonment’

RQ4: How to characterize abandonment in the web search scenario?

Page 34: Understanding User Satisfaction with Intelligent Assistants

First Query

Second Query

Third Query

>= Fourth Quey

0

1

2

3

4

5

6

Answer Box

Image SERP Visited WebSite

50

1

2

3

4

5

6

Satis

fact

ion

Leve

l

Results `Good Abandonment’

Mean of Satisfaction

Page 35: Understanding User Satisfaction with Intelligent Assistants

Search Dialogue Satisfaction

RQ5: How does query-level satisfaction relate to overall user satisfaction for the structured search dialogue scenario?

Page 36: Understanding User Satisfaction with Intelligent Assistants

Cortana: “Here are

ten restaurants near you”

Cortana:“Here are ten restaurants

near you that have good reviews”

Cortana:“Getting you direction to the Mayuri

Indian Cuisine”

User:“show

restaurants near

me”

User:“show

the best restaurants near

me ”

User:“show

directions to the second one”

SAT?

SAT?

SAT?

SAT?

SAT?

SAT?

Overall

SAT??

Page 37: Understanding User Satisfaction with Intelligent Assistants

Search Dialogue Satisfaction

RQ5: How does query-level satisfaction relate to overall user satisfaction for the structured search dialogue scenario?

Page 38: Understanding User Satisfaction with Intelligent Assistants

Satisfaction Over Different Tasks

Satisfaction Level

Weather Task

Num

ber

of

Ans

wer

s

1 2 3 4 5

Page 39: Understanding User Satisfaction with Intelligent Assistants

Satisfaction Over Different Tasks

Satisfaction Level

Weather Task Mission Task (2 sub-tasks)

Num

ber

of

Ans

wer

s

1 2 3 4 5

Page 40: Understanding User Satisfaction with Intelligent Assistants

Satisfaction Over Different Tasks

Satisfaction Level

Weather Task Mission Task (2 sub-tasks)

Mission Task (3 sub-tasks)

Num

ber

of

Ans

wer

s

1 2 3 4 5

Page 41: Understanding User Satisfaction with Intelligent Assistants

Q1: what do you have medicine for the stomach acheQ2: stomach ache medicine over the counter

Q3: show me the nearest pharmacyQ4: more information on the second one

Q5: do they have a stool softenerQ6: does Fred Meyer have stool softeners

General Search

Search Dialog

Combination of scenarios

User’s dialogue with Cortana related to the ‘stomach ache’ problem

Page 42: Understanding User Satisfaction with Intelligent Assistants

Conclusions (1)• RQ1: What are characteristic types of scenarios of use?• We proposed three main types of scenarios

• RQ2: How can we measure different aspects of user satisfaction?

• We designed a series of user studies tailored to the three scenarios

• RQ3: What are key factors determining user satisfaction for the different scenarios?

• Effort is a key component of user satisfaction across the different intelligent assistants scenarios

Page 43: Understanding User Satisfaction with Intelligent Assistants

Conclusions (2)• RQ4: How to characterize abandonment in the web

search scenario?• We concluded that to measure good abandonment we

need to investigate the other forms of interaction signals that are not based on clicks or reformulation

• RQ5: How does query-level satisfaction relate to overall user satisfaction for the search dialogue scenario?

• We looked at user satisfaction as ‘a user journey towards an information goal where each step is important,’ and showed the importance of session context

Page 44: Understanding User Satisfaction with Intelligent Assistants

Questions?• We proposed three main types of scenarios of use

• We designed a series of user studies tailored to the three scenarios

• Effort is a key component of user satisfaction across the different intelligent assistants scenarios

• We concluded that to measure good abandonment we need to investigate the other forms of interaction signals that are not based on clicks or reformulation

• We looked at user satisfaction as ‘a user journey towards an information goal where each step is important,’ and showed the importance of session context on user satisfactionQuestions?