Effects of Popularity and Quality on the Usage of Query Suggestions during Information SearchCan users be induced to take bad query suggestions because they believe many others have used the suggestions in the past?

Diane Kelly, Amber Cushing,

Maureen Dostert, Xi Niu and

Karl Gyllstrom University of North

Carolina

at Chapel Hill

Motivation & Background

• Detrimental impact of social search features• Social influences on behavior– Psychology, sociology, and economics– Recommender systems, business and marketing

• Query suggestions as a social search feature and system usage information

• Users’ abilities to identify ‘good’ suggestions

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Research Questions

1. Are users influenced by usage information associated with recommended queries?

2. Can users distinguish between high and low quality query suggestions?

3. What are users’ perceptions of the usefulness of query suggestions and usage information for open search tasks?

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

• Use of query suggestions as idea tactics– A move to help users generate new approaches or

solutions to information search problems– Potentially useful when user has a limited model

of the problem space, is researching a difficult topic, and/or exhausts own ideas for queries

Motivation & Background

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Research Questions

4. What is the relationship among topic difficulty, users’ willingness to take recommendations and their abilities to distinguish between high and low quality queries?

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Method

• Laboratory experiment with experimental search system

• Twenty-three undergraduate subjects • Four assigned search topics (15 minutes each)• Closed collection of newspaper articles (3GB,

or about 1 million documents)• 8 query suggestions per search topic

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Method

Low Quality Queries

High Quality Queries

20-40 High Usage Information

0-9 Low Usage Information

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Outcome Measures & Analysis

• Suggestion Usage/Selection (Binary)• Post-Search Evaluations (5-point scale, 1=low;

5=high)– Query Quality– Confidence in Rating– Willingness to Recommend to Others– Topic Difficulty

• Exit Interview and Manipulation Check• Analysis: Logistic regression, t-tests, open-coding

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: General Usage of Suggestions

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Source Number (%)

Subject Queries 425 (59%)

Suggested Queries 297 (41%)

Total Queries 722

Number of Subject Queries that Matched Query Suggestion

106

Results: Popularity

Were subjects influenced by the usage information associated with recommended queries?

No. Usage information (popularity) was not a significant predictor of selection behavior.

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: PopularityWhat about subjects’ post-search

evaluations of query quality? Was there a difference in ratings according to popularity?

No. Subjects’ mean ratings were similar.

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Popularity Mean (SD)

Quality Low 2.87 (1.15)

High 2.93 (1.18)

Confidence Low 2.90 (1.11)

High 2.93 (1.08)

Recommend Low 2.84 (1.19)

High 2.88 (1.23)

Results: Query Quality

Could subjects distinguish between high and low quality query suggestions?

Yes. Query quality was a significant predictor of selection behavior.

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: Query QualityQuery Quality Mean (SD)

Quality* Low 2.80 (1.18)

High 3.00 (1.15)

Confidence Low 2.84 (1.09)

High 2.99 (1.09)

Recommend* Low 2.74 (1.20)

High 2.89 (1.20)

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

*p<.05

What about subjects’ post-search evaluations of query quality? Was there a difference in ratings according to actual query quality?

Yes. Subjects rated high quality queries significantly higher than low quality queries.

Results: Topic Difficulty

What was the relationship among topic difficulty and subjects’ willingness to take recommendations?

Yes. Subjects were significantly more likely to use suggestions when the topic was hard.

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Results: Subjects’ PerceptionsWhat were subjects’ perceptions of the usefulness of query

suggestions and usage information for open search tasks?

• the usage information did not influence their selections that much and was not that important for this task

• they ignored the usage information and did their own testing• the query suggestions stimulated thinking outside the box and

helped them narrow their searches• the query suggestions were useful when they ran out of ideas

or were unsure of how to start the search

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Summary of Findings• Subjects integrated the suggestions into their

searching fairly quickly• The usage information did not influence subjects’

selection• Subjects selected more high quality queries than

low quality queries and also rated these higher• Subjects took significantly more suggestions

when searching for difficult topics• Suggestions seemed to function as an idea tactic• Task type mediates the effects of social influence

on selection behaviors (suggestive)

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Implications

• Design– Tailor suggestions to properties of task (type,

difficulty, and stage)• Suggestion algorithms• Presentation methods

– Query performance prediction

• Method– Data collection and evaluation of suggestion use

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

2010 ACM CHI CONFERENCE, APRIL 10-15, ATLANTA, GA

UNIVERSITY OF NORTH CAROLINA at CHAPEL HILL

Thank You.

Diane Kelly, dianek@email.unc.edu