Evaluating Implicit Measures to Improve the

Search Experience

SIGIR 2003

Steve Fox

Outline

Background Approach Data Analysis Value-Add Contributions Result-Level Findings Session-Level Findings

Background

Interested in implicit measures to improve user’s search experience What the user wants What satisfies them Significant implicit measures

Needed to prove it! Two goals:

Test association between implicit measures and user satisfaction

Understand what implicit measures were useful within this association

Approach

Architecture Internet Explorer add-in Client-Server Configured for MSN Search and Google

Deployment Internal MS employees (n = 146) – work environment Implicit measures and explicit feedback SQL Server back-end

Approach, cont’d

Data Analysis Bayesian modeling at result and session

level Trained on 80% and tested on 20% Three levels of SAT – VSAT, PSAT & DSAT Implicit measures:

Result-Level Session-Level

Diff Secs, Duration Secs Averages of result-level measures (Dwell Time and Position)

Scrolled, ScrollCnt, AvgSecsBetweenScroll, TotalScrollTime, MaxScroll

Query count

TimeToFirstClick, TimeToFirstScroll Results set count

Page, Page Position, Absolute Position Results visited

Visits End action

Exit Type

ImageCnt, PageSize, ScriptCnt

Added to Favorites, Printed

Data Analysis, cont’d

Result-Level Findings

1. Dwell time, clickthrough and exit type strongest predictors of SAT

2. Printing and Adding to Favorites highly predictive of SAT when present

3. Combined measures predict SAT better than clickthrough

Result Level Findings, cont’d

Only clickthrough

Combined measures

Combined measures with confidence of > 0.5 (80-20

train/test split)

Session-Level Findings

Four findings:1. Strong predictor of session-level SAT was

result-level SAT

2. Dwell time strong predictor of SAT

3. Combination of (slightly different) implicit measures could predict SAT better than clickthrough

4. Some gene sequences predict SAT (preliminary and descriptive)

Session Level Findings, cont’d

Common patterns in gene analysis, e.g. SqLrZ Session starts (S) Submit a query (q) Result list returned (L) Click a result (r) Exit on result (Z)

PatternFrequenc

y%VSA

T%PSA

T%DSA

TAvg. VSAT Dwell Time

Avg. PSAT Dwell Time

Avg. DSAT Dwell Time

SqLrZ 117 75 15 9 64 6 11

Value-Add Contributions

Deployed in the work setting Collected data in context of web search

Rich user behavior data stream Annotated data stream with explicit judgment

Used new methodology to analyze the data ‘Gene analysis’ to analyze usage patterns Mapped usage patterns to SAT

Question(s)