click models for web search - lecture...

Evaluation Data and tools Results

Click Models for Web SearchLecture 3

Aleksandr Chuklin§,¶ Ilya Markov§ Maarten de Rijke§

[email protected] [email protected] [email protected]

§University of Amsterdam¶Google Research Europe

AC–IM–MdR Click Models for Web Search 1


Course overview

Basic Click Models

Parameter Estimation Evaluation

Data and ToolsResultsApplications

Advanced Models

Recent Studies

Future Research



This lecture

Basic Click Models



Advanced Models

Recent Studies

Future Research



What do click models give us?

General:

Understanding of user behavior

Specific:

Conditional click probabilities

Full click probabilities

Attractiveness and satisfactoriness for query-document pairs



Lecture outline

1 EvaluationLikelihoodPerplexityRanking evaluation

2 Data and tools

3 Results



Evaluation summary

Click model’s output Evaluation

Conditional click probabilities Log-likelihoodFull click probabilities PerplexityParameter values Ranking evaluation



Lecture outline




Likelihood

Likelihood measures how well a click model estimatesconditional click probabilities given observed clicks.

LL(M) =1

|S|∑s∈S

logPM

(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)Cr – binary random variable denoting a click at rank r

c(s)r – observed click at rank r in a search session s

P(Cr = c

(s)r

)– probability of observing c

(s)r in session s

P(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)– probability of observing

sequence c(s)1 , . . . , c

(s)n in session s



Likelihood

PM

(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)= PM

(C1 = c

(s)1

)· PM

(C2 = c

(s)2 , . . . ,Cn = c

(s)n | C1 = c

(s)1

)= PM

(C1 = c

(s)1

)· PM

(C2 = c

(s)2 | C1 = c

(s)1

)· PM

(C3 = c

(s)3 , . . . ,Cn = c

(s)n | C1 = c

(s)1 ,C2 = c

(s)2

)=

n∏r=1

PM

(Cr = c

(s)r | C<r = c

(s)<r

)



Likelihood: summary

LL(M) =1

|S|∑s∈S

logPM

(C1 = c

(s)1 , . . . ,Cn = c

(s)n

)LL(M) =

1

|S|∑s∈S

n∑r=1

logPM

(Cr = c

(s)r | C<r = c

(s)<r

)

Likelihood measures how well a click model estimatesconditional click probabilities given observed clicks.

LL(M) ∈ [−∞..0]



Lecture outline




Perplexity

Perplexity measures how well a click model estimatesfull click probabilities (i.e., when clicks are not observed).

pr (M) = 2− 1

|S|∑

s∈S

(log2 PM(C

(s)r =c

(s)r ))

pr (M) ∈ [1..2]



Lecture outline




Ranking evaluation

R̂el i Reli

αu1q 4

αu2q 2

αu3q 1

αu4q 4

αu5q 2

DCG =n∑

i=1

2Reli − 1

log2(i + 1)



Evaluation summary

Click model’s output Evaluation

Conditional click probabilities Log-likelihoodFull click probabilities PerplexityParameter values Ranking evaluation



Lecture outline

1 Evaluation

2 Data and tools

3 Results



Datasets

AOL2006: raw queries and clicked documents (no SERPs)

MSN2006: contains only clicked documents (no SERPs)

Workshop on Web Search Click Data (WSCD)

WSCD2012: predict document relevanceWSCD2013: detect search engine switchWSCD2014: search personalization

SogouQ

Tsinghua University: eye fixation


http://imat-relpred.yandex.ru/en/datasets

http://switchdetect.yandex.ru/en

https://www.kaggle.com/c/yandex-personalized-web-search-challenge

http://www.sogou.com/labs/dl/q-e.html

http://www.thuir.cn/group/~yqliu/publications/cikm2014-liu.7z


Dataset statistics

Dataset Queries URLs Users Sessions

AOL 2006 10,154,742 1,632,788 657,426 21,011,340MSN 2006 8,831,280 4,975,897 – 7,470,915SogouQ 2012 8,939,569 15,095,269 9,739,704 25,530,711WSCD 2012 30,717,251 117,093,258 – 146,278,823WSCD 2013 10,139,547 49,029,185 956,536 17,784,583WSCD 2014 21,073,569 70,348,426 5,736,333 65,172,853



Software

Click model packages

clickmodels project by Aleksandr ChuklinPyClick by Ilya Markov et al.

Infer.NET

General-purpose languages

OctaveMatlab


https://github.com/varepsilon/clickmodels

https://github.com/markovi/PyClick

http://research.microsoft.com/en-us/um/cambridge/projects/infernet/docs/what%20is%20infer.net.aspx


Lecture outline

1 Evaluation

2 Data and tools

3 ResultsLog-likelihoodPerplexityTraining timeLarge-scale evaluation



Experimental setup

Data

first 1M query sessions from WSCD 2012 dataset75% for training, 25% for testingrepeat 15 times, each time with next 1M sessions

PyClick

50 iterations for EM



Studied click models

CTR models: counting clicks

Position-based model (PBM): examination and attractiveness

Cascade model (CM): previous examinations and clicks matter

Dynamic Bayesian network model (DBN): satisfactoriness

User browsing model (UBM): rank of previous click



Lecture outline




Log-likelihood

RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN0.40

0.35

0.30

0.25

0.20

0.15

0.10

0.05

0.00

Log-l

ikelih

ood

Cascade model: LL = −∞Complex models (DBN, UBM) win over simple onesMany examination parameters win over few: UBM > PBMSatisfaction parameters help: DBN > PBM



Lecture outline




Perplexity

RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN1.0

1.1

1.2

1.3

1.4

1.5

Perp

lexit

y

Complex models win over simple ones

Most complex models have similar perplexity



Perplexity by rank

1 2 3 4 5 6 7 8 9 100.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

GCTRRCTR

DCTRPBM

CMUBM

DCMCCM

DBNSDBN

Picture taken from A. Grotov, A. Chuklin, I. Markov, L. Stout, F. Xumara, and M. de Rijke. A comparative studyof click models for web search. In CLEF. Springer, September 2015.



Lecture outline




Training time

RCM RCTR DCTR PBM CM UBM SDCM CCM DBN SDBN0

500

1000

1500

2000

2500

3000

3500

4000

Tim

e (

sec)

MLE is much faster than EM

PBM and UBM are fast enough compared to DBN



Lecture outline




Experimental setup

Full WSCD 2012 dataset

146,278,823 query sessions30,717,251 unique queries117,093,258 unique URLs41,275 relevance labels (for 4,991 queries)

50% for training, 50% for testing

PyClick



Log-likelihood and perplexity

Click model Perplexity Log-likelihood

DBN 1.3510 −0.2824DCM 1.3627 −0.3613CCM 1.3692 −0.3560UBM 1.3431 −0.2646

UBM is the best in terms of predicting user click behavior

UBM has the largest number of examination parameters (55)



Ranking evaluation

NDCG

Click model @1 @3 @5 @10

DBN 0.717 0.725 0.764 0.833DCM 0.736 0.746 0.780 0.844CCM 0.741 0.752 0.785 0.846UBM 0.724 0.737 0.773 0.838

CCM is the best in terms of ranking

Not covered in this corse (but covered in the book)



Lecture 3 summary

Click model’s output Evaluation Best model

Conditional click probabilities Log-likelihood UBMFull click probabilities Perplexity UBMParameter values Ranking evaluation CCM

Training time MLE-based



Course overview

Basic Click Models



Advanced Models

Recent Studies

Future Research



Up next

Practical Session 1



Acknowledgments

All content represents the opinion of the authors which is not necessarily shared orendorsed by their respective employers and/or sponsors.


click models for web search - lecture...

Documents