fan guo 1, chao liu 2 and yi-min wang 2 1 carnegie mellon university 2 microsoft research feb 11,...
TRANSCRIPT
Efficient Multiple-Click Models in Web Search
Efficient Multiple-Click Models in Web Search Fan Guo1, Chao Liu2 and Yi-Min Wang21 Carnegie Mellon University2 Microsoft ResearchFeb 11, 2009
OverviewWeb Search ActivityUser Interaction LogsSearch Engine PresentationModels for Leveraging User Feedbacks
Here we focus on click logshttp://www.untyped.com/images/weblog.jpg2OverviewWeb Search ActivityUser Interaction LogsSearch Engine PresentationModels for Leveraging User Feedbacks
LoggingLogging: extract features and log them3User Implicit FeedbacksA very rich set of candidate features
(Agichtein et al., SIGIR06)
User Implicit FeedbacksPractical considerationRecording effort (how much tweaking?)Noise elimination (e.g., dwelling time?)Would the answer change after scaling up to 10x? 100x?Click LogsQuery termcmuDocument URLwww.cmu.edu Position/Rank1Clicked or NotYes
Query termcmuDocument URLwww.cmu.edu Position/Rank1Clicked or NotYes
Query termcmuDocument URLwww.cmu.edu Position/Rank1Clicked or NotYes
Query termcmuDocument URLwww.ri.cmu.edu Position/Rank10Clicked or NotNo
Click LogsQuery termcmuDocument URLwww.cmu.edu Position/Rank1Clicked or NotYes
Time stamp01/19/2009, 4:00:01pmSession ID ...User ID ...Query termcmuDocument URLwww.cmu.edu Position/Rank1Clicked or NotYes
Query termcmuDocument URLwww.cmu.edu Position/Rank1Clicked or NotYes
Query termcmuDocument URLwww.ri.cmu.edu Position/RankM = 10Clicked or NotNo
A Snapshot
From the AOL release, plain text, would be a database for TB data8OverviewWeb Search ActivityUser Interaction LogsSearch Engine PresentationModels for Leveraging User Feedbacks
CleanupQuery normalization; fraud click elimination9OverviewWeb Search ActivityUser Interaction LogsSearch Engine PresentationModels for Leveraging User Feedbacks
Modeling10Clicks are biased
Images are copied from (Joachims et al., 2007) in ACM TOISExamination HypothesisA web document must be examined before clicked.
Relevance is defined as the conditional probability of click upon examination.
P(Click) = P(Examination) * Relevance, formulated in (Richardson et al., WWW07)given E_i, C_i is independent of past examination and clicks12Most Simple Click ModelVariable DefinitionEi: binary r.v. for examination on position i;Ci: binary r.v. for click on position i;rdi: relevance for document di on position i.
Most Simple Click ModelIndependent Click ModelP(Ei = 1) = 1P(Ci=1) = rdi Therefore we have
Cascade HypothesisProposed in (Craswell et al., WSDM08)Strict Linear order of examination and clickEi = 0 => Ei+1 = 0
Cascade Model
15Cascade ModelModeling the first click (Craswell et al., WSDM08)P(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rdiP(E1=1) =1, P(Ei+1=1|Ei=0) = 0P(Ei+1=1|Ei=1, Ci=0) = 1P(Ei+1=1|Ei=1, Ci=1) = ?
examination hypothesiscascade hypothesismodeling a single clickDependent Click ModelModeling multiple clicksP(Ci=1|Ei=0) = 0, P(Ci=1|Ei=1) = rdiP(E1=1) =1, P(Ei+1=1|Ei=0) = 0P(Ei+1=1|Ei=1, Ci=0) = 1P(Ei+1=1|Ei=1, Ci=1) = iexamination hypothesiscascade hypothesismodeling 1+ click(s)
Dependent Click Modeli (1 i < 10) are global user behavior parameters.The full picture
DCM AlgorithmsTwo sets of unknown values for estimationQuery-specific: document relevance rGlobal: user behavior parameter Approximate Maximum-Likelihood LearningMaximizing a lower bound of log-likelihoodWould be exact if the chain length is infinite
Drop the (less probable) case that user looks over the last clicked position without any click (examination depth)19Estimation Formulae
empirical CTR measured before last clicked position
empirical probability of clicked-but-not-last
DCM AlgorithmsNote that in general, relevance (relevance) depends on user behavior parameters (lambda)20DCM ImplementationKeep 3 counts for each query-document pair
Then
Least effort: 2 counts for each q-U pair and 2(M-1) global counts21OverviewWeb Search ActivityUser Interaction LogsSearch Engine PresentationModels for Leveraging User Feedbacks
Applicationapplications22Click Model ApplicationsUser-Perceived RelevanceA good candidate feature for ranking, butAs input for existing ranking algorithms v.s. human judgment
Should be careful if it is the sole featurehttp://weblogs.newsday.com/news/local/longisland/politics/blog/judges.jpg23Click Model ApplicationsSearch Engline EvaluationSummary of the overall user experienceExpected clickthrough rateBrowsing Behavior Statistics
Application in Sponsored SearchBuilding block for auction/pricing mechanism(Kempe and Mahdian, WINE 2008) (Aggarwal et al., WINE 2008)(Guo et al., WSCD09)
24
http://www.jemstar.com/myjeremy/download_files/experiment_1280x960wallpape.jpg25Click Data SetCollected in 2 weeks in July 2008.Discard query sessions with no clicks.178 most frequent queries removed.
Click Data SetAfter preprocessing: 110,630 distinct queries4.8M/4.0M query sessions in the training/test set
Training Time: ~7 min
Evaluation CriteriaTest Data Log-likelihoodGiven the document impression in the test setCompute the chance to recover the entire click vectorAveraged over different query sessions
Test Data Log-LikelihoodICM: -1.401, DCM: -1.327 (26.5% chance)
Smoothing helps29Evaluation CriteriaPredicting First/Last ClicksGiven the document impression in the test setDraw 100 click vector samples (with 1+ clicks)Compute the corresponding RMS error
First/Last Clicked Position
Examination/Click Distribution
Difference by User GoalsFrom (Guo et al., WSCD09)on a different data set
An AlternativeUser Browsing Model (Dupret et al., SIGIR08)Examine probability depends on Preceding clicked positionDistance to preceding clicked position
Comparing with DCM:Allow jump in examination (no longer strictly linear)Parameter set is an order of magnitude largerMore expensive algorithms (iterative, EM-like)
ConclusionClick Models:A principled way of integrating user click dataAll current models are based on examination hypothesisShould be scalable and incrementalDCM:built upon cascade model, for multiple-clicksintroduce a set of position-dependent to characterize the examine-next probabilityprovides a fast, simple, yet effective solutionActive area: two www papers35Open ProblemsTrade-off in effectiveness/efficiency
Evaluation (test bed and test metric)
Click-model relevance v.s. human-judged relevance
Click model for universal search?36Acknowledgement Chao Liu Yi-Min Wang
Ethan Tu Li-Wei He Nick Craswell
Co-authors and Microsoft friends http://research.microsoft.com/en-us/groups/sqcil/37Acknowledgement
Christos Faloutsos Lei Li
Thank you!Related work39