making search better by tracking & utilizing user search behavior
TRANSCRIPT
Making search better by tracking & utilizing user search behavior
Sameer Maggon
Agenda
• About Me !
• Measuring Search Quality !
• Quality Metrics based on users interactions !
• Quality Metrics based on User Tagging (labelers) !
• Improving Search Results !
• Using user behavior to improve results ranking - Example
2
3
Sameer Maggon
Built relevance based search platform for AT&T Interactive & properties including yp.com, buzz.com, yellowpages.com
Founder of Cloud based Zero Management Solution for Search
Engineering Alumni
@maggonhttp://linkedin.com/in/maggon
Consulted for numerous Startups to Fortune 500 companies around Search & Discovery.
Search seems Easy
4
Index all content Put a search box Show google like results
1 2 3
5
• How do you know that your users are finding what they are looking for? !
• How do you know what impact your one-off fix has on an aggregate? !
• Seemingly good result list to one might be irrelevant to another (e.g. mosaic)
Is It?
6
How do we measure search?
Usage Data Editorial Labeling
Collect, Analyze & Report on interactions users are having with your search functionality.
Get a set of users to mark top x results with “Relevant vs. Not” for a pre-determined sample set of searches. !Then compute specific metrics based on those.
7
Usage Data: Key Metrics to look at
• No Result Search % • Search Exits % • CTR %
!• Average Click Position • MRR (Mean Reciprocal Rank) • Clicks per Search • Paging (how deep do people have
to dig?) !
• Latency (Average, tp90 and tp95)
Aggregate & Trends
Trends
Aggregate & Trends
8
Search Analytics: Examples
9
Editorial Labeling
• Precision Recall !• DCG (Discounted
Cumulative Gain) !• nDCG (Normalized DCG)
http://en.wikipedia.org/wiki/Discounted_cumulative_gain
Relevant
Not Relevant
Relevant
Not Relevant
Not Relevant
Relevant
Not Relevantweight-age decreases as as you go down on an ordered list
10
Improving Search Results
• Popular No Result Searches - Can we use keyword stuffing? • Popular Search Exits - Eyeball outliers • Popular Searches with low CTRs • Generally improve Average Click Position / MRR via
identifying patterns
Attack Low Hanging fruit first
• Topic for some other time :)
Advanced: Learning Models
Utilizing Search Behavior to improve ranking
• Utilize Popularity (click stream) to inform search ranking (impacts CTR)
• Utilizing past search history to offer assistive features (search suggestions, related searches)
11
Using Popularity to affect search ranking
DEMO
