online expansion of rare queries for sponsored search
DESCRIPTION
Online Expansion of Rare Queries for Sponsored Search. Defended by Mykell Miller. Summary: The Short Version. - PowerPoint PPT PresentationTRANSCRIPT
Online Expansion of Rare Queries for Sponsored
Search
Defended by Mykell MillerDefended by Mykell Miller
Summary: The Short VersionSummary: The Short Version
This paper describes and evaluates a This paper describes and evaluates a method of determining which ads to method of determining which ads to display on a search engine result page. display on a search engine result page. Users input varied queries, so it is Users input varied queries, so it is beneficial to post ads pertaining to not beneficial to post ads pertaining to not only the query, but to related queries as only the query, but to related queries as well. However, previous methods of well. However, previous methods of finding these related queries and finding these related queries and transforming them into ads takes a long transforming them into ads takes a long time, and therefore are done offline. This time, and therefore are done offline. This paper describes a method that allows paper describes a method that allows some of the work to be done on the fly some of the work to be done on the fly without too much overhead.without too much overhead.
Why it’s good: The Short VersionWhy it’s good: The Short Version
UsefulUseful• Ads fund search enginesAds fund search engines• If ads were more relevant, Jared might actually click on themIf ads were more relevant, Jared might actually click on them• The method shows statistically significant improvement in The method shows statistically significant improvement in
making ads more relevant, at a low overheadmaking ads more relevant, at a low overhead InterestingInteresting
• Interestingness is subjective, but this is MY defenseInterestingness is subjective, but this is MY defense Well-writtenWell-written
• Well-organizedWell-organized• I could actually understand the math because they very clearly I could actually understand the math because they very clearly
told me what all the variables meanttold me what all the variables meant• They defined all the relevant terms and summarized all the They defined all the relevant terms and summarized all the
references so I didn’t have to read 32 other papers.references so I didn’t have to read 32 other papers. Time TravelTime Travel
• This paper is only three weeks oldThis paper is only three weeks old• A paper that was published in April cited itA paper that was published in April cited it
Now for the long version…Now for the long version…
What this paper is aboutWhat this paper is about
Broad matching is where an ad is displayed when Broad matching is where an ad is displayed when its bid phrase is similar to, but not exactly, the its bid phrase is similar to, but not exactly, the query the user inputted.query the user inputted.
What this paper is aboutWhat this paper is about
Sponsored SearchSponsored Search• A.K.A. Paid search advertisingA.K.A. Paid search advertising• On Search Engine Result PagesOn Search Engine Result Pages• All major web search engines do thisAll major web search engines do this
Context MatchContext Match• A.K.A. Contextual AdvertisingA.K.A. Contextual Advertising• On other websitesOn other websites• What we looked at last WednesdayWhat we looked at last Wednesday
More on Sponsored SearchMore on Sponsored Search The authors assume a pay-per-click modelThe authors assume a pay-per-click model
• Google, Yahoo, and Microsoft all use this modelGoogle, Yahoo, and Microsoft all use this model Bid PhrasesBid Phrases
• This is the query that will result in showing this ad.This is the query that will result in showing this ad. Bidding systemBidding system
• An advertiser pays the search company whatever it An advertiser pays the search company whatever it wants to associate its ad with a bid phrasewants to associate its ad with a bid phrase
• If an advertiser pays more, its ad gets a higher ranking.If an advertiser pays more, its ad gets a higher ranking. Example:Example:
• High Bidders pays $1,000,000,000,000,000,000,000 for High Bidders pays $1,000,000,000,000,000,000,000 for the bid phrase “Dummy Query”the bid phrase “Dummy Query”
• Low Bidders pays $1 for the bid phrase “Dummy Query”Low Bidders pays $1 for the bid phrase “Dummy Query”• When I search for “Dummy Query” I see High Bidders’ When I search for “Dummy Query” I see High Bidders’
ad first, then Low Bidders’ ad.ad first, then Low Bidders’ ad.
More on Sponsored SearchMore on Sponsored Search
System
An Advertiser Other Advertisers
An Account More Accounts
An Ad Campaign More Ad Campaigns
An Ad Group More Ad Groups
Creative Bid Phrases
Why Do This Paper?Why Do This Paper?
30-40% of search engine result 30-40% of search engine result pages have no ads on them because pages have no ads on them because Google, Yahoo, etc. don’t know what Google, Yahoo, etc. don’t know what queries are similar to the bid phrasequeries are similar to the bid phrase
Previous work has developed Previous work has developed systems that are far too inefficient to systems that are far too inefficient to use in real lifeuse in real life
My Own ExperimentMy Own Experiment
Query: Banana BreadQuery: Banana Bread Query: Nut-Free Query: Nut-Free Banana BreadBanana Bread
Query: Vegan Banana BreadQuery: Vegan Banana Bread
Why do tail queries have so few Why do tail queries have so few ads?ads?
They are often harder to interpret They are often harder to interpret than more common (head and torso) than more common (head and torso) queriesqueries
There are rarely exact matches for There are rarely exact matches for bid queriesbid queries
There is little historical click dataThere is little historical click data Search engines don’t like posting Search engines don’t like posting
irrelevant adsirrelevant ads
What does this paper accomplish? What does this paper accomplish?
Online query expansion for tail Online query expansion for tail queriesqueries
New way to index query expansions New way to index query expansions for fast computation of query for fast computation of query similaritysimilarity
A way to go from pre-expanded A way to go from pre-expanded queries to expanding related queries queries to expanding related queries on the flyon the fly
A ranking and scoring methodA ranking and scoring method
The Architecture of their systemThe Architecture of their system
Query Feature ExtractionQuery Feature Extraction UnigramsUnigrams
• Process them viaProcess them via StemmingStemming
• Taking words like “Extraction” and “Extracting” and stemming Taking words like “Extraction” and “Extracting” and stemming them to “Extract”them to “Extract”
Stop wordsStop words• Ignoring words you don’t likeIgnoring words you don’t like
PhrasesPhrases• Multi-word phrases are from a dictionary of ~10 million Multi-word phrases are from a dictionary of ~10 million
phrases gathered from query logs and web pagesphrases gathered from query logs and web pages Semantic ClassesSemantic Classes
• Developed a hierarchical taxonomy of 6000 semantic Developed a hierarchical taxonomy of 6000 semantic classesclasses
• Annotate each query with the 5 most likely semantic Annotate each query with the 5 most likely semantic classesclasses
Related Query RetrievalRelated Query Retrieval
Now we have a pseudo-query made Now we have a pseudo-query made up of features.up of features.
Compare this pseudo-query to our Compare this pseudo-query to our inverted index and pull out related inverted index and pull out related pseudo-queriespseudo-queries
Runs a system that pulls out key Runs a system that pulls out key words then calculates the similarity words then calculates the similarity using a dot productusing a dot product
Query ExpansionQuery Expansion
Q* is the set of features describing Q* is the set of features describing the original features and related the original features and related queriesqueries
The weight of a given feature in Q* is The weight of a given feature in Q* is a linear combination of its weight in a linear combination of its weight in the original and related queriesthe original and related queries
This expansion is efficient because This expansion is efficient because you’re only looking at the features in you’re only looking at the features in related queriesrelated queries
Ad Feature WeightingAd Feature Weighting
Extract the same features from the Extract the same features from the bid phrases of ad groups as from bid phrases of ad groups as from queries (unigrams, phrases, semantic queries (unigrams, phrases, semantic classes)classes)
Since the weighting from the queries Since the weighting from the queries would unfairly benefit short ad would unfairly benefit short ad groups, use the BM25 weighting groups, use the BM25 weighting scheme.scheme.
Title Match BoostingTitle Match Boosting
Increases the score of ads whose Increases the score of ads whose titles match the original query very titles match the original query very wellwell
Scoring FunctionScoring Function
The end result of all thisThe end result of all this A weighted sum of dot products A weighted sum of dot products
between features and the title match between features and the title match boostboost
Now on to the results!Now on to the results!
Test SetTest Set
Test set: 400 random rare queries from Test set: 400 random rare queries from YahooYahoo• 121 were in the lookup table, 279 were not121 were in the lookup table, 279 were not• Eliminated the 10% of rare queries that were Eliminated the 10% of rare queries that were
foreignforeign Human editors judged the top 3 ads.Human editors judged the top 3 ads.
• 3556 judgments3556 judgments The system was built off of every ad Yahoo The system was built off of every ad Yahoo
has and 100 million queries based off of has and 100 million queries based off of U.S. YahooU.S. Yahoo
MetricsMetrics Discounted Cumulative Gain (DCG)Discounted Cumulative Gain (DCG)
• ““a measure of effectiveness of a Web search engine a measure of effectiveness of a Web search engine algorithm or related applications, often used in algorithm or related applications, often used in information retrieval. Using a graded relevance scale of information retrieval. Using a graded relevance scale of documents in a search engine result set, DCG measures documents in a search engine result set, DCG measures the usefulness, or the usefulness, or gaingain, of a document based on its , of a document based on its position in the result list. The gain is accumulated position in the result list. The gain is accumulated cumulatively from the top of the result list to the bottom cumulatively from the top of the result list to the bottom with the gain of each result discounted at lower ranks.” –with the gain of each result discounted at lower ranks.” –WikipediaWikipedia
• DCG is a number; higher numbers are betterDCG is a number; higher numbers are better Precision-Recall CurvesPrecision-Recall Curves
• Precision: Fraction of results returned that are relevantPrecision: Fraction of results returned that are relevant• Recall: Fraction of relevant results that are returnedRecall: Fraction of relevant results that are returned• A way to visualize it; higher is betterA way to visualize it; higher is better
Ad Matching Algorithms TestedAd Matching Algorithms Tested BaselineBaseline
• The original, unexpanded version of the query The original, unexpanded version of the query vectorvector
Offline ExpansionOffline Expansion• Expands the original query by pre-processing Expands the original query by pre-processing
offline onlyoffline only Online ExpansionOnline Expansion
• Expands the original query by processing Expands the original query by processing online onlyonline only
Online + Offline ExpansionOnline + Offline Expansion• Expands the original query using both offline Expands the original query using both offline
and online expansion algorithmsand online expansion algorithms
Test Results: Queries not found in Test Results: Queries not found in lookup tablelookup table
Tested the baseline Tested the baseline vs online vs online expansionexpansion
The online The online expansion gave expansion gave statistically statistically significant significant improvementsimprovements
Test Results: Queries found in Test Results: Queries found in lookup tablelookup table
Tested all 4 Tested all 4 algorithmsalgorithms
Best: offline Best: offline expansionexpansion
Second best: online Second best: online + offline expansion+ offline expansion
Difference between Difference between the two was not the two was not statistically statistically significantsignificant
Test results: full setTest results: full set
Tested on all four algorithmsTested on all four algorithms Best: online + offline expansionBest: online + offline expansion Online expansion also offers statistically Online expansion also offers statistically
significant improvementsignificant improvement Even better: hybridEven better: hybrid
EfficiencyEfficiency
The table lookup takes only 1 msThe table lookup takes only 1 ms Least efficient when a query is not in the Least efficient when a query is not in the
lookup tablelookup table When a query is not in the lookup table, When a query is not in the lookup table,
there is a 50% overheadthere is a 50% overhead• This is badThis is bad
But given the small proportion of queries But given the small proportion of queries not in the lookup table, the estimated not in the lookup table, the estimated average is 12.5% overheadaverage is 12.5% overhead• This is good This is good