04/07/2009

Suhas Suhas (ss3474) COMS E6125 Web Enhanced Information Management Professor Gail Kaiser

Department of Computer ScienceColumbia University

Spring 2009

Vertical Search Engines

- Jack of all trades, master of none!Traditional Search Engine

Problems continued…

Survey affirms the need for deeper, more specific, more relevant search results Traditional search engines often do not know what the user wants based on the search query

What's the point of giving half a millions less relevant search results knowing the user will not look after 30 top results

Irrelevant Ad’s !!!

Vertical search engines

Welcome to the relatively new tier in the Internet search, an industry consisting of search engines that focus on specific slices of content

If a user decides to use a search engine which says it's a "health information search engine for doctors", instead of one that is labeled a "health information search engine for patients", the user has already helped to reduce the ambiguity of his/her search queries even before he/she type in his/her query

Users searching VSEs are typically closer to purchase or looking for particular information. In other words, if users have gotten as far as a vertical search, they’ve essentially classified themselves as interested consumers

Internal of vertical search engines

Vertical spiders

Also called an a focused crawler or topical crawler which is nothing but a web crawler that attempts to download only web pages that are relevant to a pre-defined topic or set of topics (Area of the Vertical search)

Topical crawling generally assumes that only the topic is given, while focused crawling also assumes that some labeled examples of relevant and not relevant pages are available

A focused crawler ideally would like to download only web pages that are relevant to a particular topic and avoid downloading all others. Therefore a focused crawler may predict the probability that a link to a particular page is relevant before actually downloading the page 

A possible predictor is the anchor text of links Can also use the complete content of the pages already visited to infer the similarity between the driving query and the pages that have not been visited yet. In another approach, the relevance of a page is determined after downloading its content. Relevant pages are sent to content indexing and their contained URLs are added to the crawl frontier; pages that fall below a relevance threshold are discarded

Current vertical search engines spiders address different ways of combining content- and link-based Web analyses and integrating them with graph search algorithms

Example of Traditional Search Engine results for query :- Dryer

Example of Vertical Search Engine results for query :- Dryer

Example of Traditional Search Engine results for query :- Ceramics

Example of Vertical Search Engine results for query :- Ceramics

Example of Traditional Search Engine results for query :- REM

Example of Vertical Search Engine results for query :- REM

Evaluation ( As of 04/03/09)QUERY TRADITIONAL

SEARCH ENGINE

VERTICAL SEARCH ENGINE

TIME TAKEN

BY TRADITIO

NAL SEARCH ENGINE

(SECONDS)

TIME TAKEN

BY VERTICA

L SEARCH ENGINE

(SECONDS)

PRECISION@ 10

(TRADITIONAL)

PRECISION@ 10

(VERTICAL)

Springs www.google.com www.thomasnet.com 0.24 0.14 0.3 0.9

Dryer www.google.com www.chemindustry.com 0.26 0.20 0.1 1.0

Legal Software www.google.com www.findlaw.com 0.23 0.10 0.3 0.8

Advantages of vertical search engines• Professional users can save time

• Easier to search

• Superior quality of results and greater sorting

• Spam Free

• Provide access to more of the surface web

• Provide access to the deep web

• Advertisers can generate highly relevant leads through the combination of focused demographics that specialist publishers provide and search keyword targeting

Google Loves Vertical too

Challenges for vertical search engines

Getting people to know about it

Getting them to try it. And after they try it,

Getting them to comeback and use it again

The biggest challenge however is just getting people to know that there is Vertical Search offering, and getting them to look at it!!!