Internet Research

Search Engines & Subject Directories

Search engines

• Search engines are the means by which most people search the Web.

• Common examples are Google, Altavista, Direct Hit.

Yet they don’t search the Internet

• Yet a search engine does not actually search the Web during your search.

• A search engine searches itself.

• It’s a three-step process.

1) Bots index words

• Search engines continually send out hundreds of “robots” or “bots” (or “spiders” or “crawlers” )

• Bots visit web sites, read word by word, and then index those words.

2) A database is created

• A huge database of Web sites thus is gathered and indexed by word.

• These databases can be huge, with millions of links.

3) The Interface gives you access

• Using the keywords you give it, a search engine then searches its own current index.

Interfaces are based on rankings

• Search engines return results based on a ranking system.

• Ranking is the order that files are listed when they are retrieved.

The ranking system is secret• These systems are proprietary and

often “secret.” In general:• Altavista ranks web pages higher if

your search terms are found in the first few words of the page

• Google ranks by document “popularity” with other similar searches

• Direct Hit ranks by the length of time other users spent at the site

Not even half the Web

• With all of this software and sophistication, even the best search engines cover only 40-50% of the Web.

• And they miss much else on the Internet.

Bots hit and miss

Bots miss: XML pages, pdf files Dynamically created HTML pages Frames-based pages New pages or recent updated text Some say the Invisible Web is 500

times larger than Web

Subject Directories

• A subject directory is also a database of web sites and references.

• But a subject directory is organized not by keywords but by category or subject.

Yahoo!

• Yahoo! Is the most popular subject directory.

• www. about.com takes the idea a step further with subject guides for selected topics.

Subjects are organized by people.

• Information is selected, organized and cataloged by a person, not software.

• You can usually be more assured that the search results will make sense.

You get an index of sites.

• Subject directories will not often provide you with ranked web sites.

• Instead, you will get a broad index related to your topic, divided further by subheadings.

Use for early searching.• Use a subject directory

early in your search process to learn about your subject.

• You will get fewer links of higher quality.

• When you get more specific questions, you should use a search engine.