Michelle Post, Ph.D.

What is it?

Visible vs. Invisible

How does it work?

How big is it?

Where is it going?

Invisible Web Search Engines

Invisible Web Databases

Video Training

Resources

References

2© Michelle Post, Ph.D.

“Searching on the Internet today can be compared to dragging a net across the surface of the ocean. While a great deal may be caught in the net, there is still a wealth of information that is deep, and therefore, missed. The reason is simple: Most of the Web's information is buried far down on dynamically generated sites, and standard search engines never find it” (Bergman, 2001, para. 2).

“The "visible web" is what you can find using general web search engines. It's also what you see in almost all subject directories. The "invisible web" is what you cannot find using these types of tools” (UC Berkeley Library, n.d., para. 1).

3© Michelle Post, Ph.D.

“The deep Web contains 99% of the information content of the Web; however, most of this information is contained in databases and is not indexed by search engines” (Gruchawaka, 2010, para. 1).

“If the most coveted commodity of the Information Age is indeed information, then the value of deep Web content is immeasurable” (Bergman, 2001, para. 5).

Michelle Post, Ph.D. 4

Surface Web Deep Web

Millions of web pages Over 200,000 databases

1 billion documents 550 billion documents

19 terabytes 7,750 terabytes

Broad shallow coverage Deep vertical coverage

Results contain ads Results contain no ads

Content unevaluated Content evaluated by experts

5© Michelle Post, Ph.D.

(Bergman, 2001)

“The deep Web is qualitatively different from the surface Web. Deep Web sources store their content in searchable databases that only produce results dynamically in response to a direct request” (Bergman, 2001, para. 4).

6© Michelle Post, Ph.D.

Public information on the Deep Web is currently 400 to 550 times larger than the commonly defined World Wide Web.

The Deep Web contains 7,500 terabytes of information, compared to 19 terabytes of information in the surface Web.

The Deep Web contains nearly 550 billion individual documents compared to the 1 billion of the surface Web.

More than an estimated 200,000 Deep Web sites presently exist.

Sixty of the largest Deep Web sites collectively contain about 750 terabytes of information – sufficient by themselves to exceed the size of the surface Web by 40 times. (CompletePlanet, n.d., para. 6)

Michelle Post, Ph.D. 7

“The deep Web is the fastest growing sector of the Web and it appears to be the „paradigm for the next generation Internet‟” (BrightPlanet, 2005, para. 35).

8© Michelle Post, Ph.D.

ResourceShelf -http://www.resourceshelf.com/

Direct Search -http://www.freepint.com/gary/direct.htm

Complete Planet -http://www.completeplanet.com/

Michelle Post, Ph.D. 9

Listing of Largest Deep Web Sites -http://aip.completeplanet.com/aip-engines/help/largest_engines.jsp

Directory of Open Access Journals -http://www.doaj.org/

Educator's Reference Desk - http://www.eduref.org/

Find Articles - http://www.findarticles.com/

On-Line Encyclopedia of Integer Sequences -http://www.research.att.com/~njas/sequences/

PubMed -http://www.ncbi.nlm.nih.gov/entrez/query.fcgi

Super Searchers Web Page -http://www.infotoday.com/supersearchers/

Michelle Post, Ph.D. 10

The Virtual Private Library and Deep Web -http://www.dailymotion.com/video/x9udyo_the-virtual-private-library-and-dee_news

Michelle Post, Ph.D. 11

100 Time-Saving Search Engines for Serious Scholars -http://www.onlineuniversities.com/blog/2010/03/100-time-saving-search-engines-for-serious-scholars/

Deep Web FAQ - http://aip.completeplanet.com/aip-engines/help/help_deepwebfaqs.jsp

The Invisible Web Databases -http://www.noodletools.com/debbie/literacies/information/5locate/advicedepth.html

Those Dark Hiding Places: The Invisible Web Revealed -http://library.rider.edu/scholarly/rlackie/Invisible/Inv_Web.html

White Paper: The Deep Web: Surfacing Hidden Value -http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104

Michelle Post, Ph.D. 12

Bergman, M. (2001, August). White Paper: The deep web: Surfacing hidden value. The Journal of Electronic Publishing, 7(1), doi: 10.3998/3336451.0007.104. Retrieved from http://quod.lib.umich.edu/cgi/t/text/text-idx?c=jep;view=text;rgn=main;idno=3336451.0007.104

BrightPlanet. (2005). Deep webfaq. Retrieved from http://www.brightplanet.com/

CompletePlanet. (n.d.). How large is the deep web. Retrieved from http://aip.completeplanet.com/aip-engines/help/help_deepwebfaqs.jsp

Gruchawka, S. (2010). Using the deep web. Retrieved from http://techdeepweb.com/

UC Berkeley. (n.d.). Invisible or deep web: What it is, how to find it, and Its inherent ambiguity. Retrieved from http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/InvisibleWeb.html

13© Michelle Post, Ph.D.