22c:145 artificial intelligence
DESCRIPTION
22C:145 Artificial Intelligence. Introduction to Web Mining Based on the lecture notes of Anand Rajaraman Jeff Ullman. What is Web Mining?. Discovering useful information from the World-Wide Web and its usage patterns. Web Mining v. Data Mining. Structure (or lack of it) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/1.jpg)
22C:145 Artificial Intelligence
Introduction to Web Mining
Based on the lecture notes ofAnand RajaramanJeff Ullman
![Page 2: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/2.jpg)
What is Web Mining?
Discovering useful information from the World-Wide Web and its usage patterns
![Page 3: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/3.jpg)
Web Mining v. Data Mining Structure (or lack of it)
Textual information and linkage structure Scale
Data generated per day is comparable to largest conventional data warehouses
Speed Often need to react to evolving usage
patterns in real-time (e.g., merchandising)
![Page 4: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/4.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 5: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/5.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 6: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/6.jpg)
Size of the Web Number of pages
Technically, infinite Much duplication (30-40%) Best estimate of “unique” static HTML
pages comes from search engine claims Google = 8 billion(?), Yahoo = 20 billion
Number of web sites Netcraft survey says 72 million sites
(http://news.netcraft.com/archives/web_server_survey.html)
![Page 7: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/7.jpg)
Netcraft survey
http://news.netcraft.com/archives/web_server_survey.html
![Page 8: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/8.jpg)
The web as a graph Pages = nodes, hyperlinks = edges
Ignore content Directed graph
High linkage 8-10 links/page on average Power-law degree distribution
![Page 9: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/9.jpg)
Structure of Web graph Let’s take a closer look at structure
Broder et al (2000) studied a crawl of 200M pages and other smaller crawls
Bow-tie structure Not a “small world”
![Page 10: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/10.jpg)
Bow-tie Structure
Source: Broder et al, 2000
![Page 11: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/11.jpg)
What can the graph tell us? Distinguish “important” pages from
unimportant ones Page rank
Discover communities of related pages Hubs and Authorities
Detect web spam Trust rank
![Page 12: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/12.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 13: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/13.jpg)
Power-law degree distribution
Source: Broder et al, 2000
![Page 14: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/14.jpg)
Power-laws galore Structure
In-degrees Out-degrees Number of pages per site
Usage patterns Number of visitors Popularity
![Page 15: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/15.jpg)
The Long Tail
Source: Chris Anderson (2004)
![Page 16: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/16.jpg)
The Long Tail Shelf space is a scarce commodity for
traditional retailers Also: TV networks, movie theaters,…
The web enables near-zero-cost dissemination of information about products Action moves from Hits to Niches
![Page 17: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/17.jpg)
The Long Tail More choice necessitates better filters
Recommendation engines (e.g., Amazon) In fact, page rank can be seen as a
long tail filter Tapping into the Wisdom of Crowds
![Page 18: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/18.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 19: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/19.jpg)
Extracting Structured Data
http://www.simplyhired.com
![Page 20: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/20.jpg)
Extracting structured data
http://www.fatlens.com
![Page 21: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/21.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 22: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/22.jpg)
Searching the Web
Content aggregatorsThe Web Content consumers
![Page 23: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/23.jpg)
Ads vs. search results
![Page 24: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/24.jpg)
Ads vs. search results Search advertising is the revenue
model Multi-billion-dollar industry Advertisers pay for clicks on their ads
Interesting problems What ads to show for a search? If I’m an advertiser, which search terms
should I bid on and how much to bid?
![Page 25: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/25.jpg)
Sidebar: What’s in a name? Geico sued Google, contending that it
owned the trademark “Geico” Thus, ads for the keyword geico couldn’t
be sold to others Court Ruling: search engines can sell
keywords including trademarks No court ruling yet: whether the ad
itself can use the trademarked word(s)
![Page 26: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/26.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 27: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/27.jpg)
Systems architecture
Memory
Disk
CPUMachine Learning, Statistics
“Classical” Data Mining
![Page 28: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/28.jpg)
Very Large-Scale Data Mining
Mem
Disk
CPU
Mem
Disk
CPU
Mem
Disk
CPU…
Cluster of commodity nodes
![Page 29: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/29.jpg)
Systems Issues Web data sets can be very large
Tens to hundreds of terabytes Cannot mine on a single server!
Need large farms of servers How to organize hardware/software to
mine multi-terabye data sets Without breaking the bank!
![Page 30: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/30.jpg)
Web Mining topics Web graph analysis Power Laws and The Long Tail Structured data extraction Web advertising Systems Issues
![Page 31: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/31.jpg)
Web Mining Project Lots of interesting project ideas
If you can’t think of one please come discuss with us
Data and Infrastructure Webbase data (older Stanford web crawl) Recent web crawl and server courtesy of
Kosmix
![Page 32: 22C:145 Artificial Intelligence](https://reader036.vdocuments.site/reader036/viewer/2022070503/56815585550346895dc358a7/html5/thumbnails/32.jpg)
The World-Wide Web
Our modern-day Library of Alexandria
The Web