building data integration systems for the web alon halevy google nsf information integration...
TRANSCRIPT
Building Data Integration Systems for the Web
Alon Halevy
NSF Information Integration Workshop
April 22, 2010
Without (too much) Loss of Generality
Web Enterprise, Science projects, …
Information integration ≅ data management
A Few Principles
• Data management “in situ”– Data meaning is derived from its context– Manipulate data in its natural location
• Pay-as-you-go data management– Provide services before modeling is done– Data can be about any domain
• Collaboration should be built in– Query answering is only step the first step
Alex Labrinidis
@via Facebook
Structured Data & The Web
Discover
Manage,Analyze, Combine
ExtractPublish
Hard to query, visualize, combine data across organizations
Requires infrastructure, concerns about losing control
Hard to find structured data via search engines
Data is embedded in web page, behind forms
Outline
• Surfacing the Deep Web
• Searching tables on the surface Web
• Fusion Tables: a platform for data management on the Web.
What is the Deep Web?
store locationsused cars
radio stationspatents
recipes
• Deep = not accessible through general purpose search engines– Major gap in the coverage of search engines.
Tree Search
Amish quilts
Parking tickets in India
Horses
Solution Constraints
• Can’t design a solution that requires domain engineering– (unless you can make money in that
domain!)
• Boundaries between domains are fuzzy
• Solution needs to be integrated into general web search– Can’t assume special query syntax
Surfacing the Deep Web[Madhavan et al. VLDB 2008]
• Surfacing: – Find high-quality forms– Guess good queries to submit– Put the resulting HTML pages in the index
• ~3M sites, 50 languages, 700 domains.• 1000 queries per-second get results from the
deep web.• 400K forms served per day, 800K per week• Impact mostly on the long and heavy tail of
queries
Deep Web: The Future
• Still an opportunity to go deeper into the deep web:– E.g., map the user query into a form
submission.
• Key challenge: given a keyword query, map it to forms in any domain
• Understanding the meaning of forms is still hard (e.g. content, geo constraints).
Outline
Surfacing the Deep WebSearching tables on the surface Web
• Fusion Tables: a platform for data management on the Web.
Bad table
Vertical Tables
Sub-Header Rows
Winners of the Boston Marathon (but that’s nowhere in the table)
Schema Ok, but context is subtle (year = 2006)
WebTables: Exploring the Relational Web[Cafarella et al., VLDB 2008, WebDB 08]
• In corpus of 14B raw tables, we estimate 154M are “good” relations– Single-table databases; Schema = attr labels + types– Largest corpus of databases & schemas we know of
• The Webtables system:– Recovers good relations from crawl and enables search– Builds novel apps on the recovered data
(Web-scale) Schema Collection
name e-mail|email, phone|telephone, e-mail_address|email_address, date|last_modified
instructor course-title|title, day|days, course|course-#,course-name|course-title
elected candidate|name, presiding-officer|speaker
ab k|so, h|hits, avg|ba, name|player
sqft bath|baths, list|list-price, bed|beds, price|rent
With 2.6 million schemas you can do some very interesting things.
Synonym discovery
“KR”-Based Table Search [Wu, Madhavan, Miao, Pasca, Shen]
• Ideally, we describe every table:– Class of entities it contains– Properties being modeled– Context, quality, …
• Use Web-extracted knowledge bases– Extract isa-hierarchy using patterns:– “cities such as Paris and London”– “chemical elements including hydrogen and
oxygen”
Step 1: Find “Subject” of Table
Not always the left (or first non-number column)
Step 2: associate classes with subjectChemical elements
Most of the time, the class labels are not in attribute name
Leveraging Web-extracted Ontologies
• Given a query, e.g., (country, GDP)– Rank tables about countries that have GDP
somewhere in the schema. – Very high precision (~90%)
• Next challenge: understand binary properties and binary relationships.
• Domain specialization: – System should improve if given ontologies in a
particular domain.
25
Combine Search, Extraction, Cleaning and Integration
[Cafarella, Koussainova, H., VLDB 2009],
• Try to create a database of all“VLDB program committee members”
Outline
Surfacing the Deep WebSearching tables on the surface WebFusion Tables: a platform for data
management on the Web.
Data Management for the Web Era
• Integrate seamlessly with the Web:– Search, maps, …
• Easy to use:– Much broader user base, pay-as-you-go– Very simple data integration
• Provide incentives for sharing data
• Facilitate collaboration
Fusion Tables – our current attempt[Madhavan, Gonzalez, Langen, Shapley, Shen]
We store and leverage a large collection of tables.
Incentive
Incentive, Pay-..-Go
Coffee Production
Coffee Consumption
Seamless integration with other web tools
Toilet heat map…
Database functionality on map
Collaboration
Table Search
Show up in search results!
Data Integration
Merged Table
Carries attribution from both base tables. Owners maintain control of their own data.
Fine Grained Discussions
Example Uses of Fusion Tables
• Tracking potholes in Spain• Displaying bike routes (MTBGuru)• State of California statistics• Government data from data.gov• Data about voting locations in the USA• Brazilian beaches• Chicago homicides• Most requested pop songs by year
Conclusions
• Information integration “in situ”– Blur the boundary between structured and
unstructured data
• Combine search, extraction, cleaning and integration into a single experience
• Pay-as-you-go: introduce complexity as needed– Serve enterprises without IT depth
• OpenII – an open-source platform for information integration.
References• Fusion Tables:
– tables.googlelabs.com– SIGMOD, SOCC, 2010
• Deep-web crawling:– [Madhavan et al., VLDB 08]
• WebTables: – [Cafarella et al., VLDB 08]
• Octopus: – [Cafarella et al., VLDB 09],– [Elmeleegy et al, VLDB 09]