ire 2012 unstructured data talk
TRANSCRIPT
![Page 2: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/2.jpg)
What Am I Talking About?
• Example
• Structured Data 101
• Structured Data Continuum
• More Examples
![Page 3: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/3.jpg)
http://projects.propublica.org/drywall/
![Page 4: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/4.jpg)
http://www.propublica.org/documents/item/drywall-plaintiffs-omnibus-class-action-complaint
![Page 5: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/5.jpg)
1056. Plaintiffs - Intervenors, Robert and Tasha Lambert are citizens of Alabama and together own real property located at 541 Lynn Hurst Court, Montgomery, Alabama 36117. Plaintiffs are participating as class representatives in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1057. Plaintiff-Intervenor, Brenda Owens, is a citizen of Alabama and owns real property located at 2105 Lane Avenue, Birmingham, Alabama 35217. Plaintiff is participating as a class representative in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1058. Plaintiffs-Intervenors, Daniel and Nicole Smith are citizens of Alabama and together own real property located at 766 Tabernacle Road, Monroeville, Alabama
http://www.propublica.org/documents/item/drywall-plaintiffs-omnibus-class-action-complaint
![Page 6: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/6.jpg)
1056. Plaintiffs - Intervenors, Robert and Tasha Lambert are citizens of Alabama and together own real property located at 541 Lynn Hurst Court, Montgomery, Alabama 36117. Plaintiffs are participating as class representatives in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1057. Plaintiff-Intervenor, Brenda Owens, is a citizen of Alabama and owns real property located at 2105 Lane Avenue, Birmingham, Alabama 35217. Plaintiff is participating as a class representative in the class and subclasses as set forth in the schedules accompanying this complaint which are incorporated herein by reference. 1058. Plaintiffs-Intervenors, Daniel and Nicole Smith are citizens of Alabama and together own real property located at 766 Tabernacle Road, Monroeville, Alabama
http://www.propublica.org/documents/item/drywall-plaintiffs-omnibus-class-action-complaint
![Page 7: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/7.jpg)
541 Lynn Hurst Court,
Montgomery, Alabama 36117
37.0625, -95.677068
![Page 8: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/8.jpg)
541 Lynn Hurst Court,
Montgomery, Alabama 36117
37.0625, -95.677068
Jefferson County
![Page 9: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/9.jpg)
http://projects.propublica.org/drywall/
![Page 10: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/10.jpg)
http://projects.propublica.org/drywall/
![Page 11: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/11.jpg)
Scanned
Documents
Addresses
Google Maps
![Page 12: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/12.jpg)
Unstructured
Information
Structured Data
Visualization
Scanned
Documents
Addresses
Google Maps
![Page 13: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/13.jpg)
Unstructured
Information
Structured Data
Visualization
Scanned
Documents
Addresses
Google Maps
![Page 14: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/14.jpg)
Unstructured
Information
Structured Data
Visualization
Scanned
Documents
Addresses
Google Maps
![Page 15: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/15.jpg)
Unstructured
Information
Structured Data
Visualization
Scanned
Documents
Addresses
Google Maps
Who cares?
What is it?
![Page 16: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/16.jpg)
Who Cares?
Software
Visualization
Mashups
Store
Databases
PANDA
Analyze
Fusion tables
Excel
Databases
R/Python/Ruby
![Page 17: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/17.jpg)
Who Cares?
Software
Visualization
Mashups
![Page 18: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/18.jpg)
Who Cares?
Software
Visualization
Mashups
Tainted House Data
+ Economic Data
+ Health Stats
+ Crime Stats
+ Corruption Data
![Page 19: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/19.jpg)
Structured Data
![Page 20: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/20.jpg)
Structured Data
Attribute
Name
Data type
Consistent
![Page 21: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/21.jpg)
Structured Data
Attribute
Name
Data type
Consistent
![Page 22: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/22.jpg)
Structured Data
Attribute
Name
Data type
Consistent
![Page 23: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/23.jpg)
Structured Data
Attribute
Name
Data type
Consistent
Florida’s Lee County
has 1518 addresses
![Page 24: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/24.jpg)
Structured Data
Attribute
Name
Data type
Consistent
![Page 25: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/25.jpg)
Structured Data
Attribute
Name
Data type
Consistent
![Page 26: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/26.jpg)
Structured Data
Attribute
Name
Data type
Consistent
Numeric(integers, dollars,…)
Date/Time
Lat, Lon
![Page 27: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/27.jpg)
Structured Data
Attribute
Name
Data type
Consistent
Numeric(integers, dollars,…)
Date/Time
Lat, Lon
Structured strings(Florida)
![Page 28: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/28.jpg)
Structured Data
Attribute
Name
Data type
Consistent
FLORIDA
FL
Flroida
FloridaState
Florida’s
![Page 29: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/29.jpg)
Structured Data
Attribute
Name
Data type
Consistent
FLORIDA
FL
Flroida
FloridaState
Florida’s
5
10
1
1
1
![Page 30: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/30.jpg)
Structured Data
Attribute
Name
Data type
Consistent
![Page 31: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/31.jpg)
What Am I Talking About?
• Structured Data 101
• Structured Data Continuum
• More Examples
![Page 32: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/32.jpg)
structuredunstructured
Continuum
![Page 33: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/33.jpg)
structuredunstructured
ImagesImages
http://www.whatisstephenharperreading.ca/2010/03/01/book-number-76-one-day-in-the-life-of-ivan-denisovich-by-alexander-solzhenitsyn/
![Page 34: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/34.jpg)
structuredunstructured
ImagesImages Text Blob
1056. Plaintiffs - Intervenors, Robert and Tasha Lambert
are citizens of Alabama and together own real property
located at 541 Lynn Hurst Court, Montgomery, Alabama
36117. Plaintiffs are participating as class representatives
in the class and subclasses as set forth in the schedules
accompanying this complaint which are incorporated herein
by reference. 1057. Plaintiff-Intervenor, Brenda Owens, is a
citizen of Alabama and owns real property located at 2105
Lane Avenue, Birmingham, Alabama 35217. Plaintiff is
participating as a class representative in the
![Page 35: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/35.jpg)
structuredunstructured
ImagesImages Text Blob Email
![Page 36: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/36.jpg)
structuredunstructured
ImagesImages Text Blob Email
Re: IRE conference in Boston
June 1, 3:08PM
Subject
Date
From
![Page 37: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/37.jpg)
structuredunstructured
ImagesImages Text Blob Email Excel
![Page 38: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/38.jpg)
structuredunstructured
Images Text Blob Email Excel
![Page 39: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/39.jpg)
“It’s sunny
in texas”
structuredunstructured
Images Text Blob Email Excel
![Page 40: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/40.jpg)
“It’s sunny
in texas”
Tweet Weather Location
It’s sunny in
texas
Sunny Texas
structuredunstructured
Images Text Blob Email Excel
![Page 41: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/41.jpg)
“It’s sunny
in texas”
Tweet Weather Location
It’s sunny in
texas
Sunny (37.06,
-95.67)
structuredunstructured
Images Text Blob Email Excel
![Page 42: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/42.jpg)
You have unstructured data
What structure do I need?
Attributes with simple types
Whe
n
Ask
Find
![Page 43: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/43.jpg)
What Am I Talking About?
• Structured Data 101
• Structured data continuum
• More Examples
![Page 44: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/44.jpg)
http://www.boston.com/news/politics/specials/obama_state_of_the_union_word_cloud/
2011 State of the Union
![Page 45: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/45.jpg)
Type/Meaning
String
Name
Word
![Page 46: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/46.jpg)
Mr. Speaker, Mr. Vice President,
members of Congress,
distinguished guests, and fellow
Americans:
Tonight I want to begin by
congratulating the men and
women of the 112th Congress, as
well as your new Speaker, John
Boehner. And as we mark this
occasion, we're also mindful of
the empty chair in this chamber,
and we pray for the health of our
colleague -- and our friend --
Gabby Giffords.
It's no secret that those of us here
tonight have had our differences
over the last two years. The
debates have been contentious;
we have fought fiercely for our
beliefs. And that's a good thing.
That's what a robust democracy
![Page 47: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/47.jpg)
Mr. Speaker, Mr. Vice President,
members of Congress,
distinguished guests, and fellow
Americans:
Tonight I want to begin by
congratulating the men and
women of the 112th Congress, as
well as your new Speaker, John
Boehner. And as we mark this
occasion, we're also mindful of
the empty chair in this chamber,
and we pray for the health of our
colleague -- and our friend --
Gabby Giffords.
It's no secret that those of us here
tonight have had our differences
over the last two years. The
debates have been contentious;
we have fought fiercely for our
beliefs. And that's a good thing.
That's what a robust democracy
Word
Mr
Speaker
Vice
President
Members
Congress
Distinguished
Guests
Americans
People
Jobs
New
years
![Page 48: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/48.jpg)
Bin Laden Tweets/Sec
http://www.flickr.com/photos/twitteroffice/5681263084/
![Page 49: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/49.jpg)
Type/Meaning
Time
Name
Time
![Page 50: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/50.jpg)
Deadly Day in Baghdad
http://www.nytimes.com/interactive/2010/10/24/world/1024-surge-graphic.html?pagewanted=all
![Page 51: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/51.jpg)
Type/Meaning
Lat, Lon
Number
Name
Location
Body Count
![Page 52: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/52.jpg)
http://www.nytimes.com/interactive/world/iraq-war-logs.html?pagewanted=all
![Page 53: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/53.jpg)
http://www.nytimes.com/interactive/world/iraq-war-logs.html?pagewanted=all
14, 12
Killed in Action
Lat Lon
![Page 54: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/54.jpg)
Sentiment of NZ Earthquake
http://twitinfo.csail.mit.edu/detail/4/
![Page 55: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/55.jpg)
Type/Meaning
-1 to 1
Name
Happiness
![Page 56: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/56.jpg)
![Page 57: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/57.jpg)
Pattern
Matching
Great, 7AM
meeting
7:00AM
![Page 58: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/58.jpg)
Interpret
Meaning
Great, 7AM
meeting
Not Happy
![Page 59: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/59.jpg)
Interpret
Meaning
It’s still
new
Great, 7AM
meeting
Happy!
![Page 60: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/60.jpg)
Interpret
Meaning
It’s still
new
![Page 61: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/61.jpg)
Interpret
Meaning
It’s still
new
Lack of
context
Earthquakes
![Page 62: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/62.jpg)
Extracting meaning is
by far the most difficult
![Page 63: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/63.jpg)
What if it’s just unstructured?
![Page 64: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/64.jpg)
CrowdSourcing
Lots of humans do
tasks computers suck
at
Training
Quality Issues
![Page 65: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/65.jpg)
Dealing with Forms
![Page 66: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/66.jpg)
Dealing with Forms
![Page 67: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/67.jpg)
Entity Information
![Page 68: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/68.jpg)
Pattern Matching
• Regex
– Describe and find patterns
– Killed in action
(?P<n>\d{1,3})(\s[A-Z]{1,3})?\sKIA
![Page 69: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/69.jpg)
DBTruck demo?
![Page 70: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/70.jpg)
Structure = Super Valuable
![Page 71: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/71.jpg)
You have unstructured data
What structure do I need?
Attributes with simple types
When
Ask
Find
Structure = Super Valuable
![Page 72: IRE 2012 Unstructured Data Talk](https://reader034.vdocuments.site/reader034/viewer/2022042817/55a0f7b81a28ab972a8b45af/html5/thumbnails/72.jpg)
You have unstructured data
What structure do I need?
Attributes with simple types
When
Ask
Find
tinyurl.com/iredatatipsheet
@sirrice
Structure = Super Valuable