big data: a closer lookpkc/cs4hs/bigdata/bigdataintro.pdf · what is big data? increasingly ......
TRANSCRIPT
![Page 1: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/1.jpg)
Big Data: A Closer Look
![Page 2: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/2.jpg)
What is Big Data?
Increasingly popular search query http://www.google.com/trends/explore#q=big%20data
![Page 3: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/3.jpg)
What is Big Data?
the V’s
![Page 4: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/4.jpg)
Characteristics of Big Data
Volume
http://www.businessweek.com/articles/2014-03-06/interactive-graphic-how-big-is-big-data
![Page 5: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/5.jpg)
Characteristics of Big Data
VolumeVelocity
http://blog.lionbridge.com/enterprise-crowdsourcing/files/2013/06/big-data-blog-post-graphic.png
![Page 6: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/6.jpg)
Characteristics of Big Data
VolumeVelocityVariety
http://www.datasciencecentral.com/forum/topics/the-3vs-that-define-big-data
![Page 7: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/7.jpg)
Characteristics of Big Data
VolumeVelocityVarietyVeracity
http://www.sec.gov/Archives/edgar/data/51143/000110465913015636/g61551bii004.gif http://www.datasciencecentral.com/profiles/blogs/data-veracity http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4-Vs-of-big-data.jpg
![Page 8: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/8.jpg)
What to do with Big Data?
![Page 9: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/9.jpg)
What to do with Big Data?
Advances in Computer Science (CS) sample companies sample applications in analytics
Focus of speakers
![Page 10: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/10.jpg)
Sample Companies
EMC http://www.emc.com/campaign/bigdata/index.htm
![Page 11: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/11.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
![Page 12: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/12.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
Oracle http://www.oracle.com/us/technologies/big-data/index.html
![Page 13: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/13.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
Oracle (database software) http://www.oracle.com/us/technologies/big-data/index.html
![Page 14: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/14.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
Oracle (database software) http://www.oracle.com/us/technologies/big-data/index.html
SAS http://www.sas.com/en_us/insights/big-data.html
![Page 15: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/15.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
Oracle (database software) http://www.oracle.com/us/technologies/big-data/index.html
SAS (analytics software) http://www.sas.com/en_us/insights/big-data.html
![Page 16: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/16.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
Oracle (database software) http://www.oracle.com/us/technologies/big-data/index.html
SAS (analytics software) http://www.sas.com/en_us/insights/big-data.html
IBM http://www.ibm.com/big-data/
![Page 17: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/17.jpg)
Sample Companies
EMC (data storage hardware) http://www.emc.com/campaign/bigdata/index.htm
Oracle (database software) http://www.oracle.com/us/technologies/big-data/index.html
SAS (analytics software) http://www.sas.com/en_us/insights/big-data.html
IBM (computational hardware, database & analytics software, services) http://www.ibm.com/big-data/
![Page 18: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/18.jpg)
Sample Applications in Analytics
Automated Recommendation of products (amazon, netflix) Organization of news articles (google news)
![Page 19: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/19.jpg)
Product Recommendation
![Page 20: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/20.jpg)
Recommendation Systems
Netflix Recommending movies
Amazon Recommending products of many kinds
![Page 21: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/21.jpg)
Netflix Prize
netflixprize.com $1 million 10% improvement in accuracy over Netflix algorithm/system
(in 5 years) 51K contestants, 41K teams, 186 countries Did any team win the prize?
If so, how long did it take?
![Page 22: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/22.jpg)
Netflix Prize Data (1998-2005)
Customers 480,189 (ID: 1 – 2,649,429)
Movies 17,770 (ID: 1 – 17,770) ID, title, year
Ratings given in Training Set 100,480,507 min=1; max=17,653; avg=209 ratings per customer Rating scale: 1 – 5 Date
Ratings to predict in Qualifying Set 2,817,131
About 1 GB (700 MB compressed)
![Page 23: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/23.jpg)
Netflix Problem
M1 M2 M3 M4 M5 M6 M7 M8 M9
C1 ? 1 3 4 ?
C2 2 3 1 4 5
C3 2 5 3 3 3 4 4 1
C4 2 3 2
C5 4 1 3 3
? = unknown rating to be predicted
![Page 24: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/24.jpg)
Organizing News Articles
![Page 25: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/25.jpg)
Google News
news.google.com
![Page 26: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/26.jpg)
Google News
news.google.com News organizer from many sources Krishna Bharat, 2006 http://googleblog.blogspot.com/2006/01/and-now-news.html
“using computers to organize the world's news in real time and providing a bird's eye view of what's being reported on virtually any topic.
![Page 27: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/27.jpg)
Google News
news.google.com News organizer from many sources Krishna Bharat, 2006 http://googleblog.blogspot.com/2006/01/and-now-news.html
“using computers to organize the world's news in real time and providing a bird's eye view of what's being reported on virtually any topic.
By presenting news "clusters" (related articles in a group), we thought it would encourage readers to get a broader perspective by digging deeper into the news -- reading ten articles instead of one, perhaps -- and then gain a better understanding of the issues, which could ultimately benefit society”
![Page 28: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/28.jpg)
Google News Problem
Input News articles
Output Clusters of news articles
Algorithm (how) ?
![Page 29: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/29.jpg)
Animation of a Clustering Algorithm
http://shabal.in/visuals/kmeans/1.html
![Page 30: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/30.jpg)
Sample Applications in Analytics
Automated Spam Detection -- Ryan Stansifer DNA Sequence Analysis -- Debasis Mitra
![Page 31: Big Data: A Closer Lookpkc/cs4hs/bigdata/bigdataIntro.pdf · What is Big Data? Increasingly ... thought it would encourage readers to get a broader perspective by digging deeper into](https://reader034.vdocuments.site/reader034/viewer/2022052010/602074814bf7216d1f102361/html5/thumbnails/31.jpg)
Upcoming CS Outreach Events
Mar: teacher workshop Mar-Apr: student competition Boulder Dash Details in mid March
Jul: summer camps
cs.fit.edu/~pkc/cs4hs
Questions?