![Page 1: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/1.jpg)
Big Data – so what’s the big deal?
Jevin D. West iSchool, University of Washington
![Page 2: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/2.jpg)
What is big data?
![Page 3: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/3.jpg)
Agenda • Introductions • Big data – why should you care? • Introduction to big data • Examples • Nuances in big data • Big data at UW and in Seattle • An exercise in big data • Concerns • Big data skills • Homework • References
![Page 5: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/5.jpg)
![Page 6: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/6.jpg)
Molecular & Cell Biology
Medicine
Physics
Ecology & Evolution
Economics
Geosciences
Psychology
Chemistry
Psychiatry
Environmental Chemistry & Microbiology
Mathematics
Computer Science
Analytic ChemistryBusiness & Marketing
Political Science
Fluid Mechanics
Medical Imaging
Material Engineering
Sociology
Probability & Statistics
Astronomy & Astrophysics
Gastroenterology
Law
Chemical Engineering
Education
Telecommunication
Control Theory
Operations Research
Ophthalmology
Crop Science
Geography
Anthropology
Computer Imaging
Agriculture
Parasitology
Dentistry
Dermatology
Urology
Rheumatology
Applied Acoustics
Pharmacology
Pathology
Otolaryngology
Electromagnetic Engineering
Circuits
Power Systems
Tribology
Neuroscience
Orthopedics Veterinary
Environmental Health
A
Citation flow from B to ACitation flow within field
Citation flow from A to BCitation flow out of field
B
![Page 7: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/7.jpg)
![Page 8: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/8.jpg)
Why should you care about big data?
A shortage of 1.5 million jobs!
![Page 9: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/9.jpg)
![Page 10: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/10.jpg)
![Page 11: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/11.jpg)
![Page 12: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/12.jpg)
![Page 13: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/13.jpg)
Universities are going big
![Page 14: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/14.jpg)
What is big data?
![Page 15: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/15.jpg)
![Page 16: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/16.jpg)
![Page 17: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/17.jpg)
![Page 18: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/18.jpg)
“Yes, some of the best theorizing comes a4er collec6ng data because then you become aware of another
reality…”
Robert Shiller, Nobel Price in Economics (2013)
![Page 19: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/19.jpg)
Data Exhaust
![Page 20: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/20.jpg)
Data Exhaust: by-product of human activity
Examples: cell phone locations, purchase transactions, social media Predicting human behavior, spread of infectious disease
Barabasi et al., Nature (2008), Ginsperg et al., Nature (2009)
![Page 21: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/21.jpg)
Why big data?
• Cheaper sensors (climate research, astronomy, high energy physics, high-throughput gene sequencing, cell phones)
• Cheaper storage (4 TB, $168) • People willing to share their personal
information (Facebook, social media) • Faster communication (internet, cell phones) • Other reasons?
![Page 22: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/22.jpg)
![Page 23: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/23.jpg)
The Four A’s
• Architecture • Acquisition • Analysis • Archiving
![Page 24: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/24.jpg)
The Four V’s
• Volume • Velocity • Variety • Veracity
![Page 25: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/25.jpg)
Big Data is messy
![Page 26: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/26.jpg)
Correlation versus Causation
![Page 27: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/27.jpg)
Sampling
![Page 28: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/28.jpg)
Big Data at UW
• LSST • CS (Farecast) • Libraries (digital content) • Oceanography • Neuroscience
![Page 29: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/29.jpg)
Is there a secondary market for the data that companies are collecting?
![Page 30: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/30.jpg)
Big Data in action
![Page 31: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/31.jpg)
DJ Patil
![Page 32: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/32.jpg)
If you had access to the personal calendars of 200 million people, what could you do with it? What products
could you create?
![Page 33: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/33.jpg)
The power of meta data…
![Page 34: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/34.jpg)
hHp://qz.com/140357/what-‐your-‐facebook-‐friends-‐list-‐reveals-‐about-‐your-‐love-‐life/
![Page 35: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/35.jpg)
It’s only meta data…
hHp://kieranhealy.org/blog/archives/2013/06/09/using-‐metadata-‐to-‐find-‐paul-‐revere/
![Page 36: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/36.jpg)
![Page 37: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/37.jpg)
hHp://kieranhealy.org/blog/archives/2013/06/09/using-‐metadata-‐to-‐find-‐paul-‐revere/
![Page 38: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/38.jpg)
hHp://kieranhealy.org/blog/archives/2013/06/09/using-‐metadata-‐to-‐find-‐paul-‐revere/
![Page 39: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/39.jpg)
Betweeness centrality
Eigenvector centrality
hHp://kieranhealy.org/blog/archives/2013/06/09/using-‐metadata-‐to-‐find-‐paul-‐revere/
![Page 40: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/40.jpg)
Big data is about asking good questions
![Page 41: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/41.jpg)
Concerns
• Privacy • Probabilistic Models • Correlation • The big players own the big data • NSA • Reproducibility • What else?
![Page 42: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/42.jpg)
Enjoy the wave but be cautious…
![Page 43: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/43.jpg)
‘The Data Scientist’
Communication skills Ethical Reasoning
Information/Data Management Personnel Management
Interdisciplinary Adaptable
![Page 44: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/44.jpg)
Big Data involves people
![Page 45: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/45.jpg)
Homework
Example of big data (1) Why you think it is big data. (2) How it involves people?
![Page 46: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/46.jpg)
References
![Page 47: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/47.jpg)
“Data is increasingly digital air : the oxygen we breathe and the carbon dioxide that we exhale. It
can be a source of both sustenance and pollution.” -- Dana Boyd
D. Boyd & K. Crawford (2011) Six Provocations on Big Data. SSRN
![Page 48: Big Data – so what’s the big deal?kevinx-chiu.weebly.com/uploads/8/9/8/3/8983380/class_05... · 2018-09-06 · Business & Marketing Analytic Chemistry Political Science Fluid](https://reader033.vdocuments.site/reader033/viewer/2022043007/5f91c6c5a5cc6f4d6c5b2eca/html5/thumbnails/48.jpg)
Why should you care about big data?
Jobs
Privacy