![Page 1: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/1.jpg)
Mining Social Web APIswith IPython Notebook
Matthew A. Russell - @ptwobrussell - http://MiningTheSocialWeb.com
Austin - 9 January 2015
1
![Page 2: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/2.jpg)
Intro
2
![Page 3: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/3.jpg)
Hello, My Name Is ... Matthew
3
Background in Computer Science
Data mining & machine learning
CTO @ Digital Reasoning Systems
Data mining; machine learning
Author @ O'Reilly Media
5 published books on technology
Principal @ Zaffra
Selective boutique consulting
![Page 4: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/4.jpg)
Transforming Curiosity Into Insight
4
An open source software (OSS) project
http://bit.ly/MiningTheSocialWeb2E
A book
http://bit.ly/135dHfs
Accessible to (virtually) everyone
Virtual machine with turn-key coding templates for data science experiments
Think of the book as "premium" support for the OSS project
![Page 5: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/5.jpg)
Table of Contents (1/2)
Chapter 1 - Mining Twitter: Exploring Trending Topics, Discovering What People Are Talking About, and More
Chapter 2 - Mining Facebook: Analyzing Fan Pages, Examining Friendships, and More
Chapter 3 - Mining LinkedIn: Faceting Job Titles, Clustering Colleagues, and More
Chapter 4 - Mining Google+: Computing Document Similarity, Extracting Collocations, and More
Chapter 5 - Mining Web Pages: Using Natural Language Processing to Understand Human Language, Summarize Blog Posts, and More
Chapter 6 - Mining Mailboxes: Analyzing Who's Talking to Whom About What, How Often, and More
5
![Page 6: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/6.jpg)
Table of Contents (2/2)
Chapter 7 - Mining GitHub: Inspecting Software Collaboration Habits, Building Interest Graphs, and More
Chapter 8 - Mining the Semantically Marked-Up Web: Extracting Microformats, Inferencing over RDF, and More
Chapter 9 - Twitter Cookbook
Appendix A - Information About This Machine's Virtual Machine Experience
Appendix B - OAuth Primer
Appendix C - Python and IPython Notebook Tips & Tricks
6
![Page 7: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/7.jpg)
Designed for PedagogyBrief Intro
Objectives
API Primer
Analysis Technique(s)
Data Visualization
Recap
Suggested Exercises
Recommended Resources
7
![Page 8: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/8.jpg)
The Social Web Is All the Rage
World population: ~7B people
Facebook: 1.15B users
Twitter: 500M users
Google+ 343M users
LinkedIn: 238M users
~200M+ blogs (conservative estimate)
8
*Estimates as of early 2014
![Page 9: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/9.jpg)
Overview
Module 1 - Virtual Machine Setup (30 mins)
Module 2 - Mining Twitter (90 mins)
BREAK (15 mins)
Module 3 - Mining Facebook (60 mins)
Lunch (75 mins)
9
Module 4 - Mining LinkedIn (75 mins)
BREAK (15 mins)
Module 5 - Open/Choice (75 mins)
Module 6 - Privacy & Ethics; (15 mins)
Module 7 - Final Q&A (15 mins)
![Page 10: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/10.jpg)
Module Format
~10-15 minutes of exposition
I talk; you listen
~15 minutes of independent (or collaborative) work
You hack while I walk around and help you
~5 minutes of recap with Q&A
You ask; I try to answer
10
![Page 11: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/11.jpg)
Workshop Objective
To send you away as a social web hacker
Broad working knowledge popular social web APIs
Hands-on experience hacking on social web data with a common toolkit
Not for me talk to you for 8 straight hours
11
![Page 12: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/12.jpg)
Just a Few More Things
This workshop is...
An adaptation of Mining the Social Web, 2nd Edition
More of a guided hacking session where you follow along (vs a preso)
Wider than it is deeper
There's only so much you can do in a few hours
I'm available 24/7 this week (and beyond) to help you be successful
12
![Page 13: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/13.jpg)
Assumptions
At some point in your life, you have
Programmed with Python
Worked with JSON
Made requests and processed responses to/from web servers
Or you want to learn to do these things now...
And you're a quick learner
13
![Page 14: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/14.jpg)
Module 1: Virtual Machine Setup
14
![Page 15: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/15.jpg)
Why do you need a VM?
15
To save yourself a lot of time
Because installation and configuration management is tedious and time-consuming
So that you can focus on the task at hand instead
So that I can support you regardless of your hardware and operating system
![Page 16: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/16.jpg)
But I can do all of that myself...True...
If you would rather troubleshoot unexpected installation/configuration issues instead of immediately focusing on the real task at hand
At least give it a shot before resorting to your own devices so that you don't have to install specific versions of ~40 Python packages
Including scientific computing tools that require underlying C/C++ code to be compiled
Which requires specific versions of developer libraries to be installed
You get the idea...
16
![Page 17: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/17.jpg)
The Virtual Machine ExperienceVagrant
A nice abstraction around virtual machine providers
One ring to rule them all
Virtualbox, VMWare, AWS, ...
IPython Notebook
The easiest way to program with Python
A better REPL (interpreter)
Great for hacking
17
![Page 18: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/18.jpg)
What happens when you vagrant up?
Vagrant follows the instructions in your Vagrantfile
Starts up a Virtualbox instance
Uses Chef to provision it
Installs OS patches/updates
Installs MTSW software dependencies
Starts IPython Notebook server on port 8888
18
![Page 19: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/19.jpg)
Why Should I Use IPython Notebook?
Because it's great for hacking
And hacking is usually the first step
Because it's great for collaboration
Sharing/publishing results is trivial
Because the UX is as easy as working in a notepad
Think of it as "executable paper"
19
![Page 20: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/20.jpg)
20
![Page 21: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/21.jpg)
21
![Page 22: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/22.jpg)
VM Quick Start Instructions
Go to http://MiningTheSocialWeb.com/quick-start/
Follow the instructions
And watch the screencasts!
Basically:
Install Virtualbox & Vagrant
Run "vagrant up" in a terminal to start a guest VM
Then, go to http://localhost:8888 on your host machine's web browser
22
![Page 23: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/23.jpg)
What Could Be Easier?
A hosted version of the VM!
But only for a few hours during this workshop
Because it costs money to run these servers
Go to http://bit.ly/XXX and pick a machine
Do not share the URLs outside of this workshop!
Please don't try to hack the machines
Learn how I arrived at this setup at http://MiningTheSocialWeb.com
23
![Page 24: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/24.jpg)
Module 2: Mining Twitter
24
![Page 25: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/25.jpg)
Objectives
25
Be able to identify Twitter primitives
Understand tweet metadata and how to use it
Learn how to extract entities such as user mentions, hashtags, and URLs from tweets
Apply techniques for performing frequency analysis with Python
Be able to plot histograms of Twitter data with IPython Notebook
![Page 26: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/26.jpg)
Twitter Primitives
26
Accounts Types: "Anything"
"Following" Relationships
Favorites
Retweets
Replies
(Almost) No Privacy Controls
![Page 27: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/27.jpg)
API RequestsRESTful requests
Everything is a "resource"
You GET, PUT, POST, and DELETE resources
Standard HTTP "verbs"
Example: GET https://api.twitter.com/1.1/statuses/user_timeline.json?screen_name=SocialWebMining
Streaming API filters
JSON responses
Cursors (not quite pagination)
27
![Page 28: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/28.jpg)
Twitter is an Interest Graph
28
Roberto Mercedes
Jorge
Ana
Nina
Johnny Araya
Rodolfo Hernández
![Page 29: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/29.jpg)
What's in a Tweet?
29
140 Characters ...
... Plus ~5KB of metadata!
Authorship
Time & location
Tweet "entities"
Replying, retweeting, favoriting, etc.
![Page 30: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/30.jpg)
What are Tweet Entities?
Essentially, the "easy to get at" data in the 140 characters
@usermentions
#hashtags
URLs
multiple variations
(financial) symbols
stock tickers
media
30
![Page 31: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/31.jpg)
Data Mining = Curiosity + StatsCuriosity
Interests, desires, and intuitions
Statistics
Counting
Comparing
Filtering
Ranking
Hypothesis testing; knowledge discovery
31
![Page 32: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/32.jpg)
Histograms
A chart that is handy for frequency analysis
They look like bar charts...except they're not bar charts
Each value on the x-axis is a range (or "bin") of values
Not categorical data
Each value on the y-axis is the combined frequency of values in each range
32
![Page 33: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/33.jpg)
33
Example: Histogram of Retweets
![Page 34: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/34.jpg)
Social Media Analysis FrameworkA memorable four step process to guide data science experiments:
Aspire
To test a hypothesis (answer a question)
Acquire
Get the data
Analyze
Count things
Summarize
Plot the results
34
![Page 35: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/35.jpg)
ExercisesReview Python idioms in the "Appendix C (Python Tips & Tricks)" notebook
Follow the setup instructions in the "Chapter 1 (Mining Twitter)" notebook
Fill in Example 1-1 with credentials and begin work
Execute each example sequentially
Customize queries
Explore tweet metadata; count tweet entities; plot histograms of results
Explore the "Chapter 9 (Twitter Cookbook)" notebook
Think of it as a collection of building blocks
35
![Page 36: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/36.jpg)
Module 3: Mining Facebook
36
![Page 37: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/37.jpg)
Objectives
37
Be able to identify Facebook primitives
Learn about Facebook’s Social Graph API and how to make API requests
Understand how Open Graph protocol extends Facebook's Social Graph API
Be able to analyze likes from Facebook pages and friends
The Graph v2 API changes at substantially tamped down privacy and permissions
See https://developers.facebook.com/docs/apps/changelog
![Page 38: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/38.jpg)
Facebook Primitives
Account Types: People & Pages
Mutual Connections
Likes
Shares
Comments
Extensive Privacy Controls
38
![Page 39: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/39.jpg)
Facebook is an Interest Graph
39
Roberto Mercedes
Jorge
Ana
Nina
Johnny Araya
Rodolfo Hernández
![Page 40: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/40.jpg)
Graph API
Nodes
Things
Edges
Connections between things
Fields
Info (properties) about things
40
![Page 41: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/41.jpg)
Facebook API Explorer
41
Go to https://developers.facebook.com/tools/explorer
Really, go there right now...
![Page 42: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/42.jpg)
Graph API Explorer
42
![Page 43: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/43.jpg)
Example Graph API RequestsSocial Graph API requests
Easy to learn and use
http://graph.facebook.com/me/feed
http://graph.facebook.com/me/likes
http://graph.facebook.com/me/?fields=id,name,friends.fields(likes.limit(10))
http://graph.facebook.com/Mining-the-Social-Web?fields=id,name,about,likes
JSON responses
Traditional pagination
43
![Page 44: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/44.jpg)
44
Retrieve Your Likes
![Page 45: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/45.jpg)
Permissions Prior to Graph v2
45
![Page 46: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/46.jpg)
Permissions as of Graph v2
46
![Page 47: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/47.jpg)
Explore Facebook Pages
47
Names of pages
MiningTheSocialWeb
CrossFit
OReilly
Web URLs (OGP extensions to Facebook's Social Graph)
http://www.imdb.com/title/tt0117500
![Page 48: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/48.jpg)
Social Media Analysis Framework
Recall the same four step process to guide data science experiments:
Aspire
Acquire
Analyze
Summarize
48
![Page 49: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/49.jpg)
Exercises
Copy/paste your access token from the Graph API Explorer into the "Chapter 2 (Mining Facebook)" notebook
Execute and tinker with Examples 2-1 thru 2-6
Inspect content in your feed
Juxtapose public figures
Compare/contrast similar products/brands of interest
49
![Page 50: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/50.jpg)
Module 4: Mining LinkedIn
50
![Page 51: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/51.jpg)
Objectives
51
Learn about LinkedIn’s Developer Platform
Understand how clustering works
A fundamental type of machine learning
Be able to employ geocoding services to arrive at a set of coordinates from a textual reference to a location
Visualize geographic data with cartograms
![Page 52: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/52.jpg)
LinkedIn Primitives
Account Types: People, Groups, Companies, Jobs
And Activity Streams
Data is typically perceived as being more sensitive
Richest data source? (Think: LinkedIn's business model)
Profile descriptions from mutual connections
A little messier than it first appears
52
![Page 53: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/53.jpg)
API Requests
HTTP-based Requests
Field selector syntax
http://api.linkedin.com/v1/people/~:(first-name,last-name,headline,picture-url)
XML responses
CSV address book download
53
![Page 54: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/54.jpg)
Is LinkedIn an Interest Graph?
Fundamentally: yes. the developer API requires you to do a bit of work to model it
Less trivial to find some of the "pivots"
e.g. There's no public Skills API for developers
But the data is there (mostly in profile descriptions) for your direct connections
Companies, job titles, job descriptions
Lots of richness is tucked away in human language data
54
![Page 55: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/55.jpg)
Clustering
An unsupervised machine learning learning technique
Think: an algorithm that organizes the data into partitions
55
![Page 56: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/56.jpg)
Example: Clustered Job Titles
56
![Page 57: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/57.jpg)
3 Steps to Clustering Your Data
Normalization
Compare (similarity/distance measurement)
n-grams, edit distance, and Jaccard are common, but your imagination is the limit
Why can't you just compare everything to everything?
Dimensionality Reduction
Ideally, your clustering algorithm will mitigate the pain
k-means is among the most common clustering techniques in use
57
![Page 58: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/58.jpg)
Jaccard Similarity
58
![Page 59: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/59.jpg)
k-Means Explained
1. Randomly pick k points in the data space as initial values that will be used to compute the k clusters: K1, K2, ..., Kk.
2. Assign each of the n points to a cluster by finding the nearest Kn—effectively creating k clusters and requiring k*n comparisons.
3. For each of the k clusters, calculate the centroid of the cluster and reassign its Ki value to be that value. (Hence, you’re computing “k-means” during each iteration of the algorithm.)
4. Repeat steps 2–3 until the members of the clusters do not change between iterations. Generally speaking, relatively few iterations are required for convergence.
59
![Page 60: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/60.jpg)
k-Means: Initialize
60
![Page 61: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/61.jpg)
k-Means: Step 1
61
![Page 62: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/62.jpg)
k-Means: Step 2
62
![Page 63: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/63.jpg)
k-Means: Step 3
63
![Page 64: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/64.jpg)
k-Means: (Fast-Forward) Step 9
64
![Page 65: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/65.jpg)
Geocoding
Transforming a location to a set of coordinates
Nashville, TN => (36.16783905029297, -86.77816009521484)
A harder problem than it first appears
The Bing API is especially generous
Requires an account sign up: http://bingmapsportal.com
Use the API key with the geopy package
65
![Page 66: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/66.jpg)
Introducing: The Dorling Cartogram
66
![Page 67: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/67.jpg)
Social Media Analysis Framework
Remember: Use the same four step process to guide data science experiments:
Aspire
Acquire
Analyze
Summarize
67
![Page 68: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/68.jpg)
ExercisesFollow the instructions in the "Chapter 3 (Mining LinkedIn)" notebook to create an API connection and follow along with the first few examples
Download your connections as a CSV file from http://www.linkedin.com/people/export-settings and save them to your VM
A deviation from instructions in Example 3-6 is necessary for remote VMs
See http://bit.ly/mtsw-ch03-helper-code
Try clustering your contacts in Example 3-12
Use the python-linkedin client to tap into Activity Streams
See https://developer.linkedin.com/documents/get-network-updates-and-statistics-api
68
![Page 69: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/69.jpg)
Module 5: Choice
69
![Page 70: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/70.jpg)
Objectives
70
To work on "loose ends" or areas of interest from previous modules
To hack on code in notebooks not yet encountered
To setup the virtual machine on your own box if you haven't yet
To collaborate/talk and otherwise make the most of our togetherness
![Page 71: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/71.jpg)
Social Media Analysis Framework
Remember:
Aspire
Acquire
Analyze
Summarize
71
![Page 72: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/72.jpg)
Recommendations
Setup your own development environment if you haven't already
Appendix A
Text Mining & Natural Language Processing
Chapter 4 (Mining Google+) & Chapter 5 (Mining Web Pages)
Graph Mining
Chapter 7 (Mining GitHub)
72
![Page 73: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/73.jpg)
Module 6: Privacy & Ethics
73
![Page 74: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/74.jpg)
74
Know thy data, and know thyself
--Matthew A. Russell
![Page 75: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/75.jpg)
75
If we have data, let’s look at data. If we have opinions, let’s go with mine
--Jim Barksdale
![Page 76: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/76.jpg)
76
In God we trust. All others must bring data
--W. Edwards Deming
![Page 77: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/77.jpg)
Communication => Data
Communication
Senders
humans & machines
Messages
natural language, images, videos, etc.
Recipients
humans & machines
77
![Page 78: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/78.jpg)
Data Alchemy
Data: Documents & document fragments (text messages, etc.)
Information: "Assertions", summaries, tags, etc.
Knowledge: Aggregated, queryable information
Wisdom: “Compressed” knowledge
Gold: Money
78
![Page 79: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/79.jpg)
Machine Learning
79
A program that learns (improves) from experience (data) according to some objective
Supervised learning
Unsupervised learning
Reinforcement learning
How to do it
Program mathematical models and hope for the best...
How to do it well
Program state-of-the-art mathematical models with sufficient representative data
![Page 80: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/80.jpg)
80
Knowledge is a process of piling up facts; wisdom lies in their simplification
--Martin Fischer
![Page 81: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/81.jpg)
81
Any sufficiently advanced technology is indistinguishable from magic
--Arthur C. Clarke
![Page 82: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/82.jpg)
Is Privacy Already an Illusion?
82
Digital happenings circa 2014
The Cloud
Social Media
Deep Learning
The Internet of Things
Internet.org
![Page 83: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/83.jpg)
83
Civilization is the progress toward a society of privacy...
-- Ayn Rand
![Page 84: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/84.jpg)
84
If you have something that you don’t want anyone to know, maybe you shouldn’t be doing it in the first place.
-- Eric Schmidt, (former) CEO of Google
![Page 85: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/85.jpg)
Influences on Ethics
Capitalism, economics, & marketing
A for-profit corporation's fiduciary duty: To maximize the common stock's value
How to do it? By transacting commerce
How do it well? By advertising more effectively than competitors
How to do it really well? With highly relevant personalized ads (recommenders)
Terms of Service (ToS) - The legal extent of ethical obligations?
85
![Page 86: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/86.jpg)
Module 7: Final Q&A
86
![Page 87: Mining Social Web APIs with IPython Notebook (Data Day Texas 2015)](https://reader033.vdocuments.site/reader033/viewer/2022060205/55a209291a28abd6788b48f5/html5/thumbnails/87.jpg)
Free Stuff
http://MiningTheSocialWeb.com
Mining the Social Web 2E Chapter 1 (Chimera)
http://bit.ly/13XgNWR
Source Code (GitHub)
http://bit.ly/MiningTheSocialWeb2E
http://bit.ly/1fVf5ej (numbered examples)
Screencasts (Vimeo)
http://bit.ly/mtsw2e-screencasts
87