twitter power-user howto...visualize content and connections ad-hoc queries exploration focus on...

Twitter Power-User HOWTO

@steveblackmon

✦Visualize content and connections

✦Ad-hoc queries

✦Exploration

✦Focus on specific profiles and posts

Goals

2

✦Assemble data about your twitter network

✦Analyze and Assess

✦…

✦Profit?

3

Plan

Data Visualization

4

Data Visualization

Data Repositories

5

Why ?✦ Handles de-duplication and merging

✦ Speaks JSON and REST

✦ Scales vertically and horizontally

✦ Because

6

Why ?✦ Handles de-duplication and merging

✦ Relationships central to data model

✦ Built and optimized for graph traversal queries

✦ Because

7

Supporting

Technologies

8

Why ?

✦Micro-service friendly

✦Works the same way across OS, distribution, cloud provider

✦Build, ship, run anywhere with 3-5 shell commands (build, [tag,] push, [pull,] run)

9

Why ?✦Assemble pipelines of modular data providers and database writers

✦Normalize your social databases to community standards

✦Why waste time hunting for and wrangling new SDKs?

10

11

I exist. Who am I?

Part 1: ID

12

Capture own history

Who am I?

13

Capture userstream

Who am I?

14

Other people do too.

Who are they?

15

Part 2: EGO

Collect friend/follower network

Who are they?

16

I am but one piece of a vast

society.

What is my place?

Part 3: SUPER-EGO

17

What is my place?Research interesting accounts

DEMO19

20

21

22

23

24

25

HOWTO26

✦ Using linux?

• apt-get / yum install docker.io ✦ Using a mac?

✦ Install Boot2Docker

✦ http://boot2docker.io/ • boot2docker ssh

• ifconfig

• note the bridged IP (eth1)

27

Docker

✦docker run -d --name elasticsearch -p 9200:9200 -p 9300:9300 -e PLUGINS=mobz/elasticsearch-head,elasticsearch/marvel/latest quay.io/steveblackmon/elasticsearch:1.4.4

✦docker run -d --name kibana -p 5601:5601 --link elasticsearch:elasticsearch quay.io/steveblackmon/kibana:4.0.0

✦docker run -d --name neo4j -p 7474:7474 quay.io/steveblackmon/neo4j:2.2.0.M4

28

Installation

✦ docker run -d --name linkurious -p 3000:3000 —link neo4j:neo4j quay.io/steveblackmon/linkurious:latest

** LINKURIOUS IS NOT OPEN-SOURCE **** PURCHASE A LICENSE AT http://linkurio.us **

** 20% OFF IF YOU MENTION THIS TALK **** 30-DAY MONEY-BACK GUARANTEE **

** ACCESS TO THIS DOCKER REPO IF YOU BUY**

29

Installation (Cont.)

✦developer.twitter.com

✦mytwitterid.com

✦mkdir streams

✦cd streams

Configuration

(Part 0)

30

Configuration (Part 1)

twitter.userstream.json 1. Add twitter oauth credentials

2. Add own userid to twitter.follow

3. Set elasticsearch.hosts

4. Set graph.hostname

twitter.history.json 1.Add twitter oauth credentials

2.Add own userid to twitter.info

3.Set elasticsearch.hosts

✦docker run -d --name twitter.userstream -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.userstream.json -cp /twitter-0.2-SNAPSHOT.jar twitter.UserstreamPipeline

✦docker run -d --name twitter.history -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline

Execution (Part 1)

32

twitter.friends.json 1. Add twitter oauth credentials

2. Add own userid to twitter.info

3. Set graph.hostname

twitter.followers.json 1. Add twitter oauth credentials

2. Add own userid to twitter.info

3. Set graph.hostname

Configuration (Part 2)

33

✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline

✦docker run -v `pwd`/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.followers.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline

Execution (Part 2)

34

Configuration (Part 3)Export IDs of your connections with Cypher

•MATCH (v:page {handle:'handle'})-[:follow]-(x:page) RETURN DISTINCT SUBSTRING(x.id, 11);

twitter.connections.history.json 1.Add twitter oauth credentials

2.Add connection ids to twitter.info

3.Set elasticsearch.hosts

twitter.connections.friends.json 4.Add twitter oauth credentials

5.Add connection ids to twitter.info

6.Set graph.hostname35

✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.history.json -cp /twitter-0.2-SNAPSHOT.jar twitter.HistoryPipeline

✦docker run -v /home/ubuntu/streams:/streams quay.io/steveblackmon/twitter:latest java -Dconfig.file=/streams/twitter.connections.friends.json -cp /twitter-0.2-SNAPSHOT.jar twitter.FollowingPipeline

Execution (Part 3)

36

@steveblackmon37

FIN