SoLoGlo:An Archival and Analysis

Service forSocial, Local, and Global News

Martin Klein @mart1nkle1n

Peter Broadwell @peterbroadwell

Todd Grappone @liber8er

Sharon Farb @farbthink

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

• Collected by researchers• Donated by activists• Images, audio, video, scanned documents, social media,

web server logs

Digital Ephemera Collections

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

NewsScape

• >244,000 hours of TV news archived• Recorded 2005-present• 13 countries, 9 languages• 38 networks• Searchable by captions, on-screen text, named entities

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

Social LocalGlobal

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

Twitter’s Contribution

• Twitter has made full archive of tweets available

• Indexed, searchable

• Not accessible via API• How about deleted tweets?• Real-time capture of

embedded resources?

https://blog.twitter.com/2014/building-a-complete-tweet-index

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

Tool Reuse 1/2

Social Feed Manager (Dan Chudnov, GWU)

http://social-feed-manager.readthedocs.org/

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

Tool Reuse 2/2

twarc (Ed Summers, MITH)

http://inkdroid.org/journal/2014/08/30/a-ferguson-twitter-archive/

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

How SoLoGlo is different

I. Real-time capture and archiving of tweets, referenced URIs, and embedded resources

II. Rapid analysis, real-time opportunities

III. Collection-agnostic linking

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

SoLoGlo Collections 1/4

• Twitter dataset about Egyptian revolution• >400k tweets• 50k unique users• Tweets originated from within 200 miles around Cairo• 25% of tweets contain references to external resources (web

pages, images, videos, etc)

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

• Twitter dataset about Egyptian revolution• >400k tweets• 50k unique users• Tweets originated from within 200 miles around Cairo• 25% of tweets contain references to external resources (web

pages, images, videos, etc)

20% of references are dead

HTTP GET 200 OK

HTTP HEAD 204 No Content

SoLoGlo Collections 1/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

• Twitter dataset about Egyptian revolution• >400k tweets• 50k unique users• Tweets originated from within 200 miles around Cairo• 25% of tweets contain references to external resources (web

pages, images, videos, etc)

20% of references are dead

60% of these are not archived ?

SoLoGlo Collections 1/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

http://wayback.archive-it.org/all/20110203083908/http://yfrog.com/h02gvclj

This one

is!

SoLoGlo Collections 1/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

Need Another Example?

URIs from Ed Summer’s Ferguson dataset

https://edsu.github.io/ferguson-urls/

pink == not archived(Internet Archive)

28%

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

SoLoGlo Collection Linking

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

Rapid analysis

#keystoneXL

SoLoGlo Collections 2/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

• Collection on Charlie Hebdo shooting on 01/07/2015

• >11 million tweets containing #CharlieHebdo or #JeSuisCharlie• > 4.5 million tweets contain embedded media• 4.5 million tweets reference URIs

SoLoGlo Collections 3/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

~ 67k tweets contain lat/long coordinates

SoLoGlo Collections 3/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

• Collection on AirAsia QZ8501 crash on 12/28/2014• 7.3 million tweets containing #AirAsia or #QZ8501• 1.3 million distinct users

SoLoGlo Collections 4/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

• Collection on AirAsia QZ8501 crash on 12/28/2014• 7.3 million tweets containing #AirAsia or #QZ8501• 1.3 million distinct users

SoLoGlo Collections 4/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

SoLoGlo Collections 4/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

#8501qs

SoLoGlo Collections 4/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

#8501qs

Coined by CNN

SoLoGlo Collections 4/4

SoLoGloMartin Klein, Peter Broadwell, Todd Grappone, Sharon Farb

#IIPCGA2015, Stanford, CA, April 28th 2015

SoLoGlo Collections, Encore

http://babylon.library.ucla.edu/archived/iipc2015.html

SoLoGlo:An Archival and Analysis

Service forSocial, Local, and Global News

Martin Klein @mart1nkle1n

Peter Broadwell @peterbroadwell

Todd Grappone @liber8er

Sharon Farb @farbthink