taking society’s pulse in real-time ... - harvard university
TRANSCRIPT
![Page 1: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/1.jpg)
Taking Society’s Pulse in Real Time
Taking Society’s Pulse in Real-Time
TweetMap
Ben Lewis Center for Geographic Analysis,
Harvard
![Page 2: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/2.jpg)
TweetMap is…
• Part of the WorldMap platform
• Beginnings of a big data capability in WorldMap
• Interactive analytics for geo-referenced Twitter data
• Testing ground for new ways to scale WorldMap, in terms of visualization, analytics, and search against its large number of layers.
![Page 3: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/3.jpg)
WorldMap is…
• Designed to lower barriers for researchers who would like to use spatial technology
• Web-based, cloud hosted platform
• Made available to the world
• Open source software with a global developer community
![Page 4: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/4.jpg)
WorldMap Stats
In the past year and a half…
7,844 registered users
8,199 layers added or created
2,054 map collections created
350,000 visitors
![Page 5: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/5.jpg)
WorldMap Growth
![Page 6: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/6.jpg)
Traffic by City
![Page 7: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/7.jpg)
TweetMap
![Page 8: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/8.jpg)
What we mean by real-time
• Immediate Updates: MapD can support real-time database updates (in development).
• Interactive Analysis: Response time for query and visualization against very large datasets is fast enough let users explore them interactively.
![Page 9: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/9.jpg)
Key Features of MapD
• MapD is capable of providing real-time query/visualization access to a billion records
• Visualizations against keywords, geography, and time simultaneously
• Open source and built on cheap hardware
![Page 10: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/10.jpg)
New kind of tool for a new kind of media
![Page 11: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/11.jpg)
5 Million Human Sensors
![Page 12: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/12.jpg)
“snow” (December)
![Page 13: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/13.jpg)
![Page 14: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/14.jpg)
“rain”
![Page 15: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/15.jpg)
(rain in Arabic) ”مطر“
![Page 16: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/16.jpg)
“Dunkin”
![Page 17: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/17.jpg)
Dunkin
![Page 18: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/18.jpg)
“y’all”
![Page 19: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/19.jpg)
sick
![Page 20: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/20.jpg)
enfermo
![Page 21: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/21.jpg)
“newtown”
![Page 22: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/22.jpg)
“earthquake”
![Page 23: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/23.jpg)
Bringing in other layers for analysis
![Page 24: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/24.jpg)
“earthquake”
![Page 25: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/25.jpg)
Under development
• Real time streaming from Twitter to map
• Support for larger datasets
• Trend analysis
• Spatial overlay and regression
• Machine learning
![Page 26: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/26.jpg)
Robert Kirkpatrick of UN’s Global Pulse Initiative:
• Our challenge in international development is we are still using 3 year old data to make 5 year plans.
• There is an opportunity to harness the phenomenon of big data to provide real time feedback on where things ARE NOT working so we can take rapid action to fix it.
UN Global Pulse: Big Data for Development
![Page 27: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/27.jpg)
Many forms of big data
• These may be less accurate than official sources
• But are much faster to get
• And cheaper to collect (free?)
• SO… how to leverage speed to change the outcome?
UN Global Pulse: Big Data for Development
![Page 28: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/28.jpg)
Anonymized cell phone data can provide information on
• Mobility • Diameter of mobility and social network
• Radius of gyration
• Mobility Patterns
• Social • Degree of the social network
• Weight of the contacts, frequency of communication
• Consumption • Number of calls, call duration, SMS/MMS/voice
• Size, frequency, total number of airtime purchases
• Handset Type and Features
UN Global Pulse: Big Data for Development
![Page 29: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/29.jpg)
Which can be used in various ways
• Real-time Events: Earlier detection of anomalies and events allows rapid response to crises.
• Real-time Trends: A current analysis of population activities and dynamics supports more effective program planning and implementation.
• Real-time Evaluation: Real-time measurement of impact-related behavior change allows for rapid, adaptive course corrections in programs.
UN Global Pulse: Big Data for Development
![Page 30: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/30.jpg)
Coming soon…
+
![Page 31: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/31.jpg)
Thank you
![Page 32: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/32.jpg)
Telephone Calls
![Page 33: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/33.jpg)
![Page 34: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/34.jpg)
Airplane Flights
![Page 35: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/35.jpg)
What is meant by real-time
Global pulse definition: Information about a phenomenon available quickly enough to maintain an accurate reflection of its current state, such that effective action may be taken in response.
• Malnutrition -> Months
• Starvation -> Weeks
• Cholera -> Days
• Earthquake -> Hours
![Page 36: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/36.jpg)
Global Pulse
• We need a global, real-time public/private data commons
![Page 37: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/37.jpg)
• Risk of reidentification
• Big data is a human rights issue
• “EU vs Mark Zukerberg
– Privacy is dead and profit is king
– All reuse if forbidden without my explicit permission”
• “Global pulse sees big data as a raw public good but needs to be transformed and safely into a powerful asset for understanding what is happening in the world”
![Page 38: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/38.jpg)
Create sandbox
• Create safe space to experiment with different types of data for specific purposes
• While in parellel develop ways to protect privacy
![Page 39: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/39.jpg)
Agile Global Development (Global Pulse)
![Page 40: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/40.jpg)
World Economic Forum
Understanding the Dynamics of the Data Ecosystem
![Page 41: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/41.jpg)
Big Data is Human Rights Issue (global pulse)
• Never analyze personally identifiable information
• Never analyze confidential data
• Never seek to re-identify individuals
![Page 42: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/42.jpg)
• The way people buy credits – predicts household income
• These patterns can change which can show things
• When poor people are calling you. Reciprocity index increases with income.
![Page 43: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/43.jpg)
Men and Women use their phones differently
![Page 44: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/44.jpg)
MapD: A Massively Parallel Database
![Page 45: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/45.jpg)
• Vendors can sell info to phone carriers that estimate within a couple years how old a person is.
![Page 46: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/46.jpg)
Predicting cholora outbreaks
• flowminder
![Page 47: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/47.jpg)
Harvard healthmap
• Shows that if twitter had been used to monitor cholora would have known about it two weeks earlier.
• Rumi Chunara, Jason Andrews and John Brownstein
• http://www.ajtmh.org/content/86/1/39.abstract
![Page 48: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/48.jpg)
• Flu trends
• Dengue trends
![Page 49: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/49.jpg)
• How to get institutions to use real time data?
• Data is not replacing their staticstics but a compliment
![Page 50: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/50.jpg)
• Most of the universe is invisible
• Most valuable real time data is being gathered by companies as business asset
• Begun engaging around concept of data philanthopy – aggrgated data shared into a public commons, your data is shared with competitors
• Why would they? They might help their own customers to help pulic policy intervene
![Page 51: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/51.jpg)
Pulse labs
• Form partnerships between public sector and business and academy
![Page 52: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/52.jpg)
• Idea of dash board fed by streams of data being run through predictive models
• Socio economic weather stations
![Page 53: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/53.jpg)
• Marshall Mcluen famously declared that “the the medium is the message”.
• I want to tell you about a new kind of media called tweetmap and enlist your help in figuring out what its message is for us.
![Page 54: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/54.jpg)
Niche between desktop and web
Ease
of
Use
WorldMap
web apps
desktop apps
![Page 55: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/55.jpg)
Lowering Barriers to Collaboration
• Open registration
• Open access to data
• Open service protocols (WMS, WFS, ESRI Rest)
• Open data formats (Shape, GeoTIFF, GeoRSS, KML, Json, CSV)
• Open source code (GPL on github)
• Runs on open source operating systems (Linux) and could run on Windows
![Page 56: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/56.jpg)
Working Across Disciplines
http://www.architectmagazine.com/technology/meet-the-geodesigner.aspx
![Page 57: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/57.jpg)
Allows researchers to…
• Organize one’s own (large) mapping datasets online
• Discover other people’s (public) data • Mashup / Overlay one’s data with the data of
others • Create maps with complex symbology • Collaborate by letting several people edit the
same map • Publish data to the world or to just a few
collaborators
![Page 58: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/58.jpg)
Picasa Feeds
![Page 59: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/59.jpg)
Place Name Search (Gazetteers)
![Page 60: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/60.jpg)
Mobile: Feature Creation
![Page 61: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/61.jpg)
Built by the community
![Page 62: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/62.jpg)
TweetMap
![Page 63: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/63.jpg)
Big Data Visual
![Page 64: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/64.jpg)
![Page 65: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/65.jpg)
The best data is distributed
across many organizations
Universities, schools
Scholars outside the academy
Memory institutions
Government agencies
Businesses
![Page 66: Taking Society’s Pulse in Real-Time ... - Harvard University](https://reader033.vdocuments.site/reader033/viewer/2022042918/626aa13c4987c065b1261a8e/html5/thumbnails/66.jpg)
Working Across Disciplines
http://www.architectmagazine.com/technology/meet-the-geodesigner.aspx