geoiq stack - where 2012
DESCRIPTION
low image-resolution version of my talks from Where 2012 and FOSS4G-North AmericaTRANSCRIPT
Collaborative Open Access to Geospatial Analysis
Andrew TurnerChief Technology Officer & Neogeographer
Hurricanes Katrina and RitaGeospatial Analysis with Long Production Cycles
flickr: gregorio
in 2005 we were active in the response to Hurricanes Katrina and Rita. Back then the request came to perform a flood analysis. Analysts worked in DC for a few days and sent their results in powerpoint slides to the operations center. They then asked “where are our shelters and response centers?” Churn away a few days and send that back down. By then they were knee deep in water. How do you change the paradigm to give the tools to access data and ask questions into the hands of the people that need it. Go from question to answer in under 5 minutes.
Crisis Reporting
Realtime emergent information sources
at the same time we saw the emergence of new forms of citizen communication and reporting. No voice and no data meant people sitting on rooftops would text family in Michigan to then call the Red Cross back in Louisiana. The typical organizations had little capability to quickly adapt to these new streams and incorporate it into their response.
Our Mission: On-demand realtime interactive maps
in the hands of everyone, exploration and collaboration
Today this is dramatically different. We’ve spent the past 6 years building tools and capability for anyone to quickly and effectively access their data and devise actionable information. By leveraging the web and modern, emergent technology and an open-architecture it’s now possible to provide advanced tools to everyone - opening the conversation to easily enable collaboration when it’s needed most.
Collaboration
flickr: chadcat
Access
Visualize
Analyze
we focus on three overall capabilitiesData: networked, unbounded, standards-based, community driven, open data.Visualization: We need ways to get through this data. What do I have? What is the shape and idea of it? How is it changing? What can I do with it?Analysis: In the end, I want to make decisions. To do that, I need to ask questions. I need tools to help me easily ask questions and find answers without having to worry about the “how” of the question asking. Just let me explore.
Cross-DomainShare KnowledgeMake Smarter Decisions... Collaboratively
Organizations
Research
MediaDevelopers
Enterprise
Citizens
a few GeoIQ features...• Data upload• Data Parsing• Web Feeds• Dataset Aggregation• Geocoding• Geojoin• Data Permissions• Group Management• Data Cleanup• Metadata management• Tagging• Data export• Dataset caching• External Relational Database integration• User Authentication• LDAP Integration• Thematic Visualization• Spatial, Tabular, Statistical Analysis• User-defined analysis
• Stream updating• Mobile web interface• Native mobile applications• Drawing and annotation• Geographic Maps• User defined basemaps• Charts (Pie, Line, etc.)• REST and Javascript API Integration• Facebook integration• OGC Web Services• PNG and PDF Map Export• Multisite Catalog Federation• White Label Customization• Featured Maps• External Analysis Integration• Stats and Tracking• Graceful Map Degradation
“GIS” in all but name
Easy explorationQuick data publication
Data management and search
Metadata extraction, statisticsOne-click downloadsBaked into the web: durable URL, link to sources,
Advanced analysis through easy, online interfaces
http://geocommons.com
15m+ maps served
100k+ unique datasetsbillions of features.
4m+ downloads
1m+ geographic attributes
Start now by joining our GeoCommons community of more than 25k GeoIQ users. You can readily find, visualize, and very soon analyze the public repository of 1m+ geographic attributes, billions of features.
Collaborative Data
= 499,999,500,000
Possible Comparative Analysis Combinations?
Collaborative Data
well... close enough
Mapping Health http://geocommons.com/maps/77282
Mapping Traffic + Caffeine http://geocommons.com/maps/82402
Mapping Zombie Outbreaks
Mapping Socially
An Open Stack
flickr: alfr3do
Streaming Services
Streaming Analysis
GeoiQ Social
REST and Javascript API
GeoiQ Explorer GeoiQ Mobile
Tabular
Spatial
Image
Custom Analysis Workers
Spatial Data
ETL
Spatial Data
Managem
ent
User
Managem
ent
Distributed Object Store GeoiQ Connect
MySQ
L
PostgreSQL
Oracle
Mongo
HB
ase
Geospatial
Search Index
Hadoop
Distributed Job Queues
External Analysis
HDFS
Federated Data Network
OpenStreetMap
Map R
endering Engine
Open-Source Stack
Streaming Services
Streaming Analysis
GeoiQ Social
REST and Javascript API
GeoiQ Explorer GeoiQ Mobile
Tabular
Spatial
Image
Custom Analysis Workers
Spatial Data
ETL
Spatial Data
Managem
ent
User
Managem
ent
Distributed Object Store GeoiQ Connect
MySQ
L
PostgreSQL
Oracle
Mongo
HB
ase
Geospatial
Search Index
Hadoop
Distributed Job Queues
External Analysis
HDFS
Federated Data Network
OpenStreetMap
Map R
endering Engine
ModestMaps))&)Polymaps)
jQuery'&''Backbone.js'
Ac#onScript+&+Appcelerator+
AMQP%&%Redis%
WebSockets*Socket.io*
Ruby%on%Rails%
Lucene&&&Solr& Mapnik'
Ruby%&%OSGeo%
HBase&
PostgreSQL+
Node.js(&(MongoDB(
Open-Source Stack
Open-ArchitectureOpen-Standards
•REST, GeoJSON •OpenSearch-Geo•GeoRSS, Atom, KML•OAuth, LDAP•OGC WFS, WMS, CSW
•Usable Interface•Web Services•Download button
flickr: dolinski
Open-Data
•OpenStreetMap•TIGER/Line•GeoNames•World Bank•UN•IADB•US Government •100k+ open-source
datasets
flickr: ajturner
GeoCommons GeoCoder
GeoCommons GeoCoder
We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
issuemap.orggithub.com/geoiq/issuemap
We worked with the FCC to make this even easier. Enable Staffers and anyone to go from a Spreadsheet to a Map as quickly as possible. Then download or publish it for use in meeting, publications, and social networks.
Open Visualization Modest mapsPolymaps
Full user choice for quickly growing number of basemaps and providers. Each vary by cartographic style, data quality, language.
MapnikOpenStreetMapTileStache
developer.geoiq.com/tools/acetate/
We wanted to provide a basemap uniquely capable of visually attractive and understandable quantitative thematic visualizations. We also wanted to bring back a traditional concept of cartography by separating the layers to “Sandwich” it with the data. Together users can see the data and also the context of roads, places and other geographic data. Our UI makes this easy without having to think about it.
MapnikOpenStreetMapTileStache
developer.geoiq.com/tools/acetate/terrain
areas
roads
points
labels
We wanted to provide a basemap uniquely capable of visually attractive and understandable quantitative thematic visualizations. We also wanted to bring back a traditional concept of cartography by separating the layers to “Sandwich” it with the data. Together users can see the data and also the context of roads, places and other geographic data. Our UI makes this easy without having to think about it.
Spatial Operations
Buffer, Intersect, AggregationCorrelation, Prediction, Custom
Tools: geos, JTS, PostGIS
We want to go beyond simple visualization and allow people to ask deep questions of their data. So we built a number of common tabular, statistical, and geospatial analysis modules that are easy to use with your data.
Spa$al&Aggrega$on:+Group+data+by+arbitrary+geographic+boundaries
(Proprietary+and+confiden0al)
Spa$al&Correla$on:+Rela0onship+between+mul0ple+datasets+by+geography
(Proprietary+and+confiden0al)
Spa$al&Buffer&and&Distance&Filter:+Radius+buffer+from+a+point,+line+or+polygon
(Proprietary+and+confiden0al)
Intersec$on:+Overlap+between+two+polygonal+areas,+lines+or+points
Use&Cases:• Locate+that+a+social+media+men0ons+happened+within+a+geoCfence
• Track+mobile+traffic+entering+a+zip+code
(Proprietary+and+confiden0al)
User;Defined&Analysis:+Custom+equa0ons+via+intui0ve+wizard+interface
(Proprietary+and+confiden0al)
Traceable&Analyses:+Analysis+and+Filter+versions+can+be+tracked+back
All data and analyses are tracked and linked to their history and sources. Think “Github for Data”.
Realtime Social Media Indicatorshttp://geocommons.com/maps/51058
And we’re working on more than just traditional, static, analysis. We can analyze emergent and realtime streams of data.
Deploy to the Edgeflickr: oddboy
The “cloud” offers amazing capability to publish and share across the web. But many users need to work in remote and disconnected locations. Because we use open-source stack and our own technology we can easily deploy to any online or offline systems.
We’ve deployed servers to work offline in Jalalabad, Afghanistan as part of the #beer4data program that enables field collaboration of data sharing. The full offline stack can geocode, visualize and analyze data locally and optionally push the data up to the public web when there is connectivity.
Deploy to the Edges
GeoIQ Enterprise
GeoCommons
Enterprise Databases
Business Operations
GeoIQ Cloud Web!
Intranet!
Open-Source Deployment
Workers Data Store
Deployed as a Single VM
App Servers
Data Stores
Clustered Cloud Deployment
We use the same installers and virtual machine to on-demand deploy a new front-end server, or back-end analysis worker or data server.
Publish(to(mobile(
Make(a(map(in(GeoIQ(
Layer(Control( Filter((by(Loca;on( Field(Annota;ons(
Track(Field(Annota;ons((
Real(;me(upda;ng(
SelfCService(Mobile(App(Crea;on(in(Minutes(
And beyond scaling horizontally, we can push to the real edge of mobile devices. Users can create and publish a map in minutes that enables mobile access to maps and even realtime mobile collaboration.
Distributed Operations Search and Rescuehttps://vimeo.com/30435757
Combined together, these tools allow for on-demand access to emergent information, realtime streams, analysis, and field reporting.
Organizations
Research
MediaDevelopers
Enterprise
Citizens
By providing a common open platform for collaboration, members of many communities can contribute data and insight to reach common consensus.
Development Aid http://maps.worldbank.org
The World Bank is sharing the locations and impact of their projects throughout the world. The data is explorable and downloadable.
IADB MapAmericas http://www.iadb.org/mapamericas
The Inter-American Development Bank has launched a similar initiative linking project funding with on the ground outputs, photos, videos, and stories.
Climate Science WWF Climascope
The information that has traditionally been limited to expert only access is now in the hands of anyone to ask detailed important questions of quality scientific data. Citizens can ask about the impact of global phenomenon on their local environments.
Expert & Citizen Sourced Science Data
http://marine.tokelau.grida.no/
Organizations such as the United Nations Environmental Programme Grid-Arendal are gathering more scientific data and opening the tools for visualization and analysis to the world.
Social Governance
Beyond traditional science, government and organizations can ask relevant questions on urban planning and society by combining social media information. For example, investigating the crowding and ‘popularity’ of metro transit locations compared with location checkins.
Join the Communityhttp://geocommons.com
GeoCommons is the public repository to explore this data - or find out how to use GeoIQ within your organization to get better insight and information.
Collaborative Open Access to Geospatial Analysis
Andrew TurnerChief Technology Officer & Neogeographer
Collaborative Open Access to Geospatial Analysis
Andrew TurnerChief Technology Officer & Neogeographer
slideshare.net/ajturner github.com/geoiqgithub.com/geocommons