lost in the clouds

Download Lost In The Clouds

Post on 26-Dec-2014




4 download

Embed Size (px)




  • 1. Lost in the Clouds An example of a schemaless database application in the cloud SlipstreamUSA :: March 25, 2009
  • 2. The Project OpenStreetMap A free map of the planet Free geospatial data Licensed under CC-BY-SA Created from Crowd-sourced contributions 90,000 registered users 1,000+ contributors per day Used for: Wikipedia Flikr OpenCycleMap.org OpenPisteMap.org YourNavigation.org Mobile devices Constantin Litvak
  • 3. Coverage CC-BY-SA 2.0 OpenStreetMap
  • 4. The OpenStreetMap Database Server In the cloud MySQL Ways: 24,956,171 Nodes: 305,956,583 Tags: 3.6 billion highway=motorway name=Broadway foo=bar CC-BY-SA 2.0 Image Ojw Data OpenStreetMap
  • 5. Interfaces API For updates (1,000 users per day) Queries of up to 0.25 degrees Weekly Planet Dump 100Gbyte XML file Changes Feed of changes Daily / Hourly / Minutely One file per minute (~30Kb compressed) CC-BY-SA 2.0 Image Ojw Data OpenStreetMap
  • 6. Problem Statement Problem Want to be able to query the data in bigger chunks and to do it more selectively Solution Cloud based schemaless database with simple to use query interface and lots of indexes CC-BY-SA 2.0 Image Maning Sambale Data OpenStreetMap
  • 7. XAPI Service GT.M or Cache 250Gbyte database Nodes: 30Gb Node tags: 115Gb Node indexes: 70Gb Ways: 21Gb Way tags: 10Gb Way indexes: 18Gb Apache + serverLink Licensed as AGPL CC-BY-SA 2.0 Image Maning Sambale Data OpenStreetMap
  • 8. XAPI Service In the cloud Queries of up to 100 degrees Query by tags and tag values Scaleable REST style interface http://xapi.openstreetmap.org/api/0.6/* [tourism=hotel] [bbox=-75.7,40.0,-75.4,40.2] Import planet dump Import minute diff CC-BY-SA 2.0 Image Ojw Data OpenStreetMap
  • 9. XAPI Service Geospatial Index quadstrings adaabcdcabaadab
  • 10. XAPI Service ^way(27016525)=quot;adaabcdcabaadabquot; ^way(27016525,1)=296138118 ^way(27016525,2)=296138119 ^way(27016525,3)=296138120 ^way(27016525,4)=296138121 ^way(27016525,5)=296138118 ^waytag(27016525,quot;addr:housenumberquot;)=2 ^waytag(27016525,quot;buildingquot;)=quot;yesquot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27016525)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028298)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028299)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028326)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27028327)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035972)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035973)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035974)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035975)=quot;quot; ^wayx(quot;buildingquot;,quot;*quot;,quot;adaabcdcabaadabquot;,27035984)=quot;
  • 11. Deployment Currently 5 instances: London Paris San Diego Zurich Amsterdam (Wikimedia-de) More servers needed: Internet connectivity 300Gb disk storage CC-BY-SA 2.0 Image Nikolaj Data OpenStreetMap
  • 12. Wikipedia: Query to Map {{osm-query |way=name=Arizona Canal}} CC-BY-SA 2.0 OpenStreetMap
  • 13. Internals Debian Apache web-server REST URI represents a resource Xpath flavoring serverLink WebLink emulator GT.M or Cache POM AGPL
  • 14. XAPI Future Load Sharing / balancing More Xpath-like queries JSON output Generic indexing service More servers
  • 15. Thank you Thank you Questions? CC-BY-SA 2.0 Image ITO! Data OpenStreetMap