Osmose

SoTM 2013

Osmosequality assurance tool

Birmingham 2013-09-06Frdric Rodrigo (c)left 2013 - CC-BY-SA v3.0

http://osmose.openstreetmap.fr

Context

QA toolSince 2008 by Etienne Chov

Maintained by Jocelyn Jaubert and Frdric Rodrigo

Sources available on gitorioushttps://gitorious.org/osmose/

Ecosystem

No duplicate analysis from KeepRight and Osm Inspector

But integrate good idea from minor other tools

Idea from Osmose integrated into JOSM validator by don_vip (building intersection, powerline...)

Support

France mainland and oversea

HOT activated countries

All full French speaking countries and Quebec

Switzerland, Belgium, Luxembourg and Iceland

Hiring for community to support local server for new countries

The Osmose Stack

Frontend

Backends

API

Backends Input

XML or PBF export

Not all supported country are available at Geofabrik

OSM-FR generate also exporthttp://download.openstreetmap.fr/extracts/

Diff available

Need for Osmose

Analyser - SAX

Process one primitive at time

Each analyser is a pluginCheck node, way and relation

Can report errors

Use external dataWikipedia tree species

Running filter on language and country

Analyser - SAX

Format: number, date, postcode, wikipedia...

Naming: abbreviation, too many space, uppercase...

Bag tag: ref in name, typo, deprecated...

Ensure multiple tags coherence from wiki rules:multipolygon, parking lanes...

Inconsistency between tags: toll and fee, opposite cycleway without oneway...

class Structural_UnclosedArea(Plugin): def init(self, logger): self.errors[1100] = { "item": 1100, "level": 3, "tag": ["geom", "fix:imagery"], "desc": T_(u"Unclosed area") }

def way(self, data, tags, nds): if "area" not in tags or tags["area"] == "no": return if nds[0] != nds[-1]: return [(1100, 0, {})]

Analyser - Osmosis

Osmosis, a lossless schema

Multiple objects checking at once

Geospatial analysis

DBLot of indices

create/drop DB: at each Osmose run, lake of disk sppace

Keep DB: Quebec (diff mode)

Analyser Osmosis Non spatial

Done by topology

Roundabout analyserWrong highway level on roundabout

Missing oneway on link highway

Roundabout shortcut

Roundabout crossing

Analyser Osmosis - Geospatial

RoundaboutMissing junction=roundabout

Reverse roundabout

Deadend by highway levelSmall gap in highway level continuity

Power lineIsolated tower

Unconnected powerline

Missing tag power=tower

Predictive position of unmapped power=tower

Analyser Osmosis - Convergent

SoundexPhonetic index

Index all name, way kind + way name

Compare numerous name again similar with low frequency

Automatic typo on tags detectionNot work

Analyser - Merge

OpenData IntegrationIntegration vs Importation

Help contributor to add external data

Row external data set (csv) + Osmosis DB

MappingColumn to tag

Complex mapping

Analyser code is just configurationEasy to add new data set

Analyser - Merge

Conflation by ref

Conflation by distance

DetectOSM object without ref

OSM object not found in OpenData set

OpenData object not found in OSM

OpenData by OSM as sub product

Analyser Merge

French school

Wikipedia

Level crossing

self.csv_encoding = "ISO-8859-15"self.osmTags = { "amenity": "post_office" }self.osmRef = "ref:FR:LaPoste"sself.osmTypes = ["nodes", "ways"]self.sourceSRID = "4326"self.defaultTag = { "amenity": "post_office", "operator": "La Poste", "source": "data.gouv.fr:LaPoste 01/2013" }self.defaultTagMapping = { "ref:FR:LaPoste": "identifiant", "name": "nom", "wheelchair": lambda res: "yes" if self.bool[res["autonome_fauteuil_roulant"]] else "limited" if self.bool[res["pas_ressaut"]] else "no" }self.conflationDistance = 1000

Analyser OSMBin

OSMBin: Osmose proprietary DB formatFlat storage, access in O(1)

Multipolygon / boundary closed fast checking

Analyser External tools

OpenStreetBugClose OSB entries when close it on Osmose

Need support OSM Notes

Configuration

Configuration by extract (country and smaller part)

List of analysersLanguage: mainly french specific

Local way of mapping

Source available: Bing hires, French cadastre...

OptionsProjection

Driving side

Language

Boundary clipping

Extract have buffer on boundaryDuplicate error

False error (e.g. unconnected)

Clip on exact boundary polygon after analyse

About Time

Full run every 2 days

Take 80% of availible CPU

Diff mode

Speedup by running analyser only on change

Sax

Osmosischanged tables : edited nodes, ways, relations and transitive dependencies

Keep DB between runs

Not all can be run in diff modeError based on other OSM data (clustering, connection...)

Full run every month to keep safe

The Osmose Stack

Frontend

Backends

API

Errors

Backends send errors to centralized Frontend

Errors report are by Analyser and Country (aka source)

Error file replace all errors of his kind on frontendExcept on diff mode

Error

Source (country, analyser)

IdentityClass

Subclass

MetainfoLevel: 1, 2, 3 as severity

Item number: display category and subcategory

TagsOther classificatio:, by nature of OSM object, highway, waterway, name...

What is require for fix: chair, imagery, survey

Error fixes

Analyser can propose some error fixes

Add, delete or modify many tag at once

Create a new node with tags

The Osmose Stack

Frontend

Backends

API

Frontend - Map

SelectorLevels of severity

Items

Other options available on URL (see API)

MarkersStatus: open, close, false-positive

JOSM remote

JOSM fix

Wiki help

LayersHeatmap

Debug map

Stats

Errors listing

History graph

By userRSS

Update Matrix

France - Level 1

Add new analyser

Natural fix over the time

Heavy fix

Haiti All errors

Correct error with locals

France Cadastre building

overlap

New gapdetection

Heavy fdix

French OpenData School

Bugs

OpenData Update

The Osmose Stack

Frontend

Backends

API

API

External fixing tools

Stats

JOSMconfidential plugin

Quality Assurance Tools script

API doc on WikiURL parameters apply any where in the frontend

Hardware

Backendsosm3.openstreetmap.fr HOT and Quebec

osm5.openstreetmap.fr France

osm6.openstreetmap.fr France

Iceland server: Iceland

Frontendosm103.openstreetmap.fr osmose.openstreetmap.fr

Conclusion

osmose.openstreetmap.fr o.osmfr.org

https://gitorious.org/osmose/

We need youTranslation: frontend UI and backend errors

Run your own Osmose local backend server

Errors correction!

Long list of analysers request on french Trac

Localise language or country specific analyserMain st Main street

le-de-France - Total time 3h17Colonne E

Download2

Osmosis Import50

Indexing35

SAX Analyser43

Osmosis Analyser67