implementation of strategies using open source software in ... · pdf fileautomatic...

Implementation of strategiesusing open source softwarein a process chain forautomatic delineation ofINSPIRE elevation datasetsalong watercourses

Christian Schwartzecodematix GmbH

INSPIRE Conference 2017 | Friday, September 8th

InformationstechnikzentrumBund* (ITZBund, Ilmenau,Germany)

▪ IT service provider for mainly federalagencies

▪ hosting and operation of ITadministrative processes, SWdevelopment and support

▪ initiator of this project (and otherINSPIRE-related tasks)

* German Federal Centre for Information Technology

The “Who” question…codematix GmbH (Jena,Germany)

▪ dynamic company specialized oncustomer tailored SW develpoment,web services and training

▪ IT services: system design,implementation and optimization ofGIS enterprise solutions; geospatialdata processing, infrastructures andsupport

3D-Datenarchiv▪ BfG: Bundesanstalt für Gewässerkunde *

▪ intranet application: digital preservationof 3D topography data▪ central archiving, searching and

provision▪ terrain and riverbed data along federal

waterways and coastal regions▪ data types: profiles, breaklines,

measurement/bearing data, grid points,TINs… = INSPIRE data types ?

* the German Federal Institute of Hydrology

Preliminary review▪ identification of INSPIRE relevant data in

3D-Datenarchiv application▪ (open source) software approaches and

geodata transformation concepts to meetrequirements of INSPIRE elevation theme(EL, Annex II)▪ methods to provide processed data in form

of view/download services

Basic situation▪ identified representation types

in INSPIRE:ElevationGridCoverage andContourLine▪ consolidated and separately

provided data basis: ASCII filescontaining high resolutionelevation data▪ hundreds of xyz-formatted text

files (~220GB)▪ for a total of 19 federal

waterway sections▪ preprocessing needed (INSPIRE

encoding requirements)

(1) Configurable,automated andperformant workflow

E.g. using Python Stetl or QGISProcessing Framework

Main goals of ASCII elevation datatransformation

(3) Further processinginto cartographicproducts or services

E.g. DEM, contour lines orhillshade maps

(2) Output morepopular GIS compatiberaster formats

Enables a more flexible datausage, e.g. with GeoTiffs

Geospatial data processing workflows ( 1 )

Stetl ?▪ ETL means: Extract – Transform – Load▪ Stetl means: Streaming ETL▪ open source framework focussing on

transformation of (complex) mass data(e.g. rich GML) using processing chainsbased on data packages (streaming)

http://stetl.readthedocs.io


Stetl !▪ no graphical model builder, but…▪ rather scripting models with Python than graphical drag & drop

of single components▪ uses existing ETL tools, e.g. OGR, XSLT▪ speed-optimized

SOURCE INPUTINPUT FILTERFILTER FILTERFILTER OUTPUTOUTPUT TARGET…

Stetl Python modules

e.g.:stetl.fileinputstetl.ogrinput…stetl.myinput

e.g.:stetl.xsltfilterstetl.gmlfeatureextractor…stetl.myfilter

e.g.:stetl.dboutputstetl.ogroutput…stetl.myoutput

e.g.: GML e.g.: Database

Elevationmass data

DEM /Shapefile /GML …

*. xyz*. xyz*. xyz


Stetl used for ASCII elevation data▪ inputs: xyz data files with columns separated by x

whitespaces (Tripel format)▪ outputs: digital elevation models (GeoTiff),

hillshades (GeoTiff), contour lines(Shapefile/INSPIRE GML)▪ …and in between: Stetl packages , here: text

chunks

*. xyz src.utils.DgmwReader

src.utils.DgmwReader

src.utils.RealCsvConverter

src.utils.RealCsvConverter

src.utils.Rasterizersrc.utils.

Rasterizer… …Stetl packages:ASCII xyz

Stetl packages:Real CSV

*. xyz*. xyz*. xyz*. tif

JOB 1

src.utils.MapCreator

src.utils.MapCreator

JOB 2

DEM

Hillshade

ContourLines


Stetl process configuration▪ define overall process chain(s): input,

filter/processing and output moduleschains = …

▪ instantiate (your) input/outputcomponents or filterclass = src.utils.Rasterizer

▪ configure (your) class, e.g. Rasterizerclean = 1epsg = {epsg_in}


A Rasterizer with Stetl▪ takes current CSV chunk from data

stream▪ uses GDAL/OGR on each CSV

chunk, e.g:

▪ outputs individual tilesin GeoTiff formatas input for „JOB 2“


Stetl: Results▪ created datasets/files as a basis

for INSPIRE WMS (deegreesoftware)http://atlas.wsv.bund.de/inspire/el/wms

▪ data download via INSPIRE ATOMfeedhttp://atlas.wsv.bund.de/inspire/atom/hoehen.xml


QGIS ProcessingFramework▪ includes advanced analysis algorithms,

e.g. native QGIS functions, GRASS-GIS/SAGA modules, R, GDAL/OGR▪ goal: optimizing/improving contourline

visualization in view service (WMS)

1. threshold-based removal of very shortcontour lines (small „rings“)

2. additional vector attributes, e.g. for linelabels and line classification

3. visually attractive lines by means ofsmoothing algorithms (e.g. „Snake“)


QGIS contour line model

uses e.g. followingalgorithms (=modelcomponents):• GRASS-GIS for

line removal andsmoothing• Python Field

calculators• own Python

scripts

ResultingINSPIREWMS for EL

Thank you!

implementation of strategies using open source software in ... · pdf fileautomatic...

Documents