relational approach with lidar data with postgresql
TRANSCRIPT
![Page 1: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/1.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
Manage LiDAR
data with
PostgreSQL
is it possible?
www.2ndquadrant.com
![Page 2: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/2.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
Is the relational approach valid for LiDAR?Is the relational approach valid for LiDAR?
• The relational approach to the data:– data organized in tuples– tuples are part of tables– tables are related to each other through constraints (PK, FK, etc.)
• If the number of tuples grows:– indexes allow to reduce the complexity of a search to ~O(logN)...– ...but they must be contained in RAM! – OTHERWISE: the relational approach start to fail...
![Page 3: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/3.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
PostgreSQL & LiDAR dataPostgreSQL & LiDAR data• PostgreSQL is a relational DBMS with an extension for LiDAR data:
pg_pointcloud
• Part of the OpenGeo suite, completely compatible with PostGIS– two new datatype: pcpoint, pcpatch (compressed set of points)
• N points (with all attributes from the survey) → 1 patch → 1 record
– compatible with PDAL drivers to import data directly from .las 32b 32b 32b 16b
https://github.com/pgpointcloud/pointcloud
![Page 4: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/4.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
Relational approach to LiDAR data with PGRelational approach to LiDAR data with PG
• GiST indexing in PostGIS• GiST performances:
– storage: 1TB RAID1, RAM 16GB, 8 CPU @3.3GHz,PostgreSQL9.3
– index size ~O(table size)– Index was used:
• up to ~300M points in bbox inclusion searches• up to ~10M points in kNN searches
LiDAR size: ~O(109÷1011) → few % can be properly indexed!
http://www.slideshare.net/GiuseppeBroccolo/gbroccolo-foss4-geugeodbindex
![Page 5: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/5.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
A new index in PG: Block Range INdexingA new index in PG: Block Range INdexing 8kB8kB8kB8kB
data must be physically sorted on disk!
index node → row
index node → block
less specific than GiST!
Really small!
8kB8kB8kB8kB8kB8kB8kB8kB
8kB8kB8kB8kB
(S. Riggs, A. Herrera)
![Page 6: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/6.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
BRIN support for PostGIS datatypesBRIN support for PostGIS datatypes
8kB8kB8kB8kB
G. Broccolo, J. Rouhaud, R. Dunklau
3rd May, PGDay @ FOSS4G.NANO kNN SUPPORT!!
![Page 7: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/7.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
The LiDAR dataset: the ahn2 projectThe LiDAR dataset: the ahn2 project
• 3D point cloud, coverage: almost the whole Netherlands– EPSG: 28992, ~8 points/m2
• 1.6TB, ~250G points in ~560M patches (compression: ~10x)– PDAL driver – filter.chipper
• available RAM: 16GB
• the point structure: X Y Z scan LAS time RGB chipper
32b 32b 32b 40b 16b 64b 48b 32b
the “indexed” part (can be converted to PostGIS datatype)
(thanks to Tom Van Tilburg)
![Page 8: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/8.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
Typical searches on ahn2 - Typical searches on ahn2 - x3d_viewerx3d_viewer
• Intersection with a polygon (PostGIS)– the part that lasts longer – need indexes here!
• Patch “explosion” + NN sorting (pg_PointCloud+PostGIS)
• constrained Delaunay Triangulation (SFCGAL)
![Page 9: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/9.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
All just in the DB...All just in the DB...
WITH patches AS (
SELECT patches FROM ahn2
WHERE patches && ST_GeomFromText(‘POLYGON(...)’)
), points AS (
SELECT ST_Explode(patches) AS points
FROM patches
), sorted_points AS (
SELECT points,
ST_DumpPoints(ST_GeomFromText(‘POLYGON(...)’))).geom AS poly_pt
FROM points ORDER BY points <#> poly_pt LIMIT 1;
), sel AS (
SELECT points FROM sorted_points
WHERE points && ST_GeomFromText(‘POLYGON(...)’)
)
SELECT ST_Dump(ST_Triangulate2DZ(ST_Collect(points))) FROM sel;
![Page 10: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/10.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
...and with just one query!...and with just one query!
WITH patches AS (
SELECT patches FROM ahn2
WHERE patches && ST_GeomFromText(‘POLYGON(...)’)
), points AS (
SELECT ST_Explode(patches) AS points
FROM patches
), sorted_points AS (
SELECT points,
ST_DumpPoints(ST_GeomFromText(‘POLYGON(...)’))).geom AS poly_pt
FROM points ORDER BY points <#> poly_pt LIMIT 1;
), sel AS (
SELECT points FROM sorted_points
WHERE points && ST_GeomFromText(‘POLYGON(...)’)
)
SELECT ST_Dump(ST_Triangulate2DZ(ST_Collect(points))) FROM sel;
![Page 11: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/11.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
patches && polygons - GiST performancepatches && polygons - GiST performance
GiST 2 d
GiST 26GB
index buildingindex size
polygonsize timing
~O(m) ~40ms
~O(km) ~50s
~O(10km) hours
searches based on GiST
index not contained inRAM anymore
(~5G points → ~3%)
![Page 12: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/12.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
patches && polygons - BRIN performancepatches && polygons - BRIN performance
BRIN 4 h BRIN 15MB
index building index size
polygonsize timing
~O(m) ~150s
searches based on BRIN
![Page 13: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/13.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
patches && polygons - BRIN performancepatches && polygons - BRIN performance
BRIN 4 h BRIN 15MB
index building index size
polygonsize timing
~O(m) ~150s
searches based on BRIN
How data was inserted...
LASLASLASLASLASLAS
chipper
chipper
chipper
PDAL driver(6 parallel processes)
ahn2
![Page 14: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/14.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
patches && polygons - BRIN performancepatches && polygons - BRIN performance
CREATE INDEX patch_geohash ON ahn2_subset
USING btree (ST_GeoHash(ST_Transform(Geometry(patch), 4326), 20));
CLUSTER ahn2_subset USING patch_geohash;
~150s → ~800ms [radius ~O(m)]
• x20 slower than GiST searches• x200 faster than Seq searches
– (x1000 faster in ~O(100m) searches)
(http://geohash.org/)
![Page 15: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/15.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
is the drop in performance acceptable?is the drop in performance acceptable?
BRIN searches = x20 GiST searches
= x10÷x100 GiST searches
= x10÷x100 GiST searches
patch explosion+
NN sorting
constrainedtriangulation
![Page 16: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/16.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
is the drop in performance acceptable?is the drop in performance acceptable?
BRIN searches = x20 GiST searches
= x10÷x100 GiST searches
= x10÷x100 GiST searches
patch explosion+
NN sorting
constrainedtriangulation
![Page 17: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/17.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
ConclusionsConclusions
Can be the relational approach to LiDAR data valid withPostgreSQL? – Yes!
• GiST indexes are fast, but can manage just a real small portionof the dataset
• BRINs are quite slower, but generally do not represent a realbottleneck
– Make sure that data has the same sequentiality of .las
![Page 18: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/18.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
~$ whoami~$ whoami
Giuseppe Broccolo, PhDGiuseppe Broccolo, PhD PostgreSQL & PostGIS consultantPostgreSQL & PostGIS consultant
@giubro gbroccolo7
gbroccolo gemini__81
![Page 19: Relational approach with LiDAR data with PostgreSQL](https://reader031.vdocuments.site/reader031/viewer/2022013109/58ae45aa1a28abad338b5ab7/html5/thumbnails/19.jpg)
2ndQuadrant Italia Giuseppe Broccolo – [email protected]
FOSS4G.NA 2016Raleigh, NC4th May 2016
Creative Commons license
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License
http://creativecommons.org/licenses/by-nc-sa/2.5/it/© 2016 2ndQuadrant Italia – http://www.2ndquadrant.it