Data Parallel Quadtree Indexing and Spatial Query Processing of Complex Polygon Data on GPUs

Jianting Zhang1,2 Simin You2, Le Gruenwald3

1 Depart of Computer Science, CUNY City College (CCNY)2 Department of Computer Science, CUNY Graduate Center3 School of Computer Science, the University of Oklahoma

CISE/IIS Medium Collaborative Research Grants 1302423/1302439: “Spatial Data and Trajectory

Data Management on GPUs”

Outline•Introduction & Background

•Application: Large-Scale Biodiversity Data Management

•Data Parallel Designs and Implementations

• Polygon Decomposition

• Quadtree Construction

• Spatial Query Processing

•Experiments

•Summary and Future Work

Parallel Computing – Hardware

A

B

C

Thread Block

CPU Host (CMP)

Core

Local Cache

Shared Cache

DRAM

Disk SSD

GPU

SIMD

PCI-E

Ring Bus

Local Cache

Core ... Core

Core Core

GDRAM GDRAM

Core Core Core... ...

MIC

PCI-E

T0 T1

T2 T3

4-Threads

In-Order

16 Intel Sandy Bridge CPU cores+ 128GB RAM + 8TB disk + GTX TITAN + Xeon Phi 3120A ~ $9994

ASCI Red: 1997 First 1 Teraflops (sustained) system with 9298 Intel Pentium II Xeon processors (in 72 Cabinets)

•Feb. 2013 •7.1 billion transistors (551mm²) •2,688 processors •4.5 TFLOPS SP and 1.3 TFLOPS DP•Max bandwidth 288.4 GB/s•PCI-E peripheral device•250 W (17.98 GFLOPS/W -SP)• Suggested retail price: $999

What can we do today using a device that is more powerful than ASCI Red 17 years ago?

GeoTECI@CCNYCCNY Computer Science LAN

Microway

Dual 8-core128GB memoryNvidia GTX TitanIntel Xeon Phi 3120A8 TB storage

DIY

*2

SGI Octane III

Dual Quadcore48GB memoryNvidia C2050*28 TB storage

Dual-core8GB memoryNvidia GTX Titan3 TB storage

Dell T5400

Dual Quadcore16GB memoryNvidia Quadro 6000 1.5 TB storage

Lenovo T400s

Dell T7500

Dual 6-core24 GB memoryNvidia Quadro 6000

Dell T7500

Dual 6-core24 GB memoryNvidia GTX 480

Dual Quadcore16GB memoryNvidia FX3700*2

Dell T5400

DIY

Quadcore (Haswell) 16 GB memoryAMD/ATI 7970

Quadcore8 GB memoryNvidia Quadro 5000m

HP 8740w

HP 8740w

CUNY HPCC

KVM

“Brawny” GPU cluster

“Wimmy” GPU cluster

Web Server/Linux App Server Windows

App Server

...building a highly-configurable experimental computing environment for innovative BigData technologies…

Computer Architecture

Spatial Data Management

How to fill the big gap

effectively?David Wentzlaff, “Computer Architecture”, Princeton University Course on Coursea

Parallel Computing– Languages & Libraries

http://www.macs.hw.ac.uk/cs/techreps/docs/files/HW-MACS-TR-0103.pdf

Thrust

Bolt CUDPP

boost

GNU Parallel Mode

Source: http://parallelbook.com/sites/parallelbook.com/files/SC11_20111113_Intel_McCool_Robison_Reinders.pptx

Data Parallelisms Parallel Primitives Parallel libraries Parallel hardware

Outline•Introduction & Background

•Application: Large-Scale Biodiversity Data Management

•Data Parallel Designs and Implementations

• Polygon Decomposition

• Quadtree Construction

• Spatial Query Processing

•Experiments

•Conclusions and Future Work

Managing Large-Scale Biodiversity Data

SELECT aoi_id, sp_id, sum (ST_area (inter_geom)) FROM( SELECT aoi_id, sp_id, ST_Intersection (sp_geom, qw_geom)

AS inter_geom FROM SP_TB, QW_TB WHERE ST_Intersects (sp_geometry, qw_geom))GROUP BY aoi_id, sp_idHAVING sum(ST_area(inter_geom)) >T;

http://geoteci.engr.ccny.cuny.edu/birds30s/BirdsQuest.html

Indexing “Complex” Polygonshttp://en.wikipedia.org/wiki/Simple_polygon http://en.wikipedia.org/wiki/Simple_Features

Problems in indexing MBRs: •Inexpensive yet inaccurate approximation for complex polygons•Low pruning power when polygons are highly overlapped

“Complex” Polygons: •Polygons with multiple rings (with holes)•Highly overlapped

http://xlinux.nist.gov/dads/HTML/linearquadtr.html

Indexing “Complex” Polygons

(Zhang et al 2009)

(Zhang 2012)

Hours of runtimes on birds range maps by extending GDAL/OGR (serial)

Fang et al 2008. Spatial indexing in Microsoft SQL Server 2008. (SIGMOD’08)

Using B-Tree to index quadrants, but it is unclear how the quadrants are derived from polygons

Outline•Introduction & Background

•Application: Large-Scale Biodiversity Data Management

•Data Parallel Designs and Implementations

• Polygon Decomposition

• Quadtree Construction

• Spatial Query Processing

•Experiments

•Conclusions and Future Work

Parallel Quadtree Construction

Parallel Query Processing

DFS BFS

Parallel Polygon Decomposition

(0,2)

(1,3)

(1) All operations are data parallel at the quadrant level; (2) Quadrants may be at different levels and come from same or different polygons; (3) Each GPU thread process a quadrant; (4) Accesses to GPU memory can be coalesced for neighboring quadrants from

same polygons

Observations:

•Quadtree Construction

•Spatial Query Processing

Experiment SetupSpecies distribution data•4062 bird species in the West Hemisphere•708,509 polygons •77,699,991 vertices

Polygon Group

num of vertices range

total num of polygons

total num of points

1 10-100 497,559 11,961,3892 100-1,000 33,374 8,652,2783 1,000-10,000 6,719 20,436,9314 10,000-100,000 1,213 33,336,083

• Dual 8-core Sandy Bridge CPU (2.60G) • 128GB memory • Nvidia GTX Titan (6GB, 2688 cores) • Intel Xeon Phi 3120A (6GB, 57 cores)• 8 TB storage

• CentOS 6.4 with GCC 4.7.2, TBB 4.2, ISPC 1.6, CUDA 5.5

• All vector initialization times in Thrust on GPUs are counted (new versions of Thrust allow uninitialized device vectors)

• Performance can vary among CUDA SDKs

Runtimes (Polygon Decomposition)

G1 (10-100) G2 (100-1000)

G3(1000-10000)

G4(10000-100000)

Comparisons with PixelBox* (Polygon Decomposition)

 (milliseconds) 10-100 100-1000 1000-10000 10000-100000Proposed 451 2303 46215 193714PixelBox*-shared 2260 15732 338695 1686879PixelBox*-global 866 6023 124031 948560

PixelBox*: modifying and extending the PixelBox algorithm [5] to decompose single polygons (vs. computing sizes of intersection areas of pairs of polygons) and handle “complex” multi-ring polygons

PixelBox*-shared: CUDA implementation using GPU shared memory for stack • DFS traversal with a batch size of N • N can not be too big (shared memory capacity) or too small (GPU is underutilization

if less than warp size) PixelBox*-global: CUDA implementation using GPU global memory for stack

• DFS traversal with different batch sizes • Coalesced global GPU memory accesses are efficient

Proposed technique: Thrust data parallel implementation on top of parallel primitives• BFS traversal with higher degrees of parallelisms• Data parallel designs (using primitives) simplify implementations• GPU shared memory is not explicitly used and is more flexible • Coalesced global GPU memory accesses are efficient• But: large memory footprint (for the current implementation)

Summary and Future Work• Diversified hardware makes it challenging to develop efficient

parallel implementations for complex domain specific applications across platforms.

• The framework on data parallel designs on top of parallel primitives seems to be a viable solution in the context of managing and querying large-scale geo-referenced species distribution data

• Further understand the advantages and disadvantages of data parallel designs/implementations on parallel hardware (GPUs, MICs and CMPs) through domain specific applications.

• More efficient polygon decomposition algorithms (e.g. scanline based) using parallel primitives

• System integration and more applications

• Experiments on 4000+ birds species distribution data have shown up to 190X speedups for polygon decomposition and 27X speedups for quadtree construction over serial implementations on a high-end GPU.

• Comparisons with PixelBox* variations that are natively CUDA implementations have shown that efficiency and productivity can be achieved simultaneously based on the data parallel framework using parallel primitives

Q&A

jzhang@cs.ccny.cuny.edu

http://www-cs.ccny.cuny.edu/~jzhang/