using redis for geospatial applications · using redis for geospatial applications redistimeseries...
TRANSCRIPT
![Page 1: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/1.jpg)
Using Redis for Geospatial Applications
RedisTimeSeries RedisAI
Rock PereiraCUGOS Fall Fling
Oct 06, 2019
![Page 2: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/2.jpg)
1 Intro: Data Analysis Vs App Dev
Data Analysis
Data used is old
Larger size
Stored on disk
Model building / development
App Development
Data used is fresh / recent
Smaller size
Stored in memory
Model prediction / serving
![Page 3: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/3.jpg)
1 Intro: SQL Vs NOSQL
SQL
Stored as records in tables
Complex SQL queries
ACID Transactions
eg. PostgreSQL, MySQL
NOSQL
Stored as Key-Value, Documents, or Graphs
Can query, but no joins
Eventual consistency
eg. Redis, MongoDB, Neo4J
![Page 4: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/4.jpg)
2 Redis: Introduction
Redis = Remote Dictionary Service
Redis is a Data Structure server:
Strings of binary safe bytesListsHashesSets eg. “students” ::: {“Alice”, “Bob”, “Carla”}Sorted Sets eg. “student:ages” ::: {(“Bob”, 19), (“Carla”, 22),(“Alice”, 25)}
Add (“Dan”, 20): The sorted set stays sorted by the score.“student:ages” ::: {(“Bob”, 19), (“Dan”, 20), (“Carla”, 22),(“Alice”, 25)}
![Page 5: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/5.jpg)
2 Redis: Geospatial
Locations, (lon, lat) are stored as sorted sets.
The score in the sorted set cannot be a tuple.
So the (lon, lat) tuple is hashed to a single value
“cities” ::: {(“Olympia”, “a8b65e..”), (“Tacoma”, “1bc3e4..”), (“Seattle”, “3df24a..”)}
![Page 6: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/6.jpg)
2 Redis: Use Cases
Caching: To manage fresh dataMessage broker: Redis has Pub SubReal-time analyticsSession management: eg Shopping cartsGaming: Leaderboards & User activityTime-series logsJob and Queue ManagementRide-Hailing: Matching customers with nearby taxis
![Page 7: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/7.jpg)
Data Prep – Spark Workflow
Raw dataUTM Zone 10
Longitude [-126, -120]36 files, csv, 125 GB
Sort by time
VesselMetadata, csv
VesselMotion
Split
Time-sortedMotion
Aggregate byDevice ID (mmsi)
Time series array of Vessel Tracks for
each vessel (3 years)
Filter to 3 months
Save as parquet
Partitioned on year, month, day
Exploratory data for use
on a PC
3 years data for modeling
after data exploration
Save on Blob Storage for
querying
Convert UTC to local time. Then partition
Save as parquet
Puget SoundLat [47.00, 49.45]
Lon [-123.75, -122.00]
Filter by lat, lon
CSVSpark DataFrame
Parquet
![Page 8: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/8.jpg)
![Page 9: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/9.jpg)
![Page 10: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/10.jpg)
![Page 11: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/11.jpg)
![Page 12: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/12.jpg)
![Page 13: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/13.jpg)
![Page 14: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/14.jpg)
![Page 15: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/15.jpg)
![Page 16: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/16.jpg)
![Page 17: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/17.jpg)
![Page 18: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/18.jpg)
![Page 19: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/19.jpg)
![Page 20: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/20.jpg)
![Page 21: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/21.jpg)
![Page 22: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/22.jpg)
![Page 23: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/23.jpg)
![Page 24: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/24.jpg)
![Page 25: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/25.jpg)
![Page 26: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/26.jpg)
![Page 27: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/27.jpg)
![Page 28: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/28.jpg)
![Page 29: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/29.jpg)
![Page 30: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/30.jpg)
![Page 31: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/31.jpg)
![Page 32: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/32.jpg)
![Page 33: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/33.jpg)
![Page 34: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/34.jpg)
![Page 35: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/35.jpg)
![Page 36: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/36.jpg)
![Page 37: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/37.jpg)
![Page 38: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/38.jpg)
![Page 39: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/39.jpg)
![Page 40: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/40.jpg)
![Page 41: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/41.jpg)
![Page 42: Using Redis for Geospatial Applications · Using Redis for Geospatial Applications RedisTimeSeries RedisAI Rock Pereira CUGOS Fall Fling Oct 06, 2019. 1 Intro: Data Analysis Vs App](https://reader035.vdocuments.site/reader035/viewer/2022062607/6038ee79123e9372e2664748/html5/thumbnails/42.jpg)