arcgis geoevent server - overview - esri...• ingest high velocity real-time data into arcgis •...
TRANSCRIPT
ArcGIS GeoEvent Server – OverviewThomas Paschke
1
2
3
4
5
Agenda
GeoEvent Development
GeoEvent Server
Real-Time & Big Data @ 10.5.x
Visualization
Road Ahead (10.6)
R&D Project Trinity & Other6
GeoEvent Development1
GIS DataWhat has happened, what is happening, what will happen
The ‘current’ snapshot is outdated almost as soon as it’s created…
Credit: iStockphoto/chris_lemmens
Real-Time GIS RequirementsIngestion, Analytics, Notifications and Alerting
Requirement #1
Requirement #2
Continuous
AnalysisFeatures
Position
AlertInside
Boundary
Requirement #3
Applications
ArcGIS Serverwith real-time capabilities
ingest visualize
analyze store
real-time & big data
BYO-RDBMSGeoEvent
Server
Web GISwith real-time capabilities
Web GIS
ingest visualize
analyze store
real-time & big data
thousands3K e/s
3K e/s 200 e/s
BYO-RDBMSGeoEvent
Server
X
X
200 e/s
Stream Service
Web GISwith real-time capabilities
Web GIS
ingest visualize
analyze store
real-time & big data
thousands3K e/s
3K e/s 200 e/s
GeoEvent
Server
X
X
200 e/s
relational
data store
Web GISwith real-time capabilities
Web GIS
spatiotemporal
big data store
IoT
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
4K e/s
GeoEvent
Server
GeoEvent Server2
ArcGIS Enterprisewith real-time capabilities
ArcGIS
Enterprise
GeoEvent
Server
IoT
you can create
your own
connectors
Ingestionuse an existing input connector
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Poll an ArcGIS Server for Features
Poll an external website for GeoJSON, JSON, or XML
Receive Features, GeoJSON, JSON, or XML on a REST endpoint
Receive RSS
Receive GeoJSON or JSON on a WebSocket
Receive Text from a TCP or UDP Socket
Subscribe to an external WebSocket for GeoJSON or JSON
Watch a Folder for new CSV, GeoJSON, or JSON Files
Ou
t o
f th
e B
ox
REST
.csv
WS
WS
HTTP
Es
riG
all
ery
ActiveMQ
CAP
Exploitation Support Data
Cursor-on-Target
RabbitMQ
NMEA 0183
MQTT
Sierra Wireless (RAP)
KML
Kafka
Trimble (TAIP)
Pa
rtn
er
Ga
llery
CompassLDE
enviroCar
GNIP
FAA (ASDI)
exactEarth AIS
Waze
Valarm
Networkfleet
OSIsoft *
*
*
*
Zonar *
Amazon IoT
Azure IoT
you can create
your own
connectors
Ingestionconfigure a input connector by pairing a transport & adapter together
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Feature-JSON
Connectors Transport Adapter
GeoJSON
Receive Feature-JSON from Kafka
Receive GeoJSON on a REST endpoint
Feature-JSON
Adapters
GeoJSON
JSON
RSS
Text
XML
CAP
Cursor-on-Target
GeoMessage
NMEA
Sierra Wireless (RAP)
Trimble (TAIP)
VMF
Es
riG
all
ery
Waze
Feature Service
Transports
File
HTTP
HTTP+BasicAuth
HTTP+OAuth
TCP
UDP
Waze
WebSocket
Es
riG
all
ery Amazon IoT
Azure IoT
IRC
Kafka
MQTT
Kafka
HTTP
RabbitMQ
you can create
your own
connectors
Disseminationuse an existing output connector
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Ou
t o
f th
e B
ox
Add or Update a Feature
Publish Text to a UDP Socket
Send a Text Message
Send an Email
Push Text to an external TCP Socket
Push GeoJSON or JSON to an external WebSocket
Push GeoJSON or JSON to an external Website
Send an Instant Message
Send Features to a Stream Service
Write to a CSV, GeoJSON, or JSON File .csv
WS
im
HTTP
Add a Feature to a spatiotemporal big data store
Update a feature in a spatiotemporal big data store
ActiveMQ
Es
riG
all
ery
Cursor-on-Target
Hadoop
Kafka
MongoDB
MQTT
RabbitMQ
Amazon IoT
Azure IoT
GeoEvent Server Java SDKCreate your custom GeoEvent components
Applying Real-Time Analytics
• A GeoEvent Service defines the flow of GeoEvents
- The Filtering and Processing steps to perform
- what input(s) to apply them to
- and what output(s) to send the results to
GeoEvent Services
Performing continuous analytics in ArcGISProcessors
You can create
your own
processors.
• You can perform continuous analytics on events as they are received using a processor.
GeoEvent Extension
Inp
uts
Ou
tpu
ts
GeoEvent Services
Buffer Creator
Convex Hull Creator
Difference Creator
Envelope Creator
Field Calculator
Field Enricher
Field Mapper
Geotagger
Incident Detector
Intersector
Projector
Simplifier
Symmetric Difference
Track Gap Detector
Field Reducer Union Creator
Ou
t o
f th
e B
ox
Add XYZ
Es
riG
all
ery
Bearing
Ellipse
Event Volume Control
Extent Enricher
Field Grouper
GeoNames Lookup
Range Fan
Reverse Geocoder
Service Area Creator
Symbol Lookup
Track Idle Detector
Unit Converter
Visibility
Motion Calculator Query Report
Additional Real-Time Spatial Analytics
• Spatial operators:
Spatial condition evaluation
inside outside
enter exit
intersects disjoint
touches contains
crosses equals
overlaps within
Additional Real-Time Spatial AnalyticsExample – Impending weather notifications
Additional Real-Time Spatial AnalyticsExample – Convoy separation alerting
Additional Real-Time Spatial AnalyticsGeoFence scope
• All GeoFences• Any GeoFence
• Specify a GeoFence scope: Any or All
Additional Real-Time Spatial AnalyticsExample – Territory adherence alerting
GeoEvent Server REST API documentationNew at 10.5.1
Real-Time & Big Data @ 10.5.x3
ArcGIS Enterprisewith real-time & big data capabilities
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
4K e/s 10Ks e/s
GeoAnalytics
Server
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big Data
GeoAnalytics
Server
Writing analytic results
• GeoAnalytics Server has the ability to:
- perform analytics against data sources in the SBDS
- and write the analytic results to the SBDS
from GeoAnalytics Server
IoT
new Web GIS
layers
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
GeoAnalytics
Server
big datafile shares
featureservice
.shp
shapefiles
.csv
text files
Hadoopfile system
Hive
Writing analytic results
• GeoAnalytics Server has the ability to:
- perform analytics against files in an external big data file share
- and write the analytic results to the spatiotemporal big data store
from GeoAnalytics Server
new Web GIS layers
Web GIS layers
Visualization4
On-the-fly aggregations
• The spatiotemporal big data store enables you to aggregate data on-the-fly
- aggregating on-the-fly accommodates for real-time data
- you can perform exploratory analysis via any combination of space, time and attribute queries
- toggle from aggregation to raw feature view
of observations
Preparing data for on-the-fly aggregations
• as data is written to a data source in the spatiotemporal big data store:
- up to four types of spatial indices are supported: geohash, square, pointy & flat hexagon/triangle
- this is in addition to a temporal index on the time field
- and an inverted index on each of the attribute fields
spatial indexing
geohash
square
trianglepointy
triangleflat
hexagonflat
hexagonpointy
Two patterns
GeoEvent Server
Your
Applications
Stream Layer
Map Services
Feature Services
Feature Layer
feature layers
Update a Feature
Add a Feature
EGDB/
SBDS
Polling
(Pull)
Stream Services
Getting Real-Time Data into Web Apps
• Feature layers pull from feature services
- Web apps poll to get periodic updates
- Must be backed by an enterprise geodatabase (EGDB) or spatiotemporal big data store (SBDS)
• Stream layers subscribe to stream services
- Web apps subscribe to immediately receive data
- Low latency and high throughput
Send Features to a Stream Service
ArcGIS Enterprise
• Ingest high velocity real-time
data into ArcGIS
• Perform continuous analytics
on events as they are received
• Store observations in a
spatiotemporal big data store
• Run batch analytics on
stored observations
• Visualize high velocity &
volume data:
- as an aggregation
- as discrete features
• Notify those who need to
know about patterns of
interest
with real-time & big data capabilities
DesktopWeb Device
visualization
live & historic
aggregates & features
map & feature servicestream service
live features
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
storage
ingestion
analytics
GeoAnalytics
Server
analytics
Road Ahead (10.6.x)5
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
spatiotemporal
big data store
Big Data
GeoAnalytics
Server
IoT
RESILIENT environment
• GeoEvent 10.5
- Resiliency (high availability) & scalability is only possible if User’s “bring their own gateway”.
- Barrier to entry is HIGH & typically requires a professional services engagement for success.
bring your own
gatewayGeoEvent
Server
ArcGIS Enterprisewith real-time & big data capabilities
ArcGIS
Enterprise
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
GeoEvent
Server
gatewaygateway gateway
• GeoEvent 10.6
- Provides users with a resilient & scalable Real-Time GIS deployment OUT-OF-THE-BOX.
GeoEvent Server10.6 features
• RASP (Reliability, Availability, Scalability, Performance)
- Kafka as a new gateway service (dedicated process that runs Apache Kafka + Zookeeper in it’s own
life-cycle).
- Create a multi-machine GeoEvent Server site
- Uses ArcGIS Server framework for creating multi-machine site
GeoEvent Server10.6 features
GeoEvent Server10.6 features
GeoEvent Server10.6 features
GeoEvent Server10.6 features
GeoEvent Server10.6 features
• RASP (Reliability, Availability, Scalability, Performance)
- Kafka as a new gateway service (dedicated process that runs Apache Kafka + Zookeeper in it’s own
life-cycle).
- Create a multi-machine GeoEvent Server site
- Uses ArcGIS Server framework for creating multi-machine site
• 3rd party library updates for Apache Karaf, Elasticsearch & Spark
• Official Esri positioning on IoT (first time presented at UC, see session this afternoon)
• New and updated Connectors to Azure IoT, Kafka, Waze, GTFS, OAuth2 HTTP inbound
Spatiotemporal big data store10.6 enhancements
• Enhanced data retention purge rules and feature deletion
- with ability to specify a WHERE clause
• On-the-fly aggregation supports polygon, polyline & multipoint
Spatiotemporal big data storeOn-the-fly aggregation supports polygon, polyline & multipoint
Spatiotemporal big data store10.6 enhancements
• Enhanced data retention purge rules and feature deletion
- with ability to specify a WHERE clause
• On-the-fly aggregation supports polygon, polyline & multipoint
• Added space-time volume on-the-fly aggregation support
projecton-the-fly & interactive space-time aggregation
• leverage existing geohash, square & hexagon aggregation capabilities of
the spatiotemporal big data store to construct on-the-fly space-time volumes.
- enables exploratory & interactive analysis of real-time & historic observation data
ArcGIS
Enterprise
spatiotemporal
big data store
IoT
GeoEvent
Server
ArcGIS API for
JavaScript
& WebGL
client-side rendering ofspace-time volumes via WebGL
lodType=flatHexagon&lod=2&time=1474675200000,1475279999000
&timeInterval=1&timeUnit=hour
on-the-fly & interactive space-time aggregation feature service
space-time volume (based on a geohash aggregation) response
project
feature servicelodType=geohash&lod=2
geohash aggregation response
lodType=geohash&lod=2&time=1474675200000,1475279999000&timeInterval=1&timeUnit=hour&f=json
an array of time-slices of aggregated features
&returnGeometry=false &returnGeometry=false
ArcGIS EnterpriseTechnologies
ArcGIS
Enterprise
spatiotemporal
big data store
Big DataIoT
GeoAnalytics
Server
GeoEvent
Server
gatewaygateway gateway
R&D Project Trinity, Polly & Banzai6
project Polly
• as data is written to the spatiotemporal big data store:
- a spatial index for polygon aggregation is continuously updated
spatial indexing: polygon
polygonstate
polygoncounty
polygon aggregation (based on a county polygon index)polygon aggregation (based on a state polygon index)
visualization on-the-fly aggregation capabilities
Spatiotemporal,into geohash, square or hexagon
Spatial, into geohash, square or hexagon
Spatiotemporal, into polygons
Spatial, into polygons
• project Trinity is a capability that
enables existing customers and
prospects with the ability to:
- Reliably ingest, analyze & store up to
millions of sensor events per second.
- Quickly visualize, replay &
explore observations that
are billions of features.
- Perform fast analytics
on billions of observations.
an ArcGIS Enterprise bundled with massive real-time & big data capabilities
Sensors
project Trinity
DesktopWeb Device
ArcGIS Enterprise with real-time & big data
spatiotemporal
archive
real-time
batch
sources
hubs
project Trinity
on Amazon C2S
on Amazon
on Azure
Internet of your Things
bringing geospatial insights to your
DesktopWeb Device
ArcGIS Enterprise with real-time & big data
sources hubs real-time analytics spatiotemporal archive batch analytics
8 cores,28 GB ram,200 GB ssd
8 cores,28 GB ram,800 GB ssd
16 cores, 56 GB ram, 1 TB ssd
2 cores,4 GB ram,50 GB ssd
2 cores,10 GB ram,200 GB ssd
4 cores,14 GB ram,150 GB ssd
8 cores, 28 GB ram, 400 GB ssd
project Trinity
f( ) Sources + Hubs +
Real-Time Analytics f( )Spatiotemporal Archive
+ Batch Analytics
30 nodes480 cores, 1.6 TB ram, 30 TB ssd
50 nodes400 cores, 1.4 TB ram, 20 TB ssd
project Trinity
on Amazon C2S
on Amazon
on Azure
bringing geospatial insights to your
DesktopWeb Device
Internet of your Things
sources hubs real-time analytics spatiotemporal archive batch analytics
ArcGIS Enterprise with real-time & big data
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
project Trinity
1 1 11 11 1 1 1 1
1 1 11 1 1 1 1 1 1
2 2 2 2 2 2 2 2 2 2
2 2 2 2 2
3 3 3 4 4 5 6 6 6 66 67 7 6 67 7
1 1 1 1 1 12 2 2 3 3 4
ArcGIS
Enterprise
880 cores, 3TB memory, 50TB storage
project Trinity
240 cores, 1.1TB ram, 18TB storage
ArcGIS
Enterprise
monitor admin
Trinitytask manager
monitor admin
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
8 cores
28 GB ram
400 GB ssd
big data analytic pipelining & scheduling
Banzai
project Trinity
Trinity
Source ingestion
Eventhub
Real-Time analytic
Spatiotemporalarchive
Batchanalytic
Mapservice
0 to many
a lightweight
“GeoEvent Input”
distributed micro-service
0 to few
a lightweight
“GeoEvent event hub”
distributed micro-service
0 to many
a lightweight
“GeoEvent Service”
distributed micro-service
0 to few
a lightweight
“spatiotemporal big data store”
distributed micro-service
0 to many
a lightweight
“GeoAnalytics”
distributed micro-service
0 to many
a lightweight
“map & feature service”
distributed micro-service
Thank You to Our Generous Sponsor