arcgis geoevent server: applying real-time analytics€¦ · 2018 esri european developer summit...
TRANSCRIPT
ArcGIS GeoEvent Server: Applying Real-Time Analytics
Thomas Paschke (Germany) & Daniel Cronin (UK)
AgendaReal-Time DevelopmentWorking with Real-Time DataVisualizing Real-Time DataArcGIS as an IoT PlatformBest Practices & TroubleshootingRoad-Ahead & Summary
1
2
3
4
5
6
Real-Time (& Big Data) Development1
GIS DataWhat has happened, what is happening, what will happen
The ‘current’ snapshot is outdated almost as soon as it’s created…
Credit: iStockphoto/chris_lemmens
Real-Time GIS RequirementsIngestion, Analytics, Notifications and Alerting
Requirement #1
Requirement #2
Continuous Analysis
FeaturesPosition
AlertInsideBoundary
Requirement #3
Applications
Distributed Batch Analysis forlarge amounts of data
ArcGIS Enterprise with real-time & big data GIS capabilities
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
ArcGISEnterprise
spatiotemporalbig data store
GeoAnalyticsServer
IoT Big Data
GeoEventServer
ArcGIS Enterprise with real-time & big data GIS capabilities
ingest visualize
analyze store
real-time & big data
millions4K e/s
4K e/s 10Ks e/s
ArcGISEnterprise
spatiotemporalbig data store
GeoAnalyticsServer
IoT Big Data
GeoEventServer
Distributed Batch Analysis forlarge amounts of data
Working with Real-Time Data2
Ingestion of real-time dataGeoEvent Server: input connectors
GeoEvent Server
Inpu
ts
Out
puts
GeoEvent Services
Poll an ArcGIS Server for Features
Out
of t
he B
ox
Esri
Gal
lery
ActiveMQ
Part
ner G
alle
ry
Amazon IoTAzure IoT
CompassLDEenviroCar
exactEarth AIS
FAA (ASDI)
GNIP
Networkfleet
OSIsoft
Valarm
Waze
AIS
Common Alerting Protocol
Cursor-on-Target
Exploitation Support Data
GTFS
Kafka
KML
MQTT
NMEA 0183
RabbitMQ
Sierra Wireless (RAP)
Trimble (TAIP)
Poll an external website for GeoJSON, JSON, or XMLHTTP
Receive Features, GeoJSON, JSON, or XML on a REST endpointREST
Receive GeoJSON or JSON on a WebSocketWS
Receive RSS
Receive Text from a TCP or UDP Socket
Subscribe to an external WebSocket for GeoJSON or JSONWS
Watch a Folder for new CSV, GeoJSON, or JSON Files.csv
AIS
GeoEvent Server
Inpu
ts
Out
puts
GeoEvent Services
Dissemination of real-time dataoutput connectors
Out
of t
he B
ox
Add or Update a Feature
Esri
Gal
lery
Amazon IoT
Azure IoT
ActiveMQ
Cursor-on-Target
Hadoop
Kafka
MongoDB
MQTT
RabbitMQ
Publish Text to a UDP Socket
Push GeoJSON or JSON to an external WebsiteHTTP
Push GeoJSON or JSON to an external WebSocketWS
Push Text to an external TCP Socket
Send a Text Message
Send an Email
Send an Instant Messageim
Send Features to a Stream Service
Write to a CSV, GeoJSON, or JSON File .csv
Add a Feature to a spatiotemporal big data store
Update a feature in a spatiotemporal big data store
Disseminationconfigure a new output connector by pairing an adapter & transport together
Connector = Adapter + Transport
GeoEvent Server
Inpu
ts
Out
puts
GeoEvent Services
Feature-JSON
Connectors Transport Adapter
GeoJSON
Receive Feature-JSON from Kafka
Receive GeoJSON on a REST endpoint
Feature-JSON
Adapters
GeoJSON
JSON
RSS
Text
XML
CAP
Cursor-on-Target
GeoMessage
NMEA
Sierra Wireless (RAP)
Trimble (TAIP)
VMF
Esri
Gal
lery
Waze
Feature Service
Transports
File
HTTP
HTTP+BasicAuth
HTTP+OAuth
TCP
UDP
Waze
WebSocket
Esri
Gal
lery Amazon IoT
Azure IoTIRC
Kafka
MQTT
Kafka
HTTP
RabbitMQ
you can createyour own
connectors
Extending GeoEventSoftware Development Kit (SDK)
Demo: Ingestion & Visualization
Streaming analyticsGeoEvent Server: geoevent services
input1 output1buffer
geofences
filter(spatial)
input2 output2
geofences
geotagger
output3motioncalculator
geometry inside Zones/.*
geometry inside Zones/.*
• A GeoEvent Service configures the flow of real-time data,- the filtering and geoevent processing steps to perform,- the input(s) where data comes from and the output(s) to which results are sent.
Filters
• Allow event records to pass only if a conditional expression evaluates TRUE
• The expression can use attributes or geometries you have established as geofences
• You can use Boolean logic (AND, OR, NOT) to combine expressions
Spatial Operators
• Spatial operators:
geofences
inside outside
enter exit
intersect disjoint
touches contains
crosses equals
overlaps within
you can createyour own
processors
Streaming analytics, policies & orchestrationGeoEvent Server: processors
GeoEvent Server
Inpu
ts
Out
puts
GeoEvent Services
Buffer Creator
Convex Hull Creator
Difference Creator
Envelope Creator
Field Calculator
Field Enricher
Field Mapper
Geotagger
Incident Detector
Intersector
Projector
Simplifier
Symmetric Difference
Track Gap Detector
Field Reducer Union Creator
Out
of t
he B
oxAdd XYZ
Esri
Gal
lery
Bearing
Ellipse
Event Volume Control
Extent Enricher
Field Grouper
GeoNames Lookup
Range Fan
Reverse Geocoder
Service Area Creator
Symbol Lookup
Track Idle Detector
Unit Converter
Visibility
Motion Calculator Query Report
GeoEvent ServerBeispiel: Konvoy Trennung
Demo: Processing Real-Time Data
Visualizing Real-Time Data3
Visualizationchoosing a service type: stream service, feature service, map service
Stream LayerMap Layer
Feature Layer
• Stream layers in apps subscribe to stream services to immediately visualize observations.- Does not require storage, is low latency, cannot be replayed.
• Map & Features layers in apps periodically poll to visualize most current observations.- Backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS).- History can be retrieved & queried for playback.
desktop
device
web
ArcGISEnterprise
Stream Service
Send Features to a Stream Service subscribe (push)
polling (pull)
Map ServiceFeature Service
Add a Feature to a BDSUpdate a Feature in a BDS
spatiotemporalbig data store
Add or Update a Feature
EGDBGeoEventServer
Stream LayerNow supported in Pro! (2.2 onwards)
Demo: Using StreamLayer in a JS WebApp
GeoEvent Admin REST APIScript based administration
Demo: Using the Admin REST API
ArcGIS as an IoT Platform4
ArcGIS as an IoT Platformenabling geospatial insights with your IoT solution
• An ArcGIS based IoT Platform & Enterprise consists of the following capabilities:
Sensors
Actuators
Devices(or Things)
Env
iro
nmen
t
IoT PlatformEdge Enterprise
Visualization&
Dashboards
Ingestion
StreamingAnalytics
Data Store
BatchAnalytics
Actions &Intelligence
Policy & Orchestration
DeviceManagement
ManagementConsole
- Ingestion
- Streaming Analytics & Policies
- Actions (including Actuation)
- Data Store
- Batch Analytics
- Management Console
- Device Management- Visualization
- Dashboards
ArcGIS as an IoT Platformenabling geospatial insights with your IoT solution
Sensors
Actuators
Devices(or Things)
Env
iro
nmen
t
IoT PlatformEdge Enterprise
DeviceManagement
Operations Dashboard for ArcGISInsights for ArcGIS
Esri Story MapsArcGIS Earth
ArcGIS OnlineArcGIS Pro
Collector for ArcGISWeb AppBuilder for ArcGIS
AppStudio for ArcGIS
dashboards
• An ArcGIS based IoT platform consists of the following capabilities:- Ingestion:
- Streaming Analytics & Policies:
- Actions (including Actuation):
- Data Store:
- Device Management: missing functionality in ArcGIS, 3rd party solution must be integrated such as Azure IoT
- Batch Analytics:
- Management Console:
- Visualization:
- Dashboards:
spatiotemporalbig data store
data store
GeoEventServer
ingestion
actions
analytics
GeoAnalyticsServer
analytics
ArcGISEnterprise
policies & orchestration
management console
visualization
GeoEvent server input connectors
GeoEvent Services
GeoEvent output connectors
spatiotemporal big data store
GeoAnalytics Server
Enterprise (Portal & Server Manager)
Map & Feature Services
Operations Dashboard, Insights, AppBuilder
ArcGIS as an IoT Platformenabling geospatial insights with your IoT solution
Edge Enterprise
DeviceManagement
Operations Dashboard for ArcGISInsights for ArcGIS
Esri Story MapsArcGIS Earth
ArcGIS OnlineArcGIS Pro
Collector for ArcGISWeb AppBuilder for ArcGIS
AppStudio for ArcGIS
dashboards
Sensors
Actuators
Devices(or Things)
Gateways
Env
iro
nmen
t
CoAP
MQTT
AMQP
HTTP
legacy
MQTT, HTTP
IoT Platform
ingestion
Ingestion
StreamingAnalytics
Data Store
BatchAnalytics
Actions &Intelligence
Policy & Orchestration
ArcGISEnterprise
GeoEventServer
GeoAnalyticsServer
spatiotemporalbig data store
analytics
policies & orchestration
management console
data store analytics
visualization
ingestion
actions
• Complementing an IoT platform with ArcGIS
ArcGIS as an IoT Platformenabling geospatial insights with your IoT solution
Edge Enterprise
Ingestion Actions &Intelligence
Policy & Orchestration
DeviceManagement
SpatiotemporalCapabilities
visualization
Operations Dashboard for ArcGISInsights for ArcGIS
Esri Story MapsArcGIS Earth
ArcGIS OnlineArcGIS Pro
Collector for ArcGISWeb AppBuilder for ArcGIS
AppStudio for ArcGIS
dashboards
Sensors
Actuators
Devices(or Things)
Gateways
Env
iro
nmen
t
CoAP
MQTT
AMQP
HTTP
legacy
MQTT, HTTP
IoT Platform
ingestion
ingestion
actions
streaminganalytics
data store batchanalytics
• Complementing an IoT platform with ArcGIS
Best Practices & Troubleshooting5
Performance, Resiliency & Scalabilityfactors that influence throughput
Input event counts don’t always tell the whole story
input1 output1buffer
geofences
filter(spatial)
input2 output2
geofences
geotagger
output3motioncalculator
geometry inside Zones/.*
geometry inside Zones/.*
filter(for input1)
filter(for input2)
1000 e/s
2000 e/s
1000 e/s
2000 e/s
2000 e/s
1000 e/s
1000 e/s
1000 e/s
1000 e/s
1000 e/s
1000 e/s
2000 e/s 3000 e/s4000 e/s
Service Design Considerationswhich would you choose?
Service A Service B
Service Design Considerationswhich would you choose?
When possible, pre-filter the input data before ingesting.
Each “branch” in a service contains the same event data.In this example, with three branches, it is creating 3X the volume of data.
Service Design Considerationsnot all components are created equally
A
B
C
Which of these services will process the fastest? Slowest?
Service Design Considerationsnot all components are created equally
A
B
C The first service only contains components that are utilizing the internal service cache, which allows for the fastest processing.
Service Design Considerationsnot all components are created equally
A
B
C The second service utilizes the local ArcGIS Server’s Geometry service to modify the incoming events.These types of requests are typically very quick but can be impacted by geometry complexity.
Service Design Considerationsnot all components are created equally
A
B
C The third service utilizes Network Analyst to return a “drive time” polygon which can significantly impact throughput.
Road Ahead & Summary6
10.6.1 key improvements
• Significant Performance improvements over 10.5.x and 10.6- Configurable multi-threaded writing to the spatiotemporal big data store- Single machine peak throughput velocity up to 10,000 events per seconds (e/s)
• Multi-machine deployments for improved resiliency with linear scalability- Up to 5 machines practically supporting up to 50,000 e/s
• Integrated Public Key Infrastructure (PKI) Authentication (Defense / Intel)
• Improved User Interface for Processor configuration
• ArcGIS Monitor Support
• Stream Layer Support in Pro 2.2 (New render support for published Web Maps)
GeoEvent Server
GeoEvent Server
• Improved User Experience- Embedded Help- Expanded Documentation- Short Training Exercises
• Additional Support for Stream Services- Removing Platform Services Dependency- Dedicated Input Connector- Improved Integration with New Products
• Automatic Configuration Backup to Azure Blob / Amazon S3 bucket
• Quality Focused Improvements
10.7 major epics
Road Ahead content is provided for informational purposes only and is subject to change
Spatiotemporal big data store10.7.x major epics
• Enhance GeoEvent Manager and REST API with ability:- to export data on demand: including Delimited Text or Parquet to S3/Azure Blob- to import data on demand: including Delimited Text or Parquet from S3/Azure Blob
• Z Value Support for Polyline, Polygon, and Multi-Point Geometries
• Additional Symbology Support in ArcGIS Enterprise / Online JavaScript Web Map
Road Ahead content is provided for informational purposes only and is subject to change
• Ingest high velocity real-time data into ArcGIS.
• Perform continuous analytics on events as they are received.
• Store observations in a spatiotemporal big data store.
• Visualize high velocity & volume data:
- as an aggregation- or as discrete features.
• Notify about patterns of interest.
DesktopWeb Device
visualization
live & historicaggregates & features
map & feature servicestream service
live features
ArcGISEnterprise
GeoEventServer
spatiotemporalbig data store
storage
ingestion
analytics
SummaryGeoEvent Server – real-time analytics for your ArcGIS Enterprise
Summaryself-paced training and resources
• Step-by-Step Tutorials, free to download- Introduction- Stream services- Spatiotemporal Big Data Store- Notifications
• Blogs and discussions on the forum- http://links.esri.com/geoevent-forum
• Video recordings of technical workshops- http://www.esri.com/videos
Thank You to Our Sponsors