arcgis geoevent server: applying real-time analytics...receive rss receive text from a tcp or udp...
TRANSCRIPT
Stefan Jung, Thomas Paschke
ArcGIS GeoEvent Server:
Applying Real-Time
Analytics
Agenda
Real-Time Development
Working with Real-Time Data
GeoEvent Server SDK
Visualizing Real-Time Data
Summary
1
2
3
4
5
Real-Time (& Big Data) Development1
GIS DataWhat has happened, what is happening, what will happen
The ‘current’ snapshot is outdated almost as soon as it’s created…
Credit: iStockphoto/chris_lemmens
Real-Time GIS RequirementsIngestion, Analytics, Notifications and Alerting
Requirement #1
Requirement #2
Continuous
AnalysisFeatures
Position
AlertInside
Boundary
Requirement #3
Applications
ArcGIS Enterprise with real-time GIS capabilities
ArcGIS
Enterprise
spatiotemporal
big data storeGeoEvent
Server
Stream Service
ArcGIS Enterprise with real-time & big data GIS capabilities
ArcGIS
Enterprise
spatiotemporal
big data store
IoT
GeoAnalytics
Server
Big Data
GeoEvent
Server
Working with Real-Time Data2
GeoEvent ServerReal-Time Analytics
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Ingestion of real-time dataGeoEvent Server: input connectors
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Poll an ArcGIS Server for Features
Ou
t o
f th
e B
ox Poll an external website for GeoJSON, JSON, or XMLHTTP
Receive Features, GeoJSON, JSON, or XML on a REST endpointREST
Receive GeoJSON or JSON on a WebSocketWS
Receive RSS
Receive Text from a TCP or UDP Socket
Subscribe to an external WebSocket for GeoJSON or JSONWS
Watch a Folder for new CSV, GeoJSON, or JSON Files.csv
Subscribe to a Kafka Topic for JSON, GeoJSON, Text
Esri
Gallery
Azure IoT
Amazon IoT
ActiveMQ
RabbitMQ
GTFS
NMEA 0183
Kafka
KML
MQTT
…
Pa
rtn
er
Ga
lle
ry
CompassLDE
enviroCar
exactEarth AIS
FAA (ASDI)
Networkfleet
OSIsoft
Valarm
Waze
…
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Dissemination of real-time dataoutput connectors
Ou
t o
f th
e B
ox
Add or Update a Feature
Add or Update a Feature to a SBDS
Send Features to a Stream Service
Push data to an external TCP Socket
Push data to a UDP Socket
Push data to an external Website
Push data to an external WebSocketWS
HTTP
Send an Instant Messageim
Send a Text Message
Send an Email
Write to a CSV, GeoJSON, or JSON File .csv
Write data to a Kafka Topic
Esri
Gallery
Amazon IoT
Azure IoT
ActiveMQ
RabbitMQ
Hadoop
Kafka
MongoDB
MQTT
…
Disseminationconfigure a new output connector by pairing an adapter & transport together
Connector =
Adapter + Transport
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Feature-JSON
Connectors Transport Adapter
GeoJSON
Receive Feature-JSON from Kafka
Poll GeoJSON on an external Website
Feature-JSON
Adapters
GeoJSON
JSON
RSS
Text
XML
NMEA
Sierra Wireless (RAP)
Trimble (TAIP)
VMF
Esri
Gallery
Waze
Feature Service
Transports
File
HTTP
HTTP+BasicAuth
HTTP+OAuth
TCP
UDP
Waze
WebSocket
Esri
Gallery Amazon IoT
Azure IoT
Kafka
MQTT
Kafka
HTTP
RabbitMQ
you can create
your own
connectors
Demo:
Ingestion & Visualization
Streaming analyticsGeoEvent Server: geoevent services
input1 output1buffer
geofences
filter(spatial)
input2 output2
geofences
geotagger
output3motion
calculator
geometry inside Zones/.*
geometry inside Zones/.*
• A GeoEvent Service configures the flow of real-time data,
- the filtering and geoevent processing steps to perform,
- the input(s) where data comes from and the output(s) to which results are sent.
Spatial Operators
• Spatial operators:
geofences
inside outside
enter exit
intersect disjoint
touches contains
crosses equals
overlaps within
GeoEvent ServerBeispiel: Bereichsüberprüfung und Alarmierung
GeoEvent ServerBeispiel: Konvoy Trennung
Demo:
Incident Detection of Trains
you can create
your own
processors
Streaming analytics, policies & orchestrationGeoEvent Server: processors
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Buffer Creator
Convex Hull Creator
Difference Creator
Envelope Creator
Field Calculator
Field Enricher
Field Mapper
Geotagger
Incident Detector
Intersector
Projector
Simplifier
Symmetric Difference
Track Gap Detector
Field Reducer Union Creator
Ou
t o
f th
e B
ox
Add XYZ
Event Volume Control
Range Fan
Bearing
Feature To Point
Event Joiner
Esri
Gallery
Ellipse
Extent Enricher
Field Grouper
GeoNames Lookup
Reverse Geocoder
Service Area Creator
Symbol Lookup
Track Idle Detector
Unit Converter
Visibility
Motion Calculator Update Only
Spatial Query
Field Splitter
HTTP Handler Event Counter
Extending GeoEventSoftware Development Kit (SDK)
GeoEvent SDK3
Creating a custom GeoEvent Processor
You can create your own custom connectors and processors using the GeoEvent
Software Development Kit (SDK)
Your processors
Your
connectors
GeoEvent Server
Inp
uts
Ou
tpu
ts
GeoEvent Services
Your
connectors
Getting Started
• Java Development Kit (JDK 8+)
• Apache Maven (Build Management Tool)
• Eclipse (IDE)
Install and Configure
GeoEvent Server SDK
• api: JavaDoc content associated with GeoEvent Server SDK
• repository: Local maven repository
• samples: Sample connectors (and processors)
• GeoEvent Developer Guide
Define your own Processor
Input Processor OutputFilter
Which data passed the filter?
Define your own Processor
Input Processor OutputFilter
Output 2
Define your own Processor
Input Processor OutputFilter
Logging
Demo:
Custom GeoEvent Processor
GeoEvent Admin REST APIScript based administration
Visualizing Real-Time Data4
Stream Layer
Map Layer
Feature Layer
• Map & Features layers in apps periodically poll to visualize most current observations
- backed by an enterprise geodatabase (EGDB) or a spatiotemporal big data store (BDS)
- history can be retrieved & queried for playback and analysis
• Stream layers in apps subscribe to stream services to immediately visualize observations
- does not require storage, low latency, no playback
ArcGISEnterprise
Stream Service
Send Features to a Stream Service subscribe (push)
polling (pull)
Map ServiceFeature Service
Add or Update a Feature
EGDBGeoEvent
ServerAdd a Feature to a BDS
Update a Feature in a BDS
spatiotemporalbig data store
Apps
Desktop
APIs
choosing a service type: stream service, feature service, map service
Visualization
Demo:
StreamLayer in ArcGIS Pro
Summary6
• Ingest high velocity real-time
data into ArcGIS.
• Perform continuous analytics
on events as they are received.
• Store observations in a
spatiotemporal big data store.
• Visualize high velocity &
volume data:
- as an aggregation
- or as discrete features.
• Notify about patterns of
interest.
DesktopWeb Device
visualization
live & historic
aggregates & features
map & feature servicestream service
live features
ArcGIS
Enterprise
GeoEvent
Server
spatiotemporal
big data store
storage
ingestion
analytics
SummaryGeoEvent Server – real-time analytics for your ArcGIS Enterprise
Summaryself-paced training and resources
• Step-by-Step Tutorials, free to download
- Introduction
- Stream services
- Spatiotemporal Big Data Store
- Notifications
• Blogs and discussions on the forum
- http://links.esri.com/geoevent-forum
• Video recordings of technical workshops
- http://www.esri.com/videos
