oracle stream analytics - simplifying stream processing

BASEL BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH

Oracle Stream AnalyticsSimplifying Stream Processing

29.9.2016 – DOAG 2016 Big Data DaysGuido Schmutz

Guido Schmutz

Working for Trivadis for more than 19 yearsOracle ACE Director for Fusion Middleware and SOACo-Author of different booksConsultant, Trainer, Software Architect for Java, SOA & Big Data / Fast DataMember of Trivadis Architecture BoardTechnology Manager @ Trivadis

More than 25 years of software development experience

Contact: [email protected]: http://guidoschmutz.wordpress.comSlideshare: http://www.slideshare.net/gschmutzTwitter: gschmutz

Oracle Stream Analytics - Simplifying Stream Processing2

Unser Unternehmen.


Trivadis ist führend bei der IT-Beratung, der Systemintegration, dem Solution Engineering und der Erbringung von IT-Services mit Fokussierung auf -und -Technologien in der Schweiz, Deutschland, Österreich und Dänemark. Trivadis erbringt ihre Leistungen aus den strategischen Geschäftsfeldern:

Trivadis Services übernimmt den korrespondierenden Betrieb Ihrer IT Systeme.

B E T R I E B

KOPENHAGEN

MÜNCHEN

LAUSANNEBERN

ZÜRICHBRUGG

GENF

HAMBURG

DÜSSELDORF

FRANKFURT

STUTTGART

FREIBURG

BASEL

WIEN

Mit über 600 IT- und Fachexperten bei Ihnen vor Ort.


14 Trivadis Niederlassungen mitüber 600 Mitarbeitenden.

Über 200 Service Level Agreements.

Mehr als 4'000 Trainingsteilnehmer.

Forschungs- und Entwicklungsbudget: CHF 5.0 Mio.

Finanziell unabhängig undnachhaltig profitabel.

Erfahrung aus mehr als 1'900 Projekten pro Jahr bei über 800 Kunden.

Agenda

1. Introduction to Streaming Analytics2. Oracle Stream Analytics3. Demo


Introduction to Streaming Analytics


Traditional Data Processing - Challenges

• Introduces too much “decision latency”

• Responses are delivered “after the fact”

• Maximum value of the identified situation is lost

• Decision are made on old and stale data

• “Data a Rest”


The New Era: Streaming Data Analytics / Fast Data

• Events are analyzed and processed in real-time as the arrive

• Decisions are timely, contextual and based on fresh data

• Decision latency is eliminated

• “Data in motion”


Event / Stream Processing Architecture

DataIngestion

Batchcompute

DataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

Logfiles

Social

RDBMS

ERP

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

ResultStore

Messaging

ResultStore

Oracle Stream Analytics - Simplifying Stream Processing

=DatainMotion =DataatRest9

“Lambda Architecture” for Big Data

DataIngestion

(Analytical)BatchDataProcessing

Batchcompute

ResultStoreDataSources

Channel

DataConsumer

Reports

Service

AnalyticTools

AlertingTools

Content

RDBMS

Social

ERP

Logfiles

Sensor

Machine

(Analytical)Real-TimeDataProcessing

Stream/EventProcessing

Batchcompute

Messaging

ResultStore

QueryEngine

ResultStore

ComputedInformation

RawData(Reservoir)

Oracle Stream Analytics - Simplifying Stream Processing

=DatainMotion =DataatRest

PullingIngestion

10

When to Stream / When not?


ConstantlowMilliseconds&under

Lowmillisecondstoseconds,delayincaseoffailures

10sofsecondsofmore,Re-runincaseoffailures

Real-Time Near-Real-Time Batch

“No free lunch”


ConstantlowMilliseconds&under

Lowmillisecondstoseconds,delayincaseoffailures

10sofsecondsofmore,Re-runincaseoffailures

Real-Time Near-Real-Time Batch

“Difficult”architectures,lowerlatency “Easierarchitectures”,higherlatency

Why Event / Stream Processing?


Visualize Business in real-time• Dashboards can help people to visualize, monitor and make sense of massive amount of

incoming data in real-time

Detect Urgent Situations• Based on simple or complex analytical patterns of urgent business events• Urgent because they happen in real-time

Automate immediate actions• Run in the background quietly until detecting an urgent situation (risk or opportunity)• Alerts can go to humans through email, text or push notifications or to other applications trough

message queues or service call

Oracle Stream Analytics


History of Oracle Stream Analytics

OracleComplexEventProcessing(OCEP)

OracleEventProcessing(OEP)

OracleStreamExplorer(SX)

OracleEventProcessingforJavaEmbedded

OracleStreamAnalytics(OSA)

OracleEdgeAnalytics(OAE)

BEAWeblogic EventServerOracleCQL

OracleIoT CloudService

2016

2015

2007

2008

2012

2013


OEA

• Filtering• Correlation• Aggregation• Pattern

matching

Devices / Gateways

Services

Computing Edge Enterprise

“Sea of data”

Macro-eventHigh-valueActionableIn-context

EDGEAnalytics

StreamAnalytics

FOG

• High Volume• Continuous Streaming• Extreme Low Latency• Disparate Sources• Temporal Processing• Pattern Matching• Machine Learning

Oracle Stream Analytics: From Noise to Value

• HighVolume• ContinuousStreaming• Sub-MillisecondLatency• DisparateSources• Time-WindowProcessing• PatternMatching

• HighAvailability/Scalability• CoherenceIntegration• Geospatial,Geofencing• BigDataIntegration

• BusinessEventVisualization

• Action!


Oracle Stream Analytics Platform

What it does• Compelling, friendly and visually stunning real time

streaming analytics user experience for Business users to dynamically create and implement Instant Insight solutions

Key Features• Analyze simulated or live data feeds to determine event

patterns, correlation, aggregation & filtering• Pattern library for industry specific solutions• Streams, References, Maps & Explorations

Benefits• Accelerated delivery time• Hides all challenges & complexities of underlying real-time

event-driven infrastructure


Oracle Stream Analytics – Self-Service Stream Processing!

Understanding of CQL Filtering, Correlation, Pattern: NOT NEEDED

Understanding of IT Deployment and Management: NOT NEEDED

Understanding of Development, Java, Best Practices: NOT NEEDED

Understanding of the Event Driven Platform: NOT NEEDED


Oracle Stream Analytics – Terminology

Explorer: The Application User Interface Catalog: The repository for browsing resources



Stream: incoming flow of events that you want to analyze (CSV, Kafka, JMS, Rest, MQTT, …)

Exploration: application that correlates events from streams and data sources, using filters, groupings, summaries, ranges, and more



Shape: A blueprint of an event in a stream or data in a data source. How the business data is represented in the selected stream

Map: collection of geo-fences

Reference: A connection to static data that is joined to a stream to enrich it and/or to be used in business logic and output



Pattern: A pre-built Exploration that addresses a particular business scenario in a focused and simplified User Interface

Connection: collection of metadata required to connect to an external system

Targets: defines an interface with a downstream system


Business accessibility to Geo-Streaming Analytics

Real Time Streaming Solutions face an increasing need to track "assets of interest" and initiate actions based on encroachment of boundary proximity to fixed and moving objects and other geographic, temporal, or event conditions.

Geo-Fence,Fence,Polygon

Geo-StreamingOracle Stream Analytics - Simplifying Stream Processing25

“Addvaluetoyourrealtimestreamingdatadiscoveryandanalyticsbyapplyingandincludingmathematical,statisticalanalysistotheliveoutputstream”

“Thesestreaming“Excelspreadsheets”reallydocometolife”

Expression Builder enabling calculations


Concept of Connections and their reuse in Streams


Decision Table for Nested IF-THEN-ELSE Rules


Topology View and Navigation


Relationship between Streams (Sources), References and Explorations


Demo


Oracle Stream Analytics Demo Use Case: Truck Movements

Truck DataIngestion Geo-Fencing

2016-06-0214:39:56.605|98|27|MarkLochbihler|803014426|WichitatoLittle RockRoute 2|Normal|38.65|-90.21|5187297736652502631

{"timestamp":"2016-06-0214:39:56.991","truckId":99,"driverId":31,"driverName":"RommelGarcia","routeId":1565885487,"routeName":"SpringfieldtoKCViaHanibal","eventType":"Normal","latitude":37.16,"longitude":"-94.46","correlationId":5187297736652502631}

RecklessDrivingDetector

NEAR

ENTER

TruckDriver

DashboardMovement MovementJSON

RecklessDriver


Continuous Ingestion in Stream Processing

DBSource

BigData

Log

StreamProcessing

IoT Sensor

EventHub

Topic

Topic

REST

Topic

IoT GW

CDCGW

Conn

ectCDC

DBSource

Log CDC Native

IoT Sensor

IoT Sensor

33

DataflowGW

Topic

Topic

Queue

MQTTGW

Topic

DataflowGW

Dataflow

TopicRE

ST33FileSourceLog

Log

Log

Social

Native


Topic

Topic

Apache Kafka – High-volume messaging system

Distributed publish-subscribe messaging system

Designed for processing of high-volume, real time activity stream data (logs, metrics collections, social media streams, …)

Topic Semantic

does not implement JMS standard!

Initially developed at LinkedIn, now part of Apache

Kafka Cluster

Consumer Consumer Consumer

Producer Producer Producer


Demo: Oracle Stream Analytics


Summary


Native Stream Processing => OEP server

Ingestion

Event Source

Event Source

Event Source


IndividualEvent

PPPPPPPPPPPP

Micro-Batch Stream Processing => Spark Streaming

Ingestion

Event Source

Event Source

Event Source


PPPPPP

Summary

Oracle Stream Analytics leverages the capabilities found in Oracle Event Processing (OEP)

Empowering Business users to gain insight into real-time information and take appropriate actions when needed => makes stream processing accessible

Makes Stream/Event Processing less technical => “Excel spread sheet” on Streams

Part of Oracle IoT Cloud Service

Support Spark Streaming as a deployment platform for Streaming ML

Interesting road map: Rule Engine, Machine Learning, Extensible Patterns


Oracle Stream Analytics on Docker

Oracle Stream Analytics 12.2.1 Documentation

Oracle Stream Analytics 12.2.1 Download


Guido SchmutzTechnology Manager

[email protected]
