soa12c launch 5 event processing shmakov eng cr

Copyright © 2014, Oracle and/or its affiliates. All rights reserved. 1

SOA Suite 12.1.3 Field Enablement

Oracle Event Processing Enabling Fast Data and The Internet of Things Technical Presentation


Modern Computing Challenges

Solution Product Overview

Oracle Event Processing Applications

Continuous Query Language (CQL)

Internet of Things (IoT)

What is Fast Data?

12c Features

Agenda

Oracle Event Processing 12c Sales Enablement


The application evaluates if the roaming asset (device) is within a certain

distance of the business. If it is, the system will generate an event to send the

customer a promotional message.

This powerful example encompasses the integration of JMS, Real Time Spatial

Analysis and Big Data integration

Mobile Marketing - OEP in Action: DEMO


Modern Computing Challenges

Does the problem you want to solve have any one or more of the following

conditions?

Requires high-throughput and low-latency processing

Continuously streaming data (Log Files, converting Batch to Real Time)

Real-time correlation between multiple incoming data sources

Time-sensitive alerts, aggregations and calculations

Needs to look for patterns in the data stream (Spatial Analysis)

Identify when some event(s) should have happened and did not.

Data does not need to be stored, if there is nothing of interest in it

Problem is more easily solved by analyzing before storing in DB

Conditions for Event Processing Opportunities



Solution involves one or more of the following:

High Volume

Continuous Streaming

Sub-Millisecond Latency

Disparate Sources

Time-Window Processing

Pattern Matching

Business Event Visualization

Oracle Event Processing

OEP

Streaming

Event Data

Alerts,

Actions

Filtering,

Pattern Matching,

Missing Events,

Aggregations,

Correlations,

Calculations,

Geo-Spatial



Inverted Database

Data is ‘static’

Queries are ‘dynamic’

• Data (event) is ‘dynamic’

• Queries are ‘static’



POJO

App Frameworks

Spring Services

Stream Management

Core Event Infrastructure

Complex Event Processor

Real-Time Kernel

Data Cartridges

Extended Event Infrastructure

Cluster Management HTTP Pub/Sub Engine

Event Repository

Coherence

Foundation Services

Security Logging

Processes Streaming Data In-Memory & in

Real-Time (274 OSGi Bundles)

Event Analysis: Pattern Matching, Missing

Events, Aggregations, Correlations,

Calculations, Coherence Integration,

Spatial Functions, Big Data Integration

Performs Aggregations and Correlates

Multiple Streams of Data

Integrates easily with other Oracle

products: Big Data (Hadoop & NoSQL) –

FAST DATA, SOA (EDN), Coherence,

Spatial, RTD, etc.

Str

ea

min

g E

ve

nt D

ata

Alerts, Actions

Solution Overview: Oracle Event Processing (OEP)

OSGi


Solution Overview

Filtering, Correlation & Aggregation

New stream filtered for specific criteria, e.g. stock price > $22

Process events from different streams together

Scrolling, time-based window metrics, e.g. average # of stock trades in

the last hour

Detect Absence of Events and Missing Events

Event “A” NOT followed by Event “B” within 10 minutes

Event “A”, Event “B” should occur next, but Event “C” occurs instead



Perform Calculations in Real-Time

Perform Calculations on Data From Sensors:

Use last X seconds or minutes of data OR

Use last X data points

Smooth out “noise” in the data


Input

Adapter Channel

Input

Adapter Channel

Business

Logic (CQL)

Channel

Channel

Channel

Output

Adapter

Output

Adapter

Oracle Event Processing Application

DB

Input adapters connect to data sources Channels help control the flow of data and can be tuned for optimal performance Databases and Coherence caches can be referenced directly in CQL processors CQL processors contain filtering, correlation, aggregation and pattern matching business logic Output adapters send data and alerts to downstream systems and business processes

Coherence

Business

Logic (CQL)

Business

Logic (CQL)


High-Performance In-Memory Data Processing

Input

Adapter Data

Input

Adapter Data

Channel Business

Logic (CQL)

Channel

Data

Data

Analytics

Channel Business

Logic (CQL)

Enrich Output

Adapter Data Data Data Data

Analytics: Continuously Sliding Time Windows of Streaming Data, Filtering,

Correlations, Calculations, Aggregations, Pattern Matching, Missing Event Detection,

Spatial Analysis, etc.

Enrichment: Integrate with data from DB, Coherence, NoSQL, Hadoop etc.



Administration & Monitoring

View the data flow of the application

“Event Processing Network” (EPN)


Administration & Monitoring

Monitor throughput and latency between any two nodes in

the application


Oracle Event Processing & Coherence

Challenges Benefits

Handle and correlate events in real-

time, including support for multiple

patterns:

• Pre-processing (buffer OEP)

• Within OEP (to cache reference data)

• Post OEP (to expose processed events to

consuming apps)

• High throughput for storing data

• Aggregation and event querying

• Pattern implementation flexibility

combining two complementary

technologies

Data

Grid

OEP

Consolidated

& in-context

Data


Simple Filtering

SELECT *

FROM inputChannel [NOW]

WHERE eventValue > 10

Continuously calculate the last hour sales by store

SELECT SUM(amount) as salesTotal, storeID

FROM inputChannel [range 60 minutes]

GROUP BY storeID

Calculate the average of the last 2 stock ticks by stock symbol

SELECT AVG(stockPrice) as avgPrice, stockSymbol

FROM inputChannel [PARTITION BY stockSymbol ROWS 2]

GROUP BY stockSymbol

Sample CQL Queries

Filter for events meeting specific

threshold values

Running total of up- to-the-

moment sales by store

Average of the last 2 stock ticks

by symbol


CQL view that joins a cache containing

reference data to an individual streaming

event to provide additional context for further

processing.

Sample CQL Queries

Cache Join Query


Pattern Matching

Powerful concept that allows identification of complex event patterns

Defined as regular expressions

PATTERN (X+ Y+)

1 or more X events … … followed by 1 or

more Y events


CQL view filters events for use by a query.

CQL query that starts when an OFF event

occurs and waits 10 seconds to make sure

that it is not followed by an ON event before

sending the OFF event downstream.

Pattern Match Query


Find passengers stuck in security when their flight reaches “final boarding”.

SELECT

stuck.reservationLocator,

'STUCK' as state

FROM PassengerStateEventChannel MATCH_RECOGNIZE (

PARTITION BY

reservationLocator

MEASURES

Entered.reservationLocator AS reservationLocator

PATTERN (CheckIn Entered NotExited*? Final)

DEFINE

CheckIn AS state = 'CHECKIN',

Entered AS state = 'ENTERED',

NotExited AS state != 'EXITED',

Final AS state = 'FINAL'

) AS stuck

Sample CQL Queries

Find passengers who are stuck in

security when their flight is in the

“FINAL BOARDING” process.


Flight is in final boarding!

Find passengers stuck in

security.

Pattern Matching


Find users with no status update for 4 hours

SELECT

M.userWithNoStatus

FROM StatusEventChannel MATCH_RECOGNIZE (

PARTITION BY

user

MEASURES

user AS userwithNoStatus

INCLUDE TIMER EVENTS

PATTERN (A B)

DURATION 4 HOURS

DEFINE

A AS A.user is not null

) AS M

Sample CQL Queries

Missing Events: Finds users who report

a status, but fail to report another

status during the next 4 hours.


A market leader in security and also the innovation leader in

their industry that continuously offer to its customer’s new Smart

Home products and services.

An inventory and anti-jaming system connected through 3G

network to the alarm reception system, devices send keep Alive

(KA) signal in real-time providing Inventory system data and

manage device status (discovered, available, unavailable, error)

through the reception of the KA events. All this will be

connected to business applications and CRM.

Greater real-time insight into customer security assets and

immediate detection of competitive equipment tampering

ensuring customer retention and satisfaction.

Home security Alarm system equipment disruption monitoring

CO

MP

AN

Y

SO

LU

TIO

N

BE

NE

FIT

S

Deal:

• Oracle Event Processing

• Coherence


Real Time Tracking via

GPS and resource

evaluation in relation to

virtual geographical areas

Real Time Dynamic

definition of virtual

geographical complex

(polygon) areas

Real-Time Spatial Geo-Fencing Solutions


Track school buses on a map

SELECT bus.busId as busId, bus.seq as seq,

com.oracle.cep.cartridge.spatial.Geometry.createPoint(8307, bus.longitude,

bus.latitude) as geom

FROM BusPosStream as bus

Alert when the bus arrives at the bus stop

SELECT systimestamp() as incidentTime, bus.busId as busId, busstop.seq as stopSeq

FROM BusPosGeomStream[NOW] as bus, BusStopRelation as busstop

WHERE CONTAIN@spatial(busstop.geom, bus.geom, 100.0d) = true and bus.busId =

busstop.busId

Sample CQL Queries

Track a bus on a map

Determine when the bus is near

the bus stop


Point-in-Polygon Matching OEP detects entry into a pre-defined area


Nearest Neighbor OEP can find the nearest place of interest (e.g. WiFi Hotspot)


Dynamic Query Generation Business Users Modify Processing in Real-Time

Simple forms can be used to allow users to

add, modify or delete CQL queries at run-

time without stopping the application or

the server allowing existing processing to

continue running while business users

dynamically make modifications.


Dynamic Query Generation


Internet of Things (IoT)


Solution Overview: Oracle Event Processing (OEP)

Architecture designed and Optimized for Java

Enabled Devices/Gateways (197 OSGi Bundles)

High-speed real-time device data capture and

analysis helps mitigate the risk of down-time and

helps with continuous process improvement

Enables local real-time process monitoring to

detect degradation of performance, thus helping

with initiation of alerts or requests for operations

and maintenance actions

Significantly reduces noise to signal ratio of data

reaching the server, enabling savings in

bandwidth and scalability costs

Faster time to market for devices and easy over-

the-air updates post-deployment. OSGi


Oracle Event Processing for Java Embedded

Runs on devices capable of running

Java SE Embedded

Key component of Internet of Things

(IoT)

Smaller, lighter-weight version of the

same OEP product

No need to learn new skills to

program application for devices

OEP


OEP

OEP

OEP

OEP

OEP

OEP in Device to Data Center Deployments Real-time local data analysis for real-world event data

• Upstream nodes

perform basic filtering

and aggregation

• Larger servers

downstream perform

complex combining and

correlation across

multiple streams

• OEP in embedded

devices allows initial

processing to be

handled by less

powerful devices near

the origin/edge.

Event Flow

Edge devices Servers Gateways Enterprise Apps

OEP

OEP

OEP

OEP


Oracle Fast Data for Big Data

Fast Data Big Data Infrastructure

Oracle BI Foundation Suite

Oracle Real-Time Decisions

Endeca Information Discovery

Business Analytics

Oracle Event Processing Oracle Big Data

Connectors

Oracle Data Integrator

Oracle

Advanced

Analytics

Oracle

Database

Oracle

Spatial

& Graph

Apache Flume

Oracle GoldenGate

Oracle

NoSQL

Database

Cloudera

Hadoop

Oracle R

Distribution


1. Big data ≠ Infinite storage Yes, storage is cheap but it helps to have clean data, with context and less redundancy

2. Hadoop is batch-oriented and there is inherent latency "With the paths that go through Hadoop [at Yahoo!], the latency is about fifteen minutes […] it will never be true real-time. " * Raymie Stata, Yahoo! CTO (June 2011)

Some Challenges Working with Big Data

: http://www.theregister.co.uk/2011/06/30/yahoo_hadoop_and_realtime/

minutes


1. Filter out noise (ex: data ticks with no

change), add context (by correlating

multiple sources), increase relevance

1. Identify certain critical conditions as you

insert data into the warehouse

Move time-critical analysis to front of process

Get Ahead of the Curve

Filter out,

correlate

Use Event Processing Techniques


Fast Data Getting Ahead of the Curve

Big Data

minutes ms

Fast Data

His

toric

al d

ep

th: d

ee

p

His

toric

al d

ep

th:

sh

allo

w

Example:

analysis of traffic

patterns and

congestion times

for urban

planning

Example:

monitoring of traffic

cameras to ensure

given license plates

are not in use on

multiple vehicles

Add “depth” to your fast data by

merging output of MapReduce

to stream processing


COMPANY OVERVIEW

• Motorola Solutions connects people through technology • Motorola Solutions serves both enterprise and government

customers with core markets in public safety government agencies and commercial enterprises

• Global presence with 23,000 employees worldwide in 65 countries with sales in over 100 countries

CHALLENGES/OPPORTUNITIES

• Provide a Fast Data intelligence layer for Big data streaming video feeds • Event Processing analyzes Real Time Video meta-data for

distinct patterns of interest to Government agencies and law enforcement

• Centralized Exalogic implementation

PROJECT OBJECTIVES

• Projects worldwide using a reusable scalable

architecture supporting 2800 or more cameras

• Streaming video meta data interfaces with

IOmniscient processing for Face recognition and

License plate monitoring

• Event patterns for duplicate plates within

temporal period beyond distance capabilities

• Speeding analysis with driver recognition

RESULTS

• Ongoing partner relationship delivering

planned projects in Mexico, Oman

• Prototyping successfully completed with live

video feed data

• Foundational technology for Smart Cities

platform solutions

How Much Data Will Humans Create & Store

This Year (2011)? 1.8 zettabytes being created and replicated (Streaming Video)

57.5 billion 32 GB iPads, How much is that? About $34.4 trillion worth.

http://mashable.com/2011/06/28/data-infographic/

Motorola Smart IPVS : Vehicle Capacity & Flow Control

WARNING:

Suspect License

plate – CABO .

MEXICO CITY

WARNING: Driver

Criminal Record




JDeveloper IDE for OEP 12c


JDeveloper IDE for OEP 12c

• Create Event Processing Network • Graphically develop an OEP application

• Drag EPN components to canvas and configure

• Deploy to Configured OEP Servers • Define OEP Servers and Deployment Profiles


Improved Installer

• Installer and Domain Creation

Wizard Improvements

• Mac OS X is supported for

development.


New and Improved Adapters New File Adapter

• Doesn’t require the “Load

Generator Utility”.

• Easily configure all the required

parameters when defining the

adapter in Jdev.


CQL Enhancements Variable Duration For Pattern Match Queries

Duration for the pattern

match query can be

specific to each event.


CQL Pattern Templates Starts the Query Logic for the Developer


List of 12c Enhancements • CQL enhancements

• Spatial Improvements

• Application compiler

• Application and CQL testing

• New and Improved adapters

• Public Cartridge API

• Improved Installer

• Improved Coherence Cartridge

• Support for JDeveloper

• EDN integration

• Business Rules integration

• Performance optimizations


Request

Request

Event Data Event Data

SOA Composite

Instances

OEP Application

SOA

EPN

EDN Integration SOA Composite vs. Event Processing Network


Customer1

$600

Customer1

$600 Customer1

$700

> $1000 per

customer in 24 hours

Customer1

$700

EDN Integration


DEMO


Oracle Event Processing (OEP) Summary High-Volume Low-Latency Event Processing Infrastructure

Event Processing Network (EPN)

Light-weight Java Application Server (embeddable)

Easily Customizable

Integrate with existing infrastructure and other Oracle Products (e.g.

Coherence, Business Activity Monitoring, Database, Big Data

Appliance, Data Mining, Spatial, NoSQL Database etc.)

Time Management & Pattern Matching

Continuously Perform Calculations Over Time Windows

Partition Event Streams By Key Values

Perform Complex Pattern Matching, Detect Missing Events

Adjust Core Business Logic in Real-time without Redeploying


Fast Data Business Integrator and User Tooling

• Fast exploration of Real

Time Streams

• Fast definition of Real

Time Event Patterns

• Fast Testing and

Deployment of Projects

• Fast Export to

Development Environment

for more elaborate

prototyping


Summary: When To Use OEP

1 Business Logic Layer for Event-Driven/Coherence Applications

2 High-Volume Business Activity Monitoring Applications

3 Real-Time Spatial Applications

4 Fast Data: Real-Time Requirements with Big Data Infrastructure

5 High Volume Batch to Real-Time Conversion Projects

6 Internet of Things (IoT): Processing Data On and From Devices

7 Pattern Matching / Missing Events / Alerting / Fraud Detection

soa12c launch 5 event processing shmakov eng cr

Technology

dynamic data event

oracle andor

event b

minutes of data

event c

minutes event

data flow

insteadoracle event