systemübergreifende prozessüberwachung mit event stream processing

Systemübergreifende Prozessüberwachung mit Event Stream ProcessingThomas Dücker, SAP SchweizJuni, 2013

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 2

Agenda

Herausforderung und Trends

Übersicht Operational Process Intelligence Übersicht Beispiele Szenario

Mehrwert für Sie

Weitere Informationen


Event Stream Processing

Capture Disseminate & Decide Response & Action

ESP


Traditional Approaches to Event Stream Processing

• DB application (OLTP)— Pull not push (i.e. not event driven) or— Requires triggers – performance degradation / maintenance nightmare— latency in seconds

• Custom application (C/C++ / Java)— Specialized, high development / maintenance cost— Slow to change, unresponsive to the business

Applicationor

Dashboard

High-SpeedData

Queries /Results

Database Application

High-SpeedData

“Black-box”Application

Alerts /Actions

Custom Application


The SAP Approach – Sybase ESP

•Combines advantages of customized application with ease of visual data flow diagramming or SQL•Incoming data is processed as it arrives, according to the model rules and operations•Publish streaming results to other models, apps, message bus, dashboard, etc.

?

Input Streams PUBLISH

Market Data

Orders, Trades

ConsoleInput

Reporting Tools

Trading Systems

Visual / SQLAuthoring

Reference Data

SAP HANA / Sybase IQ

Dashboards

Message Bus

Output StreamsSUBSCRIBE


CEP Product Convergence

Aleri 3.2

Sybase ESP 5.0

Sybase ESP 5.1

Sybase CEP R4

Coral8 v5.6.5

Sept 2011

Sept 2012

Optional

(maintenance only)

Sybase ESP 5.1 SP1Dec 2012


Advantages of Sybase ESP Approach

Rapid Application Development•Reduce/eliminate dependence on specialist

programming skills for process•Cut implementation/deployment time•Broad out of the box connectivity

Non-intrusive Deployment•Event-driven integration with existing systems•Unify existing disparate data models

Analyze Events as they occur•Continuous Insight•Respond Immediately


Event Stream Processing Scenarios

Capital MarketsReact to incoming ticks to determine trading strategies, monitor the success of the strategies by calculating PnL in real-time; monitor risk in real-time.

RetailProvide clickstream analysis for online customer assistance / alternatives, use web scraping to react to competitor price changes



EnergyMonitor and optimize energy usage using Smart Metering to ensure QoS

Utilities / ManufacturingService Status Monitoring and Process Control using SCADA / OSISoft



TelecomsMonitor events for QoS, use DPI to detect fraudulent use

Police / Homeland SecurityANPR event correlation for traffic monitoring, security

Click icon to add pictureClick icon to add picture

ESP Architektur


ESP Project Architecture

Publish Subscribe

Data Streams

Authoring Tools

Adapters

I/O G

ateway

Security

Security

Adapters

Stream Processor+ Multi-threaded+ 64 bit - Suse 11 / Redhat 6 / Solaris 10 (Sparc / i86) Win x64+ Low latency+ Optional persistence

C+C(XML-RPC)

Operational Console

Security (PAM, SSL, RSA, Kerberos)

Data Streams

MemStore

LogStore

I/O G

ateway

Data Streams Data Streams

LogStore Command & Control Studio


Out of the Box Connectivity

In Process

• JMS, MQ, TIBCO• ATOM (RSS)• TCP Sockets• Databases via JDBC /

ODBC• Files• MS Excel RT• SMTP (out)

External

• Market Data(Reuters, Bloomberg, IDC, ACTIV, NYSE Tech)

• FIX• OpenAdaptor Java

Framework• Replication Server• IQ / HANA

PubSub SDK

• C/C++• Java• .NET


ESP Cluster Architecture I

Controller

Manager

Manager

Manager

Manager

Project Server

Controller Controller

ManagerNodes Cache

Project Server Project Server


• Manager Node − Deploys project(s) to server(s) − Maintains heartbeat with project(s) to detect failure− Manages failover− In multi node, managers are run in a cache so can failover to manage other

controller nodes

• Controller Node− Basically acts as a launch daemon for Manager to start project(s) in a Container

ESP Cluster Architecture II


• Single node cluster – − Single manager/controller node runs multiple projects on a single machine− Detects when a project fails and tries to restart it− This is the default for project development

• Multi node cluster − Multiple equi-peer managers and controllers run multiple projects on multiple machines− Used to scale out projects across multiple machines− Allows for server failure – promotes failover recovery and data redundancy − Allows for Highly Available “active:active” deployment of projects within a Data Centre

ESP Cluster Architecture III


• Three clusters created by install script: -

• Studio Cluster (single node) on 9786 : – − Used for Studio Development − Can run multiple projects − Located in $ESP_HOME/studio/clustercfg

• User-defined Cluster (single node) on 19011− Located in $ESP_HOME/cluster/nodes/node1

• Example multi-node Cluster on 19011 - 19014− Located in $ESP_HOME/cluster/examples

ESP Installed Clusters

ESP Key Concepts


• Analyst-level skills (Excel, VBA)• Easy to understand complex models• No need to learn language syntax

• Rapid programming• Easy to use language (CCL – derived from

SQL)• Modular, project based approach

Visual Dataflow Authoring CCL Authoring

Authoring Options – Eclipse Plugin


Visual Dataflow Authoring


CCL Authoring


Procedural Logic - SPLASH


IMPORT 'CreateCompleteChannel.ccl';----------CREATE MODULE CreateCompleteChannelIN MessageIn, CacheInOUT MessageOutBEGIN

// Module CodeEND;

LOAD MODULE CreateCompleteChannel AS Channel_AIN MessageIn = MessageA, CacheIn = CacheAOUT MessageOut = MessageOutA;

LOAD MODULE CreateCompleteChannel AS Channel_BIN MessageIn = MessageB, CacheIn = CacheBOUT MessageOut = MessageOutB;

ESP Modularity


Project Code, Compiled Code & Project Resources

• Project Source Code .CCL (Continuous Computation Language)− Used for schema, stream, window, method definitions compiled into

• Project Executable .CCX (Continuous Computation eXecutable)

• Project Resource File .CCR (Continuous Computation Resource)− An xml file used for in process adaptor, cluster, bindings, failover, HA definitions –

allows for easy migration between DEV, TEST and PROD


Constructing ESP Projects

•Build model

•Connect to data feeds using publish adaptor

•Connect to output devices via subscribe adaptor

ESP Project Container

PositionsConsole

Trades

Tick data feedD

B R

eplicated Data


• Events are retained as records in memory with a primary key so that further events may modify the state of the record - either update or delete it.

• A stateful event must have PK attribute(s) set.• A stateful event MUST have an associated operation code (opcode), one of INSERT, UPDATE, UPSERT,

DELETE. If no opcode is provided, ESP defaults it to UPSERT. • When an event occurs it applies the opcode through the data flow model depending on the opcode.

An insert in a source stream may become an update to an aggregate or join stream.• The full set of unique (insert) events to be retained MUST fit in memory.• Records in memory can be removed either by setting a retention window, or by publishing a DELETE

event for that key.• In ESP, stateful events are implemented using a WINDOW

Stateful Events


• An event is retained only while it is being processed. When an event moves from one stream to another it is removed from the first stream so it cannot be updated.

• All events are treated as insert.• A stateless event ignores any PK attribute(s) and always outputs an insert • In ESP stateless events are implemented with a STREAM

Stateless Events


ESP implements stateful and stateless events using 3 element types• WINDOW : used for stateful events. A WINDOW maintains state for a user defined period•STREAM : used for stateless events. A STREAM does not maintain or react to state•DELTA : used to process stateful events, but without maintaining state

• All can be INPUT, LOCAL, OUTPUT• INPUT subscribe to input adaptors i.e. events are published to an INPUT. They are the entry points

for events to enter an ESP project. • LOCAL perform processing logic. They can NOT be published or subscribed to• OUTPUT are derived elements which perform processing logic which CAN be subscribed to

ESP Elements


• WINDOWS must have a PK set• Can have an optional retention period set• Maintain state in memory or persisted to disk. i.e. State can be recovered in the event of system

failure• Can publish to STREAM, WINDOW or DELTA• Only a WINDOW can be used for aggregation• At lease one side of a join MUST be a WINDOW (unless using SPLASH). The side which maintains

state depends on the type of the join.• Use WINDOW when developing models as they help debugging

Windows (stateful)


• Stream• Cannot be persisted so any data is lost in the event of system failure• Can NOT publish to DELTA stream• Can publish to a WINDOW, but WINDOW must aggregate the input• Delta• Can only be LOCAL, OUTPUT• A stateless element (a STREAM) that understands and processes state (a WINDOW)• Can only use a WINDOW as a source • Must have a PK set• As it does not store events it can NOT be used for aggregation.• Cannot be persisted so any data is lost in the event of system failure

Streams (stateless)


Example: Simple Position Maintenance

AverageTrades Window

Symbol*Last PriceWeighted Avg PriceLast Time

Aggregate

Positions By Book Stream

BookId*Symbol*Current PositionAverage Position

Join

Trades Stream

Symbol*Time*PriceShares

Position Window

BookId*Symbol*Shares Held

Total Book Window

BookId*Current PositionAverage Position

Aggregate


MSFT 12:05:23 32.45 200

IBM 12:05:24 57.87 5000

CSCO 12:06:01 20.04 100

AveragePrices Window

Symbol*Last PriceWeighted Avg PriceLast Time

Aggregate

Individual Positions Stream

BookId*Symbol*Current PositionAverage Position

Join

Position Window

BookId*Symbol*Shares Held

Book Positions Window

BookId*Current PositionAverage Position

Aggregate

Trades Stream

Symbol*Time*PriceShares

MSFT 12:06:02 36.55 900

MSFT 32.45 33.01 12:05:23

IBM 57.87 56.90 12:05:24

CSCO 20.04 17.00 12:06:01

MSFT 36.55 34.00 12:06:02

IBM 57.87 56.90 12:05:24

CSCO 20.04 17.00 12:06:01

Book1 MSFT 3000

Book1 IBM 700

Book2 MSFT 1000

Book1 MSFT 97350 99030

Book1 IBM 40509 39830

Book2 MSFT 32450 33010

Book1 139539 138860

Book2 32450 33010

Book1 MSFT 109650 102000

Book1 IBM 40509 39830

Book2 MSFT 36550 34000

Book1 150159 141830

Book2 36550 34000

Example: Simple Position Maintenance

ESP Solution Architektur


Event Processing Solution Architecture

• Events don’t happen in isolation

• Most Event Stream Scenarios need access to historic data• Pattern Detection• Risk Calculation• Optimisation algorithm• To ensure sub millisecond response times use in memory e.g. HANA

• Many Event Stream Scenarios need access to master data• Data written directly to OLTP system• Updates to reference, semi-static data• For sub second updates without polling use log trawler e.g. Replication Server


In-database Algorithms

Analytical data

Historic data

Reference data

Real-time ANPR Alerting Framework

Event Stream Processor

CEP Engine

Inpu

t A

dapt

er

Output

Adapter

In-Memory

HANA

SQL

SQL

SQL

Monitoring

BusinessObjects DataService

ExternalDatabases

Business Inteligence

BO Predictive AnalysisPredictive models(Ad-hoc analysis)

OCR Radars


Real-time Retailing FrameworkES

PH

AN

A

Detect Event

Validate Event

Check Results Output

Run Optimization Algorithm

Manage Exception

Phase 1:Event Detection & Validation

Phase 2:Processing

Phase 3:Validation & Exception

Phase 4:Output

Straight Through Processing

Is competitor price<95% of our

price?

Is optimum price<95% of current our price?

GU

I


Integration with Hana I

• Generic input / output ODBC adapter• Data can enter via a source stream


Integration with Hana II

• Can be called from a FlexStream using SPLASH for event driven query: -

ON TRADE { if (TRADE.Price <> optimalPrice) { [ string s; | ] rec; vector (typeof(rec) v := new vector(typeof(rec)); getData(v,'HanaDSN','CALL DPTL.sp_price_opt (?, ?, ?, ?)', TRADE.Id, TRADE.Datetime, TRADE.Sector, TRADE.Price); if (not (isnull(v))) { newrec := v[0]; } // rest of method }}


Trading SystemsTrading Systems

Showing:• Concentration limits• Clustering• Outliers• Etc…

Trading SystemsMarket Data

Trading SystemsRisk

Sensitivities

Bus

• Cleansing• Normalisation• Enrichment

Real-time Aggregation of Risk and P&L

CEP

Continual Mark t o Market

HANA

• Scenario Shifts• VAR calculations

Analytics:• Heatmaps• Drill-downs• Reports

By:• Instrument• Industry• Country• Debt Rating• Counterparty• VaR Time Horizon• Etc…

Transformation of granular data

Real-time Risk Framework

ASE

• Reference Data

Replication Server


Oil & Gas - Energy Efficiency Reporting Framework

ESP (Event Stream Processor)

ECC (IS-U)

OKONon SAP (top level MES)

Oracle

Oracle

BO 4.0

SOI2 server

Panopticon

RS(Replication

Server)

HANA

RA#1

RA#2

SOI2

Replication flowEvent flowSCADA SCADA SCADA

RSHANA ODBC

systemübergreifende prozessüberwachung mit event stream processing

Documents

sap affiliate company

sap schweizjuni

statefulstateless event

discardedstateful event

sqlincoming data

sybase cep r4coral8

trading strategies

realtime monitor risk