WSO2 Complex Event Processor 3.0.0

An overview of upcoming features

By S. Suhothayan

Associate Technical Lead, Team Lead CEP.

About WSO2

•  Providing the only complete open source componentized cloud platform –  Dedicated to removing all the stumbling blocks to enterprise agility –  Enabling you to focus on business logic and business value

•  Recognized by leading analyst firms as visionaries and leaders –  Gartner cites WSO2 as visionaries in all 3 categories of application

infrastructure –  Forrester places WSO2 in top 2 for API Management

•  Global corporation with offices in USA, UK & Sri Lanka

–  200+ employees and growing

•  Business model of selling comprehensive support & maintenance for our products

150+ globally positioned support customers

Outline

!  Scenarios of Event Processing !  WSO2 CEP Server & SOA integrates !  The Siddhi Runtime CEP Engine. !  High availability, Persistence and Scalability of

WSO2 CEP !  How CEP can be combined with Business

Activity Monitoring (BAM). !  Demo

CEP Is & Is NOT!

!  Is NOT! o  Simple filters

•  Simple Event Processing •  E.g. Is this a gold or platinum customer?

o  Joining multiple event streams •  Event Stream Processing

!  Is ! o  Processing multiple event streams o  Identify meaningful patterns among streams o  Using temporal windows

•  E.g. Notify if there is a 10% increase in overall trading activity AND the average price of commodities has fallen 2% in the last 4 hours

WSO2 CEP Server

!  Enterprise grade server for CEP runtimes !  Supports several transports (network access) !  Supports several data formats !  Support for multiple CEP runtimes !  Governance !  Monitoring !  Tools (WSO2 Dev Studio)

WSO2 CEP Architecture

Siddhi CEP Runtime !  Apache License, a java library, Tuple based event model !  Supports distributed processing !  Supports multiple query models

•  Based on a SQL-like language •  Supports

!  Partitions !  Filters !  Windows !  Joins !  Ordering !  Output Rate Limiting !  and others

CEP Event Adaptors

!  Is an adaptor for receiving and publishing events !  Has the configurations to connect to external endpoints !  Its many-to-many with CEP engine

CEP Event Adaptors Support for several transports (network access) and data formats ●  SOAP/WS-Eventing

XML messages ●  REST

JSON messages ●  JMS

Map messages XML messages Text messages JSON messages

●  SMTP (Email) Text messages JSON messages XML messages

●  Thrift - WSO2 data format High Performant Event Capturing & Delivery Framework supports Java/C/C++/C# via Thrift language bindings

WSO2 Event

CEP Event Adaptors ●  Cassandra (from CEP 3.0.0)

Map messages ●  Fix (from CEP 3.0.0+)

Map messages ●  MYSQL (from CEP 3.0.0)

Map messages ●  HBase (from CEP 3.0.0+)

Map messages

& Event adaptors are pluggable !

CEP Event Builders

•  Event builder converts different input event types to a type compatible with the execution plan.

•  Subscribes to an Input Event Adaptor to listen for events and sends a converted WSO2 Event or Basic Event to the Execution Plan

•  Receives events in different formats and exposes those input streams as stream definitions

•  Has a one to many relationship with execution plans in execution plan.

CEP Execution Plan

●  Is an isolated logical execution unit

●  Each execution plan has a set of Queries Input & Output stream mappings.

●  Its one-to-one with a CEP Backend Runtime Engine

●  It deals with Siddhi processing engine.

CEP Event Formatter

Event formatter does the inverse – listens to events coming from event processor and sends converted events to Event adaptors.

There are 5 types of output mapping types are available

●  Map ●  Text ●  WSO2Event ●  XML ●  JSON

Monitoring (Event Tracer & Event Statistics) !  Provides real-time statistical visual illustrations of

request & response counts per time based on CEP server, execution plan, transport adaptor, event builder and formatter.

Writing CEP Queries (using

Siddhi Runtime)

Siddhi Queries !  Filters and Projection !  Windows

o  Events are processed within temporal windows. (e.g. for aggregation and joins)

Time window vs. length window. !  Joins - Join two streams !  Event ordering - Identify event sequences and

patterns !  Event Partitions !  Event Tables

Filters

!  Filters the events by conditions, use to detect simple condition

!  Conditions o  >, <, = , <=, <=, != o  contains, instanceof o  and, or, not

!  Example

from <stream-name> [<conditions>]* select <attributes>

insert into <stream-name>

from cseEventStream[price >= 20 and symbol==’IBM’] select symbol, volume insert into StockQuote

Window from <stream-name> [<conditions>]#window.<window-name>(<parameters>)

select <attributes> Insert into <stream-name>

Types of Windows ●  (Time | Length) (Sliding| Batch) windows ●  Type of aggregate functions ●  sum, avg, max, min

Example from cseEventStream[price >= 20]#window.lengthBatch(50) select symbol, avg(price) as avgPrice group by symbol having avgPrice>50 insert into StockQuote

Join

!  Use to join two streams based on a condition. There must be at least one window defined

!  Unidirectional – event arriving only to the unidirectional stream triggers join

!  Example

from <stream>#<window> [unidirectional] join <stream>#<window> on <condition> within <time>

insert into <stream>

from TickEvent[symbol==’IBM’]#window.length(2000) join NewsEvent#window.time(5 min)

on TickEvent.symbol==NewsEvent.company select * insert into JoinStream *

Pattern

!  Use to Check condition A happen before/after condition B.

!  Can do iterative checks via “every” keyword. !  Here with “within <time>”, SIddhi emits only events

that are within that time of each other !  Example

from [every] <condition> → [every] <condition> … <condition> within <time> select <attributes> insert into StockQuote

from every (a1 = purchase[price < 10] ) -> a2 = purchase [price >10000 and a1.cardNo==a2.cardNo] within 1 day select a1.cardNo as cardNo, a2.price as price, a2.place as place insert into potentialFraud

a1 x1 k5 a2 n7 y1

Sequence

!  Regular Expressions supported o  * - Zero or more matches (reluctant). o  + - One or more matches (reluctant). o  ? - Zero or one match (reluctant). o  or – either event

!  Here we have to refer events returned by * , + using square brackets to access a specific occurrence of that event

from <event-regular-expression> within <time> select <attributes> Insert into <stream>

from a1 = requestOrder[action == "buy"], b1 = cseEventStream[price > a1.price and symbol==a1.symbol]+, b2 = cseEventStream[price <b1.price] select a1. symbol as symbol, b1[0].price as firstPrice, b2.price as orderPrice insert into purchaseOrder

a1 b1 b1 b2 n7 y1

Event Paritions

Partition types can be one of two types •  Variable Partitions - Partitions are created by

the discrete values that are encountered for a variable

define partition StockSymbol by StockStream.symbol

define <partition-id> by <partition-type> (,<partition-type>)*

•  Range partitions - Partitions are created according to predefined ranges of variables

define partition stockVolume by range volume < 10 as 'SMALL', range volume > 10 and volume < 100 as 'MEDIUM', range volume > 100 as 'LARGE'

Event Tables define table <table-name> (<attribute-name> <type> {, <attribute-name> <type>}*) ( from <table-type>.<datasource-name>:<database-name>.<table-name>)?

Event tables can be used in the same manner as an event stream, with the difference being that events sent to an event table being persisted to a data source. CEP supports event tables for

•  In Memory •  Relational

o  MySQL o  H2

define table cseEventTable(symbol string, price int, volume float) from MYSQL.cepDataSource:cepdb.cepEventTable0

Working with Event Tables

from <stream> (select <attribute-name> (,<attribute-name>)* )? insert into <table-name>

Inserts the selected attributes from the input stream into the event table.

from cseEventCheckStream[symbol==cseEventTable.symbol in cseEventTable] insert into outStream;

For update and delete

from <stream> update <table-name> (on <condition>)?

from <stream> delete <table-name> (on <condition>)?

 

!  We compared Siddhi with Esper, the widely used opensource CEP engine

!  For evaluation, we did setup different queries using both systems, push events in to the system, and measure the time till all of them are processed.

!  We used Intel(R) Xeon(R) X3440 @2.53GHz , 4 cores 8M cache 8GB RAM running Debian 2.6.32-5-amd64 Kernel

Performance Results

 

Simple filter without window

Performance sending event within same JVM

from StockTick[prize >6] return symbol, price

 

State machine query for pattern matching

Performance sending event within same JVM

From f=FraudWarningEvent ->

p=PINChangeEvent(accountNumber=f.accountNumber) return accountNumber;

 

Performance Sending Events over the network !  Here we publihsed data from two client publisher

nodes to the CEP Sever node and sent the triggered notifications of CEP to a client subscriber node.

!  To test the worsecase sinario, 100% of the data published to CEP is recived at the subscriber node after processing (No data is filtered)

!  We used Intel® Core™ i7-2630QM CPU @ 2.00GHz, 8 cores, 8GB RAM running Ubnthu 12.04, 3.2.0-32-generic Kernel, for running CEP and used Intel® Core™ i3-2350M CPU @ 2.30GHz, 4 cores, 4GB RAM running Ubnthu 12.04, 3.2.0-32-generic Kernel, for the three client nodes.

HA/ Persistence !  Ability to recover

runtime state in the case of a failure.

!  Enables queries to span lifetimes much greater than server uptime.

!  Takes periodic snapshots and stores all state information to a scalable persistence store (Apache Cassandra).

!  Supports pluggable persistent stores.

Scaling !  Vertically scaling

o  Can be distributed as a pipeline !  Horizontally scaling

o  Queries like windows, patterns, and Join have shared states, hence hard to distribute!

o  Use distributed cache (Hazelcast) to achieve this •  shared memory and batch processing

Event Recording

!  Ability to record all/some of the events for future processing

!  Few options o  Publish them to Cassandra cluster using WSO2 data

bridge API or BAM (can process data in Cassandra with Hadoop using WSO2 BAM).

o Write them to distributed cache o  Custom thrift based event recorder

Integration with WSO2 BAM

Data Receiving Data Analyzing Data Presentation

Data Publishing

CEP Role within WSO2 Platform

DEMO

Scenario !  Monitoring stock exchange for game changing

moments !  Two input event streams.

o  Event stream of Stock Quotes from a stock exchange

o  Event stream of word count on various company names from twitter pages

!  Check whether the last traded price of the stock has changed significantly(by 2%) within last minute, and people are twitting about that company (> 10) within last minute

Input events !  Input events are JMS Maps

o  Stock Exchange Stream Map<String, Object> map1 = new HashMap<String, Object>(); map1.put("symbol", "MSFT"); map1.put("price", 26.36); publisher.publish("AllStockQuotes", map1);

o  Twitter Stream Map<String, Object> map1 = new HashMap<String, Object>(); map1.put("company", "MSFT"); map1.put("wordCount", 8); publisher.publish("TwitterFeed", map1);

Queries

Queries from allStockQuotes[win.time(60000)] select symbol,price, avg(price) as averagePrice

group by symbol having ((price > averagePrice*1.02) or (averagePrice*0.98 > price )) insert into fastMovingStockQuotes

from twitterFeed[win.time(60000)] select company as company, sum(wordCount) as words group by company having (words > 10)

insert into highFrequentTweets

from fastMovingStockQuotes[win.time(60000)] as fastMovingStockQuotes join highFrequentTweets[win.time(60000)] as highFrequentTweets on fastMovingStockQuotes.symbol==highFrequentTweets.company

select fastMovingStockQuotes.symbol as company, fastMovingStockQuotes.averagePrice as amount, highFrequentTweets.words as words

insert into predictedStockQuotes

Alert !  As a Email Hi

Within last minute, people being twitting about {company} {words} times, and the last traded price of {company} has changed by 2% and now being trading at ${amount}.

From CEP

!  As a SMS

Useful links !  WSO2 CEP 3.0.0

http://ec2-54-224-94-128.compute-1.amazonaws.com/chunk-02/N-25_09_2013/wso2cep-3.0.0.zip

!  WSO2 CEP http://wso2.com/products/complex-event-processor/

!  CEP Performance Info http://srinathsview.blogspot.com/2013/08/cep-performance-processing-100k-to.html

!  Distributed Processing Sample With Siddhi CEP and ActiveMQ JMS Broker. http://suhothayan.blogspot.com/2012/08/distributed-processing-sample-for-wso2.html

!  Creating Custom Data Publishers to BAM/CEP

http://wso2.org/library/articles/2012/07/creating-custom-agents-publish-events-bamcep

!  WSO2 BAM 2.3.0 http://wso2.com/products/business-activity-monitor/

Engage with WSO2

•  Helping you get the most out of your deployments •  From project evaluation and inception to development

and going into production, WSO2 is your partner in ensuring 100% project success

Questions?

Thank you.