stream’and’complex’eventprocessing’ discovering’exis7ng...
TRANSCRIPT
Stream and Complex Event Processing Discovering Exis7ng Systems: esper G. Cugola E. Della Valle A. Margara Politecnico di Milano Univ. della Svizzera Italiana [email protected] [email protected]
Agenda
• Introduc7on • Describing Events • Event Stream Analysis • Event PaDern Matching • Combina7ons • Resources
2 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on Esper • MoDo • ’s Event Stream and Complex Event Processing soKware turns large volume of disparate real-‐7me event streams into ac7onable intelligence.
• Esper • Event Processing for Java
• Nesper • Event Processing for .Net
Stream & Complex Event Processing -‐ Introduc7on 3
Introduc7on
Esper Architecture
4 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Esper Features at a glance 1/4 • Efficient Event Processing • Con7nuous queries, filtering, aggrega7ons, joins, sub-‐queries
• Comprehensive paDern detec7on • Pull and Push • High performance, low latency
5 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Esper Features at a glance 2/4 • Extensible Middleware • Java, .Net, Array, Map or XML events • Run7me statement management • API or configura7on driven • Plug-‐in SDK for func7ons, aggrega7ons, views and paDern detec7on extensions
• Adapters: CSV, JMS in/out, API, DB, Socket, HTTP • Data distribu7on service for data push management and service layer
6 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Esper Features at a glance 3/4
7 Stream & Complex Event Processing -‐ Introduc7on
• Rich Web-‐Based User Interface • Real-‐7me event displays: Eventlet technology allows customizable and interac7ve con7nuous displays
• CEP engine management • Design EPL Statements • Drill-‐down and browser script • integra7on
Introduc7on
Esper Features at a glance 4/4 • HA enabled (EsperHA) • Per statement configura7on • Transient combinable with fully resilient behaviour
• Hot standby API, hot backup • Highly op7mized and fast data storage technology • Engine state RDBMS storage op7on
8 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on Event Stream and Complex Event Processing
• Design con7nuous queries and complex causality rela7onships between disparate event streams with an expressive Event Processing Language (EPL).
• EPL statements are registered into (N)Esper and con7nuously executed as live data streams are pushed through.
9 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Rapid development and deployments • EPL has a "SQL look alike” • EPL statement matches trigger plain Java or .Net/C# objects for real-‐7me customized ac7onable intelligence.
• (N)Esper is pure Java/.Net and can run standalone or embedded into exis7ng middleware systems (applica7on servers, services bus, in-‐ house systems).
10 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Running Example • Count the number of fires detected using a set of smoke and temperature sensors in the last 10 minutes
• Events • Smoke Event: String sensor, boolean state • Temperature Event: String sensor, double temperature • Fire Event: String sensor, boolean smoke, double temperature
• Condi7on: • Fire: at the same sensor smoke followed by temperature>50
11 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Query Processing Model in Esper 1/2 • The Esper processing model is con7nuous • Four abstrac7ons • Sources
– Push based – Data tuples from sensors, trace files, etc.
• Registered EPL Queries – Push Based – Con7nuously executed against the events produced by the sources
• Listeners – Receive data tuples from queries – Push data tuples to other queries
• Subscribers – Receive processed data tuples
12 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Query Processing Model in Esper 2/2 • Sources, queries, listeners and subscribers are manually connected to form graphs • Sources act as input • Subscribers act as output • EPL Queries integrate sources • Listeners propagates query results (they act internal
sources)
13 Stream & Complex Event Processing -‐ Introduc7on
EPL query
EPL query
source
source
listener
subscriber
Introduc7on
Graph for the running example
14 Stream & Complex Event Processing -‐ Introduc7on
At the same sensor Smoke=true Followed by
Temperature>50
Count(*) in (10 min)
Temperature
Smoke
Fire
subscriber
Describing events
• Possible methods: • Java classes • Maps • XML • EPL
15 Stream & Complex Event Processing -‐ Introduc7on
Describing events in Java
• Java classes are a simple, rich and versa7le way to represent events in Esper. • Follow JavaBeans-‐style geDer methods and property names
16 Stream & Complex Event Processing -‐ Introduc7on
Method Property Name
getQ() q
getQN() qn
Describing events in Java Temperature event for the running example
• Note: in this and in the following examples the 7meStamp property is not necessary
17 Stream & Complex Event Processing -‐ Introduc7on
Describing events in Java Smoke event for the running example
18 Stream & Complex Event Processing -‐ Introduc7on
Describing events in Java Fire event for the running example
19 Stream & Complex Event Processing -‐ Introduc7on
Describing events in EPL Declaring an event type via EPL create schema
• EPL allows declaring an event type via the create schema clause and also by means of the sta7c or run7me configura7on API addEventType func7ons.
• Syntax • create schema schema_name [as] (property_name property_type [,property_name property_type [,...]) [inherits inherited_event_type[, inherited_event_type] [,...]]
20 Stream & Complex Event Processing -‐ Introduc7on
Describing events in EPL Temperature event for the running example
create schema SmokeSensorEvent( sensor string, smoke boolean );
21 Stream & Complex Event Processing -‐ Introduc7on
Describing events in EPL Smoke event for the running example
create schema TemperatureSensorEvent( sensor string, temperature double );
22 Stream & Complex Event Processing -‐ Introduc7on
Describing events in EPL Fire event for the running example
create schema FireComplexEvent( sensor string, smoke boolean, temperature double );
23 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Event Processing Language (EPL) • EPL statements • derive and aggregate informa7on from one or more streams of events,
• to join or merge event streams, and • to feed results from one event stream to subsequent statements.
24 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Event Processing Language (EPL) • EPL is similar to SQL in its use of the select clause and the where clause.
• EPL statements instead of tables use event streams and a concept called views.
• Views are similar to tables in an SQL statement • They define the data available for querying and filtering.
• They can represent windows over a stream of events. • They can also sort events, derive sta7s7cs from event proper7es, group events or handle unique event property values.
25 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
EPL Syntax [insert into insert_into_def] select select_list from stream_def [as name] [, stream_def [as name]] [,...] [where search_condi8ons] [group by grouping_expression_list] [having grouping_search_condi8ons] [output output_specifica8on] [order by order_by_expression_list] [limit num_rows]
26 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Simple examples • Look for specific events • select * from TemperatureSensorEvent where temperature>50
• Aggregate several events • select avg(temperature) from TemperatureSensorEvent
27 Stream & Complex Event Processing -‐ Introduc7on
Introduc7on
Trying out esper online hDp://esper-‐epl-‐tryout.appspot.com/epltryout/mainform.html
28 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Try them out with this sequence of events
SmokeSensorEvent={sensor='S1', smoke=false}
TemperatureSensorEvent={sensor='S1', temperature=30}
t=t.plus(1 seconds)
SmokeSensorEvent={sensor='S1', smoke=true}
TemperatureSensorEvent={sensor='S1', temperature=40}
t=t.plus(1 seconds)
SmokeSensorEvent={sensor='S2', smoke=false}
TemperatureSensorEvent={sensor='S1', temperature=55}
29 Stream & Complex Event Processing -‐ Introduc7on
Advancing 7me manually is only required by the online interface, normally 7me pass by on its own :-‐)
Event Stream Analysis
Windows Type Syntax Descrip:on Logical Sliding win:7me(8me period) Sliding 7me window extending
the specified 7me interval into the past.
Logical Tumbling win:7me_batch(8me period[,op8onal reference point] [, flow control])
Tumbling window that batches events and releases them every specified 7me interval, with flow control op7ons.
Physical Sliding win:length(size) Sliding length window extending the specified number of elements into the past.
Physical Tumbling win:length_batch(size)
Tumbling window that batches events and releases them when a given min-‐ imum number of events has been col-‐ lected.
30 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Logical Sliding windows
31 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Logical Sliding windows: example • Query
select avg(temperature) from TemperatureSensorEvent.win:7me(4 sec)
• Execu7on trace
32 Stream & Complex Event Processing -‐ Introduc7on
Esper, when using logical sliding
windows, reports as soon as a new event
arrives and an old one expires
At: 2001-‐01-‐01 08:00:00.000 Statement: Out Insert
Out-‐output={avg(temperature)=30.0} At: 2001-‐01-‐01 08:00:01.000 Statement: Out
Insert Out-‐output={avg(temperature)=35.0}
At: 2001-‐01-‐01 08:00:02.000 Statement: Out Insert
Out-‐output={avg(temperature)=41.66}
At: 2001-‐01-‐01 08:00:03.000 Statement: Out Insert
Out-‐output={avg(temperature)=45.0} At: 2001-‐01-‐01 08:00:04.000 Statement: Out
Insert Out-‐output={avg(temperature)=50.0}
At: 2001-‐01-‐01 08:00:04.000 Statement: Out Insert
Out-‐output={avg(temperature)=51.25}
Event Stream Analysis
Logical Tumbling windows
33 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Logical Tumbling windows: example • Query
select avg(temperature) from TemperatureSensorEvent.win:7me_batch(4 sec)
• Execu7on trace
34 Stream & Complex Event Processing -‐ Introduc7on
At: 2001-‐01-‐01 08:00:04.000 Statement: Out Insert
Out-‐output={avg(temperature)=45.0}
Esper, when usin
g logical tumbling
windows, reports
only when the
window closes
Event Stream Analysis
Physical sliding windows
35 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Physical sliding windows: example • Query
select avg(temperature) from TemperatureSensorEvent.win:length(5)
• Execu7on trace
36 Stream & Complex Event Processing -‐ Introduc7on
Esper, when usin
g physical
sliding windows,
reports as soon
as a new event ar
rives
At: 2001-‐01-‐01 08:00:00.000 Statement: Out Insert Out-‐output={avg(temperature)=30.0} At: 2001-‐01-‐01 08:00:01.000 Statement: Out Insert Out-‐output={avg(temperature)=35.0} At: 2001-‐01-‐01 08:00:02.000 Statement: Out Insert Out-‐output={avg(temperature)=41.66}
At: 2001-‐01-‐01 08:00:03.000 Statement: Out Insert Out-‐output={avg(temperature)=45.0} At: 2001-‐01-‐01 08:00:04.000 Statement: Out Insert Out-‐output={avg(temperature)=47.0}
Event Stream Analysis
Physical Tumbling windows: example • Query
select avg(temperature) from TemperatureSensorEvent.win:length_batch(5)
• Execu7on trace
37 Stream & Complex Event Processing -‐ Introduc7on
Esper, when usin
g physical
tumbling windows, re
ports only
when the window
closes
At: 2001-‐01-‐01 08:00:04.000 Statement: Out Insert Out-‐output={avg(temperature)=47.0}
Event Stream Analysis
Controlling Repor7ng • The output clause is op7onal in Esper • It is used • To control the rate at which events are output • to suppress output events.
• Syntax • Output [[all | first | last | snapshot] every output_rate [seconds | events]]
38 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Controlling Repor7ng: examples • Controlling the sliding in logical and physical windows • select avg(temperature) from TemperatureSensorEvent.win:7me(4 sec) output snapshot every 2 sec
• select avg(temperature) from TemperatureSensorEvent.win:length(4) output snapshot every 2 events
39 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching
• Event paDerns match when an event or mul7ple events occur that match the paDern's defini7on.
• PaDerns can also be temporal (7me-‐based). • PaDern matching is implemented via state machines.
40 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern atoms • Filter expressions specify an event to look for. • TemperatureEventStream(sensor="S0", temperature>50)
• Time-‐based event observers specify 7me intervals or 7me schedules. • 7mer:interval(10 seconds) • 7mer:at(5, *, *, *, *)
41 Stream & Complex Event Processing -‐ Introduc7on
Every 5 minutes
Event PaDern Matching Types of operators • Operators that control paDern finder crea7on and termina7on: every, every-‐dis8nct, [num] and un8l
• Logical operators: and, or, not • Temporal operators that operate on event order: -‐> (followed-‐by)
• Guards are where-‐condi7ons that filter out events and cause termina7on of the paDern finder, such as 8mer:within, 8mer:withinmax and while-‐expression
• Note: PaDern expressions can be nested arbitrarily deep by including the nested expression(s) in () round parenthesis.
42 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern example • Query
select a.sensor from paDern [every ( a = SmokeSensorEvent(smoke=true) -‐> TemperatureSensorEvent(temperature>50, sensor=a.sensor) where 7mer:within(2 sec) ) ]
• Execu7on trace At: 2001-‐01-‐01 08:00:02.000 Statement: Out Insert Out-‐output={a.sensor='S1'}
43 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern operators: every • The every operator indicates that the paDern sub-‐expression should restart when the sub-‐expression qualified by the every keyword evaluates to true or false.
• Without the every operator the paDern sub-‐expression stops when the paDern sub-‐expression evaluates to true or false
• Every 7me a paDern sub-‐expression within an every operator turns true the engine starts a new ac7ve sub-‐expression looking for more event(s) or 7ming condi7ons that match the paDern sub-‐expression.
44 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern operators: every • This paDern fires when encountering an A event and then stops looking: • A
• This paDern keeps firing when encountering A events, and doesn't stop looking: • every A
45 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern operators: every ( A -‐> B ) • Events • A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• PaDern • every ( A -‐> B )
• Results • Detect an A event followed by a B event. At the 7me when B occurs the paDern matches, then the paDern matcher restarts and looks for the next A event. 1. Matches on B1 for combina7on {A1 , B1} 2. Matches on B3 for combina7on {A2 , B3} 3. Matches on B4 for combina7on {A4 , B4}
46 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern operators: every A -‐> B • Events • A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• PaDern • every A -‐> B
• Results • The paDern fires for every A event followed by a B event. 1. Matches on B1 for combina7on {A1 , B1} 2. Matches on B3 for combina7on {A2 , B3} and {A3 , B3} 3. Matches on B4 for combina7on {A4 , B4}
47 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern operators: A -‐> every B • Events • A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• PaDern • A -‐> every B
• Results • The paDern fires for an A event followed by every B event. 1. Matches on B1 for combina7on {A1 , B1} 2. Matches on B2 for combina7on {A1 , B2} 3. Matches on B3 for combina7on {A1 , B3} 4. Matches on B4 for combina7on {A1 , B4}
48 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching PaDern operators: every A -‐> every B • Events • A1 B1 C1 B2 A2 D1 A3 B3 E1 A4 F1 B4
• PaDern • every A -‐> every B
• Results • The paDern fires for every A event followed by every B event. 1. Matches on B1 for combina7on {A1 , B1} 2. Matches on B2 for combina7on {A1 , B2} 3. Matches on B3 for combina7on {A1 , B3}, {A2 , B3} and {A3 , B3} 4. Matches on B4 for combina7on {A1 , B4}, {A2 , B4}, {A3 , B4}
and {A4 , B4}
49 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching Limi7ng sub-‐expression life7me 1/3 • As the introduc7on of the every operator states, the operator starts new sub-‐expression instances and can cause mul7ple matches to occur for a single arriving event.
• New sub-‐expressions also take a very small amount of system resources and thereby your applica7on should carefully consider when sub-‐expressions must end when designing paDerns. Use the 8mer:within construct and the and not constructs to end ac7ve sub-‐expressions.
• Note: the data window onto a paDern stream does not serve to limit paDern sub-‐expression life7me.
50 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching Limi7ng sub-‐expression life7me 2/3 • Events • A1 A2 B1
• PaDern • every A -‐> B
• Results • {A1 , B1} and {A2 , B1}
• Events • A1 A2 B1
• PaDern • every A -‐> (B and not A)
• Results • {A2 , B1} • The and not operators cause the sub-‐expression looking for {A1, B?} to end when A2 arrives.
51 Stream & Complex Event Processing -‐ Introduc7on
Event PaDern Matching Limi7ng sub-‐expression life7me 3/3 • Events
• A1 received at to+ 1 sec
• A2 received at to+ 3 sec• B1 received at to+ 4 sec
• PaDern • every A -‐> B
• Results • {A1 , B1} and {A2 , B1}
• Events • A1 received at to+ 1 sec
• A2 received at to+ 2 sec• B1 received at to+ 3 sec
• PaDern • every A -‐> (B where 7mer:within(2 sec) )
• Results • {A2 , B1} • The where 8mer:within operators cause
the sub-‐expression looking for {A1, B?} to end aKer 2 seconds.
52 Stream & Complex Event Processing -‐ Introduc7on
Chaining Event PaDern Matching and Stream Analysis Example • The insert into clause forwards events to other streams for further downstream processing.
• Query insert into FireComplexEvent select a.sensor as sensor, a.smoke as smoke, b.temperature as temperature from paDern [every ( a = SmokeSensorEvent(smoke=true) -‐> b = TemperatureSensorEvent(temperature>5, sensor=a.sensor) where 7mer:within(2 sec) ) ] ;
• Downstream query select count(*) from FireComplexEvent .win:7me(10 sec)
53 Stream & Complex Event Processing -‐ Introduc7on
Chaining Event PaDern Matching and Stream Analysis Example: execu7on trace At: 2001-‐01-‐01 08:00:01.000 Statement: Stmt-‐4 Insert FireComplexEvent={sensor='S1', smoke=true, temperature=40.0} Statement: Out Insert Out-‐output={count(*)=1}
54 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
The EPL alone is not enough … Configura7on cepConfig = new Configura7on(); cepConfig.addEventType("TemperatureSensorEvent", TemperatureSensorEvent.class.getName()); cepConfig.addEventType("SmokeSensoEvent", SmokeSensorEvent.class.getName()); String query = "<<a query>>"; EPServiceProvider cep = EPServiceProviderManager.getProvider("myCEP", cepConfig); EPRun7me cepRT = cep.getEPRun7me(); EPAdministrator cepAdm = cep.getEPAdministrator(); EPStatement cepStatement = cepAdm.createEPL(query); cepStatement.addListener(new CEPListener()); See also HelloWorldEsper in the Esper ready to go pack on the course Website
55 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Listening to EPL query results 1/3 The interface for listeners is com.espertech.esper.client.UpdateListener. Implementa7ons must provide a single update method that the engine invokes when results become available
56 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Listening to EPL query results 2/3 • The engine provides statement results to update listeners by placing results in com.espertech.esper.client.EventBean instances. A typical listener implementa7on queries the EventBean instances via geDer methods to obtain the statement-‐generated results.
57 Stream & Complex Event Processing -‐ Introduc7on
Event Stream Analysis
Listening to EPL query results 3/3 For instance, the following code prints each new event received public class CEPListener implements UpdateListener { public void update(EventBean[] newData, EventBean[] oldData) { for (EventBean e : newData) { System.out.println("Event received: " + e.getUnderlying()); } } } NOTE: similar code can be used to access the events that are exi7ng the window (oldData), see also next slides.
58 Stream & Complex Event Processing -‐ Introduc7on
Resources • Download Esper (for Java) • hDp://www.espertech.com/esper/download.php
• Download Nesper (for .net) • hDp://www.espertech.com/esper/nesper_download.php
• Quick start • hDp://www.espertech.com/esper/quickstart.php
• Tutorial • hDp://www.espertech.com/esper/tutorial.php
• Ques7ons on EPL • hDp://www.espertech.com/esper/solu7on_paDerns.php
• Documenta7on • hDp://www.espertech.com/esper/documenta7on.php
59 Stream & Complex Event Processing -‐ Introduc7on
Acknowledges
• Large part of the content of are taken from • EsperTech: “Event Stream Intelligence Con7nuous Event Processing for the Right Time Enterprise Products Data Sheet”
• EsperTech: "Reference Documenta7on Version: 4.2.0"
60 Stream & Complex Event Processing -‐ Introduc7on