event processing and stream reasoning with etalis

FZI F

ORS

CHUN

GSZ

ENTR

UMIN

FORM

ATIK

Event Processing and Stream Reasoning with ETALISDarko Anicic

AIFB Graduiertenkolloquium, KIT

AIFB Graduate Colloquium, KIT

07.04.2023 © FZI Forschungszentrum Informatik 2

MOTIVATIONEvent Processing & Stream Reasoning


Real Time Information Processing

Amount of generated digitalised information

Business pressure to detect relevant RT situations

Time to react

Ensure high throughput computing

Ensure timeliness and low-latency

Leave more time for appropriate reactions

Event Processing


Shifting Event Processing Toward More Intelligent Event Processing

EP

iEP


Shifting Reasoning Toward Stream Reasoning

Reasoning

Stream Reasoning


Scenario: Finding a Path Between Two Cities

Deductive reasoning can solve the

problem


Scenario: Traffic Monitoring

Event Processing solves the problem


Scenario: Finding a Path in Real Time

Stream Reasoning solves the problem


INTRODUCTIONPreliminaries & Related Work


Event Processing

Event-driven interactions vs. request-response interactions Asynchronous interactions vs. synchronous interactions Information Push vs. Information Pull Events as a means to declare changes

the principle of decoupling

With respect to Logic Programming...

An event is defined as an occurrence within a particular system or domain. It is something that has happened, or is contemplated as having happened in that domain [Etzion and Niblett, EPIA’10].

Event Processing is computing that performs operations on events. Common event processing operations include reading, creating, transforming, and deleting events [Etzion and Niblett, EPIA’10].


Overview of ETALIS Approach

ETALIS Language

Execution Model

EPN in ETALIS

Retraction in EP

Out-of-Order EP

EP-SPARQL

Implementation Evaluation ETALIS in Use

Practical Considerations

ETALIS Extensions

ETALIS Foundation


Related Work

Active Databases

Event Processing Systems

Logic-Based Approaches

Retraction in EP

Out-of-Order EP

Semantic-Based Approaches

• HiPAC [McCarthy et al.’89]• Ode [Gehani et al.‘92]• SAMOS [Gatziu et al.‘92]• Snoop [Chakravarthy et

al.‘94]• Sentinel [Chakravarthy‘97]• SnoopIB [Adaikkalavan et

al.‘06]

• TelegraphCQ [Chandrasekaran et al.’03]

• Amit [Adi et al.’04]• Cayuga [Demers et al.’07]• CEDR [Barga et al.’07]• Semantics of sliding windows

[Krämer et al.’09]• ZStream [Mei et al.’10]• SASE [Wu et al.’06,

Gyllstrom et al.’08, Agrawal et al.’08]

• Esper, Coral8, StreamSQL, CCL etc.

• Truth Maintenance System [Doyle’78 and ‘79]

• Borealis [Carney et al.’02]• Replay-based revision

[Maskey et al.’02]• Consistent streaming [Barga

et al.’02]

• Event stream processing with out-of-order data arrival [Li et al.’07]

• Consistent streaming [Barga et al.’02]

• Sequence Pattern Query Processing over Out-of-Order Event Streams [Liu et al.’09]

• Speculative out-of-order events [Brito al.’09]

• XChangeEQ [Bry et al.’07]• Homogenous reaction rules

[Paschke et al.’07]• Statelog [Lausen et al.’96]• ECA rules with process

algebras [Behrends et al.’96]• Incremental Reasoning on

Streams and Rich Background Knowledge [Barbieri et al.’10]

• Prova [Kozlenkov et al.’06]

• Introducing time into RDF [Gutierrez et al.’07]

• SPARQL-ST [Perry et al.’07]• Temporal SPARQL [Tappolet

et al.’09]• stSPARQL [Koubarakis et

al.’10]• C-SPARQL [Barbieri et al.’10]• Streaming Knowledge Bases

[Walavalkar et al.’08]• Streaming SPARQL [Bolles

et al.’08] • Incremental Reasoning on

Streams and Rich Background Knowledge [Barbieri et al.’10]


Research Questions

Can we devise a uniform formalism to formally express both, complex event patterns and background knowledge for EP and SR?

How to effectively use logic inference to derive complex events in a timely fashion (in an event-driven fashion)?

By realising EP with concepts from LP, can we detect more real time situations that are otherwise undetectable with sole EP?

Would an LP approach for EP be extensible enough for specific requirements found in EP?

Do we need to compromise on performance, to get in return detections based on events patterns and background knowledge?



ETALIS Language

Execution Model

EPN in ETALIS

Retraction in EP

Out-of-Order EP

EP-SPARQL



ETALIS Extensions

ETALIS Foundation


Design Principles and Requirements (I)

Formal declarative semantics Patterns describe what situations need to be detected, and do not

specify possible ways of detecting them, the order of execution etc. Point-based vs. interval-based temporal semantics

Interval-based semantics enables richer semantics possible inconsistencies encountered with point-based events, e.g., e1

before (e2 before e3) Seamless integration of events with queries

Databases are often used in enriching events with additional data Support for query processing (including recursive queries)

Seamless integration of events with domain knowledge To enable reasoning about events and knowledge Derivation of implicit information in order to propose recommendations,

or to accomplish event classification, clustering, filtering


Role of Logic in Event Processing

Declarative semantics to ground well defined behaviour of event-based systems

On-the-fly adaptive: everything is data (patterns can be as easy changed and adapted as data)

Justifications: why did an event occur? Why didn’t it occur? Reasoning about events (over time, space, context, their relations

and constraints): Contradicting complex events/situations; Detection of not yet fulfilled complex patterns (e.g., 80% fulfilled event); Event retraction (revision) and out-of-order events.

Detection of complex events, states, situations of interest, and further controlling reactive behaviour (actions/reactions) triggered by detected events;

Pattern rule management: consistency checking, minimal set of pattern rules, correctness of pattern rules etc.


Design Principles and Requirements (II)

Event-driven incremental reasoning Events derived in timely fashion and in the asynchronous push mode

Expressivity Support for various event processing agents Support for various other features for event-driven applications

Set at a time vs. event at a time processing Computation is performed whenever a relevant event occurs

Simplicity and ease-of-use Rules: data extraction, event composition (event hierarchies), temporal

and causal relations, aggregations, non-monotonic features Extensibility

Capability to support extensions


ETALIS Language for Events - Syntax

Event rule is defined as a formula of the following shape:

where p is an event pattern containing all variables occurring in

A predicate name with arity n, t(i) denotes terms

t is a term of type boolean

q is a nonnegative rational number

BIN: SEQ, AND, PAR, OR, EQUALS, MEETS, STARTS, or FINISHES


ETALIS: Interval-based Semantics


ETALIS Language for Events - Semantics


Complexity Properties depend on the conditions put on the formalism’s signature

EXPTIME-complete, without further restrictions [Dantsin et al. (2001)] The formalism is decidable and tractable if both C, and the arity of functions

and predicates, is bounded

Function free Horn logic is hard for PTIME Function symbol “materialisation” can be done in polynomial time There are polynomially many static ground atoms There are polynomially many event ground atoms to be possible entailed

ELE: Complexity Properties

Dantsin, E., Eiter, T., Gottlob, G., & Voronkov, A. (2001). Complexity and expressive power of logic programming. ACM Computing Surveys, 33(3), 374–425.


ETALIS: Operational Semantics (SEQ)

a SEQ b SEQ c → ce1

((a SEQ b) SEQ c) → ce1

3. Binarization

2. Decoupling

a SEQ b → ie

ie SEQ c → ce1

4. Event-driven backward chaining (EDBC) rules

1. Complex event pattern


Order of Rule Execution: SEQ & AND

c SEQ b → ab SEQ a → ie

c SEQ b → ab AND a → ie

OP1 OP2

c b

iea


Properties of EDBC Rules

A simple model: two-input intermediate goals (events) Goals are automatically asserted by rules as relevant events occur Goals are persisted over a period of time “waiting” to support

detection of a more complex goal Goals are unique Only goals useful w.r.t the given patterns are computed Rules are executed backwards, but they exhibit a forward chaining

behaviour During rule evaluation, no backtracking occurs


An Example


Examples of Iterative and Aggregative Patterns

The k-fold sequential execution of an event a:

A length-based window of size n:

A sum over an unbound event stream until a threshold value is met:


Event Processing Network

Source: O. Etzion und P. Niblett. Event Processing in Action

07.04.2023 28

Event Processing Agents

Event Processing

Agent

Filter Transformation

Translate

Enrich

Project

Aggregate Split Compose

Pattern detection

Source: O. Etzion und P. Niblett. Event Processing in Action


Event Filtering in ETALIS Language



ETALIS Language

Execution Model

EPN in ETALIS

Retraction in EP

Out-of-Order EP

EP-SPARQL



ETALIS Extensions

ETALIS Foundation


Event Retractions in ETALIS

Events are often assumed to be immutable and therefore always correct

In some situations however revisions are required: an event was reported by mistake, but did not happen in reality an event was triggered and later revoked due to a transaction failure

Events used for transaction monitoring and auditing may be retracted when a transaction fails.

As recognised in [Ryvkina et al. ICDE’06], event stream sources may issue revision tuples that amend previously issued events

ETALIS takes revision tuples into account and produce correct revision outputs.

ETALIS: Operational Semantics (rSEQ)

Standard EDBC rules for SEQ

Additional EDBC rules for retraction

Rules to save ie1 and enable its retraction


Processing Out-of-Order Events

Events are often assumed to be totally ordered Delays caused by network latencies, sensor and machine failures Delayed events are known as out-of-order events

ce1 ← stock(Agent1, “GO”, Pr1,Vol1) SEQ stock(Agent2, “GO”, Pr2,Vol2) WHERE Pr1*1.20<Pr2.ce2 ← stock(Agent1, “MS”, Pr1,Vol1) SEQ stock(Agent2, “MS”, Pr2,Vol2) WHERE Pr1>1.20*Pr2.

Missing complex events due to out-of-order stream: stock(agent1, “GO”, 100,10) SEQ stock(agent2, “GO”, 125,10);

False positives complex event ce2 due to an out-of-order event.


ETALIS: Operational Semantics (outSEQ)

Additional EDBC rules for out-of-order


EP-SPARQL: Toward Real-Time Semantic Web

Event Processing (EP)

hand

les

Rapidly changing datarepresented as events

Semantic Web technologies including

hand

les

Static or slowly evolving background knowledge

EP SPARQL EP-SPARQL

• Temporal relatedness• Semantic relatedness• Stream reasoning


EP-SPARQL - Syntax

Operators: FILTER, AND, UNION, OPTIONAL, SEQ, EQUALS, OPTIONALSEQ, and EQUALSOPTIONAL

getDURATION() yields a literal of type xsd:duration giving the time interval associated to the graph pattern

getSTARTTIME() and getENDTIME() retrieve the time stamps of type xsd:dateTime of the start and end of the interval;

Extends SPARQL to enable event-based processing by taking into account temporal situatedness of triple assertions.

Syntactical and semantic downward-compatibility to plain SPARQL.


EP-SPARQL - Semantics


EP-SPARQL Example: Traffic Monitoring



ETALIS Language

Execution Model

EPN in ETALIS

Retraction in EP

Out-of-Order EP

EP-SPARQL



ETALIS Extensions

ETALIS Foundation

ETALIS: System Diagram

Parser

Execution

Compiler

Auxiliary components

EP-SPARQL: System Diagram

Parser

Compiler

Execution

RDFS Parser and Compiler


ETALIS Interfaces


Performance Evaluation - Settings

Intel Core Quad CPU Q9400 2,66GHz, 8GB of RAM running Windows Vista x64

SWI Prolog version 5.6.64 YAP Prolog version 5.1.3 Esper 3.3.0 All tested engines ran in a single dedicated CPU core Output generated from all tests is validated Data sets:

Stream generator creates time series data with probabilistic values Streams with stock data from Google Finance and Yahoo Finance sensor readings from the National Data Buoy Center (NDBC) subclass relations computed with the Ethan Plants ontology to explore routes in Milan we use the Milan ontology GeoNames ontologies to identify important geographic locations

affected by weather observations detected in our use case


Common Operators I

25 50 75 1000

5000

10000

15000

20000

25000

30000

35000

Esper 3.3.0 P-SWI P-YAP

Event stream size x 1000

25 50 75 1000

5000

10000

15000

20000

25000

30000

Esper 3.3.0 P - SWI P - Yap


Test patterns:


Common Operators II

Test patterns:

25 50 75 1000

5000100001500020000250003000035000400004500050000

Esper 3.3.0 P-SWI P-Yap


2.5 5 7.5 100

100020003000400050006000700080009000

10000

Esper 3.3.0 P-SWI P-Yap



Iterative Event Patterns

Test patterns:

Supply chainpaths between 100

and 5000Check the path from the beginning or from

the last event


Extensions I

Test patterns:

SEQ AND PAR OR0

5000

10000

15000

20000

25000

30000

35000

Revision Flag off Revision Flag on

Operator

Th

rou

gh

pu

t (1

00

0 x

Eve

nts

/Se

c)

Retraction in EP with ETALIS

0.0

10.0

20.0

30.0

40.0

50.0

0% 10% 20% 33%

Percentage of out-of-order

Thro

ughp

ut (1

000

x Ev

ents

/Sec

)

Out of order In order

Out-of-order EP with ETALIS


Scenario: Stream Reasoning Evaluation

A Goods Delivery system in the city of Milan An agent delivers goods to a certain location While visiting a location, the system “listens” to

traffic events related to the next location Inaccessible routes are recomputed on-the-fly

5 10 15 200

200

400

600

800

1000

1200

1400

1 Visitor 10 Visitors

Number of locations

Co

ns

um

ed

tim

e in

ms

5 10 15 200

200400600800

1000120014001600

1 Visitor 10 Visitors

Number of locations

Co

ns

um

ed

Me

mo

ry in

kB


ETALIS in Use

• Event Processing in Action, by Opher Etzion and Peter Niblett

• An EPN implemented in ETALIS

The Fast Flower Delivery in EPIA

• Collaborative work on drug design • ETALIS: extending SOA with EDA

The Drug Discovery in SYNERGY

• MesoWest sensor network• Analysing sensor data over

time and geographical space

On the Live Measurements of

Environmental Phenomena


SUMMARYConclusions and Outlook



ETALIS Language

Execution Model

EPN in ETALIS

Retraction in EP

Out-of-Order EP

EP-SPARQL



ETALIS Extensions

ETALIS Foundation RR’10, RuleML’11(I), REDS’10, AAIJ’11

RuleML’11(II), PADL’11, WWW’11

SWJ’11


Research Questions

Can we devise a uniform formalism to formally express both, complex event patterns and background knowledge for EP and SR? ETALIS Language for Events

How to effectively use logic inference to derive complex events in a timely fashion (in an event-driven fashion)? EDBC rules

By realising EP with concepts from LP, can we detect more real time situations that are otherwise undetectable with sole EP? The approach pays off when background knowledge is evolving

Would an LP approach for EP be extensible enough for specific requirements found in EP? Retraction in EP, out-of-order EP, EP-SPARQL

Do we need to compromise on performance, to get in return detections based on events patterns and background knowledge? It depends how big and complex background knowledge is


Outlook: Event-Driven Dynamic Processes


Thank you for your attention!

References:

Anicic et al. Real-Time Complex Event Recognition and Reasoning – A Logic Programming Approach, the Applied Artificial Intelligence journal, Special Issue on Event Recognition. 2011.

Anicic et al. Stream Reasoning and Complex Event Processing in ETALIS, Semantic Web Journal, Special Issue: Semantic Web Tools and Systems. 2011.

Anicic et al. Retractable Complex Event Processing and Stream Reasoning, RuleML 2011.

Anicic et al. ETALIS: Rule-Based Reasoning in Event Processing, Chapter in Reasoning in Event-based Distributed Systems, 2010. Springer.

Anicic et al. A Declarative Framework for Matching Iterative and Aggregative Patterns against Event Streams, RuleML 2011.

Anicic et al. Event-driven Approach for Logic-based Complex Event Processing, The IEEE International Conference on Computational Science and Engineering 2009.

Anicic et al. EP-SPARQL: A Unified Language for Event Processing and Stream Reasoning, WWW 2011.

Anicic et al. A Rule-Based Language for Complex Event Processing and Reasoning, RR 2010

Fodor, Anicic, Rudolph. Results on Out-of-Order Event Processing, PADL 2011.

Xu, N.Stojanovic, Lj.Stojanovic, Anicic, Studer. An approach for more efficient energy consumption based on real-time situational awareness, ESWC 2011.

N.Stojanovic, Milenovic, Xu, Lj.Stojanovic, Anicic, Studer. An intelligent event-driven approach for efficient energy consumption in commercial buildings: smart office use case, DEBS 2011.

Apostolou, Stojanovic, Anicic. Responsive Knowledge Management for Public Administration: an Event-Driven Approach, IEEE Intelligent Systems, Special Issue: Transforming E-government & E-participation. 2009.

event processing and stream reasoning with etalis

Technology