towards unified and native enrichment in event processing systems

© Copyright 2010 Digital Enterprise Research Institute. All rights reserved.

Digital Enterprise Research Institute www.deri.ie

Towards Unified and Native Enrichment in

Event Processing Systems Souleiman Hasan, Sean O’Riain, Edward Curry

Digital Enterprise Research Institute, National University of Ireland, Galway, Ireland

In proceedings of The he 7th ACM International Conference on Distributed Event-Based Systems June 29 - July 3, 2013, Arlington, Texas, USA

[email protected] http://www.StefanDecker.org/


Talk Overview

n  Introduction ¨  IoT, Cyber-Physical Systems

¨  Event Incompleteness

n  Current Approaches n  Proposed Approach

¨  Challenges for Enrichment

¨  Proposed Model

¨  Implications

n  Linked Data Instantiation ¨  Evaluation

n  Summary and Future Directions

2/30


Big Data & IoT


Cyber-Physical Systems

Smart City Smart Grid

Smart Building Smart Enterprise


Event Processing Systems

n  Three dimensions of decoupling

n  Removal of explicit dependencies between event producers and consumers ð  Scalable deployment

n  Information exchange only by Events

n  (Eugster et al., 2003)

Space

Time

Synch Producer Consumer

5/30


Problem – Event Incompleteness

n  Event producers and consumers are decoupled ð  Event producers may have very little knowledge about

consumers information needs

Environmental Sensors (Event Observers/Producers)

{(type= "energy consumption") and (floor= “first floor") and (consumption="high")}

(type, "energy consumption”)

(device, "heater x”)

(consumption, "high”) Event Processing Engine

Business User

6/30


Dimensions of Incompleteness

n  Event Format: lacks syntactical structure ¨  E.g. plain text against conjunctive subscription

n  Event Semantics: events lack a reference scheme ¨  E.g. schema-less tuples

n  Lack of Background Knowledge ¨  E.g. complementary information exists in external DB

n  Incompleteness Addressable by Transformation ¨  E.g. transforming amounts of multiple measurement units

n  Temporal Segmentation ¨  E.g. Complementary information exists in past or future

events

7/30


Current Approaches

8/30


Event Enrichment



(consumption, "high”)




(room, “202e”)

(floor, “second floor”)

9/30

DEVICE ROOM FLOOR Color

heater x 202e second floor white

heater y 313 third floor blue

Meta Data

IoT Heater Event

Enriched Event


n  Dedicated agents to complement events

Agent-based Event Enrichment

Producer

Producer

Rule1

Event Processing Agent

Enricher

Enricher

Consumer

10/30


n  Pros ¨ Events complete with respect to consumer’s need

¨ Low false positives/negatives rate

n  Cons ¨ Ad-hoc and external to event processing engines

¨ Difficult to develop and maintain enrichment logic

¨ Difficult to optimise enrichment process

Agent-based Event Enrichment

11/30


Proposed Approach

12/30


Proposed Approach

n  We need ¨ Event enrichment to be integrated into the event

processing paradigm as a core task of event processing engines

n  Proposal ¨ Unified declarative language for event processing

and enrichment

¨ Enrichment element as a declarative specification for engine to enrich events with complementary information items

13/30


Unified and Native Enrichment


Challenges for Enrichment

1.  Determination of Enrichment Source (ES) ¨ E.g. BMS relational database via a connection

string “Server=www.example.com/rdbms;Database=BMS-DB;”.

2.  Retrieval of Information Items from the Enrichment Source ¨ E.g. SQL query against a query interface

3.  Finding the Complementary Information for an Event in the Enrichment Source ¨ E.g. recursively join on the ID attribute in the

LOCATIONS table and query for n times.

15/30


4.  Fusion of Complementary Information with the Event ¨  E.g.

¨  E.g.

Challenges for Enrichment







(room, “202e”)

(floor, “second floor”)







(location, “202e, second floor”)

16/30


Produce or Consumer-side

n  Producer side enrichment ¨ Producers can provide enrichment elements with

events

¨  Identify enrichment source and mechanism

n  Consumer side enrichment ¨ Have better knowledge of completeness from

their perspective

¨ Proposed here as a unified element with the subscription matching element

17/30


¨ Enrichment Element – ENRICH FROM identify the enrichment source(s)

– RETRIEVE BY specifies retrieval mechanism for atomic information items

– FIND BY dictate retrieval of information items from the enrichment source(s)

– FUSE BY define fusion approach to integrate retrieved data

¨ Matching Element – as in current matching languages

Unified Subscription

18/30


Formal Model

ES e

MVS(U)

U

HVS(U)

A1 A2

B1

A3

AB1

B2 B3

AB2 AB3

W=e ES∩The universe U, the event e, the enrichment source ES, the world W, the enrichment view HVS, and a matching view MVS

19/30


Formal Model

n  Successful Enrichment ¨ Completeness

n  Minimal Successfully Enriched Event ¨ Precision

n  Approximately Minimal Successfully Enriched Event ¨ Cost to turn an enriched event into a minimal

successfully enriched event

(More details on formal definition in paper)

20/30


Implications

n  Sharing & Re-usability of Enrichment Elements ¨ Expert consumers can share enrichment knowledge

with less informed consumers

n  Distribution of Enrichment ¨ Distribute enrichment process across nodes to

achieve an optimal overall completeness

n  Approximation in Event Processing Engines ¨ Matching over partially complete events would need

to account for still missing information

21 of 30


Linked Data Instantiation

22/30



23/30

Event Model Enrichment Source Model Enrichment source is accessible by dereferencing URIs associated with it as per Principles of Linked Data



n  Unified Subscription Language ENRICH FROM http://www.myenterprise.org/devices

RETRIEVE BY ‘DEREF’

FIND BY ‘Spreading Activation’

‘UniformWeightsAllAdjacent’

FUSE BY ‘UNION’

{?event rdf:type ont:EnergyConsumption.

?event (?p){3} building:SecondFloor.}

24/30


Evaluation Dataset

n  English Dbpedia (1st of August 2012) n  Constructed from instances of class

dbpedia-owl:Event

n  Total of 24,000 events ¨  “Football Match”, “Race”, “Music Festival”, “Space Mission”,

“Election”, “10thcentury BC Conflicts”, “Academic Conferences”, “Aviation Accidents And Incidents In 2001”, etc

n  Each event contains one triple in the form: ¨  <eventURI, rdf:type, dbpedia-owl:Event>.

n  Enrichment Source: Whole Dbpedia dataset ¨  Potnetially 250 million triples for enrichment

25/30


Evaluation – FIND BY

n  FIND BY - Spreading Activation Strategies

n  UniformWeightsAllAdjacent ¨ Activation spreads equally to all adjacent nodes

n  UniformWeightsRandomAdjacent ¨ Activation spreads equally to random set of

adjacent nodes

n  DifferentWeightsSemRel ¨ Activation spreads based on semantic relatedness

to terms in matching element of subscription

26/30


Evaluation – Matching Elements

n  Path-shaped graph used to generate matching elements

Matching Element 1 ?event rdf:type dbpedia-owl:Event.

?event (?p){1} bpedia:England_national_football_team

2 ?event rdf:type dbpedia-owl:Event. ?event (?p){2} dbpedia:Queens_Park_Rangers_F.C..

3 ?event rdf:type dbpedia-owl:Event. ?event (?p){3}

dbpedia: Loftus_Road. 4 ?event rdf:type dbpedia-owl:Event.

?event (?p){4} dbpedia: Fulham_F.C..

27/30


n  SemRel (unified approach) performs best

n  Less effective with complex subscriptions

n  Completeness and precision not weighted equally

Enrichment Results

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

1 2 3 4

F5Score

Subscription

UniformWeightsAllAdjacents UniformWeightsRandomAdjacentsDifferentWeightsSemRel

28/30


Summary and Future Directions

29/30


Summary and Future Directions

n  Event enrichment can be integrated as a core task of event processing engines ¨ Unified enrichment logic with event subscription

logic

¨ Native enricher tackles incompleteness before matching

n  Future Work ¨ Formalizing the enrichment language element

¨ Leverage commonalities among multiple consumers for enrichment optimization

30 of 30


Hasan, S., O’Riain, S., and Curry, E. 2013. “Towards Unified and Native Enrichment in Event Processing Systems,” In 7th ACM International Conference on Distributed Event-Based Systems (DEBS 2013), Arlington, Texas, USA: ACM, pp. 171–182. http://www.edwardcurry.org/publications/DEBS2013_Enrichment.pdf

Further Reading

towards unified and native enrichment in event processing systems

Technology