context-aware event stream analytics · worcester polytechnic institute acknowledgement 35...
TRANSCRIPT
![Page 1: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/1.jpg)
CAESAR: Context-Aware Event Stream
Analytics in Real time
Olga Poppe, Chuan Lei,
Elke A. Rundensteiner, and Dan Dougherty
March 18, 2016
1
![Page 2: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/2.jpg)
Worcester Polytechnic Institute
CEP engine
Complex Event Processing
The same workload of independent event queriesis continuously evaluated
2
𝑄1, 𝑄2, 𝑄3Primitive events Complex events
![Page 3: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/3.jpg)
Worcester Polytechnic Institute
Application Context
• Event compositions signify application contexts
• Most event queries are appropriate only in certain contexts
• They can be safely suspended otherwise
Examples of application contexts:
• Emergency management: normal, crowded, fire
• Health care: safe, warning, violation
• Algorithmic trading: hold, buy, sell
• Financial fraud: approved, suspicious, fraud
3
![Page 4: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/4.jpg)
Worcester Polytechnic Institute
Traffic Management Use Case
4
• 140 hours idling in traffic due to congestion in 10-worstU.S. traffic corridors per year [The Wall Street Journal]
• Health cost of $18 billion due to traffic noise and pollutionin the USA's 83 largest urban areas in 2010 [USA Today]
• 1.24 million deaths due to traffic injuries worldwide in2010 [Wikipedia]
![Page 5: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/5.jpg)
Worcester Polytechnic Institute
Traffic Management Contexts
5
Accident Congestion Clear
Goal is to leverage application contexts to speed up system responsiveness
Accident warning
Route re-computation
Toll notification
Route re-computation
Statistics
Local services
![Page 6: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/6.jpg)
Worcester Polytechnic Institute
Challenges
• Rich semantics
─ Complex conditions implying a context
─ Unknown and unbounded context duration
─ Multiple inter-dependent event queries
• Readable specification
• Real time responsiveness
6
![Page 7: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/7.jpg)
Worcester Polytechnic Institute
State-of-the-art Approaches
7
CEP Systems(Esper,
StreamInsight)
CAESARBusinessModels
(BPMN, UML)
Expressive event
queries
Application contexts
Context-aware
optimizations
![Page 8: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/8.jpg)
Worcester Polytechnic Institute
Contributions & Outline
8
CAESAR system:
• Graphical model
• Context-aware algebra
• Context-driven optimization techniques
• Execution infrastructure
Performance evaluation
![Page 9: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/9.jpg)
Worcester Polytechnic Institute
Outline
CAESAR Model
9
![Page 10: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/10.jpg)
Worcester Polytechnic Institute
Context-aware Event Stream Analytics
10
![Page 11: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/11.jpg)
Worcester Polytechnic Institute11
Context-aware Event Stream Analytics
![Page 12: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/12.jpg)
Worcester Polytechnic Institute12
Context-aware Event Stream Analytics
![Page 13: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/13.jpg)
Worcester Polytechnic Institute13
Application Contexts
![Page 14: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/14.jpg)
Worcester Polytechnic Institute14
Context Deriving Queries
![Page 15: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/15.jpg)
Worcester Polytechnic Institute15
Context Processing Queries
![Page 16: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/16.jpg)
Worcester Polytechnic Institute
Context-aware Event Queries
16
![Page 17: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/17.jpg)
Worcester Polytechnic Institute
Outline
CAESAR Algebra
17
![Page 18: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/18.jpg)
Worcester Polytechnic Institute
Context-preserving Plan Generation
18
![Page 19: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/19.jpg)
Worcester Polytechnic Institute
CAESAR Algebra Operators
1. Context initiation 𝐶𝐼c 𝐼,𝑊
2. Context termination 𝐶𝑇c 𝐼,𝑊
3. Context window 𝐶𝑊𝑐 𝐼,𝑊
4. Filter 𝐹𝐼𝜃(𝐼)
5. Projection 𝑃𝑅𝐴,𝐸(𝐼)
6. Event pattern 𝑃(𝐼)
19
![Page 20: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/20.jpg)
Worcester Polytechnic Institute
Runtime Context Maintenance
20
Context bit vector 𝑊:Context types:
Time stamp 𝑊. 𝑡𝑖𝑚𝑒
0 1 0 0 1 0 0 0 0 0
𝑐a, cb, … cz
• Updated by the context initiation & termination operators
• Accessed by the context window operator
• Synchronized by the time driven scheduler
![Page 21: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/21.jpg)
Worcester Polytechnic Institute21
Translation from Query Set to Algebra Plan
DERIVE Toll(c.id, c.sec, 5)
PATTERN NewCar c
CONTEXT congestion
DERIVE NewCar(s.id, s.xway, s.dir, s.seg, s.lane, s.pos, s.lane)
PATTERN SEQ(NOT Position f, Position s)
WHERE f.sec+30=s.sec AND
f.id=s.id AND
f.lane≠′ exit′
CONTEXT congestion
![Page 22: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/22.jpg)
Worcester Polytechnic Institute
Outline
CAESAR Optimizer
22
![Page 23: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/23.jpg)
Worcester Polytechnic Institute
CAESAR Optimizer Overview
23
Problem statement:
Given a workload of context-aware event queries,our optimization problem is to find an optimized queryplan for this workload with minimal CPU cost.
Context-aware optimization techniques:
• Context window push down strategy• Context workload sharing algorithm
![Page 24: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/24.jpg)
Worcester Polytechnic Institute
Context Window Push Down Strategy
24
Performance benefits:• Suspension of irrelevant operators• Context-driven stream routing
![Page 25: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/25.jpg)
Worcester Polytechnic Institute
Context Workload Sharing Algorithm
25
![Page 26: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/26.jpg)
Worcester Polytechnic Institute
Context Workload Sharing Algorithm
26
![Page 27: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/27.jpg)
Worcester Polytechnic Institute
Context Workload Sharing Algorithm
27
![Page 28: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/28.jpg)
Worcester Polytechnic Institute
Outline
CAESAR Infrastructure
& Experiments
28
![Page 29: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/29.jpg)
Worcester Polytechnic Institute
CAESAR Architecture
29
![Page 30: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/30.jpg)
Worcester Polytechnic Institute
Experimental Setup
30
Execution infrastructure:
Java 7, 1 Linux machine with 16-core
3.4 GHz CPU and 48GB of RAM
Data sets:
• Linear Road stream benchmark (LR) [1]
3 roads=1.7GB
• Physical Activity Monitoring real data set (PAM) [2]
1.6GB
[1] A.Arasu et al., Linear Road: A stream data management benchmark. VLDB’04[2] A.Reiss et al., Creating and benchmarking a new data set for physical activity monitoring. PETRA’12
![Page 31: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/31.jpg)
Worcester Polytechnic Institute
Context-aware Event Stream Analytics
31
For 7 roads, context-aware (CA) event stream analytics is 9-fold faster than context-independent (CI) approach.
![Page 32: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/32.jpg)
Worcester Polytechnic Institute
Context-aware Event Query Sharing
32
If 30 context windows of length 15 minutes process 4 event queries each and overlap by 15 minutes, workload sharing wins 6-fold.
![Page 33: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/33.jpg)
Worcester Polytechnic Institute
Outline
Conclusions
33
![Page 34: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/34.jpg)
Worcester Polytechnic Institute
Conclusions
34
• CAESAR is first context-aware CEP system
• Graphical context-specification model
• Context-aware algebra
• Context-driven optimization techniques
• Execution infrastructure
• 8-fold speed up on average
![Page 35: Context-aware Event Stream Analytics · Worcester Polytechnic Institute Acknowledgement 35 •Advisors: Elke A. Rundensteiner, Dan Dougherty •Collaborator: Chuan Lei •DSRG group](https://reader030.vdocuments.site/reader030/viewer/2022040901/5e7150f25884e842a77b8ff8/html5/thumbnails/35.jpg)
Worcester Polytechnic Institute
Acknowledgement
35
• Advisors: Elke A. Rundensteiner, Dan Dougherty
• Collaborator: Chuan Lei
• DSRG group at WPI
• EDBT reviewers
• NSF grants IIS 1018443 and IIS 1343620