Download - SQL Server 2008 R2 StreamInsight
SQL Server 2008 R2 StreamInsight
Complex Event ProcessingEvent Stream Processing
Who Am I?
• SQL Server MVP• SQL Server Consultant• Joint author on Wrox Professional SSIS book• www.SQLDTS.com and www.SQLIS.com• Specialise in Moving Data• @allanSQLIS (twitter)
Agenda
• Use Cases & Challenges• Formulating Declarative Queries• Windows in Time• Event Flow Debugging• Demos
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
request
response
Eventoutput streaminput
stream
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
request
response
Eventoutput streaminput
stream
What is CEP?
Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec > Tens of thousands of events/sec
request
response
Eventoutput streaminput
stream
What is CEP?
Who might need CEP
• Fraud Detection• Real-Time Trade Risking• Algorithmic Trading/Betting• Meter throughputs– Oil, Gas, Water, Electricity– Use to drive alarms, alerts etc
Standing Queries
Query Logic
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock ticker, news feeds Event stores & Databases
Pagers &Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
StreamInsight Application at Runtime
.NETC#
LINQStreamInsight Application Development
InputAdapters
OutputAdapters
StreamInsight Engine
Query Logic
Query Logic
StreamInsight Platform
1
2 34
Query Expressiveness
• Selection of events (filter)• Calculations on the payload (project)• Correlation of streams (join)• Stream partitioning (group and apply)• Aggregation (sum, count, …) over event
windows• Ranking over event windows (topK)
Query Expressiveness
var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 };
• Projection
Query Expressiveness
var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 };
• Projection• Filter
Query Expressiveness
var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w };
• Projection• Filter• Correlation (Join)
Query Expressiveness
var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) };
• Projection• Filter• Correlation (Join)• Aggregation over windows
• Projection• Filter• Correlation (Join)• Aggregation over windows• Group and Aggregate
Query Expressiveness
var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };
Windowing
• Hopping– Tumbling
• Snapshot• Count
Time Windows
Time
Hopping Window
Time Windows
Time
Tumbling Window
Snapshot Windows
Snapshot Windows
Debugger
• Has its own debugger• You cannot live without it (believe me)• Set breakpoints• Filter
• A wonderful tool
DEMOS
•Tour•Trace Reader Live•Trace File Reader•Query Demo (Joining)•Debugger Tool (How it will save you)