SQL Server 2008 R2 StreamInsight

Complex Event ProcessingEvent Stream Processing

Who Am I?

• SQL Server MVP• SQL Server Consultant• Joint author on Wrox Professional SSIS book• www.SQLDTS.com and www.SQLIS.com• Specialise in Moving Data• @allanSQLIS (twitter)

Agenda

• Use Cases & Challenges• Formulating Declarative Queries• Windows in Time• Event Flow Debugging• Demos

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

request

response

Eventoutput streaminput

stream

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

request

response

Eventoutput streaminput

stream

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

Data Rate Hundreds of events/sec > Tens of thousands of events/sec

request

response

Eventoutput streaminput

stream

What is CEP?

Who might need CEP

• Fraud Detection• Real-Time Trade Risking• Algorithmic Trading/Betting• Meter throughputs– Oil, Gas, Water, Electricity– Use to drive alarms, alerts etc

Standing Queries

Query Logic

Event sources Event targets

`

Devices, Sensors

Web servers

Event stores & Databases

Stock ticker, news feeds Event stores & Databases

Pagers &Monitoring devices

KPI Dashboards, SharePoint UI

Trading stations

StreamInsight Application at Runtime

.NETC#

LINQStreamInsight Application Development

InputAdapters

OutputAdapters

StreamInsight Engine

Query Logic

Query Logic

StreamInsight Platform

1

2 34

Query Expressiveness

• Selection of events (filter)• Calculations on the payload (project)• Correlation of streams (join)• Stream partitioning (group and apply)• Aggregation (sum, count, …) over event

windows• Ranking over event windows (topK)

Query Expressiveness

var result = from e in inputStream select new { id = e.id, W = (double)e.intW / 10 };

• Projection

Query Expressiveness

var result = from e in inputStream where e.id > 3 select new { id = e.id, W = (double)e.intW / 10 };

• Projection• Filter

Query Expressiveness

var result = from eLeft in inputStream1 join eRight in inputStream2 on eLeft.id equals eRight.id select new { id = eLeft.id, diff = eLeft.W - eRight.w };

• Projection• Filter• Correlation (Join)

Query Expressiveness

var result = from win in inputStream.TumblingWindow( TimeSpan.FromSeconds(10)) select new { avg = win.Avg(e => e.W) };

• Projection• Filter• Correlation (Join)• Aggregation over windows

• Projection• Filter• Correlation (Join)• Aggregation over windows• Group and Aggregate

Query Expressiveness

var result = from e in inputStream group e by e.id into eachGroup from win in eachGroup.TumblingWindow( TimeSpan.FromSeconds(10)) select new { eachGroup.Key, avg = win.Avg(e => e.W) };

Windowing

• Hopping– Tumbling

• Snapshot• Count

Time Windows

Time

Hopping Window

Time Windows

Time

Tumbling Window

Snapshot Windows

Snapshot Windows

Debugger

• Has its own debugger• You cannot live without it (believe me)• Set breakpoints• Filter

• A wonderful tool

DEMOS

•Tour•Trace Reader Live•Trace File Reader•Query Demo (Joining)•Debugger Tool (How it will save you)