Microsoft StreamInsight

Allan MitchellKonesans Limited

Who am I?• Allan Mitchell• Joint author on “First to Market” SSIS Book by Wrox• Websites

– www.SQLIS.com – www.SQLDTS.com– www.konesans.com

• StreamInsight and SQL Azure Advisory Boards• Microsoft SQL Server MVP• Twitter: allanSQLIS• eMail: allan.mitchell@konesans.com

Agenda

• What is Stream Processing• Understanding Streaming Data

– Differences to RDBMS• Scenarios• StreamInsight Architecture• Concepts• Typical Patterns

What is Event Stream Processing

Query

• Database loads data from disk• You query database• Data is stale

What is Stream Processing

• Continuous processing of event streams• from multiple sources• based on a declarative query language• with near-zero latency

StreamInsight

Query

Understanding Streaming Data

Using an RDBMS:

Q. Given a time interval tell me the amount of Electricity units used per household per time interval from now until I say stop.

Not EasyRequires requerying/polling

Need to store the end of the last interval

Understanding Streaming Data

This is the streaming data paradigm in a nutshell – ask questions about data in flight.

Answering with StreamInsight

• Create a window over the event stream• Partition/Group By Household• SUM() readings within window

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

What is CEP?

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications

Event-driven Applications

Query Paradigm

Ad-hoc queries or requests

Continuous standing queries

Latency Seconds, hours, days Milliseconds or lessData Query Rate

Hundreds of events/sec

> Tens of thousands of events/sec

What is CEP?

Relational Database Applications

Financial trading Applications

Scenarios for Event-Driven Applications

12

Aggregate Data Rate (Events/sec.)

Latency

0 10 100 1000 10000 100000 ~1million

Months

Days

hours

Minutes

Seconds

100 ms

< 1ms

Operational Analytics Applications, e.g., Logistics,

etc.

Manufacturing Applications

Monitoring Applications

StreamInsight Target Scenarios

Data Warehousing Applications Web Analytics Applications

Who might need CEP

• Fraud Detection• Real-Time Trade Risking• Algorithmic Trading/Betting• Meter throughputs

– Oil, Gas, Water, Electricity– Use to drive alarms, alerts etc

• Telemetry Data

Windowing

• Hopping– Tumbling

• Snapshot• Count

Time Windows

Time

Hopping Window

Time Windows

Time

Tumbling Window

Snapshot Windows

Snapshot Windows

StreamInsight Key Features• Runtime environment for continuous and event-driven

processing• In-memory processing, no persistence• Declarative query language: LINQ• .NET APIs for Input/Output Adapters• Flexible deployment models• Extensibility: Aggregates, Operators• Rich diagnostic & debugging interface

Standing Queries

Query Logic

Event sources Event targets

`

Devices, Sensors

Web servers

Event stores & Databases

Stock ticker, news feeds

Event stores & Databases

Pagers &Monitoring devices

KPI Dashboards, SharePoint UI

Trading stations

StreamInsightApplication Development

InputAdapter

s

OutputAdapter

sStreamInsight

Engine

Query Logic

Query Logic

StreamInsight Platform

12 3

4

StreamInsight Application at Runtime

Microsoft StreamInsight

21

CEP Engine

Output Adapters

Input Adapters

Event

Standing Queries

Event sources Event targets

`

Event

Event

Event

Event

Event

Event

C_ID C_NAME C_ZIP

Event

Event

Static reference data

CEP Application Development

Development experience with .NET, C#, LINQ and Visual Studio 2008/10

CEP platform from Microsoft to build event-driven applications

Event-driven applications are fundamentally different from

traditional database applications: queries are

continuous, consume and produce streams, and compute

results incrementally

Flexible adapter SDK with high performance to

connect to different event sources and sinks

The CEP platform does the heavy lifting for you to deal

with temporal characteristics of event stream data

Central Concepts• Events

– User Payload + temporal properties– Type based on .NET structure or class– Point / Interval / Edge

• Streams– Sequence of events with specific type– Flow into standing queries

• Queries– Composed of query operators– Apply desired semantics on events– In-memory, event-triggered, incremental computations

• Adapters– Produce / Consume event streams (Connection Manager)

Typical Query & Analytics Patterns

• Introduce additional event fields• Filter out events• Grouping by one or more event properties• Aggregation for each event group over a set (a window)• Rank events within windows• Correlate and synchronize event streams• Check for absence of events• Create result only when input changes• Remove duplicates• Reduce event rate, resample stream• Enrich events with reference data• Collection of assets may change over time

LINQ Query Examples

LINQ Example – GROUP&APPLY, WINDOW:from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };

LINQ Example – JOIN, PROJECT, FILTER:from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == “foo”select new { e1.f1, e2.f4 };

Join

FilterProject

Grouping

WindowProject &Aggregate

Extensibility

• Need for domain-specific extensions– Integrate with functionality available in existing libraries

• User-defined operators, functions, aggregates• Code written in .NET, deployed as .NET assembly• Window-based semantics for UDOs, UDAs

– Receive a set of events– Produce a value / set of events

Demo

• Adapter model• Windows• Grouping• Altering event lifetime• Extending

For More Information

• Download:http://go.microsoft.com/fwlink/?LinkId=160598

• StreamInsight MSDN documentation:http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx

• Blogs:http://blogs.msdn.com/streaminsight/http://www.SQLIS.com/

• Forum:http://social.msdn.microsoft.com/Forums/en-US/streaminsight

• StreamInsight MSDN portal:http://msdn.microsoft.com/en-us/ee476990.aspx

• Samples:http://streaminsight.codeplex.com

• Twitter:http://twitter.com/streaminsight