John PlummerTechnical Specialist Data PlatformMicrosoft Ltd

StreamInsightComplex Event Processing (CEP) Platform

3

What is CEP?

3

Complex Event Processing (CEP) is the continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with near-zero latency.

Database Applications Event-driven Applications

Query Paradigm Ad-hoc queries or requests Continuous standing queries

Latency Seconds, hours, days Milliseconds or less

Data Rate Hundreds of events/sec Tens of thousands of events/sec or more

request

response

Eventoutput streaminput

stream

4

Example CEP Scenarios

4

Dat

a St

ream

Stream Data Store & Archive

Event Processing Engine

Dat

a St

ream

Asset Specs & Parameters

Power, Utilities:• Energy consumption• Outages• Smart grids• 100,000 events/sec

Visual trend-line and KPI monitoringBatch & product managementAutomated anomaly detectionReal-time customer segmentation Algorithmic tradingProactive condition-based maintenance

Web Analytics:• Click-stream data• Online customer

behavior• Page layout• 100,000 events /sec

Manufacturing:• Sensor on plant floor• React through device

controllers• Aggregated data • 10,000 events/sec

• Threshold queries• Event correlation from

multiple sources• Pattern queries

Lookup

Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds

Financial Services:• Stock & news feeds• Algorithmic trading• Patterns over time• Super-low latency• 100,000 events /sec

5

“Sweet Spot” for Complex Event Processing

Relational Database Applications

Financial Trading Applications

Aggregate Data Rate (Events/sec)

0 10 100 1000 10000 100000 higher

Months

Days

Hours

Minutes

Seconds

100 ms

< 1 ms

Operational Analytics Applications (e.g., Logistics)

Manufacturing ApplicationsMonitoring Applications

CEP Target Scenarios

Data Warehousing Applications Web Analytics Applications

6

Complex Event Processing

"Intelligence isn't just about knowing what is happening. It's about looking at the patterns in real time. If business people want to truly optimize their resources, they must examine patterns in real time.”

- Mike Gualtieri - Forrester

SITUATION TODAYSQL SERVER 2008 R2 STREAMINSIGHT

Pressure to handle large volumes of complex data in real time

The size and frequency of data make it challenging to store for data mining and analysisGrowing need to monitor, analyze and act on the data in motion

Process large volume of events across multiple data streams in less than a secondManage your business through historical data mining and continuous insights.

Built in support for different types of event handling and rich query semantics

7

MSFT CEP Engine

Reference data

APPLICATIONInput A

dapte

rs

EVENT SOURCES

Devices, Sensors

Web servers

Event stores & Databases

Stock tickers & News feeds

Outp

ut A

dapte

rs

EVENT TARGETS

Event stores & Databases

Pagers & Monitoring

devices

KPI Dashboards, SharePoint UI

Trading stations

Platform from Microsoft for continuous and incremental processing of event streams from multiple sources based on declarative query and pattern specifications with low latency

StreamInsight

Bloomberg.com

8

Events expose different temporal characteristicsPoint in time eventsInterval events with fixed durationInterval events with initially unknown duration (Edge)

Rich payloads capture all properties of an event

t1 t4t3t2 t5Time

Payl

oad/

val

ue

ab

c d e

Events

9

Event TypesEvents in Microsoft’s CEP platform use the .NET type systemEvents are structured and can have multiple fieldsFields are typed using the .NET framework types CEP engine provisioned timestamp fields capture all the different temporal event characteristicsEvent sources populate time stamp fields

Timestamps/Metadata

LongpumpID

StringType

StringLocation

Doubleflow

Doublepressure

… … … … … …

10

Event Streams & AdaptersA stream is a possibly infinite sequence of events

Insertions of new eventsChanges to event durationsExamples: time series sensor data, log entries

Stream characteristics:Event/data arrival patterns

Steady rate with end-of-stream indication Examples: files, tablesIntermittent, random, or in burstsExample: retail scanners, web, weather telemetryOrder of arrival of events does not match the order of theirapplication timestamps

Input & Output Design Time for dealing with Event Streams10

11

Core CEP Query EngineHosts “standing queries”

Operators consume and produce streams

Queries are composable

Query results are computed incrementally

11

• Query instance management:• Submit, start, stop

• Runtime statistics

• Takeaway: CEP engine does the heavy lifting for you when processing temporal event data

CEP Engine Output Adapters

Input Adapters Event

Standing Queries

Event

Event

Event

Event

Event

Event

Event

Event

12

Developer Experience with the CEP Platform

CEP development environmentBuilds on top of .NETMicrosoft Visual Studio as IDEApplications written in C#Queries written in LINQSupports various deployment models for CEP applications

Embedded applicationsServer applications

12

LINQ:

var queryFilter = from c in TestEventStreamwhere c.Field1 > 1select c;

13

StreamInsight Demo…

14

StreamInsight – More Information

StreamInsight - http://www.microsoft.com/sqlserver/2008/en/us/R2-complex-event.aspx

Team Blog - http://blogs.msdn.com/streaminsight/

Forum - http://social.msdn.microsoft.com/Forums/en-US/streaminsight/threads

© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED

OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.