Complex Event Processing:

Power your middleware with

StreamInsight

Mahesh Patel (Microsoft)

Amit Bansal (PeoplewareIndia.com)

Agenda

• The Value of Timely Analytics

• The challenges / Scenarios

• Introduction to Complex Event Processing

• Introducing Microsoft StreamInsight

• Brief introduction to BizTalk Server

• Understand the key scenarios that StreamInsight can

unlock for BizTalk

• Map out the architectural, integration requirements for

BizTalk + StreamInsight.

The value of timely analytics

Current Products for analytics

Current Products for analytics

Introduction to CEP

• Complex Event Processing (CEP) is an architecture style

based on the principles of Event Driven Architecture

(EDA).

• Continuously processing a high-volume stream of events

from various event sources

• Events are logically grouped in sequences (typically

known as streams) based on a time defined criteria such

as an interval.

• Event streams are processed through a series of queries

Capabilities of a CEP Application

Event Driven

Continuous Queries

Time Sensitive

Low Latency

High Volume

The value of timely analytics

Scenarios

Stream Data Store & Archive

Event Processing Engine

Dat

a St

ream

Asset Specs & Parameters

Power, Utilities:

• Energy consumption

• Outages

• Smart grids

• 100,000 events/sec

Visual trend-line and KPI monitoring Batch & product management Automated anomaly detection Real-time customer segmentation Algorithmic trading Proactive condition-based maintenance

Web Analytics:

• Click-stream data

• Online customer behavior

• Page layout

• 100,000 events /sec

Manufacturing:

• Sensor on plant floor

• React through device

controllers

• Aggregated data

• 10,000 events/sec

• Threshold queries • Event correlation from multiple

sources • Pattern queries

Lookup

Asset Instrumentation for Data Acquisition, Subscriptions to Data Feeds

Financial Services:

• Stock & news feeds

• Algorithmic trading

• Patterns over time

• Super-low latency

• 100,000 events /sec

Dat

a St

ream

Introducing StreamInsight

DEMO

CEP using StreamInsight

• Traditional messaging applications:

• “High signal” – every message is business

relevant

• Rich data schemas – sets of related

information

• Workflow centric and transactional

• Emerging data sources and trends:

• Machine born data is growing at a very

rapid rate

• Not all of this data is business relevant

(low signal/noise ratio)

• How do we:

• Identify and extract business relevant

events from streaming data?

• Take action on these business insights

(line of business, human workflow, etc)

Messages and Events

LOB

Large Messages Low Volume

High Volume

Small Events

Web

Services

B 2 2B SWIFT

RFID

Stock Feeds

Electrical Grids

Operational Data

Health Care

CRM

Send Port

MessageBox

Orchestration Receive Port

Message

Maps

Receive Pipeline

Adapter (HTTP)

Message

LOB

Host Host

Send Port

Adapter (File)

Map

Send Pipeline

Component

Basic BizTalk Architecture

Component

Host

Management DB Tracking DB

Message Agent

Message Agent

Message Agent

Scenarios

• Good BizTalk + StreamInsight scenarios:

• Streaming data/event sources – high noise to

signal ratio

• Need to analyze and extract business

relevant events

• Integrate and transact with line of business

applications.

Scenarios

• Manufacturing (Oil and Gas) • Monitor plant floor processes

• Identify business relevant events

• Transact with LOB apps

• Utilities • Automated metering – handle flood of data from

utility meters

• Identify business relevant events

• Transact with LOB apps

• The data feeds change, the question changes – the core scenario doesn’t!

OSIsoft PI & StreamInsight

• OSIsoft PI - Industry leading process data

management and historian.

• Integrating and leveraging StreamInsight

• OSIsoft shipping StreamInsight adapter for PI.

• Stream data in and out of PI

• Extract powerful business insights from PI

data

• Integrate with LOB applications using BizTalk

Server

OPC & StreamInsight (with

MatrikonOPC)

• OPC –Industry standard for process

automation and data exchange

• MatrikonOPC – Industry leader in OPC

interfaces

• Developed StreamInsight adapters for their OPC

implementation

• Extract powerful business insights from OPC

data

• Integrate with LOB applications using BizTalk

Server

Automated Metering

• Contoso Utilities is rolling out smart electrical meters to

their customers.

• Meters report power usage (wattage, voltage) at a “point in time”.

• Data is consolidated into a back end application used for billing

and forecasting.

• The volume of data has forced Contoso to only read values from

meters every hour

• They want to:

• Monitor the data more closely, compressing the point stream to an

interval stream before loading into the back end application.

• i.e. if the data does not change by more than 2% over a 10 minute

window, keep the average value.

• Capture spikes and threshold events, route to a process control

application for further inspection.

Automated Metering • Collect high volume streaming data from smart meters

• Data consists of timestamp, source ID and payload (power in Watts)

• Sample operations on the streaming data:

• Consolidate stream of points events into 5 minute “chunks”; record average, count and standard deviation.

• Detect anomalies (threshold 10%), collect last 5 minutes worth of data for that meter and all meters fed from that substation

• Need to link meter ID with reference data (location, substation)

• Results must be integrated into line of business application

Timestamp 3/5/2010 14:23

Power 1020

Meter ID 112322

Data consolidation

(point -> interval)

Anomaly Detection

Reference data (location, hierarchy)

Batch to Message

Batch to Message

Line of Business

Application

Smart Meters

Automated Metering

Timestamp 3/5/2010 14:23

Power 1020

Meter ID 112322

Data consolidation

(point -> interval)

Anomaly Detection

Reference data (location, hierarchy)

Batch to Message

Batch to Message

Line of Business

Application

Smart Meters

StreamInsight

Meter Input Adapter

(StreamInsight)

ConsolidationQuery

Threshold Detection Query

MessageBuilderOutputAdapter

BizTalk Server

WCFMSMQAdapter

ReceivePort

OrchestrationSendPort

LOBAdapter

QueryResults (XML)

QueryResults (XML)

Query Result Format (XSD)

Query Result Format (XSD)

1

1

2

2

3

3

4

5

4

5

Architecture – Message Integration

WC

F A

dapte

rs

WC

F A

dapte

rs

BizTalk Host

Orchestration

Business

Activity

Monitoring

Business

Rules

Engine

Message Box

StreamInsight

XmlPayload Output

Adapter

Queries

Input Adapters

XSD Schema

XSD Schema

Batched QueryOutput

Batched QueryOutput

1

2

3

1 StreamInsight extracts insights from real time data feeds, feeds aggregate information to BizTalk.

2 A strongly typed XML schema is inferred from the StreamInsight query

3 Batched and correlated query results are delivered to a BizTalk receive port.

• StreamInsight is an in-memory event processing engine.

• Do not deploy StreamInsight instance on the same hardware as your

BizTalk host instances.

• Leverage StreamInsight-local MSMQ for message transfer.

BizTalk

Hosts

SQL Servers

for BizTalk

MSM

QStreamInsight

Instance

Deployment – Message Integration

Summary BizTalk Server and StreamInsight together can unlock powerful scenarios:

• High volume data processing

• Message integration

Leverage StreamInsight to:

• Consolidation and analyze incoming data streams

• convert event sets into messages

• Deliver to BizTalk receive port

Leverage BizTalk Server to:

Integrate these insights into line of Business applications

Other potential alignment points:

• Business Activity Monitoring (BAM). Tap into BAM event events stream for rich analytics and event detections

• Business Rules Engine (BRE). Leverage BRE to define event pattern rules

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in

the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft

must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any

information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.