financial services technology expo microsoft streaminsight for financial services a microsoft point...
TRANSCRIPT
Financial Services Technology Expo
Microsoft StreamInsight for Financial Services
A Microsoft Point of View Presentation
Hilton New York Hotel
New York, NY
June 22-24, 2010
2
Microsoft StreamInsight for Financial Services
Torsten GrabsLead Program ManagerMicrosoft StreamInsightemail: [email protected]
Microsoft StreamInsight
Operational Intelligence PlatformSources
Devices, Sensors
Web servers
Stock tickers & News feeds
Data Bus Caching Processing VisualizationDistribution
Static Reports
Mining, Validation, “What-If” Scenarios
Operational Dashboard(Ticking - Snapshot)
Reporting Dashboard(Refreshed)
Messag
e
Bu
s
Operational Analytics
Automated Decisions
In-memory Database
Intra-Day Cubes
Historic Cubes
ETL
ETL
Reference Data
Cache
Refresh(Push)
Refresh (Push)
Re-compute (Pull)
Service Broker
Operational Analytics
Queries designed over offline/historical data“Backtesting” of modelsAbility to work over vast amounts of offline data
Queries deployed to online/live feedsTens of thousands of events/messages per secContinuous processing – standing queriesTimely results – low latency
Queries reason about timeWindows over timeSequences of events
The Need for an Event-Driven Platform
5
Query results need to reflect important changes in business reality immediately and enable responses to them with minimal latency
Database Applications Event-driven Applications
Query Paradigm
Ad-hoc queries or requests
Continuous standing queries
Latency Seconds, hours, days Milliseconds or less
Data Rate Hundreds of events/sec Tens of thousands of events/sec or more
Query Semantics
Declarative relational analytics
Declarative relational and temporal analytics
request
response
Eventoutput stream
input stream
Overview: Microsoft StreamInsight
6
StreamInsight Engine
Output Adapters
Input Adapters
Event
Standing Queries
Event sources Event targets
`
Devices, Sensors
Web servers
Event stores & Databases
Stock tickers & News feeds
Event
Event
Event
Event
Event
Event
C_ID C_NAME C_ZIP
Event stores & Databases
Pagers & Monitoring devices
KPI Dashboards, SharePoint UI
Trading stations
Event
Event
Application at Runtime
Static reference data
.NETC#
LINQApplication Development
Event Types
StreamInsight events in use the .NET type systemEvents are structured and can have multiple fieldsFields are typed using the .NET framework types StreamInsight engine provisioned timestamp fields capture all the different temporal event characteristicsEvent sources populate time stamp fieldsAll calculations done based on “business time”
Timestamps/Metadata
LongpumpID
StringType
StringLocation
Doubleflow
Doublepressure
… … … … … …
Event Streams & AdaptersA stream is a sequence of events
Defined over a .NET typePossibly infinite
Stream characteristics:Event/data arrival patterns (steady, bursty)Out of order events: Order of arrival of events does not match the order of their application timestamps
AdaptersReceive/get events from the data sourceEnqueue events for processing in the engineInsertions of new eventsChanges to event durations
8
StreamInsight Query FeaturesOperators over streams
Calculations (PROJECT)Correlation of streams from different data sources (JOIN)Check for absence of activity with a data source (EXISTS)Selection of events from streams (FILTER)Stream partitioning (GROUP & APPLY)Aggregation (SUM, COUNT, …) Ranking and heavy hitters (TOP-K)Temporal operations: hopping window, sliding window
Extensibility – to add new domain-specific operators
LINQ Query Examples
LINQ Example – GROUP&APPLY, WINDOW:
from e3 in MyStream3group e3 by e3.i into SubStreamfrom win in SubStream.HoppingWindow( FiveMinutes,ThreeSeconds)select new { i = SubStream.Key, a = win.Avg(e => e.f) };
LINQ Example – JOIN, PROJECT, FILTER:
from e1 in MyStream1join e2 in MyStream2 on e1.ID equals e2.IDwhere e1.f2 == “foo”select new { e1.f1, e2.f4 };
Join
FilterProject
Grouping
Window
Project &Aggregate
StreamInsight Licensing Guidance
• Data rate: Tens of thousands of events every second
• Latency: Sub-second
Standard and Enterprise
Datacenter
• Data rate: Thousands of events per second
• Latency: seconds
StreamInsight is part of Microsoft SQL Server 2008 R2• Standard & Enterprise edition for embedding in many ISV applications• Standard & Enterprise edition for deployments on the edge• Data Center edition for server deployments
Data Sources
Aggregation & Correlation
Devices
SensorsWeb servers
Feeds
Stream Insight
Stream Insight
Stream Insight
Stream Insight
Stream Insight
Stream Insight
Stream Insight
Stream Insight
Stream Insight
Complex Analytics & Mining
Actual mileage will vary
Recap: Microsoft StreamInsight
12
StreamInsight Engine
Output Adapters
Input Adapters
Event
Standing Queries
Event sources Event targets
`
Event
Event
Event
Event
Event
Event
C_ID C_NAME C_ZIP
Event
Event
Static reference data
CEP Application Development
Development experience with .NET, C#, LINQ and Visual Studio 2008 or 2010
CEP platform from Microsoft to build event-driven applications
Event-driven applications are fundamentally different from
traditional database applications: queries are
continuous, consume and produce streams, and compute
results incrementally
Flexible adapter SDK with high performance to
connect to different event sources and sinks
The CEP platform does the heavy lifting for you to deal
with temporal characteristics of event stream data
For More Information
StreamInsight main page & download : http://www.microsoft.com/sqlserver/2008/en/us/R2-complex-event.aspx StreamInsight blog: http://blogs.msdn.com/streaminsight/ StreamInsight MSDN documentation: http://msdn.microsoft.com/en-us/library/ee362541(SQL.105).aspx StreamInsight E-clinics on Microsoft e-learning https://www.microsoftelearning.com/eLearning
14
More Microsoft Sessions
Tuesday June 22nd, 2:30 PM Madison Room: Cloud Computing In Financial Services
Wednesday June 23rd, 8:00 AM Green Room – 4th Floor Breakfast: Microsoft CRM for Financial Services
Wednesday June 23rd, 2:15 PM Petit Trianon Room – Real Time Risk Management with Microsoft’s StreamInsight
Meet with our Cloud Computing, StreamInsight and HPC Product teams at the Green Room, 4th Floor
Follow us on Twitter @FinServMSFT
Visit us at www.microsoft.com/financialservices
More Microsoft Sessions
Let us know about any topic you would like us to follow up on!
Write the topic on the back of your business card
Drop it off in the bowl on your way out of the session
You will be automatically entered to win a 16GB Zune HD!*
*Winner need not be present to win. Winners notified at the end of the SIFMA conference.