innov-10 progress® event engine™ technical overview prashant thumma principal software engineer

INNOV-10Progress® Event Engine™Technical Overview

Prashant ThummaPrincipal Software Engineer

2 INNOV-10 Progress Event Engine Technical Overview© 2005 Progress Software Corporation

Agenda Slide

Event Stream Processing (ESP) Progress ESP products Progress Event Engine (PSEE) PSEE Performance PSEE Integration


Event Stream Processing (CEP)

ESP is about Operational Business Data– Capture– Query– Analysis

Characteristics– High Volume Streaming Data– Time Critical Analysis– Consistent and Reproducible Results– Highly Available

Simple Structured Data– Data is not Relational or complex Object model

What is ESP?


Event Stream Processing (ESP)

Event Processing Language– Data Driven– Flexible– Complete

Event Persistence Engine– High Performance

Analytical Capabilities– Temporal Data Analysis– Inbound and Outbound Event Processors

High Availability Modeling, Deployment and Development Tools

ESP Platform Requirements


Event Stream Processing (CEP)

Financial Markets– Program Trading– Testing Trading Strategies

RFID– Inventory Logistics– Asset Tracking– Theft Detection

Manufacturing– Capture and analysis of data from PLCs

Telecommunications– Cell Phone Call Tracking– Multimedia Services Billing– Network Management

Scientific– Sensor Data

ESP Applications


Agenda Slide



Progress ESP Products

Event Cache for JMS

Adapter

Event Server

-<!– simple.xml -->

- <event>

+ <info1>

+ <info2>

</event>


- <event>

+ <info1>

+ <info2>

</event>


- <event>

+ <info1>

+ <info2>

</event>

XML Configuration FilesJava API

Progress® Event Engine™ (PSEE)

C++ API


- <event>

+ <info1>

+ <info2>

</event>


- <event>

+ <info1>

+ <info2>

</event>


- <event>

+ <info1>

+ <info2>

</event>

XML Configuration Files

Apama


Agenda Slide



Progress Event Engine

It’s all about Event Data– Collection, storage, management and query

What is Event Data?– Streaming, constantly flowing information

Stock Trades, RFID, IP Packets, Mfg PLC’s– Categorized

Stock tick symbol, RFID tag ID, IP address– Typically Time Ordered

Trade time, RFID scan time

What is it for?



3. Query

query

client client client

Event Cache

Service

1. CollectEvent Data

Feed store

Convert external events to PSEE events

2. OrganizeStructures data for fast retrieval

sort/mergePersistent Storage

disk disk

disk disk

Data Flow



High Performance, Scalable– Collection Engine

Store tens of thousands events per second 20 TB+ database size Index data is ~1% of database size

– Query Engine High Performance Pipelined Architecture Extensible Functionality

– Simultaneous Query and Collection Writes do not block reads!

Why PSEE?



Transparent Data Management– Event Persistence

Requires no coding to persist events

– Transactional Consistent Data Hidden and Managed by PSEE

– Tunable Latency “Real-time” defined by you!

Why PSEE?



Simple API Supports Multiple Languages

– Java and C++ language support Multiple Platform Support

– Window, Solaris, Linux

– Heterogeneous deployment

Why PSEE?



Functor: A unit of work in PSEE– 3 Main types (Source, Process, Sink)– Built in library of Functors– Customer coded

Pipeline: A sequence of Functors– Collect, process, transform event streams

Task Manager– Manages pipeline execution

Events: Input and Output of Pipelines

Components



PSEE Event– XML schema

– Supports All C++ Primitive Types

Event Definition

<EventDefinitions bundleName=“TABundle” …> <Event name=“Trade” type=“TimedCategorizedEvent”> <Member name=“price” type=“Float32”/> <Member name=“volume” type=“UInt32”/> </Event></EventDefinitions>


class MyCollectorFunctor : public Functor {public:

void initialize(FunctorContext& ctx);void handle(FunctorContext& ctx);void finalize(FunctorContext& ctx);

private:// functor data variables

}


Implements Event Stream Processing Logic Interface For Accessing Events

Functor


Progress Event EngineCollector Pipeline

Event-Data File

for Bulk Loading

Event

Stream

StoreRealtime

Functor

Real-Time

Data Feed

Series

(PSEE Database)

MyCollector

Functor


Progress Event EngineQuery Pipeline

Series

(PSEE Database)

Retrieve

Functor

Filter

Functor

Sort

Functor

Event

Stream

Event

Stream

Event

Stream

MyXMLwriter

Functor



Collector Functors– StoreRealtime

– StoreBulkload

Query Functors– Retrieve

– Filter

– Sort

– Count

– Group Count

Predefined Functors


Progress Event EngineScalable Deployment

OSES

Qu

ery

Ad

apte

r

PS

EE

OSES

Co

llect

ion

Ad

apte

r 2

Co

llect

ion

Ad

apte

r 1 PS

EE

Raw Series

Optimized db’s for query

Series database

OSES

Qu

ery

Ad

apte

r

PS

EE

Qu

ery

Ad

apte

r

OSES

PS

EE

VMMA & CFA Scalability

MessagingScalability


Agenda Slide



PSEE Performance

Event storage (events/sec)– Real-time collector – pipeline using

StoreRealtime functor– Bulk load collector – pipeline using

StoreBulkload functor Event retrieval (events/sec)

– Threaded execution of query pipeline Random query Query on several (randomly chosen)

categories

Performance Metrics


PSEE Performance

Database Characteristics– 160 Million Events

– 20 Byte Event

– 10000 Categories

System Configuration– Sun Fire-V440

– Solaris 8

– 4 CPUs (1GHz)

– 8 GB Memory

Performance Testing Environment


PSEE Performance

Real-time collector– 94,062 events / second

Bulk load collector– 191,152 events / second

Query– 213 Queries / second

– 10000 events / query

– 2,134,560 events / sec

Performance Testing Results


Agenda Slide



D I S C L A I M E R

Under Development

This talk includes information about potential future products and/or product enhancements.

What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here.

D I S C L A I M E R


PSEE Integration

Provides Event Persistence for Apama Initial Implementation for Customer

– Tick Database for Back-Testing Trading Strategies

– Query Interface Using Apama Event Language

Future– Extend Apama Event Language to support

Persistence– Seamless Configuration via Apama

Management Interfaces

Integration with Apama


PSEE Integration

PSEE on the bus using Event Cache Event Cache Features

– Native JMS Message Based Query

– Results are Java Objects

– Batched for Optimal Performance

Integration with Sonic


PSEE Integration

PSEE based RFID Accelerator provides 4GL Interface

Web Services based Implementation RFID Accelerator as a reference

implementation Further 4GL Integration

Integration with 4GL


In Summary

Exponential growth of event data requires ESP solutions

PSEE + Apama is an ESP solution

Based on proven technologies Simple! Infrastructure problems

are handled by PSEE It can be integrated into existing

business systems


Questions?


Thank you for your time!

innov-10 progress® event engine™ technical overview prashant thumma principal software engineer

Documents

event datacollection

event streamstask man

analysis of data

organizestructures data

psee events2

rfid tag id

rfid scan timewhat

pseetunable latencyrealtime