event driven real time analytics

7/30/2019 Event Driven Real Time Analytics

http://slidepdf.com/reader/full/event-driven-real-time-analytics 1/46

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : [email protected] W: www.rittmanmead.com

Jon Mead, Rittman MeadSeptember 2012

Event Driven Real Time Analytics

Sunday, 30 September 12

http://www.rittmanmead.com/


mailto:[email protected]





Introductions

• Jon Mead

‣CEO/co-founder of...

•Rittman Mead Consulting

‣Oracle BI & DW Consultancy

‣Gold Partner

‣

Long(est) running Oracle BI blog‣ Annual BI Forum

‣OBIEE Oracle Press book

•Customer-facing FTSE listed

‣UK based and leading

‣ Internet based‣Retail based









The point of thispresentation isto give you an

idea of how toapproach a realtime eventdriven BIsystem using

Oracle's currenttoolset.

Agenda

•Understanding the Project

‣ Legacy architecture

‣Proposed architecture

‣Reporting requirements

•Technical Infrastructure

‣Hardware and Software

•

Data Warehouse Architecture‣ Adopting the Oracle reference architecture for real

time

•Design Challenges

‣De-queuing

•Operational

‣ODI Logging‣Multi-threading and scalability

•Further thoughts

‣Middleware or memory based applications









Understanding the Project











Business Goal

• Part of a major re-architecture program

• Covering ERP, CRM and BI

Driver: single view of customer

Delivered by: channel consolidationinto single enterprise data warehouse

• Data migration• Enterprise Architecture

• Enterprise Service Bus

• Real-time reporting• Legacy reporting

• BAU reporting

• Revenue and Profit• Liability and risk

• Up/cross-sell










Legacy Architecture

• Legacyarchitectureconsisted of twocompletelyseparate

systems

• Retail storedshop basedtransactions

• Online storedtransactionsgenerated online

Retail trading systems

Online trading systems



24 hour batch (DTS)

Retail Data Warehouse

SQL Server 2005

6TB

24 hour batch (DTS)

Online Data Warehouse

SQL Server 2008

3TB











Proposed Architecture



Enterprise Architecture


Real-time feed

TIBCO QueueReal Time Data

Warehouse

Exadata

Real-time feed

Real-time feed -

transactions

Real-time feed -

reference data

DR

Exadata

ODI

ODI


SQL Server 2005

6TB


SQL Server 2008

3TB

D a t a m i g r a t i o

n

- o n

c e o f f O

D I

D a t a m i g r a t i o n

- o n c e o f f

O D I












Proposed Architecture





Real-time feed

TIBCO QueueReal Time Data

Warehouse

Exadata

Real-time feed

Real-time feed -

transactions

Real-time feed -

reference data

DR

Exadata

ODI

ODI


SQL Server 2005

6TB


SQL Server 2008

3TB

D a t a m i g r a t i o

n

- o n

c e o f f O

D I

D a t a m i g r a t i o n

- o n c e o f f

O D I




Current state to future

state includes a data

migration









Need tounderstand thedifferent driversfor each of these needs and

the valueprovided by realtime reportingduring the

running of hightransactionevents

Reporting Requirements

•Real time monitoring

‣Risk and liability‣Profit and loss

• Analytic reporting

‣Consolidated analytics

‣ Legacy reporting

•Operational reporting

‣Detail level

‣Support analytical reports

‣Drill through









Technical Infrastructure











Volumetrics

• Initially processing data from 2500 shops, scaling to capacity

‣ 8 TB of migrated data‣Processing 1.8 million transactions a day

‣Processing 4,000 reference data items a day

‣ Approximately 9 million transaction rows being processed a day

‣ All transactions read from a TIBCO queue

‣ Approximately 200,000 reference data changes a day

‣ 30,834 transaction processing cycles a day (one every ~2.8s)

‣ 2,701 reference data cycles a day

‣ 680,000 recalculations a day

•Online transactions will follow

‣ 2 million transactions a day‣Comparable downstream figures









Exadata and ODI

•Standard X2-2 Quarter Rack

‣ 2 compute nodes‣ All databases split across the nodes

•The storage is configure in dual redundancy mode to offer up about 9TB of usablespace, however, we use a couple of TB of that for backups and archive redo logs

•The flash storage has been set up as 250GB on each node as a local cache and 110GBfrom each being used to provide a 160GB flash disc.

•The database version is 11.2.0.3 and the client have the tuning and diagnostics packand Heterogeneous Services on top of the usual Exadata software set.

‣Both the 11.2.0.2 and 11.2.0.3 Oracle Homes still exists.

•ODI Agents for UAT and PROD running off Node 2

•Running up to 30 ODI Agents for PROD to get the speed to read off the TIBCO queues.Each agent running with 512MB with the calling agent running 1GB.









Exadata and ODI

NODE 1

DEV01

DEV02

NFT

PROD

NODE 2

Compute Nodes

OracleDatabases

ODI AGENTS

DEV/ NFT

ODI AGENTS

PROD

UAT

ODIInstalls

PROD ODI

WORK

SCHEMA

UAT

REPOSITORY

PROD

REPOSITORY

UAT ODI

WORK

SCHEMA

UAT

SSD









Exadata and ODI

NODE 1

DEV01

DEV02

NFT

PROD

NODE 2

Compute Nodes

OracleDatabases

ODI AGENTS

DEV/ NFT

ODI AGENTS

PROD

UAT

ODIInstalls

PROD ODI

WORK

SCHEMA

UAT

REPOSITORY

PROD

REPOSITORY

UAT ODI

WORK

SCHEMA

UAT

SSD

Currently only one Exadata

server available, so shared

platform









Data Warehouse Architectur e











Key Drivers

•Part of integrated Enterprise Architecture

•The enterprise data model was designedand developed in Enterprise Architect by themiddleware architects

•The architects wanted to base the approach

on the Oracle Reference Data Warehousearchitecture

•There were different reporting needs for real time and business as usual reporting

•Write performance was likely to be as big afactor as read performanc









Oracle Reference Architecture

•Simplified view of Oracle’s

Data Warehouse Reference Architecture

•Enterprise Architecture wasXML based

Active Data Warehouse

Staging

Foundation

O

B I E E

PerfromanceAudit and

Reconciliation

Analysis

Operationaland real-

time










Oracle Reference Architecture

•Simplified view of Oracle’s

Data Warehouse Reference Architecture

•Enterprise Architecture wasXML based

Active Data Warehouse

Staging

Foundation

O

B I E E

PerfromanceAudit and

Reconciliation

Analysis

Operationaland real-

time


One of design drivers was that the

foundation layer reflected the

enterprise data model









Limitations

•The ODS would reflect the

enterprise architecture‣Non-database centric view

•Considerable processing to getdata into ODS

‣Data processing from Stagingto Foundation was too

complex to support SLAs•Performance layer also to

reflect existing more datawarehouse structures









Limitations

•The ODS would reflect the

enterprise architecture‣Non-database centric view

•Considerable processing to getdata into ODS

‣Data processing from Stagingto Foundation was too

complex to support SLAs•Performance layer also to

reflect existing more datawarehouse structures

Result was non-performant and

unusable structures for real-time

reporting









Alternative Architecture

• Split the processing between real-time and BAU

‣Process 1: Staging to Performance (real-time)

‣Process 2: Staging to ODS to Foundation

ODI

STG_xxx

STG_xxx

STG_xxx

STG_CTL

Staging

T I B C O

ODI

Microbatch ETL

BETBETSLIPDecompositionand Aggregate

tables

PerformanceFoundation

ODI Real time ETL

Real timeETL

O B I E E

SQL

SQL

Real time

query

Near realtime

BETBETSLIPDimension and

fact tables

ODI

Microbatch ETL

SQL

Near realtime

BETBETSLIP

3NF tablesBETBETSLIP

3NF tables

• Independentcontrol of either process

• Mechanism tohandle peaks indata

• Needed to ensureconsistencybetweenprocesses









Design Challenges











De-queuing

•Concern that ODI would not be able to de-queue

‣ A lot of fluctuations, depending on events

•XML messages were verbose

‣ Large amount of processing time for each batch of messages

•Scalability provided by creating more agents

‣What would the limitations be in terms of RAM

‣What would the limitations be in terms of connections

‣What would the limitations be in terms of management









2 Queues

•Queue are unstructured so

the data can arrive in anyorder

‣Difficulty of processingbusiness logic

‣ Timestamps not alwaysaccurate

‣Keys not always present

•One of most challengingareas of the project

‣Often need to do manuallookup of keys

Transactions

STG_CTL

T I B C O

Real timeETL

Referencedata

Recycle

STG_xxx

STG_xxx

STG_xxx

Solution: recycle

mechanism









High number of writes

•The real-time write process generated a very high number of writes

•Exadata optimised for bulk reads•Contention for REDO logs (see also the ODI Logging)

•Exadata configured for more of an OLTP system than Data Warehousing system

‣However both share the same server

•Resolution: lots of work by the DBAs to optimise database configuration

Isn't this a little bit like an

OLTP system?









Operational Challenges











ODI Parser

•XML Parser

‣ The XML data definition files were dynamically generated.‣ The current version of the XML Parser does not do a double pass of the definition file

‣ Any referenced complex definitions needed to be defined in the order they wereaccessed

‣ The software generating the XML definition files did not do this

•Resolution

‣Build a Java program to re-parse the XML data definition file and output a correctlyordered version

‣ This is a once per release process

‣ This behaviour if fixed in 11.1.1.7 of ODI (I think)









De-queuing Performance

•ODI struggled to keep up with, or fell behind the queue at peak times

‣Volumes of messages were not regular

•We also found agents failing

‣Hence we needed a resumption mechanism









Scaling agents

•Because of the failing agents, we couldn’t just

increase their number

•Set up parent agents

‣One for each queue

‣One for monitoring and maintenance scripts

•Each parent agent ran a number of child agents

‣Each child agent was actually two agents

‣Second agent acted as redundancy

‣ Agents killed after 50 executions

P1 P2 M&M

A1 A2 A3 A4 C6

A3 A4

C3 C4

A1 A2

A1 A2

C1 C2

Q1 Q2









Connections:

Every time anODI agent readfrom thequeue a newconnectionwas createdand destroyed.

There didn’tseem to beany pooling.

Impacts of Multiple Agents

•Memory

‣Parent agent 1024MB‣Child agent 512M

•Total number of agents used

‣ 3 parents

‣ 18 child (primary) and 18 child (secondary)

‣ Total: 39 = 21504MB (approx 21GB)

•However we didn’t get anywhere near linear scaling

‣Max TPS = 176

‣Max queue TPS = 480

•Second option is to increase the number of queues

‣Split by functional area









ODI Logging

•ODI Logging

‣ The ODI processes create 900GBof log files a day

‣ODI logging needs high IOPS

‣Exadata, by default not allocatingenough IOPS resource to the ODIlogging

‣ODI logging then becomes alimiting factor on the databaseperformance

‣ Target is SNP_SESS_TASK_LOG

‣ Log writer process cannot keepup

‣Number of active processes meanthe database will be performingas hard as it can and more activitywill slow everything down.

!









ODI Logging

•ODI Logging

‣ The ODI processes create 900GBof log files a day

‣ODI logging needs high IOPS

‣Exadata, by default not allocatingenough IOPS resource to the ODIlogging

‣ODI logging then becomes alimiting factor on the databaseperformance

‣ Target is SNP_SESS_TASK_LOG

‣ Log writer process cannot keepup

‣Number of active processes meanthe database will be performingas hard as it can and more activitywill slow everything down.

!

ODI does the same logging but the volumepreserved is reduced with lower levels of logging.

So in fact, lower levels of logging could be more

IO demanding as more data is deleted.










SNP_SESS_TASK_LOG

•The most demanding

SQL statement on thesystem is and always hasbeen the update toSNP_SESS_TASK_LOG

•This table holds 3 CLOB

columns. The update is"lazy", all columns areupdated each time. Thuseach update canpotentially update:

‣ the table

‣

three clob indexes‣ three clob tables









ODI Logging Impact

• Initially the system was

throttled on theSNP_SESS_TASK_LOG.

•ODI IOPS demand wasmaxing out the physical discIOPS capability of the box

•MoveSNP_SESS_TASK_LOG andSNP_SESS_TASK to a new ASM diskgroup created fromthe SSD storage in Exadata

!









ODI Logging Impact

• Initially the system was

throttled on theSNP_SESS_TASK_LOG.

•ODI IOPS demand wasmaxing out the physical discIOPS capability of the box

•MoveSNP_SESS_TASK_LOG andSNP_SESS_TASK to a new ASM diskgroup created fromthe SSD storage in Exadata

!

Simple solution for the ODI

Logging problem is to movethe database to another

server









ODI temporary tables

•The ODI real-time processing

creating a large number of I$ andother internal tables

•Once the processing around theseis complete, they are put theRecycle Bin

•The Recycle Bin become either

full or unmanageable









ODI temporary tables

•The ODI real-time processing

creating a large number of I$ andother internal tables

•Once the processing around theseis complete, they are put theRecycle Bin

•The Recycle Bin become either

full or unmanageable

There is a wider issue here

that affects scalability of the

whole solution, discussed innext slides









Future Challenges











Accessing datain memoryreduces the I/Oreading activity

when queryingthe data whichprovides faster and more

predictableperformancethan disk

Scalability

•The main bottleneck we are experiencing is I/O

‣High number of writes to the database‣ODI $ internal tables

‣ODI logging

•We should address this problem by making better use of memory

•We also have a constraint on the amount of memory Exadata can provide the agents

‣ Any allocated memory has the opportunity costof not be used by the database

•We should also explore other ‘logical’ approachesto solving this problem









Process Flexibility

•The source system splits the data into real

time and batch‣ESB provides 2 separate queues

•The XSD is the same for both queues

•The processing for both queues isconstantly running in a loop

‣ The batch queue is much larger than the

real time queue.‣ The foundation layer requires data from

both

•The data from each queue lands in thesame stage tables partitioned by thequeue name

•Entire process controlled by maintainingBATCH_IDs

SRC SystemsFeed

Non Critical Data Real Time Data

Stage Schema

DQProcess

DQProcess

Event StageTables

STG_ Tables

Performance Schema

Foundation Schema

Reporting Tables

ODS LoadProcessing

Real Timeprocessing

Foundation LayerTables

CTL_EVENTbatch_idbatch_typeODS_processedRTF_processedSTG_processed

CDC ProcessCDC Process









In-Memory Processing

•Will require re-writing of the Knowledge Modules

‣Should also persist connections

•Option 1: Remove the writes to the $ tables and attempt to do moreoperations on-the-fly

‣Potential loss of audit trail and reconciliation points

‣Currently all outer joins are materialised

‣Will need to perform

•Option 2: Use In-Memory database?









In-Memory Processing

•Will require re-writing of the Knowledge Modules

‣Should also persist connections

•Option 1: Remove the writes to the $ tables and attempt to do moreoperations on-the-fly

‣Potential loss of audit trail and reconciliation points

‣Currently all outer joins are materialised

‣Will need to perform

•Option 2: Use In-Memory database?

Exalytics?









Conclusion










The objectivesof the projectwhereachieved. Theresulting datais being usedon a dailybasis andproving

significantvalue to theorganisation

Conclusion

•The Oracle Reference Data Warehouse architect can

support real time event driven ETL, however it mayneed modifications

• IDO has some rough edges and kinks that need to beironed out for it to act at this kind of enterprise level

•Don’t underestimate the effort of doing a data migration

• Its important to understand the implications anddifferences of middleware centric data models andprocessing compared with databases centric ones









Questions?









Jon Mead, Rittman MeadSeptember 2012

Event Driven Real Time Analytics





event driven real time analytics

Documents