real-time business intelligence

8/6/2019 Real-Time Business Intelligence

1/35

Evolution In Information AccessReal-time Business Intelligence


2/35

Evolving to real-timedata availability

29%instantaneous

44%daily

24%weekly

3%monthly

Data: Gartner Survey of 540 organizations

2006 2006 2002 2002

55%daily

11%instantaneous

17%weekly

17%monthly


3/35

Real Time DataWarehouse DesignChallengesDr. Ralph KimballFounder Kimball Group

2005 Kimball Group / Kimball University All rights reserved.


4/35

Definition of Real Time

Real Time is anythingthat is too fast for your current ETL

When you reach the performance limitations of your current ETL architecture, you are forced tore-architect your system.



5/35

Study the Alphabet Soup

EII Enterprise Information Integration Direct access to source data in warehouse framework

CTF Capture, Transform and Flow Database to database pipe

SOA Service Oriented Architecture XML documents with WSDL payloads inside SOAP envelopes

SETL Streaming ETL Record at a time transfer MBETL Micro Batch ETL

Batch transfer every few minutes or seconds



6/35

Real Time Requires More

But when you re-architect your system, you mustinclude a whole set of other new data warehouserequirements Operational Interaction

Extreme Integration Multi-modal data, especially text search Subtransactional data streams

ComplianceSome of these conflict



7/35

Operational Interaction

Most extreme latency demand Always need the current truthNeed write privileges with locking

Push, pull, and non-eventsGood application of EII Conflicts with integration and compliance



8/35

Extreme Integration

Local and global de-duplication in real time Need multi-valued back pointers to original IDsThe wider the integration,the more difficult to support real timeWorld is rapidly moving to SOA Service Oriented Architecture

Abstract individual system details Compose coarse grain services from fine grain XML data transfer but relational targets



9/35

Multi-modal data

Especially text search Google has rewritten expectations

for content and performanceAll of the web

Deliver search result in less than 1 secondCan expect hybrid combinations of SQL and text search on same databases Fact Tables for Text Document Searching(Kimball article archive: www.kimballgroup.com) old Red Brick pattern index



10/35

Subtransactional Data Streams

Enormous volumes, up to 10 TB/day threatenconventional ETL and RDBMS modelsData streams allow sequential tracking of individualcustomers or productsWe barely understand sequential behavior analysis Most queries today look at flow at a single point

Need to look at voyages through many points New design: Step dimension



11/35

Snippet from Modeling Class:Design of a Step Dimension

Design: Add a Step Dimension to a transaction fact table:



12/35

Compliance

Maintain the custody of data Prove later that data hasnt changed and

that you computed the result correctly Show how information has been used

Column-level encryptionHash-coded data sets Conflicts with operational interaction

Where is the system of record? Compliance webinar (April) on Informatica web site



13/35

How to Proceed(Building a RT Data Warehouse)

Decompose the problem before implementing! Business Needs Compliance Data Profiling

Security Data Integration Data Latency

Archiving and Lineage End User Delivery Interfaces Available IT and End User Support Skills

Legacy Licenses2005 Kimball Group / Kimball University All rights reserved.


14/35

How to Proceed

Choose an extensible architecture good at Simultaneous updating and querying Enormous data volumes, up to TB/day Multi-modal data search

Compliance support Hardware independence

Learn, learn, learn Sybase Education Services (www.sybase.com) Kimball University (www.kimballgroup.com)



15/35



16/35

Real-timeData Integration for Real-timeBusiness IntelligenceDavid LyleVice President, Product StrategyInformatica Corporation

2005 Informatica Corporation


17/35


Real-time BI requires real-time DI

Real-time Business Intelligence requires theintegration of data in real-time

Kimball definition of real-time: Anything that is too fast for your current ETLYou need to re-architect your kitchen, but you still need a sink,oven, stove, fridge,

I f i l i DI


18/35


Informatica real-time DIto the rescue

Different appliances for your real-time kitchenEII virtual integration through query federation

Capture, transform and flow = PowerCenter Real-time optionSOA PowerCenter Connect for Web servicesPowerExchange Change Data Capture options

(Streaming ETL, Micro-batch ETL)PowerCenter Unstructured Data option


19/35


Capture, transform and flow

Always-on sessionsPowerCenter Connect for MSMQ, IBM MQSeries, Tibco,

webMethods, httpCan perform the extreme integration youre used to within thesame PowerCenter platform


20/35


Service-oriented architecture

New way of thinkingNew programming language, too

But SOA doesnt equal real-time data integrationSOA frameworks are enterprise stub-generators, leaving the dataintegration problem up to you!

PowerCenter Connect for Web servicesBuild your data integration logic as Web services, backed up bymetadata and change management


21/35


Change data capture

Approaches to dateAdd timestamps through reverse engineering

User exits that interrupt transaction streamsAdd triggers that impact application performanceFull file compares to determine deltas

Home-grown solutions technically challenging

all disruptive, all programming-centric!


22/35


Value

Simplify with one standard for all data types and latenciesExtend value of existinginvestments

Accelerate project lifecyclesReduce solution developmentand maintenance costs

Technology

Complex source accessPatented changed data captureLatency: real-time, change, batchNon-invasive, service-orientedarchitectureCodeless, visual SQL drivenEnterprise scalability

Informatica PowerExchange

Unlock complex data on demand


23/35


Real-time analytics

The reduction in data integration overhead may beits own justification for a real-time data warehouse

Less data to move, reduce the load window, keepup with increasing data volumes, smaller systemand disk sizes requiredIts a win, win, win situation!

And then if the business also has aBI requirement for real-time, thats obviously

the most important justification of all!


24/35



25/35

Business Intelligenceon DemandInformation at the right time,right place, right priceKapi Attawar Senior Director Sybase, Inc.

2005 Sybase, Inc.


26/35

Sybase solutions

2005 Sybase, Inc.

ON-DEMANDMOBILITY SERVICES

AvantGo

Mobile Business Services AvantGo Premier Hosted PortalHosted Emails

INFORMATIONANYWHERE

SQL AnywhereM-Business AnywhereTM

Mobile Link

Unwired Accelerator Unwired Orchestrator

Answers AnywhereTMMobile Sales FamilyPylonTM

AfariaMach DesktopTM

mFolioTM

RFID Enterprise/RFID AnywhereTM

HIGH-PERFORMANCEDATABASE

ASE

ENABLE BETTER DEVELOPMENTWorkSpace, PowerDesigner , PowerBuilder , PocketBuilder TM for client/server, Web and mobile

INFORMATION MANAGEMENT (DATACENTER) INFORMATION MOBILITY (DE

DATA

KNOWLEDGE

TAMING THEDATA EXPLOSION

Heterogeneous Data Services

Dynamic ArchiveMirror Activator TMDynamic ODS

Sybase IQ Auditing

Replication Server Real-Time Data Services

Enterprise InformationIntegration (EII)

Unstructured Data


27/35

Business problemCritical data in silos and in

different formsUsers need dynamic access todata in real time

Solution: Avaki EII

Cost-effective tool to developapplicationsSeamless access to disparatedata sources

Provides customers with virtualviews of: Multiple data sources Federated data access SOAIntegration and query capabilities

Business problemRapid growth of enterprise

information outside structureddatabasesThis unstructured data comesin a variety of formats emails,videos, PowerPoint, etc

Solution: ISDDProvides enhanced query andanalysis of both structured andunstructured dataAddresses both fixed andmobile sources

Recent acquisitions

2005 Sybase, Inc.

Just-in-time


28/35

Extract

DataWarehouse

Just in timeinformation architecture

OperationalData Store

ASE

Analytics

Business ObjectsCognos

Microstrategy

HyperionSPSSSAS

Crystal ReportsMicrosoft Excel

SAP BW

Applix

2005 Sybase, Inc.

Just-in-time


29/35

Transactions

Access

Avaki

Extract

DataWarehouse

Just in timeinformation architecture

OperationalData Store

ASE

Analytics

Business ObjectsCognos

Microstrategy

HyperionSPSSSAS

Crystal ReportsMicrosoft Excel

SAP BW

Applix

The business opportunityDemand for real-time business reportingand analyticsImprove service to business by movingaway from batch reporting

2005 Sybase, Inc.

Wh d I ?


30/35

What do I use?

Sybase Dynamic ODS with InformaticaOperational system overload and higher operational costs causedby data and user growthCut costs reporting from OLTP systems is very expensiveImprove performance transactions currently adversely affectonline reportingNeed both current and historical data

EII with Sybase AvakiOperational systems overload not the primary challengeOffload is not a good option too hard, data too dynamic, too muchfragmentation, mix of operational and OLAPReal-time accessibility with caching for performance and resourcemanagement

Ideal for a grid environmentMoving to an SOA architecture 2005 Sybase, Inc.

Real-time reporting


31/35

Real time reportingDynamic ODS with Informatica

ETL Tool (Informatica)

Harvest data in near real-timeSource database agnosticHigh-speed data harvestingDisaster recovery capability

Data on-demandwithout stressing theOLTP systemRisk-free approach,does not interfere withoperational systems

Dynamic ODS (Sybase)

Non-intrusive, risk freeData compression (up to 60%) for structuredand unstructured dataOptimized for fast-read access column-

based, not rowStandard SQL access to data

2005 Sybase, Inc.

Real-time reporting


32/35

ea t e epo t gEII with Avaki

EII platform (Avaki)

Real-time view of operational dataSource database agnosticHigh-speed data queries across distributedsystems

Configurable caching for improved performanceand reduced operational impact

Complex operationalenvironments

Managed approachto operationaldata access

Dynamic data transformationon-the-flyStandard SQL access to dataSupports updates and queries

Service Oriented Architeture

2005 Sybase, Inc.

DatabaseEnvironment

ETL Tool

Data Sources

Avaki Views

Streamlining mainframes


33/35

gand Teradata

Dynamic ODS customer case studiesPicknPay (South Africa) Replaced 20 node Teradata with 8 CPU Xeon environment

Major global credit card company (USA) Fraud detection investment of $4m = saving of tens of millions

Telefonica (Europe) Single view of customer - investment $4m = savings of $15m in first year Mainframe, Oracle and Teradata co-existence

HIRA (Korea) Enabled storage of five years of data on-line, improved query time from 4 days to 30 seconds Beat Teradata, DB2 and Oracle on performance and cost

Avaki customer case studies

Pfizer (USA) Daily sales dashboard real-time access to data from multiple data sources 600% improvement in analytic performance

Pacific Capital Bank (USA) Risk management single view of customer and risk scoring

Real-time access to data, transform and present in application-specific format

2005 Sybase, Inc.

Information at the right time,


34/35

g ,right place, right price

NeedsRelieve operational system overloadCut operational costsImprove transactional performanceProvide both current and historical data

Solution: Sybase Dynamic ODS and Informatica

NeedsOffload not a good optionAccess to operational system data in real-timeProvide a single virtual view of dataMergers and transitions

Solution: EII with Sybase Avaki

2005 Sybase, Inc.


35/35


real-time business intelligence

Documents