data virtualization an overview

ON-DEMAND VISIBILITY

Copyright © 2005 Composite Software, Inc. All Rights Reserved.

Ian Pestell

Director EMEA Operations

Data Virtualization an Overview

22Copyright © 2006 Composite Software, Inc. All Rights Reserved.

Is Your Data an Asset or a Liability?

“Managers spend two hours a day looking forinformation they need, and almost half thedata is useless once they get it”

Source: Accenture

“Managers spend two hours a day looking forinformation they need, and almost half thedata is useless once they get it”

Source: Accenture


Data within SOA

SOA Architectures implement Business Process Through Application Logic Operating on Data

Data from many sources Existing Legacy systems Databases, Web Services,

Files Complicated by package

applications, ERP and CRM

Data Services are often addressed late in SOA

SOA

Business

Process

Application

Logic

Business

Process

Application

Logic

LegacyData

ApplicationData

Data


Data Warehouses

RDBMS Flat FilesPackaged

Apps

Web Services

Data Integration Options

ExistingData

ConstantChange

Siloed&

Complex

DataIntegrationChallenge

BusinessSolutions

ReportingDashboards SOA

Code within the Application?

File Extracts / Database Replications?

ETL / Physical Warehouses, Marts, Stores?

EII / Virtual?


Data Consolidation / ETL

Physical data movement via scripts Data is extracted from source

systems Transformed

Data Cleansing, OLTP to OLAP Loaded into target system

Physical Data Warehouse, Mart, Operational Data Store

Very large volumes of data High degree of transformation Data is only as current as last

load

ETL

Server

ETL

Server

DataWarehouse

Data Mart/ ODS

ApplicationApplication

MDMHub

ApplicationsFiles

Databases


Data Federation / EII

Virtual data views via modelling Data is left in-situ within data

sources EII server federates and

transforms data on demand

No Data Warehouse required High degree of flexibility in adding

new data sources Lower development & operating

costs

Moderate to high data volumes Low to medium transformations Data is ‘near’ real-time

EII

Server

EII

Server

ApplicationApplication

cache

ApplicationsFiles

Databases


Solutions to Data Integration – Application Integration

Integration performed in the application logic Data sources require WS API

Vendor – bespoke Access to WS API via ESB

Supported by standards ws-orchestration, ws-choreography

Disadvantages Complex logic with Application High develop time Maintenance of change Enabling WS APIs for data High load on ESB Latency of ESB

Enterprise Service Bus

Application

Logic

LegacyData

ApplicationData

WS API WS API WS API WS API


Solutions to Data Integration – Data Services Layer

Implement Data Services Layer Provides set of reusable

data services WS access into ESB Connects to data sources

using native interfaces Integration / optimization

only done within data layer Reduces load

Enterprise Service Bus

Application

Logic

LegacyData

ApplicationData

Data Services Layer


Customers

Orders

Invoices

Trades

Positions

Vendors

Employees

POs

GLs

Data Warehouses

RDBMS Flat FilesPackaged

Apps

Web Services

You Can Integrate Data Differently

ExistingData

Real-timeInformation

Siloed&

Complex

VirtualizesVirtualDataLayer

Abstracts

BusinessSolutions

ReportingDashboards Applications

Federates


Composite’s Mission

Composite helps IT teams who need to build new applications faster,

BusinessApplications

ReportingDashboards SOA

Data Warehouses

RDBMS XML Docs Flat FilesPackaged Apps

ExistingData

by providing data services middleware that lets them access any data in the enterprise… and deliver it on demand.


What We Do

Data EngineData Engine

ForFor

Operational BIOperational BI

Operational ReportingInformation

Infrastructure

Drug Discovery Order Management Portfolio Management Inventory Management

Drug Discovery Order Management Portfolio Management Inventory Management

Single Source for all Data Standardized Access Reusable Data Services

Single Source for all Data Standardized Access Reusable Data Services

NA

Sal

es

Data ServicesData Services

(SOA)(SOA)

Access

App

Combine

Deliver

>>> On Demand <<<


Composite Software at a Glance

Privately-held, Silicon Valley firm Strong Business Model & Team

Best-of-Breed Virtual Data Integration / EII Discover, Virtualize, Abstract & Federate

Proven Business and IT Value Increased Flexibility Greater Agility Lower Costs

Public Sector

Financial ServicesFinancial Services Life SciencesLife Sciences

Top 5 I-Bank

Top 5 I-Bank

Top 5 CommBank

Top 5 Inv Bank

Top 5I-Bank

MutualFund

Life Sciences TechnologyFinancial Services Consumer/Retail

IE 2008Editor’s Award

Energy and Utility


Data Source 1Data Source 1 Data Source 2Data Source 2

Composite ViewComposite View

Step 1 - Development: Build and Publish a View (A View contains Metadata on the actual data source – format, type, location, access, etc.)

How we do it – Data Views


Data Source 1Data Source 1 Data Source 2Data Source 2

Composite ViewComposite View

1 A B

2 C D

3 E F

4 G H

1 a b

2 c d

3 e f

4 g h

1 A B a b

2 C D c d

3 E F e f

4 G H g h

Step 2 – Runtime: Access and Combine(Queries are optimized for speed and efficiency)

Step 3 – Runtime: Deliver Information(Combined data creates meaningful information)

Reporting Analytics Composite Applications

Step 1 - Development: Build and Publish a View (A View contains Metadata on the actual data source – format, type, location, access, etc.)

How we do it – Data Views


Composite Lets You Easily Construct Data Services

Access DataAccess DataCombine Data/

Model

Combine Data/Model

Request Results

Deliver ResultsDeliver Results Auto-Generated WSDLAuto-Generated WSDL

Access DataAccess DataCombine Data/Model

Combine Data/Model

Request Results

Deliver ResultsDeliver ResultsAuto-Generated WSDLAuto-Generated WSDL


•S.

Data Services

Query Processing Engine

COMPOSITE INFORMATION SERVERCOMPOSITE INFORMATION SERVERCOMPOSITE INFORMATION SERVERCOMPOSITE INFORMATION SERVER

Cache

ApplicationRDBMS Excel Web Service Flat File

Composite Information Server Architecture Easily Works Within Complex Environments

Client and Enterprise Applications

Pre-Built

Custom

Security

Security

Customer-Built

(Views, SQL Script, XQuery, Java)

Mainframe

Composite StudioComposite Studio

Modeler

Manager

Composite Web Composite Web ManagerManager

Relational SQL (ODBC/JDBC/ADO.NET)Web Services (SOAP/JMS)

APIsWS (SOAP/JMS)Relational (SQL) Java MF Adapter

Ad

min

istratio

n A

PI


Why Composite Application Data Services™

Composite Apps

ExcelBI Tools

SQL / WS

Complex API

Enterprise Data

AdaptorApplication

Views


Composite Applications

Data Services (Info Server)

XML/HTTPSOAP/HTTPCustom

AppsXML FilesFilesPackaged Apps

Transaction Services

App Server

1) Data services stand alone2) Transaction services stand

alone3) Data services integrated

with transaction services4) Data services with

orchestration using a business process management tool or ESB

5) Data services WSDL stored in a UDDI registry

Orchestration

ESB / BPMDATADATA DATA

DATA

DATADATA DATA

DATA

Data Services

Info ServerDATADATA

DATADATA

DATADATA

DATADATA

DATADATA DATA

DATA

Data Services are a Critical SOA Component

Registry

“Over 60% of SOA services will be data services”

Source: IDC

“Over 60% of SOA services will be data services”

Source: IDC


NY Investment Bank – New e-business application

Implementation of new e-business application

Initial estimate 2 man years 1 year application 1 data integration Using EAI approach


NY Investment Bank – New e-business application

Implementation of new e-business application

Initial estimate 2 man years 1 year application 1 data integration Using EAI approach

Using Composite Data services Reduces Data integration from 12

Month to 4 Months Created optimized and re-usable

data services Introduced abstraction layer

between application and data


Data Services At Work - New York Bank

DATADATA

DATADATA

DATADATA DATA

DATA DATADATA

DATADATA

DATADATA DATA

DATADATADATA

1. Prime Brokerage Reporting

2. Risk Management

3. Back Office Reconciliation

4. Portfolio Balancing

1. Prime Brokerage Reporting

2. Risk Management

3. Back Office Reconciliation

4. Portfolio Balancing

Multiple Projects

Data Required (100%) Reuse (nearly 40%) Lower Impact (75% or more) Faster Response (up to 80%)

Data Required (100%) Reuse (nearly 40%) Lower Impact (75% or more) Faster Response (up to 80%)

Business and IT Benefits

ExistingData

DataServices

Layer

Data Warehouses

RDBMS XML Docs Flat FilesPackaged Apps


CompositeInformationServer

Single View of Payroll and VariancesHusasmidjan, Inc.

Situation & Requirements: Scandinavian construction materials

retailer/wholesaler, 25 stores

Business requirement: detect and address variances between planned and actual payroll expense

Business user: Office of the CFO

Technical requirement: Expose and combine SAP R/3 and in-house budget data

Solution:

1. Create views to SAP payroll information

2. Create views to other payroll information

3. Build roll-up view for analysis

4. Expose view into budget system

Benefits: More accurate variance information sooner

Rapid application development Legacy

Rollup View

Local views normalize across different stores

and apps

Payroll data sources

Files

data virtualization an overview

Documents