Data Warehousing & Business Intelligence

Introduction

What do you think of when you hear the words Data Warehousing ?

Prithwis Mukerjee, Ph.D.

2©© Prithwis Mukerjee

Conceptual DW Definition

Data warehousing is a program dedicated to the delivery of information which advances decision making, improves business practices, and empowers workers.

3©© Prithwis Mukerjee

Data

Structu

re

Technology

Infrastr

ucture

Man

agem

ent

Business

Applicati

ons

Information

Information

Technology

Technology

Proce

ss

Proce

ss

Peopl

e

Peopl

e

The Knowledge Management Framework

4©© Prithwis Mukerjee

Database

How it all fits in ..

CRM : Customer Relationship Management

Transactional Systems ERP : Enterprise Resource Planning

SCM : Supply Chain Management

Data Warehouse

5©© Prithwis Mukerjee

Target Advertising campaigns

Strategic Initiatives

Business Processes

Functions

ProfitabilityAnalysis

Market Basket

Analysis

Product Pricing

Cross-selling and upgrade selling

Just-in-Time Inventory

Category Management

Human Resources Management

Determine Customer Lifetime Value

Predict Customer

Behavior

Management Reporting

Customer Acquisition and Retention

Typical Business Uses of the Data Warehouse

6©© Prithwis Mukerjee

Benefits of the Data Warehouse Program

Improves the way we do business and the bottom line

Revenue Stimulation & Revenue Protection

Cost Reduction and Cost Avoidance

Productivity Improvement

Profitability Enhancement

Performance Analysis

DecisionMaking

Market Response

Competitive advantage

7©© Prithwis Mukerjee

DSSs,Report writers, Excel, databases, etc.DSSs,Report writers, Excel, databases, etc.

Data FeedsData Feeds

BudgetingBudgeting

AnalysisAnalysis

Non-integrated Decision Support Architecture

Inventory System

Order System

Procurement System

Accounting System

Sales ForecastingSales Forecasting

8©© Prithwis Mukerjee

Enterprise DW/ODS

Subject oriented Data Warehouses or Data Marts

Subject oriented Data Warehouses or Data Marts

One Stop Data Shopping

One Stop Data Shopping

Basic Data Warehouse Architecture

Fewer Data FeedsFewer Data Feeds

Inventory System

Order System

Procurement System

Accounting System

9©© Prithwis Mukerjee

Performance Measures : Definition & Examples

Carefully selected set of measures derived from strategies, goals and objectives that represents a tool to communicating strategic direction to the organization for motivating change.These form the basis to plan, budget, structure the organization and to control results.

Innovation & Learning Measures

Innovation & Learning Measures

Customer MeasuresCustomer Measures

Financial MeasuresFinancial Measures

Internal Process

Measures

Internal Process

Measures

% Sales of New ProductsCustomers AcquiredCustomer Satisfaction

Market ShareROI and ROARevenue Growth

Product Time to MarketUnit Manufacturing CostDays Supply to inventory

New Product IntroductionManagement SkillsEmployee Turnover

10©© Prithwis Mukerjee

Differences between OLTP and DW

Data Access, Manipulation and UseData Organisation and IntegrationTime HandlingUsageData Structures and Schemas

Explanations ..Explanations ..

11©© Prithwis Mukerjee

Data access, manipulation and use

Data EntryTransaction OrientedConsistent use patternsData retrievals are lookups of single recordsUsers deal with one record at a timePerformance is criticalReporting is generally table lists

Data QueryBulk data orientedSpiked, uneven use patternsQueries are unpredictable, they change continuouslyData retrievals are summary and sorts of millions of recordsPerformance is relaxed (sec/min)Reporting is primary activity (on line, presented in small chunks)

OLTPOLTP DWDWDifferences between OLTP and DWDifferences between OLTP and DW

12©© Prithwis Mukerjee

Data Organisation And integration

Organized around applicationsUnintegrated dataDifferent key structuresDifferent naming conventionsDifferent file formats

Organized around subject areasIntegrated dataStandardized key structuresStandardized naming conventionsStandardized file formats

OLTPOLTP DWDWDifferences between OLTP and DWDifferences between OLTP and DW

13©© Prithwis Mukerjee

Time Handling

No time series analysisData relationships constantly changeChanges are instantaneousLimited history, 60-90 days

Twinkling Database ….

Time series analysisData is static over timeSeries of data snapshotsSnapshots create historical database, often greater than two yearsQuiet database

OLTPOLTP DWDWDifferences between OLTP and DWDifferences between OLTP and DW

14©© Prithwis Mukerjee

Usage

Place an order for a productLook up price for a productApply discountAssign shipperTrigger inventory pick-listVerify shipment of productCreate invoice for the productApply credit to sales representative

Essential to RUN the company

What type of customers are ordering this product?Who are my top 10% accounts? By name, by revenue, by profitability, by region?How are these different by customer segments? By sales rep? By store?Which shippers have the best on time delivery records ?How does this vary by shipment size? By season of year?

Essential to WATCH the company

OLTPOLTP DWDWDifferences between OLTP and DWDifferences between OLTP and DW

15©© Prithwis Mukerjee

Data Structures & Schemas

Drives out all data redundancy

Improves performance

Divides data into many discrete entitiesTables are symmetrical

Can’t tell most important, largest, which hold measures, which are static descriptors

Lots of connection paths between tables

prefers to use tables individually or in pairs

Too complex for users to understand

Data redundancy is encouraged

Improves table browsing

Subject area oriented. Groups data into categories of business measure and characteristicsTables are symmetrical

Large dominant tables

Clearly defined connection paths for table joinsSimple for users to understand and navigate

OLTPOLTP DWDWDifferences between OLTP and DWDifferences between OLTP and DW

16©© Prithwis Mukerjee

Basic Datawarehousing Topics

The Four Building BlocksDW DefinitionDW Usage and BenefitsDW Vs. the non-integrated DSS environmentPerformance Measures

Dimensional ModelingTechnical InfrastructureKnowledge Mgmt. ArchitectureIT and Business PerspectivesDW Methodology

17©© Prithwis Mukerjee

Dimensional Data Modeling

Dimensional Data Modeling techniques organize the content of the data warehouse. It structures the data according to the way users ask business questions.

18©© Prithwis Mukerjee

The Technical Infrastructure

A technical infrastructure provides the physical framework to support data acquisition, storage, access, and data management. It involves development and integration of hardware and software components.

19©© Prithwis Mukerjee

Metadata

Source Data

Purchasing Systems

General Ledger

Other Internal Systems

External Data

Sources

Data Resource Management And Quality Assurance.

Invoicing Systems Data

Extraction

Integration

and

Cleansing

Processes

Extract ODS

Purchasing

Marketing and Sales Corporate

information Product Line

Location

Translate

Attribute

Calculate

Derive

Summarize

Synchronize

Segmented

Data

Subsets

Summarized

Data

Data Warehouse Applications

Custom

Developed Applications

Data Mining

Statistical

Packages

Query Access Tools

Data Marts

Transform

Knowledge Management Architecture

20©© Prithwis Mukerjee

The Business and The IT Perspective

Business

What will it do?

What value will it bring?

How is it built?

How does it work?

Information Technology

Data Warehouse

21©© Prithwis Mukerjee

The Business Perspective of the Data Warehouse

It takes forever to get the information I need to do my jobWhen I do get it, it’s wrongWe have mountains of data, but I can’t figure out what’s importantIt takes so long to get the data that I don’t have any time left over to analyze itI want it to be easy. Just let me point and click my way to an answerI want to see my data in every possible combinationData is scattered everywhere across our organization. Where do I look ?I want a historical view of the businessI want to predict the future

Focuses on needs and usage

22©© Prithwis Mukerjee

The IT Perspective of the Data Warehouse

Organizes and stores data by subject area rather than applicationExtracts and integrates data from multiple source systems into a single databaseProvides data cleansing, summarization, and calculationUser does not create, update, or delete dataProvides snapshots of data over periods of timeSupports analytical processing, not transactional processingBuilds a technology infrastructure to support data acquisition, data storage, data access, and metadata captureFocuses on database, technology, organizational features

23©© Prithwis Mukerjee

DW Methodology

The methodology provides a detailed roadmap to organize and perform the tasks required in building the data warehouse

24©© Prithwis Mukerjee

Data Warehouse System Development Life Cycle

CONSTRUC-TION

CONSTRUC-TION

IMPLEMEN-TATION

IMPLEMEN-TATION

DESIGNDESIGNANALYSIS

ANALYSIS

PLANNING MANAGING

Business ArchitectureBusiness Architecture

Data ArchitectureData Architecture

Technology ArchitectureTechnology Architecture

Management InfrastructureManagement Infrastructure

25©© Prithwis Mukerjee

stopstopstopstop