do you really need a data warehouse senturus webinar

DO YOU NEED A DATA WAREHOUSE?

WHY PROPERLY STAGED DATA IS CRITICAL TO BI SYSTEM SUCCESS

• Introduction

• The Quick Answer

• Why Business Intelligence (BI)

• Challenges & Basic Requirements of BI Systems

• Reporting Direct from Source Systems

• Technical Solution Alternatives

• Data Warehouse Benefits

• How to Build a Data Warehouse (20,000 foot view)

• Additional Resources & Upcoming Events

• Q & A

TODAY’S AGENDA

2Copyright 2014 Senturus, Inc. All Rights Reserved

Submit questions here

Click arrow to restore full control panel

GOTOWEBINAR CONTROL PANEL

Copyright 2014 Senturus, Inc. All Rights Reserved 3

PRESENTATION SLIDE DECK ON WWW.SENTURUS.COM


CRITICAL SUCCESS FACTORS IN BI

• Architectures & Data Transformation

• BI Tools

• Methodologies & Techniques

• People & Processes

Chapters in the BI Demystified Series

CRITICAL SUCCESS FACTORS IN BI

• Architectures & Data Transformation

• Data Marts & Data Warehouses

• BI Tools

• Methodologies & Techniques

• People & Processes

Chapters in the BI Demystified Series

John

Peterson CEO & Co-Founder

Senturus

TODAY’S PRESENTER

7

WHO WE ARE

SENTURUS INTRODUCTION

Our Team:

Business depth combined with technical expertise. Former CFOs, CIOs, Controllers, Directors, BI Managers & Enterprise BI/DW Architects

SENTURUS: BUSINESS ANALYTICS ARCHITECTS


Business Intelligence Enterprise Planning Predictive Analytics

Creating Clarity from Chaos

750+ CLIENTS, 1600+ PROJECTS, 14+ YEARS


• Former Head of BI/ Lead Architect – VISA

• Former BI Architect – Jamba Juice

• Former Head of BI – Dole

• Former Chief BI Architect – Cisco

• Former BI Architect – Daimler AG

• Former Lead of IT Architecture – Paramount Pictures

• Former Head of BI – Experian

• Former Head of BI – Robert Half International

• Former Head of Training (IBM Cognos, Southern California)

• Former Controller – The GAP

• Two former CFO’s

• Several former Vice Presidents of Marketing

• Several former COO’s

• Several Former CIO’s

• Former Partner - PWC ($50million+ projects)

• Average experience = over 20 years

A Few of Our Team Members (former roles)

Deep & Pragmatic Experience

Copyright 2014 Senturus, Inc. All Rights Reserved.11

WHAT DO YOU USE FOR BI DATA

“STORAGE” TODAY?

QUICK POLL

DO YOU INTEND TO DEPLOY AN ENTERPRISE

DATA WAREHOUSE AT SOME POINT?

QUICK POLL

DO YOU REALLY NEED A DATA WAREHOUSE?

QUICK ANSWER

The short is answer is:

Almost always, YES

DO YOU REALLY NEED A DATA WAREHOUSE*?

15Copyright 2014 Senturus, Inc. All Rights Reserved.

* or Conforming Data Marts

The rest of this presentation will focus

on why…

WHY?


DATA-DRIVEN INSIGHT LEADS TO

BOTTOM LINE RESULTS

WHY BUSINESS ANALYTICS?

BUSINESS INTELLIGENCE DRIVES COMPETITIVE ADVANTAGE


11.3%

14.0%

12.1%

0.5%

9.4% 9.3% Value Integrators

All other enterprises

EBITDA5-year CAGR, 2004-2008

Revenue5-year CAGR, 2004-2008

ROIC5-year average, 2004-2008

49% more 30% more> 20x more

Source: IBM Institute for Business Value, The Global CFO Study 2010

GETTING THE RIGHT INFORMATION TO THE

RIGHT DECISION MAKERS AT THE RIGHT TIME

THE CHALLENGE

SOURCE DATA IS NOT ACTIONABLE INFORMATION


Standard

Reports (Push-Pull)

Dashboards/

Scorecards

Self-service Reporting

& Ad-Hoc Analysis

Alerts

The

Chasm ERP, CRM Data

Planning Data

De

cis

ion

s &

Acti

on

s

So

urc

e S

yste

ms o

f R

eco

rd

Other Sources

“What do you want?”

“What do you have?”

THE TYPICAL SOLUTION


Standard

Reports (Push-Pull)

Dashboards/

Scorecards


& Ad-Hoc Analysis

Alerts

ERP Data

CRM Data

Planning Data

De

cis

ion

s &

Acti

on

s

So

urc

e S

yste

ms o

f R

eco

rd

Other Sources Or more specifically….

THE TYPICAL SOLUTION* (DETAILED)


Standard

Reports (Push-Pull)

Dashboards/

Scorecards


& Ad-Hoc Analysis

Alerts

ERP Data

CRM Data

Planning Data

De

cis

ion

s &

Acti

on

s

So

urc

e S

yste

ms o

f R

eco

rd

Other Sources * Often coupled with individual acts of Macro & VLOOKUP

heroism, done infrequently and inconsistently

Excel

Powerpoint

Access

Solu

tions

Manually process in Excel

Combine multiple sources

Find, organize and align

data

Filter non-relevant data

Calculate missing

measures

Publish and distribute

reports

Use BI Tools toproduce reports

(scheduled and on-demand)

Use ETLto populate a mart/DW

(write once, run daily)

OR

But, most reports require business logic be applied to data

Pro

ble

mWHY IT PAYS TO BUILD A AUTOMATED BI SYSTEM


Save money, make money

“We just want a report”

Need

Repeat for everyreport, everymonth

Build Once

Build Once

THE REAL CHALLENGE IN A NUTSHELL

The Data has to be Transformed somewhere between the source systems and the end-user

The question is simply – WHERE ?

1. By the End-User (In Excel, etc)

2. By the Front-end BI Tool (with live queries)

3. By an Intermediate process & staging area (ETL, DW)

BUT FIRST, SOME BUSINESS INTELLIGENCE

MUST-HAVES & GIVENS

BASIC REQUIREMENTS

• Deliver a stable & user-friendly data structure– Reports will not break if source system files change– Foundation for true “Self-service” reporting and analytics

• Provide fast performance

– Especially for ad hoc reporting and interactive dashboards

• Handle multiple sources of data

– Cross-functional facts (metrics) and dimensions

• Deliver high quality, validated data

• Maintain historical data in a common format – Even if source systems change or grow– Also, maintain historical context of data (SCD’s) – Allows for trending and “as-of” analysis

A FEW UNIVERSAL BI SYSTEM REQUIREMENTS


• Provide additional ways to “roll-up” data

– Hierarchies, attributes, defined metrics

• Provide field, table & measure names that make sense to business users

• Enable pre-calculations for commonly used measures

– E.g Gross margin, ratios, special qualities (pounds, gallons, etc)

• Provide user & role based security

– Often different than authentication within OLTP environment

A FEW UNIVERSAL BI REQUIREMENTS (CONT.)


WHY NOT SIMPLY POINT THE BI TOOLS AT THE

SOURCE SYSTEMS?

ARCHITECTURAL OPTIONS

DIRECT CONNECTION TO SOURCE SYSTEM

ERP Data

Labor Data

StandardReports

Web P

ort

al

Other SourcesAd h

oc

Query

ingPlanning Data

Slic

ing &

D

icin

g

Dash

board

Auth

oring

Report

Auth

oring

Dashboards/Scorecards

Sourc

e S

yst

em

s of

Reco

rd

Thre

shold

Ale

rtin

g

Self-service Reporting& Analysis

Threshold-basedAlerts

Excel

Planning “Data Set”

Sales “Data Set”

Finance “Data Set”

HR “Data Set”

Other “Data Set”

…

• Transaction processing (OLTP) systems are optimized for Data Entry, not Reporting

– Highly normalized, atomic level data

– Few indexes

– Cryptic naming (tables, columns)

– Odd formats (e.g. Julian dates, non-decimal numbers

– Priority often given to transaction processing

• OLTP systems change over time

– System upgrades, inducing structural changes

– System migrations

– Company acquisitions bring new sources

• OLTP systems not designed for rich metadata and hierarchies

– Limited fields and flex (UD) fields

– Little to no control over uniqueness of rollups

– Dimension maintenance is tedious at best

OTHER CHALLENGES OF DIRECT CONNECTION


Performance

Usability

Stability

Usability

• Reporting queries can adversely impact OLTP data entry

– Queries are often intensive

• OLTP systems lack historical data and context

– Deleted records

– Legacy data often lost

– Only current values stored

• OLTP systems not capable of storing data from other/all sources

– Despite claims, source systems are not good repositories of other system data

– Multiple sources often don’t have common keys, structures relationships, granularity, etc.

• OLTP system security typically does not match BI needs

– Additional users and roles

– Extra licenses

– Unnecessary (& risky) access and complexity

CHALLENGES OF DIRECT CONNECTION OR REPLICATION (CONT.)


Performance

Usability

Usability

Secure

OPERATIONAL DATA SOURCE: SAP EXAMPLE

* Just a few of the over 70,000 tables in SAP R/3

A FEW OTHER PROBLEMS WITH REPORTING

DIRECT FROM SOURCE (OLTP) SYSTEMS

MORE CHALLENGES

ERP, plus…

CRM

Master data of all types

Plans, forecasts, budgets

Security data

POS or channel data

Shop floor (or equiv) data

3rd party data

Big Data

…

THERE IS ALWAYS MORE THAN ONE SOURCE


ONE PROBLEM WITH TRYING TO REPORT ACROSS SUBJECTS


Cross joins (between query subjects [Reporting View, DEPT BUDGET,

Reporting View, DEPT EXPENSE] are not permitted in the identity.

Without conformed dimensions

Upgrades

Migrations

Re-implementations

Acquisitions

…

SOURCE SYSTEMS CHANGE OVER TIME


Yet, good BI relies on historical data

trending and context

My Big Epiphany:

Business Intelligence Success Hinges on Dimensional Data

Source systems NEVER support all the rollups, attributes & hierarchies

Rollups, attributes & hierarchies changeALL the time

ROLLUPS & HIERARCHIES MUST BE ADDED

Date & Time

Financial - Departments

Financial - Chart of Accounts

Product (often multiple)

Brand

Sales Territory

Customer

Employees/Management

Supplier

Asset

Geography/Location

Etc.

A FEW HIERARCHY EXAMPLES

Company reorgs

Multiple product hierarchies

Finance version vs. Marketing version

Sales territory realignment

Management hierarchies vs. geographic territories

Pre- and post- acquisition rollups

Multiple division rollup disparities

External supplier and third-party data hierarchies vs. internal

Temporary groupings (promos, tiger teams, etc.)

A FEW CLASSIC EXAMPLES OF CHANGE

SO WHAT DO WE NEED TO DO…

TECHNICAL SOLUTION

• Separate intensive query and reporting tasks from servers & disks used by transaction processing (OLTP) systems

• Create data models and technologies optimized for query and reporting that are NOT appropriate for transaction processing.

– E.g. bit-mapped indexes, denormalized tables…

• Transform data and embed “knowledge,” roll-ups and business logic into the data structures so that non-IT users can perform “self-service BI”

• Create a single location where information from multiple source systems can be accessed and combined for reporting purposes.

SO WHAT DO WE NEED TO DO (TECHINICALLY)


• Provide a validated repository of data that has been cleaned of inaccurate or spurious data quality issues.

• Maintain a repository of historical data gathered from prior and legacy sources, as well as data that would otherwise be purged from the current transaction processing system(s).

• Allow for secured access to data for analytics without opening up access to systems where data might inadvertently be modified, or transaction processing performance hindered.

• Provide a stable platform upon which end-users can build customized reports, dashboards and analytics

– Regardless of source system gyrations over time

SO WHAT DO WE NEED TO DO… (CONT.)


Create a

Data Warehouse

IN OTHER WORDS…


THE REAL SOLUTION

1. Properly staged data Extracted

Transformed

Enhanced & Combined

Validated

Delivered

2. Good tools to “consume” and use the information Report

Monitor

Analyze

Properly Staged Data BI Tools

The Real Solution


Sourc

e S

yst

em

s of

Record

Single Version of the Truth

Data

Abst

racti

on M

odelInformation Security

ReportAuthoring

DashboardAuthoring

Slicing &Dicing

Ad HocQuerying

ThresholdAlerting

ERPData

LaborData

OtherSources

PlanningData

StandardReports


Self-ServiceReporting &Analysis

Threshold-based Alerts

Web P

ort

al

WHAT IS A DATA WAREHOUSE?

DEFINITIONS

“ A data warehouse is a subject oriented, integrated, nonvolatile, time variant collection of data in support of management's decisions"

Bill InmonBuilding the Data WarehouseJohn Wiley & Sons, Inc., 1992

Classic Definition: Data Warehouse


WHAT IS NOT A DATA WAREHOUSE?

DEFINITIONS (PART 2)

DATA WAREHOUSE = COPY OF SOURCE SYSTEM?

ERP Data

Labor Data

StandardReports

Web P

ort

al

Other SourcesAd h

oc

Query

ingPlanning Data

Slic

ing &

D

icin

g

Dash

board

Auth

oring

Report

Auth

oring


Sourc

e S

yst

em

s of

Reco

rd

Thre

shold

Ale

rtin

g



ERP Data“Warehouse”

Labor Data“Warehouse”

Other Data“Warehouse”

Planning Data“Warehouse”

Excel

Planning “Universe”

Sales “Universe”

Finance “Universe”

HR “Universe”

Other “Universes”

…

Replication

DATA WAREHOUSE = NON-INTEGRATED DATA MART SILOS?

ERP Data

Labor Data

StandardReports

Web P

ort

al

Other SourcesAd h

oc

Query

ingPlanning Data

Slic

ing &

D

icin

g

Dash

board

Auth

oring

Report

Auth

oring


Sourc

e S

yst

em

s of

Reco

rd

Thre

shold

Ale

rtin

g



Excel Spreadmart

Planning Datamart

Sales Datamart

Finance Datamart

HR Datamart

Other Datamarts

…

ETL Processes

Integrated Data BI Tools

TRUE INTEGRATED DATA WAREHOUSE


* Also known as a Star Schema

Sourc

e S

yst

em

s of

Record

ConformingBusiness ProcessDimensional Models*

Single Version of the Truth

Data

Abst

racti

on M

odelInformation Security

ReportAuthoring

DashboardAuthoring

Slicing &Dicing

Ad HocQuerying

ThresholdAlerting

ERPData

LaborData

OtherSources

PlanningData

Data

Inte

gra

tion (

ETL)

StandardReports


Self-ServiceReporting &Analysis

Threshold-based Alerts

Web P

ort

al

WHY INTEGRATED DATA MARTS OR WAREHOUSE?

Enhances Reporting Performance and Flexibility– Data Marts are organized as denormalized data structures for speed and

ease of Reporting vs. Transactional system.

– Offloads the transactional system of reporting requests

– Drill-to-detail (regardless of data location)

Enables Data Integration or Cross Business Analysis– Enables analysis across business processes and functional areas

– Allows data from multiple sources to be integrated into one source of truth with common dimensionality (GL, Planning, Payroll, Sales)

– Discussion of conformed dimensions

– Example: Budget vs. Actuals

Allows Historical Data and Trend Analysis– Captures historical perspective vs. snapshot in time. (ex.Sq ft)

– Allows shifts in sources systems seamlessly

WHY INTEGRATED DATA MARTS OR WAREHOUSE? (CONTINUED)

4. Allows for Automation of business rules & transformations to human-readable information

– Insulates Business Users from cryptic structures and changes in the source systems

– Discussion of Transformation Layers

5. Allows for Additional org/hierarchy rollups & groupingsnot provided by source systems

– ALWAYS needed, never 100% supported by sources

– Should be table driven

Manual

EffortVlookups

Fragile

Macros

WHY INTEGRATED DATA MARTS OR WAREHOUSE? (CONTINUED)

6. Flexible Architectures enables reporting flexibility, i.e. the right tool for the right job

– Robust Reports for operational needs (plus, automatic delivery)

– Cubes for analytics, what if and scenarios

– Ad Hoc Reporting

– Dashboards & Scorecards for Management

7. Empowers Business User Self Service through any of the avenues from above

– Provides the ability to drill into the “Why?”

SPECIFIC EXAMPLES OF WHEN A DATA

WAREHOUSE ADDS VALUE

BENEFITS

Aggregation

Pre-calculation

Fewer joins

Simple joins

Incremental loads

Indexing and Optimization for DW

Less logic at the reporting layer

PERFORMANCE ENHANCEMENT EXAMPLES


Nomenclature transformation

Both measures and dimensions

Lookups - Measures (e.g. cost)

Lookups – Dimensions (rich master data)

Date conversions

“Pre-computed” Date logic

Granularity matching (e.g. plan vs. actual)

Business logic application

e.g. Definition of Revenue

USABILITY & VALIDITY ENHANCEMENT EXAMPLES

ENABLE QUERIES ACROSS SUBJECT AREAS

Product ProductType

ProductCategory

ProductClass

SuperBallpoint Pen

Ballpoints Pens Education

Metal Writer Pen

Ballpoints Pens Business

Felt Great Felt Tips Markers Education

Product Supplier MaterialType

Product Category

Product Class

Super Ballpoint Pen

Acme Ballpoints Pens Plastic

Metal Writer Pen

XYZ Ballpoints Pens Metal

Felt Great Acme Felt Tips Markers Hybrid

Marketing’s Product Dimension table: Manufacturing’s Product Dimension table:

Product Product Type Product Category

Marketing Product Class

Manufacturing Product Class

Supplier

Super BallpointPen

Ballpoints Pens Education Plastic Acme

Metal WriterPen

Ballpoints Pens Business Metal XYZ

Felt Great Felt Tips Markers Education Hybrid Acme

Conformed Product Dimension table:

Befo

reAft

er

CONFORMING DIMENSIONS (EXAMPLE)

Source:

The Data Warehouse Toolkit

© Ralph Kimball, Margy Ross

John Wiley & Sons, Inc.

Maintaining accurate historical context (SCD’s)

Snapshots and balances (e.g. inventory)

Transactionless Facts (e.g. promo periods)

Trending

EXAMPLES OF SPECIAL CHALLENGES

ACCURATELY MAINTAIN HISTORICAL CONTEXT

20092010

Store #23: 8,000 sq ft

Store #23: 20,000 sq ft

Year Store ID Store Size

Revenue Rev/sq ft

2009 23 8,000 $500,000 $63

2010 23 20,000 $1,300,000 $65

Year Store ID Store Size

Revenue Rev/sq ft

2009 23 20,000 $500,000 $25

2010 23 20,000 $1,300,000 $65

Accurate Historical Context:

Report that uses Store Size attribute from ERP table:

Store gets remodeled

Incorrect !

Rich, built-in date functionality (MTD,QTD…)

Pre-calculated time intervals, and other derived metrics

Data-driven security

Reduced licensing costs

And lots more….

A FEW MORE EXAMPLES

HOW TO BUILD A DATA WAREHOUSE

-- A 20,000 FOOT VIEW

FINAL TIP

DATA WAREHOUSES WITHOUT THE NEGATIVES

• Recommendation: Don’t set out to build a data

warehouse [i.e. “Boil the ocean”]

• Instead, build a series of business process

dimensional models with conformed

dimensions

• The result will provide the benefits of a data

warehouse without you ever having done a

data warehouse project.

Business Process Dimensional Models

Date Time Store Product, etc. Qty Ext Cost Ext Amount Margin,.

Dimensions (Attributes) Measures (Metrics)

Store Key

Store ID

Store Name

Store Loc

Store

Region

Store Size

Store Age

…

Product ID

Product Name

Product Class

Product Line

Product Weight

Shipping Cost

…

Date

Year

Quarter

Month

Week

Day

Day name

…

Added (Rolled-Up)

Averaged

Calculated

…

CONFORMED DIMENSIONS = KEY TO INTEGRATION

Date Time Store Product Qty, Revenue, Gross Margin

Dimensions (Attributes) Measures (Metrics)

Store ID

Store Name

Store

District

Store

Region

Store Mgr

Store Age

…

Product ID

Product Name

Product Class

Product Line

Product Weight

Shipping Cost

…

Date

Year

Quarter

Month

Week

Day

Day name

…

Added (Rolled-Up)

Averaged

Calculated

…

Date Time Store Product Plan Qty, Plan Rev, Plan Margin

Product

Line

Measures / Facts

Amount, Quantity

Units = 10

Amount = $17,525

Cost = $8,000

District

StoreProduct

(SKU)

Product

Subclass

Channel

Calculations &

Consolidations

Margin, Roll-ups…

Quarter

Year

Month

Week

Day

Period 1

Versions

Scenarios

Actuals

Forecast

MTD

QTD

YTD

WTD

Season

Period 2

Product

Class

Territory

Sales Rep

Old RegionNew Region

DIMENSIONAL MODEL (SIMPLIFIED)

CLOSING ARGUMENTS

CONCLUSION

We agree that some Additional Repository (other than the source system) is needed.

We agree that some Transformations should be done once during ETL, not live in every query.

We agree that Rich Dimensionality and Transformation adds tremendous value to data.

We agree that it is critical to lay the Proper FoundationBEFORE you start building tons of reports, etc.

CLOSING ARGUMENTS


Therefore, we agree that:

We need a Real Integrated Data Warehouse

We need to do it Right

We can & should build it Incrementally

And we need to do it Now

CLOSING ARGUMENTS


FROM IBM AND SENTURUS

ADDITIONAL RESOURCES

PRESENTATION SLIDE DECK ON WWW.SENTURUS.COM


www.senturus.com/events

• Sept 11 Beginning Authoring Tips & Tricks in Cognos BI

• Sept 17 Houston Cognos Users Group

• Sept 18 Improving the Planning Cycle for Sophisticated Business Needs

UPCOMING EVENTS


Save $400

By registering before October 3rd and using

Senturus’ preferred customer code

G14SNTURUS, you’ll get an additional $100

off the early bird discount for a total of $400

savings.

PLUS $200 Training Credit

Receive $200 in Senturus training credit, good toward any of our 20 live

instructor-led Cognos online training classes.

PLUS $25 in Gambling Chips

At Senturus, we’re betting on you! Registrants will enjoy a welcome gift of

$25 in Mandalay Bay Hotel gambling chips upon arrival.

Save $600 on IBM Insight (formerly IOD)


http://www.senturus.com/ibm-insight-iod-2014/

*Custom, tailored training also available*

COGNOS TRAINING OPTIONS


THANK YOU

do you really need a data warehouse senturus webinar

Documents

bi datastorage

head of bi dole

head of bi experian

staged data

bi architect jamba juice

chief bi architect cisco

bi architect daimler

head of bi lead architect