holistic data warehousing sample

7/27/2019 Holistic Data Warehousing Sample

1/35

Page | i

HOLISTIC DATA WAREHOUSINGON MICROSOFT SQL SERVER 2008

A New Data Warehousing Strategy

Methodology and Guide to the FreeReady Made Template for Full

Supply Chain and Sales & Operation

Reporting

Gerry Phillips and Jane McCarthy

For-tee Too Sight PublishingMelbourne, Australia

www.42sight.com

42sight.blogspot.com


2/35

Page | ii

HOLISTIC DATA WAREHOUSING

ON MICROSOFT SQL SERVER 2008

Copyright 2010 by Gerry Phillips and Jane McCarthy

ISBN 978-0-9808742-4-2

Published by

For-tee Too Sight Publishing

49 Mowbray Drive

Wantirna South

Melbourne, Australia, 3152

www.42sight.com

42sight.blogspot.com

All rights reserved. No part of this book may be reproduced in whole or in part without

written permission from the publisher except in the case of brief quotations embodied

in reviews. For information address: For-tee Too Sight Publishing, 49 Mowbray Drive,

Wantirna South, Melbourne, Australia, 3152


3/35

Page | iii

Contents

Acknowledgements ...............................................

Imagine .................................................................

Introduction ..........................................................

Who Are We? ........................................................................................

Book Overview .......................................................................................

How this Book is Structured ...................................................................

What is a Data Warehouse? ..................................Our Definition of a Data Warehouse ......................................................

The Data Warehouse Environment ........................................................

A DW System Compared to a Transactional System ...............................

A Comparison of a Holistic DW to a Conventional DW ............................

Full Supply Chain Reporting One Page .............................................

What Data Did Our First Data Warehouse Have? ...................................

What is a Data Warehouse? Summary ................................................

The Holistic Data Warehouse Strategy ..................

The Holistic Data Warehouse Vision ......................................................

The Standard Data Warehouse Goal ....................................................

What is the Supply Chain? .....................................................................

Full Supply Chain Reporting Goals ..........................................................

Objectives of All Data Warehouses ........................................................

Minimize Inconsistent Reports and Reconcile Different Views of the Same Data Improve Quality of Data ...................................................................................


4/35

Page | iv

To Consolidated Enterprise Data from Multiple Sources and Time Periods ......... 32 Make the Data Easily Accessible and Provide Transparency ............................... 33 To Enable Common and Flexible Calendars ........................................................ 35 To Save Time on Report Preparation and Construction ....................................... 35 To Address the Weaknesses of Current Reporting Systems ................................. 36 To Empower People with Information ................................................................ 37The Additional Objectives of the Holistic DW .......................................... 38

To Enable Pre-emptive Reporting of Events that are Expected to Happen........... 38 To Enable a Single Cross Functional Report ........................................................ 39 To Offer all Supply Chain Related Information .................................................... 42 To Effortlessly Replace All Your Reporting Systems ............................................. 45 To Address Deficiencies in the Operational Systems ........................................... 46 To Be Capable of Unlimited Measures ................................................................ 48 To Allow Unlimited Product Hierarchies ............................................................. 53In Summary the Benefits of the Holistic DW............................................ 53

The Holistic DW Strategy in Summary ..................................................... 54

Our First Data Warehouse Project ......................... 55

Why the Need for a Data Warehouse? .................................................... 55

The Consultants Health Check ................................................................ 56

The Project Begins with a Simple Objective ............................................. 57

Our Data Warehouse Design Brainstorm ................................................. 57

Two Hours Later a New Methodology ................................................................ 57Other Factors That Influenced this New Methodology ............................ 61

Two Weeks Later the Prototype .............................................................. 62

Apparently we are Doing it the Wrong Way! ....................................... 63

We Decide to Research the Correct Way ............................................. 64


5/35

Page | v

The Prototypes First benefits for the Business ......................................

Senior Management Excited About the System .....................................

From Access Prototype to Microsoft SQL Test Server ............................

The Final Server and Launched Six Months After Starting ......................Switching Off the Old Reporting Systems ...............................................

SAP Story ...............................................................................................

Summary of Our First DW Project ..........................................................

Technical Section ..................................................

Introduction to the Technical Section ....................................................

Data Warehouse Infrastructure ............................

Microsoft SQL & Windows Server 64 bit ................................................

The Holistic DW Under the Hood .......................

The Translated Data Table (TDT) ............................................................

The Holistic Data Warehouse Data Table ...............................................

The Holistic Data Table vs the Star Schema Fact Table ..................................... The Overall Data Table Structure ...................................................................... The Data Table Data Elements .........................................................................The Holistic Data Warehouse Linked Tables...........................................

Smart Values in the Linked Identity Field .......................................................... Holistic Linked Table Joins ................................................................................ The Types of Linked Tables Used....................................................................... Conforming in the Holistic DW Model Linked Tables ...................................... Linked Tables in More Detail ............................................................................The Translated Data Table Linkages .......................................................

The Simple Linkages with Single Links: ..............................................................


6/35

Page | vi

The Complex Linkages with Multiple Links: ....................................................... 146 The Translated Data Table Conversion Calculation ........................................... 148 The Translated Data Table Summary ..................................................... 148

The Full Supply Chain Top Down/Bottom Up Reports ........................... 149

The Top-Down Reporting ............................................................................... .. 150 The Bottom-Up Reporting ................................................................................ 153 Full Supply Chain Reporting in Summary .......................................................... 154The Holistic Data Warehouse In a Nutshell ........................................ 154

The Rules of Data Warehousing ....................... 155

The Holistic View Are They Applicable and Why? ............................ 155

Rules That Are Applicable to the HDW .................................................. 156

Where the HDW Bends the Rules .......................................................... 162

Where the HDW Breaks the Rules ......................................................... 165

Our Bottom Line on Data Warehouse Rules .......................................... 173

How to Populate the Holistic Data Warehouse ... 174

The SQL Server SSIS Import and Export Wizard .................................. 175The Microsoft Adventure Works Holistic DW Load ................................ 184

The Initialisation Phase .................................................................................... 186 Loading the Data Table .................................................................................... 187 Loading the Item Tables ................................................................................... 216 Loading the Entity Tables ................................................................................. 228 Loading the Representative (Rep) Table ........................................................... 237 Loading the Reason Table ................................................................................ 239 Loading the Conversion Table........................................................................... 243 The Wrap Up Section of the Adventure Works Load ......................................... 248 In Summary the Adventure Works Load Into the Holistic DW ........................... 265


7/35

Page | vii

The Data Loading Process (DLP) .............................................................

ETL, ELT, ELTLT, Blah, blah, blah..................................................................... The Holistic Staging Database in SQL Server ..................................................... Using Microsoft Access as a Staging, Cleaning Tool and Data Portal ............. Adding Ancillary Data Back on the Data Source ................................................ Timing of the Loads .......................................................................................... The Batch Loading (and Deleting) Approach to DLP .......................................... There are Many Tools in Your DLP Toolbox ....................................................... The SQL Server Integration Services (SSIS) Wizard is Your Friend ...................... Wikipedia Article on SQL Server Integration Services ........................................

Slowly Changing Linked Tables/Dimensions Effective Date To and From LinksAggregations Alternate Frequency Loads .........................................

Summary of How to Populate the Holistic DW .......................................

How to Use the Holistic Template....................... 2

Reporting From the Holistic Data Warehouse ..... 2

Reconcile the Reporting .........................................................................

Queries Over the Translated Data Table (TDT) .......................................

Technical Getting the ODBC Connection to Work ........................................... Writing the Query Over the TDT .......................................................................Using Prime Report ................................................................................

Using Prime Report to Build Reports ......................................................

Reporting Over the Top Down and Bottom Up ......................................

Template Top Down/Bottom Up Spreadsheet Models ...................................... Cost Sensitivity Analysis with the Bottom Up ....................................................

Variations on the Translated Data Table (TDT) ... 3

The TDT with Selectable Date Periods....................................................


8/35

Page | viii

The Linked Calculator Table and the TDT .............................................. 309

Security Tables/ Linked Views and the TDT ........................................... 312

How the Full Supply Chain Queries Work ............ 316

The Top Down Query (TDQ) Explained ............................................... 316

The Top Down Views from an SQL Perspective ................................................. 325 Summary of the Top Down Query (TDQ) ........................................................... 339The Bottom Up Query (BUQ) Explained ............................................. 340

The Bottom Up Views from an SQL Perspective ................................................ 348 Summary Bottom Up Query (BUQ) ................................................................ 362 Summary The Full Supply Chain Queries ........................................................ 363

Appendix ............................................................ 365

Where to Find the Holistic DW Template Download ............................. 365

How to Install the Holistic Data Warehouse Template .......................... 365

How to Install the Adventure Works Demo Loads ................................. 369

Bibliography ........................................................ 373

Acronyms Used Throughout the Book ................. 375

Abbreviations ........................................................................................ 376


9/35

Page | 1

AcknowledgementsMark McCarthy for help with the word smithing, proof reading, nice breakfasts on

weekends and his support

Amanda Phillips for putting up with Gerry while he was pre-occupied with the book

Daniel Moorfield for giving us the opportunity and having faith in our capabilities to bour first data warehouse

Mark Phillips putting us on track with Microsoft SQL Server

Russell Eves for his support and help with proof reading

Ray Phillips for helping with the proof reading

Jane Wong for helping us with book ideas

For all our friends and family who kept on asking is the book done?


10/35

Page | 2

ImagineLogic will get you from A to B. Imagination will take you everywhere Albert Einstein

Imagine a real silver bullet for data warehousing:

Imagine a single data warehouse that:

can be used to store practically any information about what is going on in or outsideyour business

is one central data store for all historical, operational and forward lookinginformation

Imagine a reporting system:

that seamlessly provides all the information summarised on one report where the user could open a single spreadsheet and explore all the data has unlimited potential and infinite applications

Imagine a data warehouse implementation:

without a 6+ month planning phase where you do not have to design the model and instead use a standard template that takes a matter of hours to install the empty model that does not tax key resources that does not require a major financial strategy to implement

Imagine a business intelligence environment where:

you never need to build another data model and reporting system you do not have to wait months before new types of data are made available for

reporting

you can adapt it to changing requirements and conditions as you go without havingto re-train and re-do all the reporting e.g. business acquisitions and mergers

when a new report requirement for new type of data can be met within days a forum exists where you are able to share your ideas, ask for help and benefit from

ideas input by other users in businesses employing the same template

you have time to help users with their advanced reporting and analysis needsImagine the improvements for your people and processes where:


11/35

Ima

P a g e

managers are enlightened to what factors are actually important, now that aldata and information they were lacking previously, is readily available

people are no longer wasting time manually preparing reports people no longer blaming the lack of data for not being able to do their job prope managers transcending above the data and being able to focus on ensuring

businesss success

Imagine being able to achieve this with only intermediate skills in reporting and ana

using query language, without an IT qualification.

Imagine being able to do this with only a budget of up to $30,000 for the hardware

software

Imagine bringing your business out of the information dark ages into a new era of busi

information transparency and people empowerment within a few months

Imagine all this.... well you can now give your imagination a break as we have done it

you can too with the Holistic Data Warehouse.

These and many more outcomes are now possible with a single d

warehouse model that can handle all your current and future needs.

model is our free Holistic Data Warehousing template, on Microsoft

Server 2008.

* Links to where you can download the free template are via our website www.42sight.c


12/35

Page | 4

IntroductionAs a company grows it becomes more of a challenge to locate all the necessary information

to make calculated decisions. Until now there has been no single system that will give you all

the answers you need to proceed without spending a great deal of time and money.

Holistic Data Warehousing is a book about a new simple method for developing apowerful model over all of your Business data. We go through the strategy behind the

Holistic approach and contrast it to the more traditional and complicated data warehousing

methods. When used, this strategy and method will help you foresee your Companys future

by providing data from all aspects of the business not just the traditional historical data and

business plans. This model and methodology takes full benefit of todays computing power

and as such it achieves complexity through simplicity.

The pre-emptive reporting facet allows reporting on current operational data and plans that

provide the user the ability to analyse, project and fore tell.

Data warehouses using our template, or modelled on our design schema, are extremely

flexible and can be loaded with new information in very short time. When there is a need for

new data to come through the reporting process it can often be made available within hours

of the request and does not require many days or months of effort and rigmarole. This

provides a highly potent system that significantly empowers its users with information and

analysis when they need it.

Who Are We?

In short we are business people with no IT qualifications, starting our careers at thebeginning of the PC revolution.

At the time we started our first data warehouse project we were in the Finance & IT team at

an A$100 million Fast Moving Consumer Goods (FMCG) business in Australia, but with

quite varied and broad backgrounds. Each of us is multi-skilled and was responsible for

functions that would normally be performed by multiple people across different

departments. This business is the Australian operation of a large US$3 Billion+ food business

with head quarters in the USA and operations all over the world. This food business

manufactures most of their products using Australian and imported ingredients and sells to

the big grocery retailers, the main fast food chains and other food manufacturers. Thebusiness is very diverse with over 25 production lines, thousands of ingredients and a

portfolio of over 1000 products.

Jane, in her 20+ years in the business, has vast experience in Customer Service, Sales and

Information Technology using her business system knowledge to provide a whole slew of


13/35

Book Overview Introdu

P a g e

reporting from many different sources of data. Jane was the businesss E-Commerce

barcode systems expert. Along with this Jane was the businesss expert in our t

marketing and scan data analysis systems.

Gerry similarly has widespread experience with financial, costing, management accoun

and commercial roles over his 20+ years in the Australian operation and more recently inassociated businesses across the Asia Pacific zone and involved with international proj

originating from the USA head office. He is an Australian Certified Practicing Accoun

(CPA) with a Bachelor of Economics and the businesss expert in management reporting

financial planning. Up until 2008, Gerry developed and built the Companys financial, co

and management reporting, budgeting and forecasting systems. The Companys

forecasting systems were also developed by Gerry using varied software packages.

The as mentioned above, we are both completely devoid of any formal IT trainin

qualifications, and are both basically self taught from an IT standpoint. Even so,

Australian business had complete faith in us to deliver a data warehouse, covering the w

supply chain, quickly and on time. We were given $30,000 to spend and six month

complete it from scratch.

We are not professional authors. We are two business people that decided to write, pro

and publish a book to about a paradigm shift approach to data warehousing and busi

intelligence.

Our major goal with this book is to empower businesses that are daunted by the cost

expertise normally required when implementing a world class business intelligence sys

With the free Holistic Data Warehouse Template, the ability to implement a busi

Intelligence system is now within reach for all, with or without IT qualifications.

Even for big businesses, where they are entrenched in their current technology and w

an unconventional model might be difficult to get off the ground, the Holistic

Warehouse Template will have considerable benefits as a prototyping system in areas w

experimental efforts are difficult to cost justify. All you need is access to data, MS

developer edition (US$37) and this template, and you too, can build proof of concept

Supply Chain Reporting in only a week or two

Book Overview

This book is targeted at a widespread audience from those in Senior Management acro

those users in the business with self-taught IT skills and know-how that are intereste

embarking on an in-house data warehousing project. The primary target for this book

savvy business people like we were, starved of IT resources and tools, in a business w


14/35

Introduction Book Overview

P a g e | 6

legacy environment and too small to embark on a fully fledged business intelligence solution

with an astronomical cost.

The book is about our business intelligence strategy and our Holistic Data Warehouse model.

Throughout the book you will be exposed to our IT philosophy and this we are un-apologetic

about and realise that in many cases we break with convention.

The plan is for the book to be a part of a bigger solution that includes the website and the

online community that we would like to build and grow. Primarily the book serves as the

guide to the Holistic Data Warehouse Template and our strategy. This template if adopted

by everyone using this book and method will provide the means for you and your fellow

Holistic Data Warehouse Users to share queries and reports. This is especially relevant for

those of you in a supply chain business, in the business of buying, and/or manufacturing and

selling things.

The first chapter presents an introduction to data warehousing as a concept and establishes

at a basic level the differences between our Holistic Data Warehouse model and the

traditional models. At this early stage we give you a taste of the powerful reporting and

analytics that the Holistic DW Model, was from the beginning, designed to provide.

The next chapter covers our philosophy of data warehousing and the Holistic Data

Warehouse Strategy in contrast to the normal Data Warehousing goals and objectives. Our

strategy is one that is very ambitious, forward looking and pro-active where the normal

approach is to be subservient to business requirements, backwards looking and re-active.

We then take you through a timeline of our first data warehouse project and describe how

we formulated this methodology in the absence of any understanding of the current data

warehousing methodologies. We did not know who Ralph Kimball and Bill Imnon were (two

Data Warehousing pioneers and leading authorities in the field) until after we began our

prototype. As users building a system from scratch, we knew what we wanted to be in it;

everything!

The Technical Section of the book goes into more detail of the workings of the model and

how to use the included SQL Servertemplate. This begins with the Holistic DW Under the

Hoodchapter where we go through all the essential aspects of our model beginning with theTranslated Data Table and all of its elements. All through the book we demonstrate the

power of the Holistic DW Model by showing examples over the Microsoft Adventure Works

sample data including cost sensitivity analysis.

The next chapter of the book, the largest, takes you through the most important topic, for

our template, which is How to populate the Holistic Data Warehouse model. We go


15/35

How this Book is Structured Introdu

P a g e

through in detail the demonstration load of the Microsoftsample data into the model

documenting every step in the comprehensive load. Almost every scrap of information

would be useful for business reporting was loaded into our template.

The remainder of the book, including the appendixes, shows how to install the Holistic

Warehouse Template for SQL Server2008 and how to use it to build a data warehousany Business data. The use of our template reporting including our Prime Re

spreadsheet is demonstrated and the installation explained.

The template includes two Microsoft SQL Server Integration Services (SSIS) routines for

the 2005 and 2008 versions of sample database from MicrosoftforAdventure Works.

Our strategy with this book, template and online presence is to foster a new commu

around the template with a website where users can exchange ideas and where we

provide support back and share improvements, add-ons reports and models.

How this Book is Structured

This book starts at a very highly summarised level and slowly spirals downward through

subject areas drilling further and further into more detail. As the book progresses

concepts, commentary, explanations, documentation and diagrams become more det

and complex.

We have chosen this approach so that the reader can start at the 50,000 foot level slo

descending through greater degrees of complexity along the learning curve. This mea

you pull out before the end of the book you will still have been through most of the torelating to the Holistic Data Warehouse and our philosophy on business intellige

systems and reporting. Inevitably this spiralling approach results in some areas b

repetitive but we have tried our best to limit this and keep you, our reader, engaged.

This book is heavily illustrated to visualise the concepts that we are covering and uses m

reporting examples from the Holistic DW Model with the Microsoft Adventure W

demonstration data. Many of the concepts are abstract and much easy to explain

visuals and example reporting.


16/35

Page | 8

What is a Data Warehouse?In this chapter the conventional theory relating to a data warehouse (DW) will be

summarised, particularly our view of the Dimensional theory of data warehousing as

professed by the pioneer Ralph Kimball (a leader of data warehousing theory since the

1990s). We then briefly describe the business intelligence environment that a DW belongs

to and its part in it. This will then be contrasted to transactional systems and at a very high

level we summarise our Holistic approach to data warehousing with a comparison to the

conventional dimensional DW methodology. Finally Full Supply Chain reporting is

previewed giving a taste of the powerful reporting and analysis that we had in mind when

we first had the idea for the Holistic Data Warehouse.

Our Definition of a Data Warehouse

A data warehouse (DW) is the component of an environment that holds and makes

available a large collection of information from disparate systems in a dimensional

database structure. The overall DW environment includes the process of extracting datafrom the source systems and the tools to deliver decision support information.

A decision is the action an executive must take when he has information so

incomplete that the answer does not suggest itself Arthur William Radford

Our definition has many elements and we will break them down:

makes available a large collection of informationThis relates the aspect of a DW

where it is used to supply data for querying and analysis. The DW data store is not

suitable to support other types of systems like a transactional system. The data

structures of a DW are perfect for analysing data on a mass scale. They support big,

potentially complex queries, over significant amounts of data and sometimes

spanning lengthy periods of time. This is an environment with a few large sporadic

queries are made by users, rather than a transactional system with frequent micro

and simultaneous updates to many different tables. In summary in a daily business

cycle a DW is a write once, multiple reads, few users system where a transactional

system is a 24 hour many users multiple read/write/update system.

The ratio of users of a DW to a transactional system in a business varies and itparallels the pyramid organisational structures found in businesses. The people

further up the pyramid are the typical DW user where people near the base of the

organisations pyramid are skewed to transactional system use.


17/35

The Data Warehouse Environment What is a Data Wareho

P a g e

dimensional database structure... traditionally this refers to the Kimball

Schema for a relational database and cubes which is a term often use

describe the data repositories used in business intelligence tools. We have

version of a dimensional database in the Holistic Data Warehouse model. Th

referring to a single place rather than many disparate places. By using dimensi

structures to store data we achieve the ability to use filtering, aggregations

other techniques to organise the data for reporting, with each data elem

classified by the associated dimensions.

the process of extracting data from the source systems This aspect is the Ex

Transform Load (ETL) or as we call it the DLP Data Loading Process. This pro

is common to all data warehouse (DW) load processes as they use data

disparate sources and bring it together into the one place. The chapter called Ho

Populate the Holistic Data Warehouse from page 174 is a major part of this book

justifiably so, as the task of populating any DW is a major undertaking.

to deliver decision support information as above with the typical users of a

these people are often the decision makers of the business. The DW sys

provides data and reporting that supports the people making decisions. Of high v

is information that supports people making strategic decisions. Transacti

systems can support people making operational decisions.

However, we argue that a DW can sometimes fill information gaps in a busin

where the transactional system is lacking capabilities to support operati

decision makers at lower business levels. This is one of our objectives of the HoData Warehousing Strategy To Address Deficiencies in the Operational Syste

covered in more detail on page 46.

The Data Warehouse Environment

As seen in Figure 1 below we see at a very high level that a data warehouse (#3) sits a

your current computer systems (#1) that are used for transaction processing and recor

of business information. It is 100% reliant on these source systems for data and rarely

any facility for manual data entry and manual processing. In this diagram we see that

our Holistic methodology the data warehouse (DW) is a single data store. Most warehousing environments have multiple data stores (Data Marts).


18/35

What is a Data Warehouse? The Data Warehouse Environment

P a g e | 10

Figure 1 The data warehousing environment

We call the process that takes this data, processes and loads it into the DW DLP Data

Load Process (#2). Conventional theory refers to this as ETL Extract Transform and

Load a term we consider to be too rigid. Refer to the chapter The Data Loading Process

(DLP) on page 267.

The system(s) that deliver information from the DW to the user is the reporting front-end

(#4) and these are sometimes referred to as BI Business Intelligence. We include with

our template some Excel models that provide the basic reporting.


19/35

A DW System Compared to a Transactional System What is a Data Wareho

P a g e

Figure 2 legend to Figure 1 The data warehousing environment

A DW System Compared to a Transactional SystemWe could use a whole book chapter comparing a data warehouse (DW) system

environment to transactional systems. Instead this topic will be briefly covered in a

paragraphs. There are differences based on the purpose of each system and there

differences from a technical and architectural perspective. In Figure 3 below, at a very

level, you can see how a transactional system can be designed with a schema from

demonstration system Microsoft Adventure Works. This diagram depicts the u

arrangement of many interconnected files fulfilling multiple purposes. It is too larg

reproduce in detail in this book but we wanted to give you an impression of how com

and intertwined a transactional system can be.

DWs are purpose built to do reporting and analysis where transactional systems

obviously built to primarily process transactions. In DW literature these are often referre

with an unusual acronym of OLTP On-Line Transaction Processing. This is now a

old fashioned term because in these days are there any Offline transaction proces

systems? These would be systems where paperwork is processed in batches by key p

operators or with punch cards (where the term key punch operator is derived from).


20/35

What is a Data Warehouse? A DW System Compared to a Transactional System

P a g e | 12

Often businesses have many transactional systems using different platforms and in global

organisations these transactional systems are often located across many countries and

different languages. These factors make combined business reporting impossible without a

DW.

Figure 3 The Microsoft Adventure Works schema as an example of a

transactional database (found at Microsofts www.codeplex.com website)

Transactional systems often have low historical data retention where one of the purposes of

a DW is to retain data. However, this will become less important reason for a DW, as

hardware capabilities grow easily enabling years of history to be stored without detriment.

Performance issues are usually a primary concern driving the need for a DW. However, as

business systems become more powerful, this will be irrelevant. As computers become more

powerful the integration of data from multiple sources should be the primary reason for aDW. The DW will be organised for reporting, using business intelligence systems, whereas

the transactional system will be optimised for many concurrent users, all processing updates

to many tables and have a reputation of being notoriously slow to run Large reports.


21/35

A Comparison of a Holistic DW to a Conventional DW What is a Data Wareho

P a g e

A Comparison of a Holistic DW to a Conventional DW

Figure 4 The Holistic Data Warehouse generic multi-purpose Link structure

The Holistic Data Warehouse (HDW) represents a modern methodology of

warehousing. Simply it is one multi-purpose data warehouse model for all your bus

intelligence needs. The template is designed for a simple implementation and it adap

additional requirements without modification as you go. This is made possible by the H

having a simple generic Link structure as depicted in Figure 4 above.


22/35

What is a Data Warehouse? A Comparison of a Holistic DW to a Conventional DW

P a g e | 14

Figure 5 A Mesh of conventional and customised

data warehouses (Data Marts)

The conventional dimensional approach is to use multiple single purpose data warehouse

models, each requiring a fresh implementation and addressing a single area of the business.

In an optimised environment these models would share some of their structures and


23/35

A Comparison of a Holistic DW to a Conventional DW What is a Data Wareho

P a g e

dimension tables. When viewed as a whole this would resemble a Mesh of mode

depicted above in Figure 5. The Time Dimension and Product Dimensions are typic

dimensions that should be shared to ensure consistency in reporting from these systems

The conventional Mesh approach is used in a Microsoft sample Adventure Works

warehouse based on the simulated business Adventure Works transaction system. Adventure Works DW has a mesh of five Fact Tables & 16 Dimension Tables some of w

are shared between each model. It is a sample system that perfectly typifies

conventional approach to a DW. We use the Adventure Works data throughout this b

and have a full data load in the template to the HDW. The data from Adventure Wor

almost perfect for demonstrating and proving the capabilities of the HDW.

With conventional models the design requirements are needed up front, often with s

change governance, resulting in changes being difficult to apply. Where dimensions

shared this can reduce workload except where modifications are required. Another fa

burdening the conventional approach is that each of these models would in turn req

their own documentation and people to support them.

The conventional design was required due to the poor performance hardware of yester

where a Holistic DW system would have been prohibitively slow. Computers have

become so powerful, that even when compared those available 10 years ago, they

practically zero cost for the same performance. The speed available from todays hardw

can more than compensate for this performance difference and the additional cost of

hardware is insignificant compared to the implementation cost of a conventional sys

This is discussed in more detail later in the book.

Dimensional cube / OLAP systems are another data store option for business intellig

systems. We consider these to be in a different space to the relational database models

are used as data warehouses. However, OLAP systems often need to sit above a relat

data storage system, and often these are either data bases of the dimensional or

normalised flavour.

In summary the conventional approach to data warehousing, used since the 1990s,

proved to be very effective and powerful. Our concerns are with the cost and difficul

implementation and with the level of expertise required to build, grow and maintain thIn these modern times businesses should have higher expectations from their systems;

need an adaptable and agile data warehouse that is quick to implement and modify. T

need more powerful reporting such as Full Supply Chain Reporting; they need a Ho

Data Warehouse.


24/35

What is a Data Warehouse? Full Supply Chain Reporting One Page

P a g e | 16

Full Supply Chain Reporting One Page

We had Full Supply Chain Reporting" in mind right at the beginning when we formulated

the Holistic Data Warehouse methodology. This will be covered in detail in the Strategy

section but we will give some brief insight into it here in this introduction.

The objectives of Full Supply Chain Reporting are to provide pre-emptive, forward looking,

cross functional reports and for manufacturing businesses full drill through reporting. A

sample of a cross functional report is shown below in Figure 6.

These Sales and Operations reports provide a holistic view over the business and give them

insight into the future business and operational plans. Many businesses have analysts that

prepare these cross functional reports manually, and the process requires data from many

different sources to be consolidated on the summary reports. The Holistic DW is special in

that these reports can be made automatically available, not only for the full business and

once a month, but daily and for any level of detail desired. The report is always up-to-datewhich avoids the usual traps of out of sync information.

Our philosophy is that business should be demanding from their reporting systems a single

view over their whole business. These are the everything you wanted to know about your

business on the one page but were afraid to ask reports. Kimball refers to this type of

reporting as drill-across reporting. A normal DW requires significant investment of time

and effort in ensuring your separate reporting systems share common dimensions, the

Holistic DW Model does this reporting naturally.


25/35

Full Supply Chain Reporting One Page What is a Data Wareho

P a g e

Figure 6 A sample Total Business cross functional Sales and Operations report


26/35

What is a Data Warehouse? Full Supply Chain Reporting One Page

P a g e | 18

However, drill-across is nice to have, but for a manufacturing business the Holistic DW

goes one better with drill across AND drill through reporting The Top-Down and

Bottom-Up reports.

These reports are only applicable to manufacturing and assembly businesses as they will go

through multiple levels of production to explode the entire supply chain and includecomplete reporting on all components. That is why we refer to this as a Full Supply Chain

Top-Down report. We depict the type and structure of the information automatically

revealed by these reports below in Figure 7, where through Production In, data

components are found and then supply chain information is reported for them. The partner

report is a Bottom up which begins at a component to find where it is used and provides

complete reporting for all end products.

The Top-Down/Bottom-Up reports are included with the Holistic DW Template and they

will work automatically if implemented according to our guidelines, (found in the technical

section Reporting from the Holistic Data Warehouse). Additionally the reports are covered

in more detail in the Holistic Data Warehouse Under the HoodChapter and documented

fully with details of how they work in the last chapters of the book.

Systems that just provide siloed reporting should be a thing of the past as even existing DW

systems can be modified to provide Full Supply Chain Reporting but with varying degrees of

difficulty and cost.


27/35

Full Supply Chain Reporting One Page What is a Data Wareho

P a g e

Figure 7 The information revealed by a Holistic DW "Top-Down report


28/35

What is a Data Warehouse? What Data Did Our First Data Warehouse Have?

P a g e | 20

What Data Did Our First Data Warehouse Have?

The following diagram, Figure 8 Data within our first model on page 21, depicts all the

different types of data in our first attempt at data warehousing. This business intelligence

system was constructed, including front-end reporting, to this level detail within three

months of beginning the final model after prototyping in Microsoft Access. All made possibleby this powerful Holistic Data Warehouse model.

The first Holistic DW Model has reporting available in any or all of the following measures.

$ Value Kilograms Qty Standard Cost Latest cost 15 Std Cost elements Cost elements like:

o Raw Material costo Labour Costo Packaging cost

And 15 latest cost Elements by dateThis means we can run sales reports valued in packaging cost or labour cost in addition to

the normal sales revenue measure.

Notably we had most of this information in the Microsoft Access prototype within a monthof starting, so this is no way a significant undertaking, even for people like ourselves with

limited IT experience.


29/35

What Data Did Our First Data Warehouse Have? What is a Data Wareho

P a g e

Figure 8 Data within our first model


30/35

What is a Data Warehouse? What is a Data Warehouse? Summary

P a g e | 22

What is a Data Warehouse? Summary

In this What is a Data Warehouse?chapter we briefly defined a data warehouse and then

contrasted the Holistic Data Warehouse to the conventional type of DW. The two

contrasting approaches were described as a single Holistic Link style to a conventional

Mesh arrangement of multiple DW models with shared structures.

We define a Data Warehouse as a central data store from which all your business

intelligence reporting is based on. The broad user requirements for reporting and apparently

incompatible data normally make a DW a considerable challenge to implement. That is until

our Holistic Data Warehouse model came along.

Finally we touched on the powerful Full Supply Chain Reporting that is easily enabled by the

Holistic Data Warehouse. This reporting was envisioned when we began our first data

warehouse and in a later chapter we document our first project that gave birth to the

Holistic Data Warehousing methodology. It was a journey that took a business with basicallyno supply chain reporting capabilities and cumbersome business reporting to the leading

edge.

In the next chapter of the book we take you through the strategy that underpins the Holistic

Data Warehouse methodology and go into greater depth comparing it to the conventional

approach.

Wikipedia is a great reference on the subject of data warehousing if you are interested in

learning more about the current thinking on this topic.


31/35

Page | 23

The Holistic Data Warehouse StrategyThe vision is really about empowering workers, giving them all the information a

whats going on so they can do a lot more than theyve done in the past. Bill Gates

In this chapter we would like to first take you through our Vision, Goals, Objectives

Benefits of the Holistic Data Warehouse. We will contrast these with the Standard warehouse strategy.

The Holistic Data Warehouse Vision

One simple and easy system with minimal limitations providing the one view ovewhole business and its supply chain in which the user can use filtering technique

select which data they would like to view on their report.

Right from the beginning, with our first data warehouse, our aim has been for a s

repository and reporting front end that can be used for anything we could be thrown

and something that will provide the holistic view over the business and the supply chain

With this approach there is a bonus in that it avoids the time and effort normally associ

with planning and developing separate models for each new business requirement. We

some insight to what we define as a Supply Chain on the following page.

The Standard Data Warehouse Goal

The goal of a data warehouse is to provide business intelligence that is consisand reconciled based on operational data, decision support data and external

from multiple sources.

This is the primary goal of any data warehouse and addresses the major weaknes

business reporting that is based on many different systems. While not impossible this go

difficult to achieve with most standard data warehouses. Some will never manage to

there. Working against them is the time & effort involved in set-up and the cost.

Making data conform is difficult and there are many books and reference material wr

about how to implement data warehouses properly in order to attain this goa

consistency. Interestingly the Adventure Works Data Warehouse fails to meet this go

actually has four separate specialized models for sales reporting which to us seridiculous, and on top of that the data does not fully reconcile with some key omissions.


32/35

The Holistic Data Warehouse Strategy Full Supply Chain Reporting Goals

P a g e | 24

What is the Supply Chain?

From a very shallow perspective the Supply Chain is often considered to be just those

processes and activities between supplier and customer. However, for a manufacturing

company, we consider the Supply Chain to be much greater than this and that the chain

goes all the way back through production to the suppliers of component materials.

From Wikipedia: A supply chain... is the system of organizations, people, technology,

activities, information and resources involved in moving a product or service from supplier

to customer. Supply chain activities transform natural resources, raw materials and

components into a finished product that is delivered to the end customer. (For our view of

the supply chain the end customer is the end consumer).

A basic diagram representing the supply chain of a consumer goods manufacturing and

distribution company follows:

Figure 9 Example of a supply chain for a consumer goods manufacturing and distribution

company


33/35

Full Supply Chain Reporting Goals The Holistic Data Warehouse Stra

P a g e

Full Supply Chain Reporting Goals

1stGoal (Applicable to all businesses using the Holistic DW Model)To provide reporting that gives the user the option to see ALL the informaavailable on activities pertaining to the subject matter they are interested in.

2ndGoal (Applicable to manufacturing and assembly types of businesses)To provide reporting that enables the user to see ALL information about the activ

in the Full Supply Chain that are related to the subject matter in question.

These additional Full Supply reporting goals underlie the Holistic Data Warehouse (DW)

first relates to the often requested (but not achieved) report that tells me everything I

know on the one page. The types of supply chain questions seen below in Figure 10 b

on the next page (under 1st

Goal). These reports are usually done manually by u

consolidating data from multiple sources in their spreadsheets. This drill-across repo

was introduced in the What is a Data Warehouse? chapter.

The first Full Supply Chain Reporting goal does not necessarily need our Holistic DW M

Gerry achieved this goal, with some difficulty, with a SAP BW (Business Warehouse) w

he was the architect of the model (using a BW Multi-provider Cube). The substantial

had 400+ reporting fields and 110+ key figures resulting in complex queries and rep

While the Full Supply Chain Reporting goal was attained, the effort and cost astronomically higher than our earlier model over the legacy systems.

The second goal is accomplished with the Holistic DW Top-Down and Bottom-Up rep

built into the template. These are reports that are extremely difficult if not consid

impossible in normal DW and business intelligence (BI) systems. For instance with SAP

we tried and failed to achieve these reports. However, these Holistic DW reports do w

automatically, if populated according to our guidelines and only apply to produc

businesses as the reporting goes through the supply chain to analyse components.

These powerful reports are fully documented in the How the Full Supply Chain Queries Wchapter at the end of the book and in the largest section of this book we take you throu

detail a full documented load of the MicrosoftAdventure Works data into the Holistic

showing you how to populate the model to ensure the reports work.


34/35

The Holistic Data Warehouse Strategy Full Supply Chain Reporting Goals

P a g e | 26

Figure 10 Questions about the supply chain

How often have you asked your own BI resources for this type of reporting in vain? How

often have you been told that your report request was not possible, cant be done, too

ambitious and/or too costly? We believe that there is no BI report that is too hard. We have

produced reports from the Holistic DW within 45 minutes of hearing the users requestwhere others had failed to deliver the report after months of effort in other larger

companies with more resources. The Holistic DW can do the hard reports.

One interesting point was, that although we started with these goals in mind when

envisioning our data warehousing solution, we didnt think that these goals were anything


35/35

Objectives of All Data Warehouses The Holistic Data Warehouse Stra

P a g e

special at the time and just took them as a given. Only afterwards did we realise

ambitious nature of these goals for a normal data warehouse implementation. In a

chapter we tell the full story of our first effort.

After a few years of researching standard data warehouses our opinion is that these g

are technically very difficult (read expensive) to achieve using a traditional data warehomodel(s) and even more modern business intelligence systems. We have been to quite a

sales presentations where the sell job was for systems that provide a single view over

so-called supply chain, costing anywhere between $100,000 to around a $1 million and

was just for their services and software. But still these systems appeared to be too lim

only encompassing either the distribution supply chain without any consideration to

supplier end of the chain or the reverse, focusing on suppliers and not customers.

Objectives of All Data Warehouses

Minimize inconsistent reports and reconcile different views of the same data Improve Quality of Data To consolidated enterprise data from multiple sources and time periods. Make the Data easily accessible and provide transparency To enable common and flexible calendars To save time on report preparation and construction Address the weaknesses of current reporting systems To empower people with information

These objectives while being seen as standard data warehouse objectives are normally

difficult to achieve. However, the Holistic Data Warehouse takes these objectives q

easily into its stride. This is because it has a simple design and structure. Even with on

basic raw prototype we achieved all these objectives, using around two years worth of

within two weeks of starting our project, using the Holistic Data Warehouse framework.

We next take you through each of these standard objectives in some detail to describe

interpretation of what they mean.

Minimize Inconsistent Reports and Reconcile Different Views of the Same D

Establish one version of the truth.

Prior to our first data warehouse the business had inconsistent reporting of sales e

End Of ExtractThe Holistic Data Warehousing book can be found at Amazon.

A colour hardback version of the book will also be available

sometime early 2011.

If you have any questions or comments please lodge them via the

blog or the codeplex websites (link found at the blog)

http://42sight.blogspot.com/

holistic data warehousing sample

Documents