data warehouse automation - nweug€¦ · wherescape 3d wherescape red. today 1960s 1 person 1920s...

Using

Data Warehouse Automation

to enable

Agile Projects

Douglas Barrett(WhereScape)

How to use data warehouse automation to deliver agile analytics with the business without bypassing IT.

WhereScape

Model Automation Build Automation

discover design

document

build, refine, document, promote, operate, support

and enhance

Traditional analytics environment

Sales

Billing

The Data Warehouse is not dead!

Governed / Monitored / Reliable

One version of the truth!

Services

Financials

Data Warehouse

• IT is too slow

• Overcomplicated

• Hard to change

• Slow

REQUIREMENTS

DW framework

Profile

Logical Model

Physical Model

DB Architecture

Storage Mgmt

Index Mgmt

OLAP Design

ETL Mapping

ETL Dev

Versioning

Testing

Scheduling

Deployment

Word/Excel

Mainly in-house

solutions

BI Tools

Trillium

DataStage

PowerDesigner

Modelling

JIRA

SSMS

SSIS

SSAS

TFS

SQL Agent

Change tool

Change tool

Change tool

Change tool

Change tool

Change tool

Informatica

ErWIN

DB Management

SSIS Profile

Viewer

Documentation

Word/Excel

TFS

ETL

DW Traditional Approach

Business can not define accurate requirements

Business wants fast delivery cycles

Business wants to change requirements

often

Source data changes / New sources

Agile is Everything

Just give me access, I don’t need a data warehouse

Direct access in PowerBI / Tableau / Excel will give me what I need

See the rise of Desk top Visualization / Data Prep tools

Business goes round IT

Sales

Business goes round IT!

Sales

Desktop tools are good, but:

• No data persistence / history• Limited re-use• No IT Management

• Monitoring• Tuning• Operations• Standards• Back up

Business goes round IT!

Desktop tools are good, but:

• Quickly grows

Sales

Billing

Services

Financials

A vacuum is emerging between the business and IT

Symptom:Tableau, Qlik, PowerBI bought and managed by the business.

Business goes round IT!

IT and ETLBusiness and desktop tools

DW Framework

Profile

Physical Model

DB Architecture

Storage Mgmt

Index Mgmt

OLAP Design

ETL Mapping

ETL Dev

Version Control

Workflow

Deployment

Maintenance

Logical Model

Documentation

REQUIREMENTS

Blends development tasks1 development task instead of 6Efficient changeAgile projects

DW Better Approach - Automate

WhereScape 3DWhereScape RED

Today

1960s

1 person

1920s

3 people one car

Robots

Automation

• Productivity• Consistency

2000s

1990s

More Programmers

1980s

Programmers

Even More Programmers!

DW Automation

• Productivity?• Consistency????

12

USA I UK I NZ I Australia I South Africa

DW Architecture

Metadata

13

USA I UK I NZ I Australia I South Africa

Demo

3D• Discovery• Design

RED• Build• Document• Operate

14

USA I UK I NZ I Australia I South Africa

3D Demo

Source Target

15

USA I UK I NZ I Australia I South Africa

RED Demo