#centennium presentation data vault automation seminar #dvauto oct 6th 2011

1

CDM

Centennium Datawarehouse MethodologyStructure – Model – Generate – Knowledge PartnershipErik FransenAntoine StelmaOctober 6, 2011

2

Agenda

• Centennium BI expertisehuis• CDM

– Structure Reference Architecture– Model Data Vault– Generate Template Based

Development– Knowledge

Partnership Knowledge transfer, coaching

• CDM process

3

Centennium BI expertisehuis• Independent • We provide our customers with the right skills, at the

right time and the right way• Consultancy, ad-interim support, project

implementation and training services, but primarily acts as a BI/DWH knowledge partner for its clients

• We take or share responsibility for the execution and management of BI and DWH projects and provide second or third line support for existing implementations

• We offer an extensive portfolio of courses and training services

• We provide our customers with the knowledge and practical insights required to be(come) self-sufficient in maintaining and expanding their BI-environments

www.centennium.nl

4

Centennium Datawarehouse Methodology

• A methodology, including DWH-tools• Build or migrate datawarehouses fast, with high quality

and low cost• Automatic generation of datawarehouse based on

descriptive metadata• CDM includes:

– Template Based Development– Best practices– Quality control mechanism– Knowledge partnership

• Best Practices: Data Vault, Kimball, Reference Architecture• Quality control mechanism: extensive checklists and

documentation• Knowledge transfer by training, certification and learning

on-the-job

5 ModellingStructuring Generating

Knowledge Partnership

Centennium Datawarehouse Methodology

6

Knowledge partnership

• Goal: Self-supporting, CDM-aware customers• Training options

– BI and DWH fundamentals– CDM designer / TBD developer training– Data Vault fundamentals– Data Vault certification (Genesee Academy)– Dimensional modelling– More info on www.bi-opleidingen.nl

• On-the-job coaching, learning by doing• Centennium supports it’s customers by

(management) consultancy, assessments, projects, training and sourcing

7

Reference Architecture: Structuring

8

Template Based Development:Generating

• TBD generates all ETL and objects for– Registration layer– Presentation/Publication layer

• Repository and scripts are free of charge

• Creating the staging layer is not part of TBD but can be automated per customer

9

TBD databases

Central Data Warehouse

(CDW)

Staging (STA)

Publication

(PUB)

Repository (REP)

10

TBD repository

Central Data Warehouse

(CDW)

Staging (STA)

Publication

(PUB)

Repository (REP)

11

Functionality TBD repositorylog tables (1)

• Functionality: – Registration of all ETL processes– Insight in data logistics– Historic profile of loading processes

• Goal:– Traceability – Maintenance: localisation and identification

of all data/transactions (wrong and correct)– Linkage to source systems

PUBCDWSTA

REP

12

Functionality TBD repositorytemplate tables

• Functionality: – Central storage of template scripts for

generating tables and ETL procedures– Easy promotion to other environments

(OTAP)– Transparant

• Goal: – Version management of templates– Object (re-)creation and maintenance

PUBCDWSTA

REP

13

Functionality TBD repositorymeta data tables

• Functionality: – Defines registration and publication layer– Contains object and ETL definitions– Contains data logistics

• Goal:– Generating:

• Registration layer (Data Vault tables)• Publication layer (Star schemes) • ETL processes via ‘stored procedures’

– Object creation and maintenance

PUBCDWSTA

REP

14

Functionality TBD repositorymeta data tables

• Table ‘Repos_Table’: – Contains object names for

• Staging • Registration• Publication

– Table is filled by an application or Excel sheet

• Table ‘Repos_Mapping’: – Contains mapping of staging, registration and

presentation– Table is filled by an application or Excel sheet

PUBCDWSTA

REP

15

Registration layer

Central Data Warehouse

(CDW)

Staging (STA)

Publication

(PUB)

Repository (REP)

16

Registration layer

• Script ‘CDW 01a table hub’– Generates all hub tables in the CDW, as

defined in repository– Every hub contains the columns:

• id (primary key), <afkorting>_key, meta_laad_dts, meta_bron en meta_audit_id

– Variables from repository are applied– Once the hubs are generated, the

repository is updated– Hubs generated according to Data Vault

Standards!

PUBCDWSTA

REP

17

Registration layer

• Script ‘CDW 01a tabel hub’– Generates all hub tables in the CDW, as

defined in repository– Every hub contains the columns:

• id (primary key), <afkorting>_key, meta_laad_dts, meta_bron en meta_audit_id

– Variables from repository are applied– Once the hubs are generated, the

repository is updated– Hubs generated according to Data Vault

Standards!

PUBCDWSTA

REP

18

Registration layer

• Script ‘CDW 02a table sat’– Generates all satellite tables in the CDW, as

defined in repository– Every satellite contains the the columns:

• h_id (foreign key), meta_laad_dts, meta_laad_eind_dts, meta_bron en meta_audit_id

– Variables from repository are applied– Once the hubs are generated, the

repository is updated– Sats generated according to Data Vault

Standards!

PUBCDWSTA

REP

19

Registration layer

• Script ‘CDW 02a tabel sat’– Generates all satellite tables in the CDW, as

defined in repository– Every satellite contains the the columns:

• h_id (foreign key), meta_laad_dts, meta_laad_eind_dts, meta_bron en meta_audit_id

– Variables from repository are applied– Once the hubs are generated, the

repository is updated– Sats generated according to Data Vault

Standards!

PUBCDWSTA

REP

20

Registration layer

• Script ‘CDW 03a table link’– Generates all link tables in the CDW, as

defined in repository– Every link is connected to two or more hubs– Variables from repository are applied– Once the links are generated, the

repository is updated– Links generated according to Data Vault

Standards!

PUBCDWSTA

REP

21

Registration layer

• Script ‘CDW 03a table link’– Generates all link tables in the CDW, as

defined in repository– Every link is connected to two or more hubs– Variables from repository are applied– Once the links are generated, the

repository is updated– Links generated according to Data Vault

Standards!

PUBCDWSTA

REP

22

Registration layer

• Script ‘CDW 04a table link satellite’– Generates all link satellite tables in the

CDW, as defined in repository– Variables from repository are applied– Once the link satellites are generated, the

repository is updated– Link satellites are generated according to

Data Vault Standards!

PUBCDWSTA

REP

23

Registration layer

• Script ‘CDW 04a table link satellite’– Generates all link satellite tables in the

CDW, as defined in repository– Variables from repository are applied– Once the link satellites are generated, the

repository is updated– Link satellites are generated according to

Data Vault Standards!

PUBCDWSTA

REP

24

Publication layer

Central Data Warehouse

(CDW)

Staging (STA)

Publication

(PUB)

Repository (REP)

25

Publication layerdimensions and facts

• Fed only and directly from CDW• Conforms to Kimball’s star scheme

standards– Or any other format

• Business rules can be applied– Currently using views– Business rule editor in next release

• Type SCD I, II, etc.• Conformed dimensions when needed

PUBCDWSTA

REP

26

CDM: process• Incremental approach• Timeboxes of 4-6 weeks

100% Centennium90% Centennium 30% Customer

70% Customer40% Centennium

100% Customer10% Centennium

Phase 1 Phase 2 Phase 3 Phase 4

Increment

1-8-2011 31-12-2011

CDM: typical increment

CDM-trainin

g

Data Vault

Certification

Training and Coaching on-the-job

Supporting customer

• Typical increment ranges from 2 to 6 months• Centennium role changes from LEAD to FOLLOW• Customer is fully CDM-aware at the end of the increment• Centennium continues supporting customers through

knowledge partnership

Q & A

28

29

ABOUT CENTENNIUM

Centennium BI expertisehuis houses all the experts under one roof, hereby offering all knowledge and expertise to address the complex business intelligence issues facing our clients today

Services overview:• Consultancy• Projects • Resourcing • Education

Some of our clients: Woonbron, Albron, NZa, CAK, OBR, Vopak, several Dutch Municipalities, Aegon, Nutreco, TNO, Genzyme, Tata Steel, KPN, DELTA, IKEA, Accell, TomTom, KAS BANK, LeasePlan, Brabant Water

Facts and figures:• Founded: 1998• 45+ business intelligence

consultants

Core values:• Human Capital• In close collaboration• Objective and Independent

Expertise:• Business intelligence• Strategic, tactic and operational• Vision based on “effective BI”

Certification and Industry Recognition

All Centennium consultants are Certified BI Professionals (CBIP) awarded by TDWI.

CBIP, backed by TDWI's reputation as the leader in BI education and research, makes a strong statement that we are among the leaders in the industry—up to date with BI technologies, knowledgeable about best practices and state-of-the-art solutions, and informed of emerging trends

All Centennium Data Vault specialists and architects are certified on Data Vault modeling by Dan Linstedt, Data Warehouse industry thought leader and founder of the Data Vault modeling standard.Centennium is the business partner of Genesee Academy in organizing the Data Vault Modeling & Certification – Europe Course. We are accredited to perform the Genesee Academy Data Vault trainings.

Centennium’s customer Nutreco was awarded the world’s first Certified Data Vault Data Model in 2009 for their corporate financial and procurement data warehouse model.

32

Centennium BI expertisehuisLange Voorhout 432514 EC 's-Gravenhage Telefoon 070 31 20 370Fax 070 31 20 371URL www.centennium.nl

www.bi-opleidingen.nlwww.data-vault.nl