chapitre de montréal du tdwi - · pdf filechapitre de montréal du tdwi 13...

1

Chapitre de Montréal du TDWI

13 septembre 2010

Qu’est ce que TDWI ?

TDWI provides education, training, certification,

news, and research for executives and

information technology (IT) professionals

worldwide.

Founded in 1995, TDWI is the premier

educational institute for business intelligence

and data warehousing.

2

2

Mission du chapitre de Montréal

Informer objectivement la communauté sur les

meilleures pratiques BI et les tendances

émergentes en matière d’Intelligence d’Affaires.

Offrir de la formation sur mesure, promouvoir le

réseautage et le partage de connaissances en

tenant compte de la culture d’affaires

Québecoise.

3

Montreal Chapter Board

• Alain Bond, President

• Elise Lacoste, Vice President

• Sylvie Fréchette, Secretary

• Yvan Dupont, Sponsor Coordinator

• Stéphan Robitaille, Event coordinator

Bienvenu à nos petits nouveaux

• David-Marc Petit

• Frédéric Gingras

4

3

5

Merci à nos commanditaires!

66

Svp. / Please

Fermez la sonnerie de votre cellulaire /

Turn off your cell phone ringing

4

7

Ordre du jour / Agenda

08h00 - 08h10 Propos d'ouverture

Par Alain Bond

Président du Chapitre

Opening Remarks

By Alain Bond

Chapter President

08h10 - 09h15 Première partieIntroduction à l'approche de développement Agile

et discussion sur comment elle doit être adapté au

contexte des projets BI

Par Ralph Hughes

Architecte de systèmes en chef pour Ceregenics,

Inc

Part OneIntroduction to Agile development approach and

discussion on how it must be adapted for BI

projects

By Ralph Hughes

Chief Systems Architect for Ceregenics, Inc

09h15 - 09h30 Pause Break

09h30 - 10h20 Deuxième partieSolutions aux défis les plus communs qui

accablent souvent les nouvelles équipes Agile de

développement d'entrepôt de données

Par Ralph Hughes


Inc

Part TwoSolutions to the most common challenges that

often overwhelm new Agile warehousing teams

By Ralph Hughes


10h20 - 10h30 Mot de la finPar Elise Lacoste

Vice-présidente du Chapitre

Closing Remarks

By Elise Lacoste

Chapter vice-president

Ralph HughesChief Systems Architect for Ceregenics, Inc

Ralph Hughes is leading business intelligence programs and

projects for Fortune 500 companies in aerospace,

telecommunications, and pharmaceuticals. He holds bachelor and

master degrees from Stanford University in computer modeling and

econometric forecasting.

A certified Scrum Master (CSM) and PMI project management

professional (PMP), he has been building data warehouses since the

early 1980s and was the primary inventor of the Agile Data

Warehousing(TM) methodology.

He lives in Denver, Colorado, skiing and fly fishing between projects.

He can be reached at [email protected],

720.951.2100.

5

Agile Delivery of BI Projects

Ralph HughesChief Systems Architect for Ceregenics, Inc

[email protected]

• Agile delivery and how it differs from waterfall methods?

• To what degree can Agile accelerate our BI/DW projects?

• How can fast-moving Agile BI/DW teams ensure quality?

• How to start small and grow Agile BI/DW?

• Does Agile transform other project aspects

such as estimating and architecture?

Copyright (c) 2010, Ceregenics, Inc. 10

Topics

6


Coke vs. Pepsi

Cats vs. Dogs

Zealotry?Balanced Views Work Better

…PC vs. Mac

12

DOD-Std-2167

Applctn Rqmts

However...Waterfall *was* a Misconception!

System Reqs Design Testing OperationsCoding

Coding

Usage

“risky & invites failure”

Analysis

In the 2nd half

of Royce’s

paper

Dr. W. W. Royce, Managing the Development of Large Software Systems, 1970

…

7

Speaker’s Background

Copyright (c) 2010, Ceregenics, Inc., www.ceregenics.com 13

PharmaTelecomGovernmentOil & Gas

MA, PMP, CSM

InformaticaPL/SQL, Perl

BObj, Cognos, HyperionQlikView

TRWSonyEli Lilly Level 3Bell LabsComcastLockheed MartinAT&T BroadbandBureau of Land Management

NextelQwestSonyWolters KluwerGuidant BiomedFeed the ChildrenStanford Med Center

speed x20

cost x14

speed x4

cost x2

…

Shippable

Application

Three Major Components


2. Agile

Development1. Agile

Requirements

Management

3. Agile

Quality

Assurance

Potentially-

Shippable

Code...and Agile QA gets you

successfully out

Requirements management gets

you successfully into a project...

Agile development delivers the code

quickly...

…

8

Scrum CoreRUP Wrapper

Agile Flywheel & Its Wrapper

Copyright (c) 2010, Ceregenics, Inc.15

16

Key Innovations of Generic Agile

• Co-located Teammates .........put business & coders in the same room

• Time-Boxed Development .....................give them three weeks

• Self-Organized Teams ............let them work as best they know

• Just-in-Time Requirements ......figuring things out as they go

• Size-Based Estimation ............................using known references

• Test-Led Development ............and a tough definition of “done”

• Automated Integration Testing ..................that’s applied daily

• PM-Light ...................with only two, paper-based tools for guidance

9

17

Self-Oranized Team: Task BoardStory by Priority

Tasks Waiting

TestsWritten

Under Development

WaitingValidation

Ready to Demo

User needs to...

2 sp

Code the...

5 hr / 2

User needs to...

8 sp

User needs to...

6 sp

Code the...

8 hr / 10Code the...

24 hr

Code the...

12 hr

Code the...

10 hr / 6

#1

#2

#3

List...

Tally...

Compare...

List...

Tally...

Compare...

Business provides

“User Stories” (est.

in story points”

Team provides

coding tasks

(est. in labor hours)

…

18

PM Lite: Burndown Chart

Demo

“Tech Debt”

…

10

Does Anyone Really Do Waterfall?


• F4 Phantom

• Built via plan-driven process

like IT’s waterfall methods

• Introduced in 1960

with missiles only

• Supersonic dogfights

thought impossible...

• ...but missiles proved unreliable and unusable in short range

• Five years later, the fifth revision finally had cannon, smokeless engines to reduce visibility from the

ground, and leading-edge slats to improve maneuverability

• Essentially, designers followed iterative development over releases

• In IT, such long-term iterations denies the current project sponsor any benefit from lessons

learned

M

Release

DeploySystem

Test

Intgrn

TestBuild20%

Analysis &

DesignROM

Agile is Evolution – Not Revolution


T-Shirt

ROMT-Shirt

20%

Waterfall

Agile

Dept

Arch

1 2 3 4 5

Here the project

“out drives its headlights”

M

Deploy

ReleasePotentially-Shippable

Modules

System

Test

11

21

Agile’s Iterative Approach

• Emphasizes small efforts frequently reviewed

• De-emphasizes “getting it perfect the first time”

• “Fail fast & fix quickly”

• Recursive 80/20 rule

• Maximizes learning

Full Scope

Breadth

Depth

Offers the best chance of success when facing significant unknowns

…

22

Starting a Team

Option 1: Gradual Improvement Iterations Required

Stage Fast Slow

0: Generic Scrum 1 2

1: Pipelined Delivery 1 2

2: Sized-Based Estimation & Release Plan 2 4

3: Reference Model 2 4

4: Test-Led Development 3 6

5: Requirements Mgt & Automated Testing 3 6

Total Iterations 12 24

Elapsed Time (2- and 3-week iterations) 24 – 36 wks 48 – 72 wks

…

12

23

Pipelined Delivery Squads

24

The User Story

• Index card with one, two lines

• “As a <user role>,

I want to <action>,

so that <benefit>.”

• Reminder to have a conversation

• Property of the Product Owner

• Can be moved, prioritized, partitioned quickly

– as part of the conversation

• They are the central organizing vehicle of Scrum

M

13

DWBI Adds “Developer Story”

• Typical user story

– Add the billing data for the newly acquired company to

the revenue assurance data mart

• Team creates developer stories that are actionable:

– Stage the Western-region billing data

– Scrub customers

– Add customers to the customer master data

– Scrub addresses

– Add addresses to the LOCATION master data

– Confirm that new data now flows to data mart

Copyright (c) 2010, Ceregenics, Inc. 25M

Sized-Base Estimation: Rapid & Accurate

26

“Estimating Poker” Cards

• Cards have story points on them

• Use only the Fibonacci sequence

• Use reference stories from a past sprint

• For each new story:

• Compares to previous sprints’ stories

• Coders pick cards with new estimate

• Show them simultaneously

• Discuss the outliers

• Revote till consensus

• Most stories: 3 to 5 min

14

27

Better Estimates: The Secret Reason

Demo

Here they

could see

they were

slipping

…

28

Better Estimates: Reference Model

15

29

Ordre du jour / Agenda

08h00 - 08h10 Propos d'ouverture

Par Alain Bond

Président du Chapitre

Opening Remarks

By Alain Bond

Chapter

08h10 - 09h15 Première partieIntroduction à l'approche de développement Agile

et discussion sur comment elle doit être adapté au

contexte des projets BI

Par Ralph Hughes


Inc

Part OneIntroduction to Agile development approach and

discussion on how it must be adapted for BI

projects

By Ralph Hughes


09h15 - 09h30 Pause Break

09h30 - 10h20 Deuxième partieSolutions aux défis les plus communs qui

accablent souvent les nouvelles équipes Agile de

développement d'entrepôt de données

Par Ralph Hughes


Inc

Part TwoSolutions to the most common challenges that

often overwhelm new Agile warehousing teams

By Ralph Hughes


10h20 - 10h30 Mot de la finPar Elise Lacoste

Vice-présidente du Chapitre

Closing Remarks

By Elise Lacoste

Chapter vice-president

30

Test-Driven Development

Story by Priority

Tasks Waiting

TestsWritten

Under Developm

ent

WaitingValidation

Ready to Demo

User needs to...

2 sp

Code the...

5 hr / 2

User needs to...

8 sp

User needs to...

6 sp

Code the...

8 hr / 10Code the...

24 hr

Code the...

12 hr

Code the...

10 hr / 6

#1

#2

#3

List...

Tally...

Compare...

List...

Tally...

Compare...

16

31

Req. Mgt: Hierarchy of DWBI Needs

Analysis

Research

Reporting (incl KPIs)

Data Access

Copyright (c) 2010, Ceregenics, Inc.

Prediction

32

Agile’s Five-Levels of Planning

Product

Product

Release

Iteration

Daily

Product Owner & Execs, 1 to 2 x / yr

Vision Statement

Product Owner & Execs, 1 to 2 x / yr

Product Roadmap (calendarized)

Product Owner & Team; 3 to 4 x / yr

Release Backlog (user stories)

Product Owner & Team; every 3 to 4

weeks, Iteration Backlog (user stories)

Team; every day

Tasks and Burndown Review

(three standard questions)


Vision

Roadmap

M

…

17

33

User

Story

Requirements Mgt: A Series of Artifacts

33

Application

Use Case

Solution

Use Case

Vision

Document

Stakeholder

Request

Program

Concept

Brief

Epic

Theme

Developer

Story

Iterative

RUP / Traditional

“We going to resell

Verizon cell phones, and

we’ll need to the revenue

by month”

Problem: Business could insist IT start

at any point in these chains

“We can’t make the

partner products

program profitable if we

can’t see the gateway

data”

“Give us an estimate on

all the projects we’ve

talked about so we can

pick which ones to do

next year”

“Send over a programmer, we’ll tell

them what to build.”

“Send over a team, Mike will work

with them to create the revenue data

mart.”

“Go ask Mike how to add the

subsidiary billings to the revenue

data mart.”

“Add the 3rd party revenue flat file to

the staging layer.”

Solution: Be ready to start at any point

and retrospectively draft the RUP docs

as the project proceeds.

Mike: “I know exactly how you

should build the allocation

module.”


…

34

Automated Regression / Integration Testing

Staging

Integration 3NF

Ent. Dimensional

Dept. Dimensional

Dirty Data Missing Sets Resource Failure

Initial loadIncremental loads

Nominal Data

…

18

Automating Testing: The Scenario Cycle


36

Starting a Team

Option 1: Gradual Improvement Iterations Required

Stage Fast Slow

0: Generic Scrum 1 2

1: Pipelined Delivery 1 2

2: Sized-Based Estimation & Release Plan 2 4

3: Reference Model 2 4

4: Test-Led Development 3 6

5: Requirements Mgt & Automated Testing 3 6

Total Iterations 12 24

Elapsed Time (2- and 3-week iterations) 24 – 36 wks 48 – 72 wks

Option 2: Fast,

Test-Driven

Cultural Change

…

19

Agile vs. Really Tough DW Challenges

Situation:

• Team asked to add the newly acquired company’s

billing data to the warehouse

• Don’t know the data’s structure, attributes, or quality

• Management wants to know how long will it take and

how much is it going to cost...

• ...so they can decide what degree of integration they’ll

sponsor

38

Bonnie’s Maxim

There are no silver bullets...


...but monsters are

real.

…

20


Response:

• Agile does’t mean you can’t use all your smarts and tools!

– *Do* profile the data, calculate a ROM estimate, etc.

• “Don’t know the data’s structure, attributes, or quality....”

– Agile is the best way to proceed when facing a large unknown

– We get started after one-fifth as much analysis (i.e., 80/20 rule)

– We visualize & demo data as we go, for developer and business

– The problems will bubble up to the surface much faster

• “Management wants to know how long will it take and how much”

– Give us two iterations against the real problem, and we’ll give you

an estimate based upon real knowledge, not speculation

40


Summary: If it’s going to be “mission impossible”

...tackle it with Agile,

...because disaster will happen in iteration 1 or 2

...rather than leaving all the problems to suddenly appear

in user acceptance testing.

Agile can’t boil the ocean any better than

waterfall can, but it fail to do it faster.

41

…

21

42

Agility through Back Filling the Architecture

Staging

Historical

Archive Integration Dimensional OLAP Department

Access

Trends

360 Vision

Dash boards

1:

2:

3:

4:

5:

Single Version of the Truth

End User

1% of data

5% of data

10% of data

25% of data

Full data

Copyright (c) 2010, Ceregenics, Inc.,

www.ceregenics.com

…

(c) Ceregenics 2009 43

Integration

Source

Agile Trims Standard DW Architecture

Staging

Presentation

Front End Tools

ETL = $$$

ETL = $$$

ETL = $$$

Source

Source Archive

Integration

Presentation

Front End Tools

Config

Config

Config

Change Data Capture

Data Warehouse Appliances

Associative Query Engines

ConfigXX

XX

ETL

ETL

ETL

Maximize the

objects not built

…

22

Advanced Topics

• Agile & Project Risk

• Agile Resource Planning

• Agile Architecture?

• Bidding an Agile Project


Agile & Project Risk

Risks

– Poor business requirements

– Misunderstood requirements

– Needless requirements

– Institutionalized scope creep

– Under-employed developers

– Untested product


via

Requirements Management

pipelined early starts & task board

iterative & automated testing

M

Mitigation

…

23

Agile & Project Risk

Further Risks

• Rework

– ETL sensitive to data model changes

– Refactoring for re-usability

• Team disruption

– Re-assignment and production support

• Geographically-remote teammates

– Possible if performed carefully


Team leaders’ docs developed

during iterations -1 and 0:

• Vision Doc

• Initial Data Model

• Initial Ref. Model

Mitigation

…

Resource Planning

Advantages of Agile

• Size-based estimation fast & more accurate

• 20% LOE estimates easier to derive

• Faster to spin-up projects & minimize bench time

• Staff planning becomes “We run three teams, that’s what we do”

• Agile is fast to train

• Quick, thorough, and standard means to manage staff augmentation

Disadvantages of Agile

• Requires “agile” temperament, esp. in leadership positions

• Greatest velocity from consistent team rosters

• Emphasizes co-located teams


24

Agile Architecture?

Concerns

• “Incremental optimization” can equal “muddling through”

• “How to avoid an Apollo 13-type situation?”

Mititgations

• Wrapper process includes requirements management

• Architecture comes from departmental standards

• New & smaller programs can externally acquire an architecture

• Reserve 20% of team bandwidth for non-functional requirements


Bidding an Agile Project

Challenges

• Need 2 to 3 iterations to establish a velocity

• Product owner in command of backlog

Possible solutions

• Two-part bids: “Establish Velocity” and “Remaining Deliverables”

• Reference Architecture

• “Reference” Agile team to backlog and story point a new project

• Analyze as waterfall and reduce proportionally for Agile approach

• Requirements management to constrain backlog churn

• Any new requirement affecting scope, budget, or duration CR

• Customer’s ability to conclude early can lead a fixed-price project to T&M

...puts you on the same side of the table with them


25

Summary

• High-level view of how Agile DWBI works

• How Agile DWBI is reducing time-to-market

• Estimating and quality get better under Agile

• Start-up path for a DWBI team or shop


51

For More Information

Available via

Amazon.com and Barnes & Noble

Ralph Hughes

Ceregenics, Inc.

[email protected]

999 18th St., Ste 3000,

Denver Co 80202

720.951.2100

26

52

Questions

Prochaine rencontre

Nous sommes heureux d’annoncer que notre

prochaine rencontre sera

53

Au programme:

À venir...

27

54

Merci à nos commanditaires!

55

TDWI Membership Offer

• 10% discount on TDWI Membership

– For TDWI Chapter attendees; Limited time only

– You can renew or extend your existing TDWI membership

• Benefits

– Discounts on conferences, seminars, certification, books

– Valuable reports and publications from TDWI Research

– Web archive of research

– Inquiry Service for Team Members

• Individual and team memberships options

• Join today!

28

57

Environnement

Nous récupérerons les enveloppes de plastique

We recycle plastic name tags

58

Annonces

Informez-vous à l’avance et nous pourrons

annoncer votre événement BI.

Let us know in advance and we

could announce your BI event.

29

59

La présentation

La présentation sera envoyée

aux gens présents seulement

The presentation will only be sent to poeple who

are present today.

60

Mot de la fin / Parting words

Commentaires & suggestions à

Send comments and suggestions to

[email protected]

30

61

Merci !

Thank you!

chapitre de montréal du tdwi - · pdf filechapitre de montréal du tdwi 13...

Documents