Creating Business Intelligence Applications

Take small steps toward success or take a big step toward failure

Walter VerhoevenT & I, Database IntegrationEuropean Service Center5 Nov 2007

Zurich - The Global Insurer

• Offices in North America and Europe as well as in Asia Pacific, Latin America and other markets

• Servicing capabilities to manage programs with risk exposure in more than 170 countries

• Approximately 58,000 employees worldwide

• Insurer of the majority of Fortune’s Global 100 companies

• Net income attributable to shareholders of USD 4.5 billion in 2006

• Business operating profit of USD 5.9 billion in 2006

Business Value Of BI

• Share departmental data efficiently• Understand and improve cost structures• Capacity planning and optimization• Risk Management• Find organizational flaws• Recognize trend• Fraud and abuse detection• Customer attrition

Business intelligence is a cross-organizational initiative. The absence of a enterprise wide system will lead to more data marts and more standalone BI applications that are neither integrated nor reconciled. As a result, the organization would continue to lose the opportunity to enhance its business decisions and competitive advantage.

When You Think BI

• We plan• Data mining• Data warehousing• Multidimensional data analysis

(online analytical processing or OLAP)

• Data security• Data presentation• Data input and imports

• We usually forget• Business justification• Project planning• Data analysis• Implementation consequences• Deployment of BI systems• Maintaining BI systems • Implementing requirements

that come after the distribution and users use the system

Designing a BI Application

2 ways of implementing a release concept

Justification Planning Business analysis Design Construction Deployment

High-quality partial applicationHigh-quality partial application Low-quality inclusive applicationLow-quality inclusive application

- Requirements change over time as users use the system

- Project and development know-how increases and is re-used in the next cycle

- Smaller teams = more efficiency

- Higher customer satisfaction

- Requirements change over time as users use the system

- Project and development know-how increases and is re-used in the next cycle

- Smaller teams = more efficiency

- Higher customer satisfaction

- Bigger initial budget- Easier resource management- Less billing overhead- Less customer specific

solution = higher re-use and re-sale possibility

- Bigger initial budget- Easier resource management- Less billing overhead- Less customer specific

solution = higher re-use and re-sale possibility

VS

There are 16 Milestones In Every BI Application1. Business case assessment2. Infrastructure evaluation3. Project planning4. Requirement definition5. Data analysis6. Application prototyping 7. Modeling the business8. Designing the database

9. ELT Design10. Meta data repository design11. ELT development12. Application development13. Data mining14. Meta data repository

development15. Implementation/rollout16. Release evaluation

Just

ific

atio

n

1 - Business Case Assessment

Deliverables resulting from this step:• Strategic business goals • Objectives of the proposed BI• Statement of the business need • How BI will satisfy that need• Ramifications of not addressing the

business need and not committing to the proposed BI solution

• Cost-benefit analysis results• Risk assessment• Recommendations for the current

business process

If you skip this step:• You end up creating BI that does not

answer the strategic goals of the business

• You will fail to get a strong management commitment

• Costs of the project will not be understood and you might not get the funds needed to implement the solution

• Business problems do not get solved

Assess and identify the business needs that validate such a project and create a return on investment (ROI) strategy for it

Pla

nn

ing

2-Infrastructure Evaluation

Deliverables resulting from this step:• Standards, network and OS• Use of a development

methodology• Estimating guidelines• Scope management procedure• Issues management procedure• Roles and responsibilities• Security process• Meta data capture and delivery• Process for merging project-

specific logical data models into the enterprise logical data model

• Test process requirements• Service level agreements

If you skip this step:• It is mandatory to assess the

hardware, middleware, DBMS, and tools to ensure the BI application performs adequately.

• You will not know the knowledge skills needed to develop, maintain, and use the proposed solution.

• You will not have identified the “key players” and subject experts that are vital for the next steps.

• Your budget estimate will be off.

Understand and map out what you have. What hardware is involved, what skills you have in-house, what databases you need to extract data from, how the departments communicate with one and other, and most important; what info is where

Pla

nn

ing

3-Project Planning

• Function point analysis is a sure way to underestimate effort, budget, and resources (80% data, 20% functionality)

• Create triggers for risks identified in step 1• Quality is more important factor for success than budget, scope,

effort (time), and resources (capable and available people).• Assumptions “always” backfire.• BI is a catalyst for improved decision-making so change is good and

must be managed using change-control procedures • No 2 BI projects are alike, estimates are sure to be wrong.

Designing and implementing a BI decision-support environment is Designing and implementing a BI decision-support environment is very complicated, and BI projects are very costly. The risks of very complicated, and BI projects are very costly. The risks of undertaking such projects without adequate planning and control undertaking such projects without adequate planning and control are unacceptable.are unacceptable.

Pla

nn

ing

3 - Project Planning(continued)

Deliverables resulting from this step:• Goals and objectives• Statement of the business problem• Results from the cost-benefit and

infrastructure gap analysis• Functional project deliverables• Subject area to be delivered• Items not within the project scope• Condition of source files and databases• Team structures• Assumptions made• Constraints that need to be

implemented• Risk assessment• Critical success factors• Communication plan• Availability and security requirements

If you skip this step:• You can’t foresee the costs of

a BI application without careful planning

• Giving a delivery date is guess work without careful planning and resource allocations and commitment

• Its very easy to implement the wrong solution.

• You may never get to finish your implementation because your sponsor jumps ship due to lack of trust.

Poor planning and preparation produce poor results.

Bu

sin

ess

An

alys

is

4 – Definition of Project Requirements

Deliverables resulting from this step:

• Technical infrastructure requirements

• Non-technical infrastructure requirements

• Reporting requirements• Ad hoc and canned query

requirements• Requirements for source data,

including history• High-level logical data model• Data-cleansing requirements• Security requirements• Updated preliminary SLAs• Tools used to create the solution

If you skip this step:• Lose sight of objectives and

scope of the project.• Functionality or data are

requirements are missed.• Security issues are ignored.• Requirements are not prioritized.• Business objectives are not

targeted.

Requirements can only be defined when interviewing all parties Requirements can only be defined when interviewing all parties involved, business sponsors, business representatives, power involved, business sponsors, business representatives, power users, stakeholders, subject matter experts, IT staff. users, stakeholders, subject matter experts, IT staff.

Never skip it or combine it with Never skip it or combine it with data analysis or with application data analysis or with application

prototyping.prototyping.

Bu

sin

ess

An

alys

is

5 - Data Analysis

Deliverables resulting from this step:

• Foundation for normalized and fully attributed logical data model

• Business meta data specifications

• Data-cleansing specifications• Expanded enterprise logical

data model

If you skip this step:• You will create copies of existing

data impairments to the new BI decision-support environment.

• You will compound existing data problems.

• You will create additional redundant and inconsistent BI target databases and applications to maintain.

• You will create a decision-support system, and not a BI solution.

• Your analysis models will provide false information.

• You will not model the business.

BI is a logical data model, it is supposed to be a view on the BI is a logical data model, it is supposed to be a view on the business. It involves top-down logical data modeling and bottom-up business. It involves top-down logical data modeling and bottom-up source data analysissource data analysis and you, like all others, will be confronted with overwhelmingly poor-quality and conflicting data.

Do not judge the success of a BI Do not judge the success of a BI project by the speed with which it project by the speed with which it gets delivered, but rather by the gets delivered, but rather by the quality of its deliverable.quality of its deliverable.

Bu

sin

ess

An

alys

is

6 - Application Prototyping

Deliverables resulting from this step:

• Improved requirement specs.

• Improved budget

• Improved project plan

• Skills matrix

• Issue log

• The prototype

If you skip this step:

• The design of the database, the design of the GUI, and the BI technologies selected will not be able to meet the business requirements or expectations.

• You will not be able to validate cost, effort and resources needed to complete the proposed solution.

Ensures that everyone agrees on what is expected from the final BI Ensures that everyone agrees on what is expected from the final BI application, just make sure that the prototype does become the application, just make sure that the prototype does become the proposed solution.proposed solution.

Type

s of

pro

toty

pes

Scope & Resources

Without a prototype you will find that Without a prototype you will find that you may build a BI solution that will cost you may build a BI solution that will cost much more, and take much longer than much more, and take much longer than you expected and that the skills for a you expected and that the skills for a technology implemented are missing.technology implemented are missing.

AndAnd

You will not realize it until it is too late.You will not realize it until it is too late.

Show-and-Tell

Mock-Up

Proof-of-Concept

Visual-Design

Demo

Operational

Show-and-Tell

Mock-Up

Proof-of-Concept

Visual-Design

Demo

Operational

Bu

sin

ess

An

alys

is

7 - Modeling the Business

Deliverables resulting from this step:

• Logical data model divided into• Business model• Technical model

• Documentation of the above models (Meta-Meta data)

If you skip this step:• Ad -hoc business rules are

invented by departments that do not meet the BI model.

• Without a model the business does not know what it has and where it has it.

• Failing to standardize will cause frustration among users and they will not want to use the BI application because they do not want to be technology/subject experts

Describe the organization in terms of its business activities and the Describe the organization in terms of its business activities and the business objects on which the business activities are performed; business objects on which the business activities are performed; and ensure the correct interpretation thereof.and ensure the correct interpretation thereof.

Des

ign

8 - Designing the Database

Deliverables resulting from this step:

• Target DBMS• Physical data model • Physical BI target databases• Implementation plan• Maintenance plan• Disaster recovery plan• Skill assessment matrix

If you skip this step:• The database model could end

up being designed by developers and not database administrators.

• Database is normalized like a data entry application, killing performance.

• Poor design makes the database non-maintainable.

• Data retrieval can become impossible as dimensions are not factored in to the design.

• Non-uniqueness of primary keys is not factored in, resulting in false facts and data load errors.

BI database models have to be designed for analysis and reporting. Queries can take minutes hours or days, not milliseconds.

Des

ign

9 - ELT Design

• Load resource availability and planning its execution• Implementation strategy: Load all data in one ELT load and unload it

again to validate against the source• Prepare ELT source data in three steps: Reformat, Reconcile,

Cleansing• Design your three loads (Initial, Historical, Incremental)• Design data extraction programs• Design ELT specifications in a source-to-target mapping document

showing where and how each column transforms. • Test third-party vendor solutions or write your own• Create a “staging area” specifications document as well as the

staging area.

Vendors will promise you heaven on earth. Take your time validating their claims and know the limitations.

Deliverables resulting from this step:

• Source-to-target mapping document

• Evaluate and test ELT tools with source data

• Designed ELT process flow

• Design the ELT load programs (initial, historic, Incremental)

• ELT staging area

If you skip this step:• There is no skipping this step, it

has to be done and it will take most of the BI implementation design time.

• You need to implement the model created for the business and store it in the database.

• Cleansing the data is done here, it is paramount that data integrity and quality be implemented in this step.

Des

ign

9 - ELT Design(continued)

Des

ign

10 - Meta Data Repository Design

Deliverables resulting from this step:

• Physical meta model• Data definition language for the

meta data repository• Data control language for the

meta data repository (set of permissions and security constraints)

• Meta data repository programming specifications

• Third-party vendor evaluation

If you skip this step:• You end up developing on an

inferior, inadequate foundation.• Might have to replace the repository

for another later in the project.• Might not be able to facilitate the

proposed solution and the project will fail.

• You fail to evaluate your vendor and:• The might “extort” you when you

need a change• They might discontinue support

Remember it is a database designed to store contextual information about the business data you are allowing to be reported, analyzed, and mined. It can be a centralized or decentralized object or relational database and it can be bought or designed in-house.

11 - ELT Development

Deliverables resulting from this step:

• ETL test plan• ETL programs• ETL program library and scripts• Disaster recovery plan• Skill matrix and a know-how

retention and transfer plan• Commit budget to licensing

third-party tools and development

If you skip this step:• This step is very time

consuming but without it there is no BI data, meaning no BI.

• If quality and testing is not done comprehensively, you will end up providing false figures and logical data errors.

Co

nst

ruct

ion

You will need to implement the ELT design at some point in time and this is a difficult task, mistakes will occur and cover-ups will only cause more problems. Make sure you get your ELT processing dependencies in order and test and validate the results.

ELT in BI applications is all aboutELT in BI applications is all aboutJunk in - Gold outJunk in - Gold out

12 - Application Development

Deliverables resulting from this step:

• Application design document• Application test plan• Application programs and library

and scripts• Training materials• Disaster recovery

documentation• Maintenance plan and budget

If you skip this step:• The success of your application

design and implementation can’t be measured if the test-plan is missing.

• Staff may not have the skills to implement the proposed solution.

• Solution might not scale well as design or data access and tools do not allow it.

• You fail to obtain necessary budget funds for support as maintenance requirements are not communicated.

Co

nst

ruct

ion

This is where you start implementing the prototyping results This is where you start implementing the prototyping results together with your business “subject experts” and power users.together with your business “subject experts” and power users.

13 - Data Mining

Deliverables resulting from this step:

• Data mining database • Analytical data model• Training plan• Know-how retention and

transfer strategy• Operational budget

If you skip this step:• Not being able to get to the data. • Not being able to extract the BI out

of the tool.• Users will not understand how to

use your tool or understand the data, get frustrated and give up on it.

• The TCO and the ROI do not match or not get advertised and the completed application will be discontinued.

Co

nst

ruct

ion

Implement the management and marketing questions in a high Implement the management and marketing questions in a high quality and understandable presentation layer that is easily to quality and understandable presentation layer that is easily to navigate.navigate.

14 - Meta Data Repository Development

Deliverables resulting from this step:

• Physical meta data repository database

• Meta data repository test plan• Meta data repository programs

and library• Meta data repository production

documentation• Meta data repository training

materials• Skill matrix and know-how

retention and sharing strategy

If you skip this step:• You will have to create a

“Logical” repository in the application code based on extracting the “repository” from source data and DBMS tools.

• CASE tools are not a permanent solution and not readable by business users.

• Data is not understood, or not seen in the context it should be.

• Your business know-how may not be sharable or shared, taking the Intelligence out of your business intelligence application.

Co

nst

ruct

ion

Do you create a custom-built in-house solution or will you use a third-party product? If the latter, you will need “add-ons”. Keep in mind that the business will evolve, and so will your model.

15 - Implementation/ Rollout

Deliverables resulting from this step:

• Validated user acceptance test• Production ETL program library • Production application program

library• Production meta data repository

program library• Production BI target databases• Production meta data repository

database• Production documentation• Verified disaster recovery plan

If you skip this step:• When not done right your designs

and development might not be stable.

• You may run into problems if you can’t recover this critical multiple-million dollar project when the business infrastructure becomes dependent on it.

Dep

loy

Did you prepare for production? Have security and data preservation regulations been implemented? Does the infrastructure support the load and is it maintainable? How long does it take to do a disaster recovery?

16 - Release Evaluation

• Post-Implementation Review

• Measures of Success

• Plans for the Next Release

• Deliverables

• Post-implementation review meeting

• Action items

• If you skip this step

• Fail to advertise the implementing solution

• Lose support of sponsors for future releases

If you do not remember your past mistakes you will repeat them

Dep

loy

Questions

The only stupid question is one not asked.