Steps Towards Business Intelligence

Ahsan Kabir ,MVP

What is BI

Business Intelligence is an umbrella term thatincludes the applications, infrastructure andtools, and best practices that enable decisionmakers to make proper decisions.

• What happened?

• What is happening?

• Why did it happen?

• What will happen?

Past

Present

Future

“Understand the pulse of the

Organization”

Why BI

How BI ?

Microsoft BI Technologies

What is DW?

“…is designed specifically to be a central repository for all data in a company separated from transactional systems.”

“…is designed to be the source of analysis and reports.”

“But it’s not a copy of a source database.”

Why DW

• Central Repository

• Reduce extra load

• sources unaffected

• Empower Business Users

• Improve data quality

• Single version of the truth

1) Data volumes,

2) Real-time data,

3) New sources and types of data, and

4) Cloud-born data

But …..

Now

• The data warehouse is unable to keep up with explosive volumes.

• The data warehouse is falling behind the velocity of real-time performance requirements.

• The data warehouse is slower than desired in adopting a variety of new data sources, slowing time–to-value

• The platform costs more, while performance lags.

Planning

1. Analytical and Report requirement

2. Business Process

3. Prioritization

4. Identify Source Data

5. Dimensional Model

6. Documentation

7. Design Data Warehouse

Data Warehouse vs.Data Mart

Data Warehouse

Enterprise-wide

Union of all data marts

Data Mart

Departmental or Business line

Single business process

Kimball

Bottom-Up

Data marts

Logical data warehouse

Decentralized

Quick results, iterative approach

Inmon

Top-Down:

Enterprise data model

Centralized

Later create data marts

More upfront work but less redo

Kimball vs. InmonMethodology

Data Model

OLAP cube /Multidimensional modeling :

“…is based on the OLAP cube and is fitted with measures and dimensions”

In-memory tabular model:

“…is based on a new In-memory engine for tables “

OLAP cube/Multidimensional modeling

Fact or measure

“… are numeric and additive values “

Foreign keys

Dimension

“…Descriptive information”

Surrogate key.

Business key.

SSAS Loaded into in-Memory engine called xVelocity in-memory.

Tabular modeling allows you to create a table-based model from existing data in Warehouse and and create a relationship between models.

Data Analysis Expression (DAX) is an expression language for SSAS Tabular, which helps you create calculations and measures based on existing columns and relationships.

Tabular Model

Schema Design

The layout indicate the relationship between facts and dimensions is called a schema.

Star Schema :

For each fact entity join with single level of dimension entities.

Snowflake Schema :

If there are dimensions with large numbers of attributes, it might be necessary to break the dimensions down into sub dimension entities

Star Schema

Snowflake Schema

DEMO

Analysis Services (SSIS)

1. Develop Cubes and 2. Create dimensions and measures.3. Creating hierarchies4. MDX queries will be compiled,

parsed, and executed in the SSAS engine

ETL

“…is a program that periodically runs.”

Extract

Fetching data from the source

relational databases, web services, and

SharePoint lists.

Transform

“..Cleansing the data and converting to a

OLAP-friendly data model”

Load

“..loading data into the data warehouse

as fact and dimensions”

DEMO

Data is kept in a

specific business

line wise.

Before enter into warehouse

Data is processed

(cleansed and transformed)

Warehouse Data Marts

Users query

the data

warehouse

“…staging area is an area where we fetch data from different sources exactly as it is into our integrated database. “

Staging

Data Quality Services

Data quality issues can be divided into the following categories:

Uniqueness

Validity

Accuracy

Standardization

Completeness

Name Address City House No DoB State Country

Ahsan CDA Avenue CTG 181/1 05/11/1978 BD

Kabir RB Avn CTG 41/6 23/04/1991 DHK Bangladesh

Before

After

Accuracy Consistency Completeness Conformity

Name Address City House

No

DoB Stat

e

Country

Ahsan CDA Avenue CTG 181/1 05/11/1978 CT Bangladesh

Kabir RB Avenue DHK 41/6 23/04/1991 DHK Bangladesh

Start DQS

Knowledge Base Management

Knowledge Base Management is where you can create and manage Knowledge Base, domains, and domain rules

Data quality projects

projects apply the Knowledge Base and matching rules on an existing dataset and provide results.

Administration

Configuration and administration tasks can be performed here

Components in DQS

1. Cleansing,

Cleansing is about cleaning data based on a Knowledge Base and domains.

2. Matching,

Matching would match data based on the similarity rules and threshold defined in a Knowledge Base.

3. Monitoring

Monitoring will show the status of records during the cleansing and matching projects.

4. Profiling.

Profiling will help in creating business rules or changing the domain rules and Knowledge Base from what the existing data profiling results are.

DEMO

Technology

SSDT

“…is the integrated IDE for SSIS, SSRS, and SSAS. SSDT was formerly known as Business Intelligence Development Studio (BIDS). “

SSIS

SSIS was released with this name for the first time in 2005, but prior to that, it was named Data Transformation Services (DTS). DTS was available even in SQL Server 2000

SSRS

“is a data Visualization tools for

developing and publishing reports”

ReportServer DB

Report definition,

Snapshot,

Execution log etc.

ReportServer TempDB

Session and

Cached information.

Report Server web application

Report Manager web application

Reporting Services Configuration Manager.

Master Data Service (MDS)

“…is data shared across computer systems in the enterprise.”

“… is the dimension or hierarchy data in data warehouses and transactional systems”

“… is core business objects shared by applications across an enterprise

-The processes and technology to produce and maintain a single clean copy of master data

Customer

ABC

PQR

XYZ

Country

Europe

Norway

Sweden

Features

Domain management

models, entities, attributes, and hierarchies.

Business rules

Data validation is also provided.

Import and export master data

Data cleanup

Architecture

SQL Server database for storing data and metadata.

MDS engine read and write information to that database by : WebUI and Excel Add-ins.

MDS uses subscription views to export information from MDS to other systems

Staging mechanism to import data from other systems, which is called entity-based staging.

DEMO

Resources:

1. Microsoft SQL Server Analysis Service (SSAS)

Demo Program: Step By Step: Develop ETL Process using SQL Server Integration Services (SSIS)

https://gallery.technet.microsoft.com/Design-Cube-in-SQL-Server-49ee6e1c

2. Microsoft SQL Server Integration Service (SSIS)

Demo Program: Step By Step: Develop ETL Process using SQL Server Integration Services (SSIS)

https://gallery.technet.microsoft.com/Step-By-step-Creating-a-d0e3e71d/edit?newSession=True

3. Microsoft SQL Server Reporting Service (SSRS)

Demo Program: Step by Step SSRS Report Design Using CUBE

https://gallery.technet.microsoft.com/Step-by-Step-SSRS-Report-8de35ea8

Resources:

To know more about SQL Server 2014

https://www.microsoft.com/en-us/server-cloud/products/sql-server/Resources.aspx

Microsoft SQL Server Data Tools - Business Intelligence for Visual Studio 2013

https://www.microsoft.com/en-us/download/details.aspx?id=42313

Adventure Works 2014 Sample Databases

https://msftdbprodsamples.codeplex.com/releases/view/125550

Resources:Data Warehouse Architecture – Kimball and Inmon methodologies: http://bit.ly/SrzNHy

SQL Server 2012: Multidimensional vs tabular: http://bit.ly/SrzX1x

Data Warehouse vs Data Mart: http://bit.ly/SrAi4p

Fast Track Data Warehouse Reference Guide for SQL Server 2012: http://bit.ly/SrAwsj

Complex reporting off a SSAS cube: http://bit.ly/SrAEYw

Surrogate Keys: http://bit.ly/SrAIrp

Normalizing Your Database: http://bit.ly/SrAHnc

Difference between ETL and ELT: http://bit.ly/SrAKQa

Microsoft’s Data Warehouse offerings: http://bit.ly/xAZy9h

Microsoft SQL Server Reference Architecture and Appliances: http://bit.ly/y7bXY5

Methods for populating a data warehouse: http://bit.ly/SrARuZ

Great white paper: Microsoft EDW Architecture, Guidance and Deployment Best Practices:

http://bit.ly/SrAZug

End-User Microsoft BI Tools – Clearing up the confusion: http://bit.ly/SrBMLT

Microsoft Appliances: http://bit.ly/YQIXzM

Thanks