knowledge discovery in sap bw 3.5: the analysis process

Glen LeslieSAP Labs

Product ManagerSAP NetWeaver BW

Knowledge Discovery in SAP BW 3.5:

The Analysis Process Designer

SAP AG 2003, Title of Presentation, Speaker Name / 2

§§ Analysis Processes Analysis Processes -- OverviewOverview

§§ Analysis Process DesignerAnalysis Process Designer

§§ Analysis Process vs. Data Staging ProcessAnalysis Process vs. Data Staging Process

§§ Integration of Data Mining Workbench into APDIntegration of Data Mining Workbench into APD

§§ Introduction in Data Mining MethodsIntroduction in Data Mining Methods

§§ Summary and OutlookSummary and Outlook

Agenda


Motivation

Explore your data

thoroughly

Explore your Explore your data data

thoroughly thoroughly Recognize

relationships in your data

Recognize Recognize relationships relationships in your datain your data

Use advanced transformations and analytical

methods

Use advanced Use advanced transformations transformations and analytical and analytical

methodsmethodsPersist

Analysis Results

Persist Persist Analysis Analysis ResultsResults


Credit Score

èYou know the credit behaviour of your clients

èScore the credibility of your clients (credit rating)

Business Cases

Advanced ‘Slow Sellers’ Analysis

èGroup your customers in an ABC-analysis

èFind A-customers who did not buy during the last month in order to start a marketing campaign


Analysis Process

Gain New Insights Into

Your Data

Gain New Gain New Insights Into Insights Into

Your Data Your Data Explore meaningful relationships between

your data, which are hidden or too complex to be uncovered through pure observation

(OLAP analysis) or intuition

FeaturesuAdvanced analytical methods (data mining, …)uBlock transformationsu(Persistent) (sub-)query results


“Knowledge Discovery in Databases”

There are several divergent KDD methodologies / models. Nearly all can be summarized / synthesized into the following 5 main phases:

Task Analysis Preprocessing Data Mining Postprocessing Deployment

ü Task

ü Business Understanding

ü Problem definition

ü Analysis of requirements

ü Data Selection

ü Data Cleaning

ü Data Preparation

ü Data Transformation

ü Model Develop-ment (Training)

ü Running Models (Prediction)

ü Output Generation

ü Evaluation/ Analysis of Results

ü Deployment of results

“Knowledge Discovery in Databases” (KDD) is a process-based approach to the search for potentially high-quality knowledge in source databases


The Analysis Process Designer: KDD Compliance

Update and Analysis in SAP BW

Preparation

Selection

Transformation

x²+y²+2dx+2ey+f=0

(x,y)=F(x²,y²)

SAP BW

Campaigns-Target groups

Transfer results into & apply in

OLTPs (e.g. SAP CRM)


A Basic Analysis Process

Sources

Transformation

Target


Business Benefits

The most important potential which can be realized with this improved information are:

ü Cost reduction (TCO)

ü Revenue improvement (ROI)

ü Improved customer experience and -satisfaction

By being 100% integrated into the SAP BW, the Analysis Process Designer (incl. Data Mining Features) also guarantees, that only a single database is accessed and not different data tables in different source system. This significantly decreases interfacing problems as well as related issues with data integrity, quality and system performance.








Agenda


Analysis Process Designer Integration

Analysis Process

Target- BW- CRM

Source- BW- Flat File- 3rd Party

Transformations Data Mining- SAP- 3rd Party

BW 3.0B

BW 3.5

Pre-Process(TX: RSANWB)

Analyze(TX: RSDMWB)

Analysis Process Designer

Data Mining


The Interactive Workbench of the APD (SAP BW 3.5)

Analysis Process

Repository

Drag & DropContext Menu for Display and

Settings

Transaction:RSANWB


Analysis Process Designer

Sources Transformations

Data MiningTargets

DB Table

Flat File

Master Data

Query

Aggregate

Routine

Sort

Filter

Transpose

Drop Column

Merge

ABC Class.

Scoring

Regression

DecisionTree

3rd Party

AssociationAnalysis

Clustering

Master DataODS Object

CRM System

InfoProvider


Characteristic: Read data from an InfoObject Master Data

InfoProvider: Use an InfoCube, ODS object, or Multi-Provider as source

Query: Read data from a query

Flat File: Read data from a flat file

Data Base Tables: Read data from a database table

APD – Sources (Step 1)

Read from the following sources :

Based on the task or problem at hand, data must be provided which is statistically and semantically relevant. The APD provides a mechanism for delivering this data in an easy to use, graphical environment. Subsequent steps in the process can then be assured of a firm bases for continued analysis.


APD Data Transformation (Step 2 & Step 3)

Preparation (Step 2) :

To maintain the quality of the analysis process results a clean,complete and error free database is crucial. In order to ensure quality, the APD provides basic data operations to prepare and cleanse the raw data.

Transformation (Step 3) :

With the help of the robust transformations it is possible to discover and make hidden information relationships apparent.


Filter: Restrict the amount of data to be processed

Aggregation: Group and aggregate data according to selected fields

Join: Merge data from two different sources

Sort: Sort the data according to the selected fields

Transpose into columns: Transform flat data records into a list

Transpose into rows: Transform a list into flat data records

Hide columns: Hiding of entire columns

Use the following operations to prepare/transform your data:

APD – Data Transformation I: Preparation (Step 2)


ABC Classification: Calculation of ABC Classification

Regression: Application of linear/non-linear Regression algorithm

Clustering: Application of Clustering algorithm

Scoring: Application of “Weighted Score Table“ algorithm

Decision Tree: Application of Decision Tree algorithm

Data Mining: Application of external (non-SAP) Data Mining Models

Routine: Custom transformation via ABAP routine

Data Mining models can also transform data!

APD – Data Transformation II (Step 3)


APD – Visualization of Intermediate Results

è Display Data: the data per “process node“ can be displayed at any time in tabular format.

è Elementary Statistics: Advanced visualization methods for a quick view of basic properties and quality of the interim results per “process node”. This functionality includes histograms, distributions and basic statistical measures like means, standard deviations, correlations and visualizations.

è Calculate Intermediate Results: the interim results of each “process node” can be stored temporarily for performance reasons.

A cleansed, complete and error-free database is crucial for meaningful results in an analysis process. This is realized step-by-step within the APD. At the same time it must be possible to check each process step individually.

To facilitate the discovery process the APD provides visualization tools with which intermediate results can be easily displayed or the quality of the data can be analyzed:


APD – “Data Display“


APD – Basic Statistics


ODS Object: Load results back to a transactional ODS; from here updates into other SAP BW Data targets can be performed.

Master Data: Update InfoObject master data

OLTP System: Transfer results to a OLTP system, e.g.CRM system

Association Analysis: Training/Application of the Association Analysis algorithm

Regression: Training of linear/non-linear Regression algorithm

Clustering: Training of Clustering algorithm

Decision Tree: Training of Decision Tree algorithm

APD – Data Targets (Step 4)

Write results back to the following targets :

Data Mining Models as Data Targets :








Agenda


ETL versus APD

n ETL Processu Extraction: data procurement, that is selection of

relevant data from source systems and supply of the data work area; requires delta capabilities

u Transformation: processing and massaging data to specified structure and quality requirements of the data in the workspace

u Loading: bringing the data physically from the workspace into the Data Warehouse

n Analysis Processu Creation of new information out of existing datau Transformation of datau Writing back new data to BW database or operational

system (CRM)

Similar functionalities, but different objectives.


ETL versus APD

Source Systems Data Warehouse

ETL Process

Data Data New Data

APD Process

Data copies New Data

~ = +








Agenda


What is Data Mining?

Data Mining: an analytical approach that looks for hidden information

patterns in large databases


What is Data Mining?


Data Mining in SAP BW APD data flows

APD

Task Analysis Preprocessing Data Mining Postprocessing Deployment

ü Task

ü Business Understanding

ü Problem definition

ü Analysis of requirements

ü Data Selection

ü Data Cleaning

ü Data Preparation

ü Data Transformation

ü Model Develop-ment (Training)

ü Running Models (Prediction)

ü Output Generation

ü Evaluation/ Analysis of Results

ü Deployment of results

Step 1: Select Data

Step 2: Preparation

Step 3: Transformation

CRMother Systems (e.g. CRM)

Step 4: Store/Transfer Data

SAP BW

Step 5: Deploy DataCamp.Targetgroups

ABC Analysis

A B C


Top Five Reasons for Using SAP Data Mining

1. Fully integrated

2. Standard and out of the box functionality

3. Easy to use

4. Extensible with 3rd party tool connection

5. Delivered with best business practice scenarios

=Requirements Return

TotalCost ofOwnership


1. Fully Integrated

Web shop

Campaign

Call Center

Analyze

DeploySAP BW

SAP CRMIntelligenceConnector

AnalysisProcessDesigner

DataMining

Measure InfoProviders


2. Standard Functionality

DataMining

SAP BW

SAP CRM

n Time required to activate SAP Data Mining:

n Additional licenses required to be able to use SAP Data Mining:

00


3. Easy to Use

Drag & Drop

Map Data Flow with

Arrows

Visualize Complete Data Process inSingle Screen

Double Click on Nodes to

Get Properties


With BW3.5, SAP releases an API which 3rd party vendorscan use to connect their solution to the SAP Data Mining framework.

n User benefits from greater flexibility and choice in terms of data mining engines

n Additional algorithms and complementary functionality are offered

n Delivered connector by 3rd party minimizes cost and effort required for integration

n List of certified 3rd party vendors is available on SAP service marketplace

4. Extensible with 3rd Party (1)


Complexity

BusinessNeed

SAP OLAP

SAP Data Mining

3rd Party Data Mining (with SAP Partners)

4. Extensible with 3rd Party (2)


5. Delivered with Best Business Practice Scenarios

SAP delivers with BW ready to go and easily extensible reports, data infrastructure, and mining models for selected best business practice scenarios. The immediate benefits of using this business content are reduced implementation time and costs.

A few examples include:

n Churn Management: monitor, understand, predict and manage customer attrition behavior

n Cross Selling Analysis: discover cross-selling opportunities in your product and service portfolio

n Customer Migration Analysis: monitor changes in customer behavior by tracking segment migration over time


SAP Data Mining < BW 3.5

£ Data Mining methods have been available in SAP BW since Release 3.0B:ð ABCð Association Analysisð Regression ð Decision Treeð Clustering

£ Training and application of the SAP Data Mining Methods via a separate Workbench (Data Mining Workbench)

£ single "Data Source“ à BW Query

£ Training, Evaluation and application of Data Mining Models via "Wizard“


APD – Process Summary

Step 1: Select Data

Step 5: Deploy DataCampaigns-Target groups

CRMOther System (e.g. CRM)

Step 4: Store/Transfer Data

ABC Analysis-Customer

A B C

BW

Step 2: Prepare Data

Step 3: Transform DataSAP Data Mining Integrationè Analysis Process Designer

functions as a new, graphical Frontend for Data Mining to train and apply Data Mining models.

è Train Data Mining model (the model is a target node in the Analysis Process)

è Application of Data Mining models for prediction in Analysis Processes (the model is a transformation node in the Analysis Process)








Agenda


Data Mining Methods - Explorative

A B C n 20% of customers generate 60% of sales

Clustering

ABCClassification

AssociationAnalysis

WeightedScore Tables

……

Gender

………………

IncomeAgeCustomer

……

Score

Weight


Clustering

Cluster Distribution

Fields Used in Clustering

Distribution of Specified

Field


ABC Classification

ABC Classes

Gain Chart


Weighted Score Tables

Weighting SchemeSummary


Association Analysis

List of Cross-Selling Rules

Probability Indicators

Export to CRM


Data Mining Methods - Predictive

Historical

New Data

Historical

Historical

New Data

New Data

Train Model Evaluate

Model

Apply Model

1 2

3


?Age

Sales

Complaints

# of Products

Gender

Income

Buy

Yes

No

Complaints

Gender

Income

Data Mining Methods - Predictive

>4<=4

MF

<=30 k >30 k

Regression

Decision Trees

No

No Yes

Yes


Regression

Quality Measures

Coefficients for Linear

Regression


Decision Trees

Distribution within Specified Node

Rule within Specified Node

Preview Pane


SAP BW Advantages as a Data Mining Platform

n Tight integration with operational systemsn Reduction of processes and support staffn SAP BW Customers already own it!


Closed-Loop Analytics with SAP Data Mining

From Diagnostic Analytics with Best-of-Breed…

… To Closed-Loop Analytics with SAP.

Transforming Insight into Action !

Data Mining

Data Store

OperationalProcesses

SAP BW SAP CRM

Data Store

OperationalProcesses

SAP BW

SAP CRM

Models

Results

Best-of-Breed Data Mining

Data Store


Roles Required for Data Mining

BusinessAnalyst

StatisticianIT AdminBusinessAnalyst

Best-of-Breed Solutions SAP Solution

IT Admin


SAP Data Mining vs. Best-of-Breed Solutions

SAP Data Mining may not have as extensive breadth of algorithms as best-of-breeds products have, but if you don’t need it, why pay for it?

SAP

BoB

Coverage ofAnalyticsNeed

BoB Functionality SAP Functionality

SAP

BoB

Offered Used Offered Used

+








Agenda


Outlook

BW Integration

l Process Chain Integration

l Data Targets: Flat File, InfoCubes

l Write Back to SAP operational systems

l Variables

Enhanced Transformations

l Formula Editor

l Prognosis

l Sampling, Binning, Normalization, ...

l Boolean Algebra: Union, Subtract, ...

knowledge discovery in sap bw 3.5: the analysis process

Documents