copyright 2001 advanced strategies, inc. 1 data bridging an overview prepared for digit 07-11- 2001...

Copyright 2001 Advanced Strategies, Inc.

www.advstr.com1

Data Bridging

An Overview

Prepared for DIGIT 07-11- 2001

By Advanced Strategies, Inc.

Copyright 2001 Advanced Strategies, Inc. www.advstr.com2

What is Data Bridging?

A process that connects two or more fully specified data stores for a limited time or on an ongoing basis

D ataS tore

D ata B ridge D ataS tore

<S ource> <T arge t>


www.advstr.com3

What is Your Data Bridging Experience?


www.advstr.com4

Foundation Concepts


Why Do We Say Data Stores and Not Data Bases? A Data Store is any facility that is capable

of retaining data over time The data stores can be of any type:

– Flat files, Relational Databases, …– Manual or automated

Bridges can be theoretically built among any collection of data stores


Bridges are used:

In Data Conversions & Initial Data Loading As Ongoing Interfaces For sourcing Data Warehouses or Data

Marts ...


Bridging is Not Simple andIs Often Underestimated.

D irty D a ta

U n its

C lean bu tIncom p le te

M illions

Source(s) Target(s)Bridge

B ridg le t

Trigger

Data Bridge Basics

8/31/99 1:40:38 PM rhbBridg ingO vrV iew

C opyrigh t 2001 A dvanced S tra teg ies ,Inc .

w w w .A dvS tr.com


Key Concepts

D ata S tore

Bridges Can Be Batch

Bridges involve:Occurrences of Data Components

Bridges Can Be Real-Time


Key Concepts Continued

The target of one bridge can be the source of another

D ataS tore

D ata B ridge D ataS tore


D a ta B ridge D ataS tore



www.advstr.com10

The Bridging Process


The Data Bridging Process

A bridge involves an integrated collection of activities

Each activity performs a specific function


The functions can be performed singularly across an entire set of data occurrences;

or collectively on a given set of data occurrences

The Data Bridging Process


What are the Minimum Processes in a Bridge?

Suppose there were multiple choices for possible source stores

SalesD ata

W arehouseLoadU nload

O rderSystem (C orpora te

O ffice)

(3 R egiona lO ffices)


Adding Source Selection

Suppose we are not confident that the Source Data is pristine (enough)

S alesD ata

W arehouseS ource LoadU nload

O rderS ystem (C orpora te

O ffice)

(3 R eg iona lO ffices)


Adding Source Data Audit

Suppose the Source Data is predictably flawed.

S alesD ata

W arehouseS ource LoadA ud it U n load


O ffice)



Adding Source Data Scrubbing

Suppose the Source Data is not in the units, size or format we need

Suppose the Target Data needs to be assembled from multiple Sources, ...

S alesD ata

W arehouseS ource LoadS crubA ud it U n load


O ffice)



Adding Transformation

Suppose we need to establish an audit trail for the data; or be able to provide the user with information about the source and/or quality of the Target Data.

SalesD ata

W arehouseSource LoadScrub TransformAudit U nload


O ffice)



Adding Logging

Suppose we need to assure ourselves and our users that the process has gone as expected and that the Target Data can be trusted

SalesD ata


Log


O ffice)



Adding Validation

SalesD ata


Log Valida te


O ffice)



Sourcing Auditing Unloading Loading Scrubbing Transforming Logging Validating

Here are the possible activities in our bridging process;

Not all are always needed, but all should be carefully considered


The Data Bridging Process - The Framework

Sets o fD ata S tores and

C om ponents Aud it thedata

com ponent

Identifysources o f

data

AuditR ules R ejected

D ataC om ponent

U nload thedata

com ponent

Approved,H igher-C onfidenceD ata C om ponent

Approved,Low er-C onfidenceD ata C om ponent

U nloadedD ata

C om ponent

Scrub thedata

com ponent

ScrubbedD ata

C om ponent

Transformthe data

com ponent

Transform ationR ules

Transform edD ata

C om ponent

Load thedata

com ponent

Source toTarget

M appingU n-Transform ableD ata C om ponent

U n-Scrubab leD ata C om ponent

Log

Log sta tus,im age, e tc. Va lida te

resu ltingdata

com ponent

ScrubR ules

S ta tus, Im age, e tc.

D ata C om ponent

C orrectD ata C om ponent

E rroneousD ata C om ponent

Va lida tion R u les

SourceD ata S tore(s)

(Se lected Sources)

(C and idateSources)

TargetD ata S tore(s)

SourcingR ules U nload

R ules

LoadR ules

Logging R u les


www.advstr.com22

A Bridging MethodologyA Few Notes


How to Approach a Bridging Effort? As a Fully Formed Subproject: think “Path

To A Solution” Ideally, initiated after the target data stores

have been designed


Abbreviated Path To a Solution for Bridges Define the bridge, especially the values Analyze the Source, Target and Specify

Mapping, Bridging Rules, ... Design the Bridge Realize the Bridge Implement & QA the Bridge Use & Maintain the Bridge


Bridges Involve All Aspects of a Business Object/Data - what are the data and

relationship occurrences we are moving Process - what are the bridging activities Event - what triggers the bridging activities Location - where is the data and how do we

transport it


Socio-Political - do we need permission to access the data, are there data practices mandates we must consider, can the data be misinterpreted or misused

Enabling-Medium - are there platform conversions required; technology bridging

Bridges Involve All Aspects -- Continued:


www.advstr.com27

Summary


The Big Picture: A Summary


Data Bridging is not normally trivial or cheap

Bridging can make or break many types of projects; it should never be an after thought

A rigorous process can help make it manageable

Data Bridging can be treated as a series of subprojects

Conclusion


www.advstr.com30

Questions?

copyright 2001 advanced strategies, inc. 1 data bridging an overview prepared for digit 07-11- 2001...

Documents

advanced strategies

data bridging process

data marts

data bases

source data audit

source data scrubbing

data bridging experience

specified data stores