event management for effective data movement by design - data... · event management for effective...

IRM Consulting, Ltd. Co. © All Rights Reserved

QBD: Data Movement Event Management 1

© 2005, IRM Consulting, Ltd., Co.; Confidential & Proprietary

Quality By Design:Event Management for Effective Data

MovementIRMACIRMAC

Toronto, ON Toronto, ON –– October, 17October, 17thth 20072007

Andres PerezSenior Information Management Consultant

IRM Consulting, Ltd., Co.12415 Stable Wood

San Antonio, Texas 78249-4621(210) 413-1481 - [email protected]

QBD: Data Movement Event Management 2(© IRM Consulting, Ltd. Co.; Confidential & Proprietary)

Andres PerezInformation Quality and Information Management Consultant

He provides consulting in the disciplines of:• Program and Project Management.• Information Quality Management (TIQM® Certified).• Enterprise Information Resource Management (Information Stewardship, Information

Administration, Metadata Repository and Management• Information Architecture (Modeling, Design, Standardization, Profiling, Reverse Engineering,

Movement, and Integration (Enterprise Information Modeling, ETL and EAI)• Information Integrity audits (COBIT), Analytical Solutions (Business Intelligence, Data

Warehousing, ODS, Data Marts), Customer Relationship Management and other Large Operational Solutions (OLTP)

He conceived and implemented an information management program called “Data Certification.”Mr. Pérez is a well known speaker at data management and information quality conferences in the US

and Europe including DAMA International, Information Quality, ZIFA (Zachman Institute for Framework Advancement) and IAA (Insurance Application Architecture). His presentations encompass information resource management, information stewardship, information quality management, and enterprise information architecture.

He is the VP of Marketing for DAMA International, past member of the Board of Directors for the IAIDQ, past President of the Heart of Texas DAMA Chapter, past member of the IAA Board of Directors, and past member of the Object Management Group.

Mr. Perez is a certified TIQM® consultant and has a BS in Mechanical Engineering and Business Administration from the Universidad Autónoma de Nuevo León; Monterrey, México.

He may be contacted at: phone: +1 210-413-1481 or email: [email protected].

Andrés Pérez is an Information Resource Management Consultant and President of IRM Consulting, Ltd. Co., based in San Antonio, Texas. Mr. Pérez specializes in information resource management, information architecture and information quality management.




AgendaThe Business Case for Event Management

Purpose & ObjectivesConsiderationsPrinciples

The Data Movement Design ProcessGeneral Design ConsiderationsDetail Design ConsiderationsEnd-To-End Design

Event & Process ManagementE&PM FunctionsE&PM Application DesignEvent Logging & Parameter Maintenance

Learning From Event ManagementDefect Reporting & Prevention

Q&A


The Business Case for Event ManagementUnderstanding the Challenges of Data Movement

The scary thing is how bad it is. We’re betting our economy [sic] on an information world, and we don’t have any idea how good the information in those databases is.”

Robert Goldberg,MIT Sloan School of Management.




Purpose & Objectives

Support Moving Data with Quality and Control:Reduce information defectsIncrease confidence in the informationPromote use of the information

Provide Event Management (Audit / Monitor)Capture events (e.g., defects) & key process information in electronic formQuickly Notify Responsible PartiesProvide Reports & Support Analysis

Provide Process Management (Balance & Control)Externalize controls for ease of operationsExternalize process hierarchies for improved controlExternalize processing rules for balancing

Deliver Business ValueDesign for All Batch & Messaging ProcessesDevelop as part of EDW initiativesProvide appropriate securityEnable “value” visibility to business and IT areas


Economic Considerations

Leverage InvestmentUse Existing Web InfrastructureProvide open access (web browser)Provide appropriate security

Maintain Balance Between Value & CostDesign With Target State in MindDevelop Minimal Capabilities to deliver valueDesign to “Build as you go”




Processing Considerations

Data Movement is Usually Batch/MessagingEvents Can Have High Volumes (at any one time)Provide Trend Analysis (at one time; over time)Notify appropriate personnelEnable drill-down –– find and correct systems and data problemsProvide the ability to improve:

Understanding the specific issue (a meaningful description)Knowing the person or group to be notifiedUnderstanding the impact… Without Changing The Logging Programs


Functional ConsiderationsManage events:

Capture with little overhead (minimalist approach)Enable dynamic changes to notification & reporting details (manual entry)Provide an easy to use & access interface (GUI on web) for manual data entryProvide batch process support to load large volumes of data (e.g., file and table layouts)

Provide capability to manage process controls

Externalize process controls (e.g., the last processed post date)Externalize balancing logic (e.g., enable dynamic definition of amounts, counts and comparisons)Externalize process hierarchies (e.g., scheduler to workflow to program relationships)




Data Movement PrinciplesDon’t Do It!!!

It adds costReduces semantic valueIncreases information floatIntroduces opportunity for quality defects

“The goal of Information Management is to avoid and eliminate unnecessary data movement by providing well defined enterprise information models and well implemented enterprise-strength databases that meet the information, accessibility and performance needs of all stakeholders”

-- Larry English


Data Movement Principles (Cont’d)Well … if you must, use Data Movement to:

Bring information from, or send information to, outside data producers or consumersMove from the official record-of-origin to the official record-of-reference data storeMove from the official record-of-reference data store to an application software package database and backMove from an enterprise record-of-reference data store to the analytic or strategic (operational data store or data warehouse) databaseMove data as a one time conversion from a legacy data store to the newly architected data store.




The Data Movement Design Process

“Begin with the end in mind.”

-- Steven Covey


Design Iteration(s)

Data Movement General Design

Staging

• Source A• Source B• Source C• Source D• Source E• Source F• …

Source 4

Source 3

Source 2

Source 1

Staging

ConformedDimensions

TargetData Mart

BusinessAnalysts

Pre-Existing Flows

New Flows

Design Process Flow

Data Flow

New Flows




Requirements Gathering Matrix

Reports & Ad-Hoc Needs Data Sources

Group ID Name Data Element Transformation Table/File Data Element

1 2 3

6 5

4

5 4 3

6

1

2

A

BSource Mr. Richard Curtright)


Requirements Gathering Matrix - Sample

Plan Quantity -the quantity of product planned based on orders (excludes schedule receipts).

Plan History (New table)

Plan from XYZ System – Plan Order (current is Saturday reload; need new download on Friday with history)

PlanPlan Adherence

R-109Supply

Actual ProductionPlan

Production CasesProduction QuantityProduction UOM

Source Column Name

R-109

Production Weekly

Source Name

Set to the (Actual Production – Plan) if greater than zero; otherwise = set to zero

Over Production

Plan Adherence

R-109Supply

Based on the historical ProductionNote: the current table is dropped and reloaded; need to keep history.

Actual Production

Plan Adherence

R-109Supply

TransformationReport Field

Report Name

Report IDGroup




Inconsistent Business Meaning

Asking the same question from the lines of business of a Financial Organization such as “What’s the Customer Credit Rating?” elicits different meanings from each:

P&C Insurance: Customer’s billing account status (is the customer current?) used to determine continuance of a policy.Banking: Customer’s payment history from external sources (e.g., credit bureaus) on borrowed funds used for loan/credit card determination.Life Insurance: Customer’s credit status used to determine non-availability of funds (i.e. IRS seizures).


Business Concept Definition

Business Concept Definition

Business Concept Template

Definition A succinct, complete, business description using common terms or previously defined terms.

Related BusinessConcepts

• A short-cut or hot-link to a previously defined term, for example: Customer.

• A description or annotation of an undefined term, for example: Products or services are pending definition.

Data IntegrityRules

• A bulleted list of data integrity rules.

Unique Identifier Whenever is appropriate and or necessary, this is free form text, bullets, or whatever can be used to provide insights on the appropriate way to uniquely identify occurrences of the data entities associated with the business concept.

Status XXXXX status as of x/x/xx

Life Cycle Whenever is appropriate, this is free form text, bullets, or whatever can be used to provide insights on the states and transitions associated with the business concept.

Classification Whenever is appropriate, this is free form text, bullets, or whatever can be used to provide insights on the intrinsic classification necessary to understand the business concept.

Domain • A bulleted list of domain values. It can be all-inclusive or a sample; please indicate which approach is used.

• Can be a diagram (see Line of Business as an example)

Special Usage This is free form text, bullets, etc. can be used to highlight real-life cases of the use of the business concept.

Examples This is free form text, bullets, etc. can be used to provide real-life or mock-up illustrations of the business concept.

Issues & Concerns Proposed definition concerns:• A bulleted list of issues or concerns associated with the

definition proposed in this document.Existing definition concerns:• A bulleted list of issues or concerns associated with the

existing definition, or definitions.

Background Use free form text and bullets as needed. For specific references to existing document, use the name and date of the document.

Classification (subtypes)

Business Concept Title (term)

Definition (clear, concise, succinct, declarative)

Related Business Concepts (used to define this one)

Data Integrity Rules (business rules declaratives)

Unique Identifier (recommended keys)

Life Cycle (states the Business Concept experiences)

Status (W/D/R/P)

Domain (value constraints, ranges, value set, profile)

Special Usage (special conditions)

Examples

Issues & Concerns

Background (research, findings)

Working DraftRecommended Published

Legend:




Sample Fact & Dimensions

PRODUCT

LOCATION

PRODUCTTRANSFER

SourcePeriodModel

Weight

WEEK

CUSTOMER


Sample Data Movement

Production

Planned Transfer Extract

Planned Transfer Prorate

Create Transfer Facts

Actual Transfer Update

INVTRNSF8Actual

Transfer Extract ActualTrnsWrk

Actual Transfer

Product

Inventory

Planned Transfer

ProdTrnsDrWrk

PlanTrnsWrk

Actual Transfer

Load

Data Mart

Planned Transfer Update

Product

Plant

Week

Month

Planned Transfer

Load

Data Mart

Planned Transfer

Actual Transfer

Month

Week

Week

TRUNCATE

Week

Inventory-Plant

Drop & Reload Staging Table

History (Append) Staging Table

“Disposable” Table

Pre-Existing Dimension Table

New Fact Table

Inventory

Planning

PLNPRDTRNF7




Sample Process Context Diagram

Planned Transfer Extract

PLNPRDTRNF7

WeekDimension

ProductDimension

LocationDimension

Information Defects

Location-PlantX-Reference

MonthDimension

Process Statistics

Planned ProductTransfer


Sample Data Movement Specification

Data Movement Specification

Spec Name PPIR0006 – Planned Product Transfer Extract

Source System System Y

Load Strategy • Drop & Reload.

Status Published as of 11/30/2005

Filters

Target System System X

… … … …

Source File Source Y Target Table Target X

References • Ref A, Ref B, …

Volumetric • 3.46 M inputs per week/Condensed to 1.05 M outputs

Target Field Name Source Field Name Conversion Specs Validation

Sort

Name T-1 Name S-1 Map, convert, aggregate, etc. Must be …

Name T-2 Name S-2Name S-3

Map, convert, aggregate, etc. Must be …

Name T-3 Name S-4 Map, convert, aggregate, etc. Must be …

… … …. …… … …. …

Sample Filters:Select Transfer Category = A17, A18, A34, A42Exclude all entries with a Source or Destination of “EXCESS” or “EXP ” or ending with “_XYZ”…

Sample Validations:If not in Product Dimension, report as “1148” and discard recordIf not in Plant Dimension, report as “1149” and create a “skeleton”Dimension row…

Planned Transfer ExtractPLNPRDTRNF7

WeekDimension

ProductDimension

LocationDimension

Information Defects

Location-PlantX-Reference

MonthDimension

Process Statistics

Planned ProductTransfer




Sample Data Movement Specification (Cont’d)

270,984/wkVolumetric:Sunday NightSchedule:

Product Dimension, Location Dimension, …References:

Product TransferTarget Table:SRCMTRC107Source File:

Truncate & InsertLoad Strategy:NewType of Change

…………

Must be in Product (1148).If not, produce error report and discard the record

Set to the Key in Product for matching Item to Product Code and Brand Code.Item is structured as follows:

“P” (constant)6 digits numeric (Product Code)“-” (constant)4 digits numeric (Brand Code)

SRCMTRC107:Item Code

Product DimensionProduct CodeBrand CodeProduct Key

Product Key

ValidationTransformation SpecificationSource ColumnsTarget Columns

NoneExclude rows with Company = “OTH”Filter

ValidationSpecificationProcessing Consideration


Event & Process Management

Enabling Defect Correction& Process Controls




Batch or Message Process*

(Parameter Generator)

Parameter Generator

Event Logging

Web Application

Reports (Canned & Ad-Hoc)

E&PMDatabase

Escalation(Process Alerts)

ETL Tool Catalog(Process Logs)


(Event Consumer)

* Batch or Messaging Processes can be: Informatica, PL/SQL, UNIX Script, etc.

Event Log Purge

Metadata ExportDBMS Catalog

Publish Processing Information

Notification (Information

Defects)

Derivation & Validation


(Balancing Function)

Event & Process Management Functions


(Parameter Consumer)Parameter

Maintenance

Automated Processes


Web Application

Provides an Intuitive, Web-Based set of functions to manually maintain the E&PM Configuration:

Register Registrant AuthoritiesRegister Stewards (Access & Security)Register Data ContainersRegister ProcessesRegister Process ParametersRegister Events




Web Interface

Event and Process Management


Business Concept: RegistrantA person or organization (usually an organizational unit within the enterprise) responsible for the registration and maintenance of processes and data containers and their associated data elements and events.Registrants assign stewardship rights to people (usually workers in the organizational unit) to develop and maintain processes and data containers. This right may be revoked (e.g., when the person changes jobs or responsibilities) but the responsibility for the created objects remains with the registrant.Example: EDW (for the Enterprise Data Warehouse team)

Registrant




Registering Registrant


Business Concepts: StewardA person assigned stewardship by the registrant for development and maintenance of processes and data containers and related events.When a steward creates a new process or data container, the identifier of the associated registrant will be included in such registration. Stewards can change their association with the registrant, but objects once created remain with the original registrant.

Steward




Registering Stewards


Business Concept: ProcessA program or software component that performs a defined set of activities as prescribed by its code or structure.A process is registered when it is expected to produce registered events.Examples:

Scheduler StepInformatica Work-FlowJava Program

Special Notes:For Informatica, processes will document Work Flows and Sessionsbut not Mappings. Mappings will be documented as part of the Sessions.

Process




Registering Processes


Registering Processes




Business Concept: ParameterA processing characteristic or control variable necessary to manage and control the execution of a process.Examples:

Last Processed Date: the latest (largest) value of a column in an incoming table or a field on an incoming file containing the row or record modified date used to select only rows or records changedsince the last time the process executedDatabase location: the name of the physical location for a database or table.Record or row count: the number of records read, written, rejected, etc., by a process. A set of these parameters (one for read, one or more for written and one or more for rejected) may be used to ensure that all the records or rows that were initially received were properly processed by the end-to-end process

Parameter


Registering Process Parameters




Registering Derivations


Business Concept: Data ContainerA file or database table registered in the ABC database that contains one or more data elements.

Data Container




Registering Data Containers


Business Concept: EventA fundamental observation of physical reality of a happening, condition, or occurrence that has been previously identified as possible, has been described as part of a process, and has been registered to enable its recording.Examples of information events:

A date field was received as part of a file and it is not in a valid format (e.g., dd-mm-yyyy)A numeric field contains non-numeric dataThe end date of a period is before the start date of a periodA value considered a foreign key (or look up) is not found on the referred table or file

Examples of processing events:A file was received with fewer records than the control file indicatesThe control parameters needed by a process to execute effectively are not present

Event




Registering Events


Registering Events




Logging EventsSmall footprint tableThe “sequence” and “date” can be filled via database trigger.When a process encounters a registered condition; logs it with an InsertETL tool specific routines (reusable components) are created to standardize the logging process (e.g., an Informatica Mapplet fills “processing context” and generates an event if the insert fails)

Event Logging


Maintaining ParametersSmall footprint viewWhen a process requires to record a new parameter value it does it with an Update onlyETL tool specific routines (reusable components) can be created to standardize the parameter maintenance process (e.g., an Informatica Mapplet to update the “last process date” and generates an event if the update fails)

Parameter Maintenance




Database (Information Model)


Learning from Event Management

“Speak with data”-- Ishikawa




Event (Defectives) Reporting

Trend AnalysisPareto AnalysisProcess Control Analysis

Source Data IssuesData Movement Data Issues“No Action” Data Issues

Summary ReportsDetail Reports


Defectives Per Week – Phase I

2,839

460

1,4371,581

484

707

574

706

3670

500

1,000

1,500

2,000

2,500

3,000




Defectives Per Week – Phase II

3,783,788

0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

4,000,000


Defectives Per Week – Phase II (Second Part)

313,493

179,457

159,108

249,083

267,290

306,213

203,116204,060

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000




0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1407

1288

2005 13

1110

0117

4516

48 1101

1315

1316

1450 12

01110

919

08

Event Registration ID

Inci

dent

sCount

Pareto Analysis (Illustration Only)

63.7

78.5

25.4

1009894.288.4

Percent


Data Corrections (Source)“Missing Product to Plant Allocation”

8,331

62,459

87,604

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

90,000




Data Corrections (Source)“Missing Product to Plant Allocation” (After Initial Correction)

223 5

860

665

1,156

1,240

25

706736677

3

1,399

2,359

1,450

3,5183,602

2,650

152272

1,221

86

0

1,000

2,000

3,000

4,000


Data Corrections (Source)“Zero Line Available Hours”

557

11

194

172

190

575

518

518505

476475

406

401

0

100

200

300

400

500

600




Data Corrections (Data Movement)“Mismatching Special Forecasts”

64,531

760 577767377

62,762

48,289

48,101

48,477

48,540

48,977

50,351

52,037

51,581

0

10,000

20,000

30,000

40,000

50,000

60,000


Data Corrections (“No Action”)“Zero Pounds Per Hour”

60

686960

56

6061

57

63

61

60

0000

10

20

30

40

50

60

70




Reports: Summary (One Week’s Sample)

1,899,897Week Ending date Before Begin DateP00231169

780,513Line Availability Hours Not FoundP00161236

175,195Product-Plant Allocation Percent Not FoundP00161234

60,019Line Percent NOT Greater Than Zero P00121329

19,925Duration Value Not In Valid Value ListP00101196

947New Forecast Type Added to DimensionP00101188

CountEvent DescriptionProcessEvent ID

Source: Production Log March 2006


Reports: Detail (Small Sample)

Source: Production Log March 2006

11/6/2005CORRECTED TO DEFAULT PPIR0032 P002806-0929 | SG121000 | SAND DOLLAR DFU Level Not Available 1277

11/6/2005CORRECTED TO DEFAULT PPIR0032 P003803-0910 | SG140000 | OTHERDFU Level Not Available 1277

11/6/2005CORRECTED TO DEFAULT PPIR0032 P615914-0797 | SG612000 | 86891

Demand Division Owner Not In Restated Market Table

1176



1176

11/6/2005CORRECTED TO DEFAULT PPIR0032 P610249-0797 | SG612000 | SAND DOLLAR


1176



1176



1176

DateAction ProcessEvent Value IDEvent Description

Event ID




Total Weekly Movement Processing Time

13:50:31

65:31:38

0:00:00

12:00:00

24:00:00

36:00:00

48:00:00

60:00:00

72:00:00

Note: Not the “batch window” –includes multiple threads


Andres PerezSenior Information Management Consultant

IRM Consulting, Ltd., Co.12415 Stable Wood

San Antonio, Texas 78249-4621(210) 413-1481 - [email protected]

event management for effective data movement by design - data... · event management for effective...

Documents