loshin operationalizingdatagovernance

Operationalizing Data Governance through Data Quality Control

David Loshin Knowledge Integrity, Inc.

www.knowledge-integrity.com

1 © 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com

(301)754-6350

Linking Organization to Practice p  Mapping business value drivers to

data expectations and objectives, granting oversight and accountability, and verifying performance of compliance with corporate information policies n  Processes prescribed for operations, n  Procedures for day-to-day

observance n  Oversight for verifying compliance

© 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com

(301)754-6350

2

Accountability

Data Policies

Processes & Best Practices

Information Standards

Business Policy

Roles and responsibilities

Program management

Data policies and standards

Business intelligence

Business terminology

Data quality

Auditability

Business Value and Data Dependence


(301)754-6350

3

Expenses

Risk Management

Revenue Customer Experience

Performance

p  Business policies, corporate mission, and strategic performance objectives can be translated into dimensions of value

p  These criteria are used for prioritizing effort in relation to maximizing value

p  Data governance helps establish the relationship between value drivers and information utility

Sources of Information Policy


(301)754-6350

4

Expenses

Risk Management

Revenue Customer Experience

Performance

•  Customer lifetime value analysis

•  Voice of the customer

•  Satisfaction surveys

•  Asset productivity analysis

•  Human capital performance

•  Regulations •  Operational risk •  Market factors

•  Customer acquisition and retention

•  Investment opportunities

•  Product performance

•  Spend analysis •  Commodity risk •  Cost management

Data Management Challenges

Data Use and ReUse

Data Requirements

Reinterpreting Semantics

Measurement Triage and Remediation

Impact Assessment


(301)754-6350

5

The Data Policy Lifecycle: Actualizing Governance

p  Refinement of business requirements for facets of information utility

p  Specification for data quality measures and level of acceptability

p  Determination of functional requirements to facilitate continuous compliance


(301)754-6350

6

Determination of need

Drafting a Policy

Policy Review & Approval

Design & Development Marketing Deployment

Information Policies and Data Governance

Integrated Data

Governance

Manage User Requirements

Data Discovery

Shared Semantics

Embedded Validation


(301)754-6350

7

•  Business metadata management •  Data quality discovery and assessment •  Specifying data quality rules •  Inspection, monitoring, measurement •  Managing data lineage

Managing the Quality of Business Metadata

p  Many sources of entity concepts and business terms may conflict with each other

p  The data governance framework must facilitate the collection, documentation, and harmonization of business terms


(301)754-6350

8

Policies

System Docs

Processes

Models

Standards

Applications

Business Rules

Profiling

Etc.

Entity Concepts

Business Terms

Definition Contextual Meaning

… …




Data Discovery

p  Data Discovery enables these types of questions to be answered: n  What data sets are available? n  What entities are embedded? n  What data elements are

available? n  How is the data accessed? n  What are the quality

constraints? p  The results can be shared via

a platform for managing semantic metadata


(301)754-6350

9

Data

Integ

ratio

n

Attribute

First d 4 6 y

Last f 6 2 h

Street d 4 7 n

City a 0 2 o

State

Value Count

A 12000

I 10000

L 7655

X 3208

N 120

M 8

Data Quality Assessment

p  Analysis of data sets, records, data elements, and data values to n  Identify potential anomalies n  Determine business impacts n  Evaluate dimensions for measurement of data quality


(301)754-6350

10

Analysis

Data Quality: Expectations, Rules, and Monitoring

p  Data quality rules can be used to monitor conformance to data policies

p  Conformance can be measured, thresholded, and reported at each handoff location in the processing stream

p  Specific failures can generate events as directed by Data Quality Service Level Agreements

p  Static auditing: measurement applied to a “static” data set n  Examples: SQL queries, data profiling tools

p  Inlined monitoring: measurement performed within a process flow n  Example: edit checks, dynamic monitors

p  All measurements are compared against acceptability thresholds

p  Acceptability threshold is related to the degree of impact

11 11 © 2013 Knowledge Integrity, Inc. www.knowledge-integrity.com

(301)754-6350

Data Quality Rules: Measures and Thresholds

p  Provide specific n  Measures n  Methods of measurement n  Units of measures n  Levels of acceptability


(301)754-6350

12


(301)754-6350

13 13

Data Quality Control

p  Controls measure observance of data expectations based on information policies and corresponding data rules

p  Those rules are refined based on an analysis of the data dependencies and defined expectations

p  Controls are placed at relevant locations within the process stream

Producer Process

Consumer Process

As data is handed off between process tasks,

controls validate accuracy, completeness, consistency, timeliness against defined

expectations

Instituting Inspection Using Data Quality Rules

p  Apply tools and techniques for measuring conformance to data rules (using data profiling and data monitoring tools):


(301)754-6350

14


p  Data quality expectations are inspected and any emerging issues are identified:


(301)754-6350

15


p  Different events can be triggered by a data failure, such as notifications to data stewards:


(301)754-6350

16


p  Or logging the failure in a Data Quality Incident Management System and score card:


(301)754-6350

17


p  Effectiveness demonstrated when: n  Control events occur when data failure events take place, n  The proper mitigation or remediation actions are performed, n  The corrective actions to correct the problem and eliminate its root

cause are performed within a reasonable time frame, and n  A control event for the same issue is never triggered further

downstream p  Measurements can be aggregated over time into performance

metrics

© 2013 Knowledge Integrity, Inc.

www.knowledge-integrity.com (301)754-6350

18

Integrating Data Quality Reporting with Governance


(301)754-6350

19

Processing Stage

Processing Stage

Processing Stage

Processing Stage

Tools & Processes: Operationalizing Data Governance

p  Methods and tools for data discovery: profiling data, statistical analysis of values, and model evaluation

p  Metadata management through a central platform for knowledge capture and communication

p  End-to-end visibility of lineage for structure, semantics, and use across enterprise

p  Data Quality assessment p  Integrated data quality control p  Inspection, monitoring, and reporting


(301)754-6350

20

Questions and Open Discussion

p  www.knowledge-integrity.com

p  If you have questions,

comments, or suggestions, please contact me David Loshin 301-754-6350 [email protected]


(301)754-6350

21

www.dataqualitybook.com

www.mdmbook.com

loshin operationalizingdatagovernance

Documents