data quality sunz 2012
Post on 19-Oct-2014
1.275 views
DESCRIPTION
TRANSCRIPT
![Page 1: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/1.jpg)
1
16 February 2012
Zeeman van der Merwe Manager: Information Strategy & Planning, ACC
DATA QUALITY: Getting Investment for a Weird Area
![Page 2: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/2.jpg)
2
Why a weird area … ?
• Everyone talks about it
• Very few understand what it is about
• Lots of interpretation and confusion
• Mysticism ?
• How organisations understand & interpret data quality has bearing
• It is more about evolution
• This is a journey … that began in 2008
• With an assessment …
![Page 3: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/3.jpg)
3
How did ACC Justify
Data Quality Tools?
![Page 4: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/4.jpg)
4
State of BI in ACC - 2008
![Page 5: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/5.jpg)
5
Background: ACC in 2008
• Inconsistent statistics for Ministerials
• Inconsistent data terminology
• Duplicate/inconsistent datasets
• Inconsistent business rules
• Unmanaged data in Data Warehouse
• Uncoordinated Data Analyst community
• Minimal standards & processes
Common Interpretation: Data Quality is bad
![Page 6: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/6.jpg)
6
Intelligent Business Vision
![Page 7: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/7.jpg)
7
• Oversees all aspects of data and information management
• Exercise authority and control (planning, monitoring, and enforcement) over the management of ACC’s data and information assets.
To be achieved by:
• Defining, and communicating strategies, policies, standards, architecture, procedures, and metrics relating to data and information
• Developing regulatory procedures for data and information
• Overseeing data management projects
• Providing governance and oversight of ACC data and information related issues
• Communicating the value of ACC’s data and information assets
Why Data Governance ?
Source: ACC Data Governance Terms of Reference (Version 1)
![Page 8: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/8.jpg)
Data users have the right to know what the data means 1. The right to know the definition of the data.
2. The right to know where the data came from.
3. The right to know how the data was calculated or manipulated.
Data users have the right to know how risks to the data have
(or have not) been managed 4. The right to know what Security risks weren't eliminated.
5. The right to know what Quality risks weren't eliminated.
6. The right to know what Privacy risks weren't eliminated.
7. The right to know what Compliance requirements influenced data
processing and usage.
Data users have the right to know who made decisions about
managing the data, according to what rules 8. The right to know who made data-related decisions.
9. The right to know what decision-making checks-and-balances were in place.
10. The right to know how issues have been and will be resolved.
THE DATA USER’S
BILL OF RIGHTS
![Page 9: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/9.jpg)
9
• Data Governance The execution and enforcement of authority over the management of data assets and the performance of data functions
• Data Stewardship The formalization of accountability for the management of data resources
Steward: Old English “Sty Ward”; “Keeper of the sty”
![Page 10: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/10.jpg)
10
![Page 11: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/11.jpg)
11
Data Management
Document & Content
Management
Data
Warehousing
& Business
Intelligence
Management
Reference &
Master Data
Management
Data
Security
Management
Data
Development
Meta Data
Management
Data
Quality
Management
Data
Architecture
Management
Database
Operations
Management
Data Governance
• Specification
• Analysis
• Measurement
• Improvement
• Enterprise Data Modelling
• Value Chain Analysis
• Related Data Architecture
• Architecture
• Integration
• Control
• Delivery
• Acquisition & Storage
• Backup & Recovery
• Content Mgmt
• Retrieval
• Retention
• Architecture
• Implementation
• Training & Support
• Monitoring & Tuning
• Analysis
• Data Modelling
• Database Design
• Implementation
• Acquisition
• Recovery
• Tuning
• Retention
• Purging
• Standards
• Classification
• Administration
• Authentication
• Auditing
• External Codes
• Internal Codes
• Customer Data
• Product Data
• Dimension Mgmt
• Strategy
• Organisation & Roles
• Policies & Standards
• Projects & Services
• Issues
• Valuation
![Page 12: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/12.jpg)
12
Data Governance
Assessment
Assess maturity
• Executive awareness
• Data “ownership”
• Data “stewardship”
• Organisation structures
• Processes & procedures
Establish requirements
Develop
Data Governance
Strategy
Establish
Data Governance
Structures
Develop
Data Stewardship
Functions
Mandate
Data Governance
Strategy
Create
Data Excellence
Awareness
Data Governance
Committee
The Data Council
Define/Recommend
• Approach
• High Level Structures
• Roles/Responsibilities
• Change Management
• Budget
• Educate/Present
• Reporting line
• Agree strategy
• Get mandate to implement
• Identify Business & Technical Sponsor
Launch
Data Excellence
Educate
Organisation
“Recruit”
Data Stewards
Training &
/Mentoring
Identify members
• Subject Areas
• Responsibilities
• Identify/Appoint Implementation Team
• Appoint Principal Data Steward
Messages
• This is serious
• Long term
• Everyone responsible
By area
• Responsibility
• Impact on others
Change Management Plan
• Importance
• Commitment
Define/Agree
• Responsibilities
• Operations
• Sanctions
• Roles
• Areas
![Page 13: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/13.jpg)
13
Data Governance Council
Data Stewardship
Steering Committee
Executive
Data Stewards
Coordinating
Data Stewards
Data Stewardship Teams Business
Data Stewards
Strategic
Tactical
Operational
< 5%
< 20%
< 80 - 85%
Data Governance Office
DMBOK Data Governance Issue
Resolution
![Page 14: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/14.jpg)
14
History in ACC
• 2006 Information Governance Committee
– Failed: no terms of reference
• BI Strategy identified data governance
– CSF for Strategy
• Data Quality Effectiveness Review
• Data Quality Working Group
• Recommended Data Governance
– Received Mandate
– Formed Data Governance Committee
– Identified and appointed Data Stewards
– Formed Data Council
All because of data quality!
![Page 15: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/15.jpg)
15
ACC Executive Mandate
• Defining & implementing a data governance framework across ACC
• Management of data
• Oversee the implementation of the ACC Business Intelligence strategy where it supports the aims and objectives of data governance
• Responsible for all data quality initiatives and resolution of data issues across ACC
• Incorporating existing initiatives where appropriate
![Page 16: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/16.jpg)
16
Data Governance Structures
ACC BoardACC Board
Executive Management TeamExecutive Management Team
Data Governance CommitteeData Governance Committee
Data CouncilData Council
Data Quality
Coordination Group
Data Quality
Coordination Group“Data in Action”
Workgroup(s)
“Data in Action”
Workgroup(s)
Data Governance
Office
Data Governance
Office
Coordinates data management
initiatives across all projects
Resolves data related issues
Reviews and ratifies
recommendations
Overall responsibility for
data governance
Coordinates and supports
data governance in ACC
Data Related
Steering Committees
Data Related
Steering Committees
Governs data related projects
Chief Executive OfficerChief Executive Officer
![Page 17: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/17.jpg)
17
ACC Data Quality Guiding Principles
1. ACC will manage data as a core organisational asset with decisions made based on value and the greater good of ACC and its stakeholders
2. The Data Council is mandated as the only forum for ratifying semantics, definitions and business rules for the use of data
3. Use industry and international data standards whenever, and as current, as possible
4. Data quality will be measured across the value chain and all data consumers will have a voice in specifying data quality service level agreements
5. Business process owners will agree to and abide by data quality Service Level Agreements
6. Data masters will be the primary source for any further use of that data
7. Validate data instances and data sets against defined business rules
8. Apply data corrections at the original source, if possible
9. Data entry and system integration will be automated whenever possible with validation applied on entry
10. System user interfaces will be designed to assist and encourage data quality
11. Reference data is current and relevant
12. All data entities will be semantically unique and defined
13. Metadata will be available to all data consumers
14. All report development will be peer reviewed
![Page 18: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/18.jpg)
18
Data Quality Initiatives
Improving Client Information • Reducing number of client duplicates
created
–350,000 duplicates (60,000 P.A.)
–Staff were trained and Eos improved
–Rate reduced from 18% to 8%
• Improving NHI, Date of Death (DOD)
–Collaboration with MOH
– Improve valid NHI from 49% to 81%
–Provide DOD for 345,000 clients
![Page 19: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/19.jpg)
19
RFI: Data Quality Tools
Data Quality Tools & Processes
• To be used for:
– Analysis of data quality
– Address cleansing, standardisation and validation
– Duplicate identification and elimination
– Monitoring and managing data quality
• First application Client address standardisation and verification
![Page 20: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/20.jpg)
20
Data Quality Management is a process …
SOURCE: SAS/Dataflux
![Page 21: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/21.jpg)
21
Evaluation Criteria (Gartner) 1. Connectivity/Adapters 2. Data Quality Assessment and Visualization 3. Parsing 4. Standardization and Cleansing 5. Matching/Relationship Identification 6. Monitoring 7. Subject Area — Specific Support 8. Address Validation/Geocoding 9. International Support 10.Data Quality Workflow 11.Enrichment 12.Metadata 13.Configuration Environment 14.Deployment Modes and Runtime Environment 15.Operations and Administration 16.Architecture and Integration 17.Service — Enablement Vendor/product image, relationship with ACC & Support Total Cost of Ownership
![Page 22: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/22.jpg)
22
The Business Case
• Financial Savings
– NZ Post
– Rework
– Error recovery
• Reputational Risk Mitigation
– Minister/Public/Clients
• Process Support
– Data cleansing/verification
– Data matching/lookup (Master Data)
• Ease of development/reuse/monitoring
This is on paper …
![Page 23: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/23.jpg)
23
Not on paper • Data governance was working
– Used as a ―sanctioning committee‖
• Data quality must be addressed
– Tools will help to do this
– We must have tools
• Confusion
– Data Quality vs Text mining
– What do the tools actually do?
– How do you use them (development)
– How can they help?
• Current Image: Anything to do with data can be fixed with the tools!
![Page 24: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/24.jpg)
24
Summary
• Procurement highly influenced by:
– Non-paper justification
– Reputation of Data Governance
– Data quality is a ―big‖ problem
• Lots of confusion
– Needs lots of education/mitigation
– Evaluation panel needed lots of coaching
• The organisation was ―ready‖
• The business case … well …
![Page 25: Data quality sunz 2012](https://reader034.vdocuments.site/reader034/viewer/2022051411/544457c0b1af9f740a8b48cb/html5/thumbnails/25.jpg)
25
Useful Sites
• Data Management Association (DAMA) www.dama.org
• The Data Administrator Newsletter www.tdan.com
• The Data Governance Institute (DGI) www.datagovernance.com