data audit approach to developing an enterprise data strategy

Alan McSweeney

Data Audit Approach To Developing An Enterprise Data Strategy

1

Objective

• Define a data audit approach to creating an enterprise current data state view as part of defining an enterprise data strategy

February 18, 2015 2

Developing And Implementing An Enterprise Data Strategy

• Any enterprise data strategy of an existing and mature organisation with a substantial portfolio of applications and associated data should start with a data audit that establishes a baseline that will be one input to a data strategy

• Any new strategy needs to take into account this (possibly) substantial applications and data legacy

• Any strategy has to be implementable and operable

• There will be a current state and a future state where the future state represents the fully actualised strategy

February 18, 2015 3

Current State

Desired Long-Term

Steady State

Need to Move From Current State To Future

State In A Series Of Steps

Developing And Implementing An Enterprise Data Strategy

February 18, 2015 4

Business Objectives

Business Operational

Model

Enterprise Architecture

Solution Implementation

and Delivery

Management And

Operations

Business Processes

Required Operational

Business Systems

Business Strategy

Systems Design/

Selection

Business IT Strategy

IT Function Strategy

Enterprise Data

Strategy

Required Operational Processes

Required Infrastructure

Business Systems

Systems Design/

Selection

Information and Data

Architecture

Enterprise Data Strategy In Business And IT Context

February 18, 2015 5

Enterprise Data Strategy In Context

• An enterprise data strategy exists in a wider organisation and IT context − The organisation will have an overall IT strategy to accomplish the

organisation strategy and associated objectives − The IT function will then need its own internal IT strategy that will

structure the function in order to ensure that it can deliver on the wider organisation strategy

− The enterprise data strategy is connected to the overall IT strategy, the enterprise architecture and the internal IT strategy

− The enterprise data strategy will be implemented and operated through an information and data architecture that is part of the overall enterprise architecture

− This context is important in ensuring that the enterprise data strategy fits into the overall IT and wider organisational structure

− The enterprise data strategy exists to ultimately deliver a business benefit and contribute to the achievement of the business strategy

− The strategy must be translated into an operational framework to enable the strategy to be actualised

February 18, 2015 6

Traditional View Of Information And Data Architecture In An Enterprise Architecture Context

February 18, 2015 7


Information Systems Architecture

Data Architecture

Solutions and Application Architecture

Business Architecture

Technology Architecture

Data-Oriented View Of Information And Data Architecture In An Enterprise Architecture Context

February 18, 2015 8


Information and Data Architecture

Information Systems

Architecture

Solutions and

Application Architecture

Business Architecture

Technology Architecture

Traditional View Of Information And Data Architecture In An Enterprise Architecture Context

• Data and Information Architecture - the structure of an organisation's logical and physical data assets and data management resources – is defined as a subset of Information Systems Architecture which key applications and data that form the core of mission-critical business processes

• Data and Information Architecture manages the information of the enterprise by clarifying business relationships and enhancing the understanding of the business processes and rules implemented by the enterprise

• Data and Information Architecture links Business Processes to the Information Systems that support the processes

February 18, 2015 9

It’s All About The Data (And The Processes)

• Data needs to be organised by business process, not by application − The enterprise is the sum of its processes

• An effective data architecture is a principal driver of successful business models and therefore competitive advantage

• Providing business experts timely access to accurate data is the key factor in improving the ability of the enterprises to make effective and informed business decisions

February 18, 2015 10

Components Of An Information And Data Architecture And Associated Strategy



Data Governance Data Architecture Management

Data Development Data Operations Management

Data Security Management Data Quality Management

Reference and Master Data Management

Data Warehousing and Business Intelligence Management

Document and Content Management

Metadata Management

Components Of An Information And Data Architecture And Associated Strategy

• Data Governance - planning, supervision and control over data management and use

• Data Architecture Management - defining the blueprint for managing data assets

• Data Development - analysis, design, implementation, testing, deployment, maintenance

• Data Operations Management - providing support from data acquisition to purging

• Data Security Management - Ensuring privacy, confidentiality and appropriate access

• Data Quality Management - defining, monitoring and improving data quality

• Reference and Master Data Management - managing master versions and replicas

• Data Warehousing and Business Intelligence Management - enabling reporting and analysis

• Document and Content Management - managing data found outside of databases, including digital strategy and social media

• Document and Content Management - integrating, controlling and providing metadata


Information And Data Architecture Components And Their Functional Elements

• There are a number of functional elements associated with each of these components


Data Management Functional Elements

Goals and Principles Activities

Primary Deliverables Roles and

Responsibilities

Practices and Techniques

Technology

Organisation and Culture

Information And Data Architecture Components And Their Functional Elements

• Goals and Principles - directional business goals of each function and the fundamental principles that guide performance of each function

• Activities - each function is composed of lower level activities, sub-activities, tasks and steps that are function-specific

• Primary Deliverables - information and physical databases and documents created as interim and final outputs of each function. Some deliverables are essential, some are generally recommended, and others are optional depending on circumstances

• Roles and Responsibilities - business and IT roles involved in performing and supervising the function, and the specific responsibilities of each role in that function. Many roles will participate in multiple functions

• Practices and Techniques - common and popular methods and procedures used to perform the processes and produce the deliverables and may also include common conventions, best practice recommendations, and alternative approaches without elaboration

• Technology - categories of supporting technology such as software tools, standards and protocols, product selection criteria and learning curves

• Organisation and Culture – this can include issues such as management metrics, critical success factors, reporting structures, budgeting, resource allocation issues, expectations and attitudes, style, cultural, approach to change management


Why It Happened?

Why Is Likely To Happen In The Future?

What Is Currently Happening?

What Happened?

Every Organisation Aspires To ...


Reporting Insight/ Forecast

Monitoring Analysis

Trailing And Leading Indicators

Reporting

• Report on Gathered Information On What Happened To Understand Pinch Points, Quantify Effectiveness, Measure Resource Usage And Success

Monitoring • Gather Information In Realtime To Understand

Activities, Respond And Make Reallocation Decisions

Analysis • Understand Reasons For Outcomes and Modify

Operation To Embed Improvements

Insight and Forecast

• Quantify Propensities, Forecast Likely Outcomes, Identify Leading Indicators, Create Actionable Intelligence


Trailing Indicators

Leading Indicators

Every Organisation Needs An Effective Enterprise Data Strategy


Data Operations Management

Data Quality Management

Data Development

Metadata Management



Data Security Management

Data Warehousing and Business Intelligence Management

Data Governance

Data Architecture Management

Reporting Insight/ Forecast

Monitoring Analysis

Solid Data

Management Foundation

and Framework

} You Cannot Have This ...

... Without This

Measurement Framework Iceberg


To Do This ...

... You Need To Do This ...

... Which Requires This ...

... Which In Turn Needs This ...

... And So On ...

...

...

...

Be Able To Take Action Based on Reliable

Information

Measure What is Important

Know What Is Important In Order To

Measure It

Define Measurements

Define Consistent Units of

Measurements

Define Measurement Processes

Define Operational Framework

Define Collection Process

Define Data Storage Model

Define Transformation And Standardisation

Install Data Collection Facilities

Collect Data

Monitor Data Collection

Manage Data Collection

Validate And Store Data

Report And Analyse Stored Data

Define Reports

Run And Distribute Reports

Define Analyses

Run And Distribute Analyses

Provide Realtime Access To Collected

Data

Define Data Tools And Infrastructure

Processes Define How The Organisation Delivers Its Products And Services


Business Function

Business Function

Business Function

Business Function

Business Function

Partners

Regulators

Customers

Service Providers

Suppliers

Collaborators

Core And Extended Organisation Landscape


Business Function

Business Function

Business Function

Business Function

Business Function

Partners

Regulators

Customers

Service Providers

Suppliers

Collaborators

Core Landscape

Extended Landscape

Processes Define How The Organisation Delivers Its Products And Services

• Work – products and services - moves throughout the extended organisation landscape as it is delivered to the customer

• Data accompanies – supports, describes, enables, measures – work



Cross Functional Processes Crossing “Vertical” Operational Organisational Units To Deliver Work


Core Cross Functional Processes

• Three cross-functional processes that are common to all organisations − Product/service delivery

• From order/specification/design/selection to delivery/installation/implementation/provision and billing

− Customer management • From customer acquisition to management to repeat business to up-sell/cross-sell

− New product/service provision • From research to product/service design to implementation and commercialisation

• These processes cross multiple internal organisation boundaries and have multiple handoffs but they are what concern customers

• Cross-functional processes deliver value − Value to the customer − Value to the enterprise

• Integrated cross-functional processes means better customer service and more satisfied and more customers


Core Cross Functional Processes and Customer View

Product/Service Delivery: from order

to completion

Customer Relationship Management

New Product/ Service

Provision

The organisation sees the structure vertically and in a compartmentalised view and all to frequently does not see the customer viewpoint

The customer sees across the structure and is not concerned with but is all too often aware of the operational elements, their complexity and lack of

interoperability

Organisation Data

• Data flows within the organisation between business functions, supporting the key processes of: −Delivery of products and services

− Customer acquisition, management and retention

− Product and service development

• Enterprise data model needs to be structured to define process interactions and associated data − Feed data into processes to enable their efficient operation

− Take data from processes to allow their operation to be monitored


Organisation Information And Data Landscape

• Information and data landscape defines the operational data environment for the organisation −Operational Use

• Storage

• Manage

• Share

• Exchange

−Analytic Use • Monitoring

• Reporting

• Analysis

• Forecast


Enterprise Data Model Needs To Encapsulate Data Landscape


Enterprise Data Model

Subject Area Model

Conceptual Data Model

Enterprise Logical Data

Models


Elements

Data Steward Responsibility Assignments

Valid Reference Data Values

Data Quality Specifications

Entity Life Cycles


Generalised Enterprise Business Process Model

Business Controlling

Process

Processes That Direct and Tune Other Processes

Core Processes Processes That Create Value for the Customer

Customer Acquisition

Product Delivery

Order Fulfilment

Customer Support

Enabling Processes Processes That Supply Resources to Other Processes

Channel Management

Supply Management

Human Resources

Information Technology

Business Acquisition

Business Measurement

Process

Processes That Monitor and Report the

Results of Other Processes

Customer’s Process Needs

Supplier’s Processes

Business Environment Competitors, Governments Regulations and Requirements, Standards, Economics

Generic Enterprise Business Process Model

• Representation of the key processes within and across an enterprise − The enterprise is the sum of its processes

• Key processes require and generate data

• Data model needs represent data to and from processes



Data Collection And Measures Need To Be Linked To Key Enterprise Processes

Business Controlling

Process

Processes That Direct and Tune Other Processes

Core ProcessesProcesses That Create Value for the Customer


ProductDelivery

OrderFulfilment

CustomerSupport

Enabling ProcessesProcesses That Supply Resources to Other Processes

Channel Management

Supply Management

Human Resources

Information Technology

Business Acquisition

Business Measurement

Process

Processes That Monitor and Report the

Results of Other Processes

Customer’s Process Needs

Supplier’s Processes

Business EnvironmentCompetitors, Governments Regulations and Requirements, Standards, Economics

Number of New

Customers

Customer Turnover

Profitability Per Customer


Cost

Number of Customers Complaints

Time to Resolve

Complaints

Delivery Time

Accuracy

Number of Returns

Payment Times

Inventory

Time to Fulfil Order

Invoice Accuracy

Forecast Accuracy



Business Function

Business Function

Business Function

Business Function

Business Function

Partners

Regulators

Customers

Service Providers

Suppliers

Collaborators




• Build an enterprise data model in layers

• Focus on the most critical business subject areas − Subject Area Model

− Conceptual Data Model

− Enterprise Logical Data Models


Subject Area Model

• List of major subject areas that collectively express the essential scope of the enterprise

• Important to the success of the entire enterprise data model

• List of enterprise subject areas becomes one of the most significant organisation classifications

• Acceptable to organisation stakeholders

• Useful as the organising framework for data governance, data stewardship, and further enterprise data modeling



• Conceptual data model defines business entities and their relationships

• Business entities are the primary organisational structures in a conceptual data model

• Business needs data about business entities

• Include a glossary containing the business definitions and other metadata associated with business entities and their relationships

• Assists improved business understanding and reconciliation of terms and their meanings

• Provide the framework for developing integrated information systems to support both transactional processing and business intelligence.

• Depicts how the enterprise sees information


Enterprise Logical Data Models

• Logical data model contain a level of detail below the conceptual data model

• Contain the essential data attributes for each entity

• Essential data attributes are those data attributes without which the enterprise cannot function – can be a subjective decision


Enterprise Data Model Components

• Data Steward Responsibility Assignments- for subject areas, entities, attributes, and/or reference data value sets

• Valid Reference Data Values - controlled value sets for codes and/or labels and their business meaning

• Data Quality Specifications - rules for essential data attributes, such as accuracy / precision requirements, currency (timeliness), integrity rules, nullability, formatting, match/merge rules, and/or audit requirements

• Entity Life Cycles - show the different lifecycle states of the most important entities and the trigger events that change an entity from one state to another


Data Strategy

• High-level course of action to achieve high-level goals

• Data strategy is a data management program strategy a plan for maintaining and improving data quality, integrity, security and access

• Address all data management functions relevant to the organisation


Elements Of Information And Data Strategy

• Vision for data management

• Summary business case for data management

• Guiding principles, values, and management perspectives

• Mission and long-term directional goals of data management

• Management measures of data management success

• Short-term data management programme objectives

• Descriptions of data management roles and business units along with a summary of their responsibilities and decision rights

• Descriptions of data management programme components and initiatives

• Outline of the data management implementation roadmap

• Scope boundaries


Data Strategy

Data Management Scope Statement

Goals and objectives for a

defined planning horizon and the roles, organisations, and

individual leaders accountable for achieving these objectives

Data Management Programme Charter

Overall vision, business case,

goals, guiding principles, measures of success, critical

success factors, recognised risks

Data Management Implementation

Roadmap

Identifying specific programs, projects, task assignments, and

delivery milestones

Data Audit And Information And Data Strategy

• The objectives of the audit are to understand the current data management systems, structures and processes

• This will then feed into the development of the strategy and the identification of gaps

• Data audit views 1. Data landscape view 2. Data supply chain view 3. Data model view 4. Data lifecycle view 5. Current information and data architecture and data strategy

view 6. Current data management view


Data Landscape View


Data Landscape View

• The purpose of the Data Landscape View is to describe the entities and functional units within and outside the organisation with which the organisation interacts and to describe the interactions in terms of data flows

• This will show the participants in data flows

• These can be business units, partners, service providers, regulators and other entities

• The data landscape view can be created at different levels of details: − Level 1 – Main Interactions - Main interactions and functions associated with

the Enterprise Level − Level 2 – Business Function - Specific data exchanges of the function − Level 3 – Function - What is done within each function as a series of activities − Level 4 – Procedure - How each activity is carried out through a series of tasks − Level 5 - Sub Procedure - Detailed steps which are carried out to complete a

task


Data Supply Chain View

• The data supply chain view looks at in-bound and out-bound data paths within and outside the organisations in terms of the applications and the data that flows along the data paths

• It can be a subset or an extension of the Data Landscape View


Data Model View

• Enterprise data model is a set of data specifications that reflect data requirements and designs and defines the critical data produced and consumed across the organisation

• Data model view quantifies the status of the development and specification of the enterprise data model





Subject Area Model


Enterprise Logical Data

Models


Elements

Data Steward Responsibility Assignments

Valid Reference Data Values

Data Quality Specifications

Entity Life Cycles

Data Lifecycle View

• When analysing data, what you are really analysing is the state of the processes around its lifecycle: how well defined those processes are, how automated, how risks and controls are defined and managed


Data Lifecycle View


Data Lifecycle View

• The stages in this generalised lifecycle are: − Architect, Budget, Plan, Design and Specify - This relates to the design and specification of the data

storage and management and their supporting processes. This establishes the data management framework

− Implement Underlying Technology- This is concerned with implementing the data-related hardware and software technology components. This relates to database components, data storage hardware, backup and recovery software, monitoring and control software and other items

− Enter, Create, Acquire, Derive, Update, Integrate, Capture- This stage is where data originated, such as data entry or data capture and acquired from other systems or sources

− Secure, Store, Replicate and Distribute - In this stage, data is stored with appropriate security and access controls including data access and update audit. It may be replicated to other applications and distributed

− Present, Report, Analyse, Model - This stage is concerned with the presentation of information, the generation of reports and analysis and the created of derived information

− Preserve, Protect and Recover- This stage relates to the management of data in terms of backup, recovery and retention/preservation

− Archive and Recall - This stage is where information that is no longer active but still required in archived to secondary data storage platforms and from which the information can be recovered if required

− Delete/Remove - The stage is concerned with the deletion of data that cannot or does not need to be retained any longer

− Define, Design, Implement, Measure, Manage, Monitor, Control, Staff, Train and Administer, Standards, Governance, Fund - This is not a single stage but a set of processes and procedures that cross all stages and is concerned with ensuring that the processes associated with each of the lifestyle stages are operated correctly and that data assurance, quality and governance procedures exist and are operated


Data Audit Approach

1. Build an application landscape view, including internal and external systems and third-parties from which data may be obtained and to which data may be supplied

− The application view can be supplement with a system and infrastructure view that shows the hardware and software components behind an application

2. Layer onto this information capture, storage and flows: where and what types of information is maintained by applications and that is passed between applications

− An application is a collection of systems and infrastructure that delivers an integrated set of functions − It may or may not be necessary to document the underlying infrastructure associated with applications − This may be further complicated because the underlying infrastructure may not be isolated but may itself be part

of an application - this would be the case where the server infrastructure is virtualised and managed by virtualisation manager

3. Categorise information by a classification such as: Operational Data, Master and Reference Data, Analytic Data and Unstructured Data

4. Define the business units/functions and their use of applications

5. View the information capture, storage and flows identified above across the stages of their lifecycle

6. Identify how well the processes and their controls associated with the lifecycle stages are defined, documented and operated. This will identify gaps to be remediated

− This will then form the basis of a work plan to resolve any data-related process gaps


Data Audit Approach – Application Landscape


Application 1

Application 2

Application 3

Application 4

Application 5

Application 6

Application 7

Application 8

Application 9

Data Audit Approach – Data Capture, Storage And Transfer


Application 1

Application 2

Application 3

Application 4

Application 5

Application 6

Application 7

Application 8

Application 9

Data Audit Approach – Infrastructure And System View


Application Web Server

Database

Web Server

Application Server

Application Server

Database Server Database Server

Load Balancer Load Balancer Authentication Server

User Directory Firewall Firewall

Consists of

Classification Information By Operational Data, Master and Reference Data, Analytic Data and Unstructured Data


Architect, Budget, Plan, Design and Specify

Enter, Create, Acquire, Derive, Update, Integrate, Capture

Secure, Store, Replicate and Distribute

Preserve, Protect and Recover

Archive and Recall

Delete/Remove

Implement Underlying Technology

Present, Report, Analyse, Model

Define, Design, Implement, Measure, Manage, Monitor, Control, Staff, Train and Administer,

Standards, Governance, Fund

Operational Data

Analytic and Derived Data

Unstructured Data

Master and Reference

Data

Business Functions And Application Use


Application 1 Application 2 Application 3



Business Function 1

Business Function 2

Business Function 3

Business Function 4

Information Capture, Storage And Flows Identified Above Across The Stages Of Their Lifecycle






Archive and Recall

Delete/Remove





Data Type 1

Data Type 3

Data Type 4

Data Type 2

Identify How Well The Processes And Their Controls Associated With The Lifecycle Stages Are Defined






Archive and Recall

Delete/Remove





Data Type 1

Data Type 3

Data Type 4

Data Type 2

Identify How Well The Processes And Their Controls Associated With The Lifecycle Stages Are Defined

• Provides a baseline of the status of data processes in the organisation

• Identify gaps to be remediated

• This will then form the basis of a workplan to resolve any data-related process gaps


Current Information and Data Architecture And Data Strategy and View

• Review current information and data architecture and implementation and operational under the key component areas



Data Governance Data Architecture

Management

Data Development Data Operations

Management

Data Security Management Data Quality Management


Data Warehousing and Business Intelligence

Management


Metadata Management

Current Data Management View

• The data strategy components and the functional elements are be combined to create a view of all the potential elements of an operational data strategy implementation and operational framework

• Not all of these facets will have the same importance

• Each of these facets will also be in a different state of effective operation

• You can create a high-level representation of the state of data management strategy and its implementation


Data Management View – Components And Functional Elements

Goals and Principles

Activities Primary Deliverables

Roles and Responsibilities


Technology Organisation and Culture

Data Governance Data Architecture Management Data Development Data Operations Management

Scope of Each Data Management Function Data Security Management Data Quality Management Reference and Master Data Management Data Warehousing and Business Intelligence Management Document and Content Management Metadata Management


Goals and Principles

Activities Primary Deliverables

Roles and Responsibilities


Technology Organisation and Culture

Importance Current

State Importance

Current State

Importance Current

State Importance

Current State

Importance Current

State Importance

Current State

Importance Current

State

Data Governance

Data Architecture Management

Data Development

Data Operations Management

Data Security Management

Data Quality Management


Data Warehousing and Business Intelligence

Management


Metadata Management

Data Management View – Importance and State


= High Importance

= Medium Importance

= Low Importance

= Good State

= Medium State

= Poor State

Data Management View – Importance and Status

• Coding of data management components and functional elements

• Understand their importance and current state of implementation and operation


Data Audit Views And Results

• Data Landscape View – quantify and understand where data exists

• Data Supply Chain View – quantify and understand data exchanges and interfaces

• Data Model View – quantify and understand the development and specification of the enterprise data model

• Data Lifecycle View – identify how well the processes and the controls associated with the lifecycle stages are defined

• Current Information And Data Architecture And Data Strategy View – identify current information and data architecture and implementation and operational under the key component areas

• Current Data Management View – quantify the relative importance and current state of implementation and operation of data management components and functional elements


Data Audit Views And Results

• Gives a comprehensive view of the current state, desired future state and gaps/deficiencies

• Provides a current state view within the context of a future state

• Ensures that any information and data architecture and strategy is based on evidence

• Enables a realistic workplan to be developed and worked through to achieve the desired results

• Approach can be applied to the entire enterprise or functional component


Now All That Is Left Is The Implementation And Operation



More Information

Alan McSweeney

http://ie.linkedin.com/in/alanmcsweeney

http://ie.linkedin.com/in/alanmcsweeney