Data Value Mapping

Promoting Behaviours which ensure

Higher Quality Data

Presenter

Martin Doyle

Data Value Mapping

• `Everyone who’s

involved in

mapping data to or

from CRM or AX

Who’s it

for?

• How to ‘Engage’

the whole business

in the value of

data

What you

will learn?

Martin Doyle

Data Quality Improvement Evangelist

Who are DQ Global?

Who are

we ?

What

do we

do ?

How

do we

do it ?

What’s in

it for our

clients ?

Who is the Data Value Map for?

Ted Friedman,

Senior Research

Analyst, Strategic

Data Management

Team, Gartner Inc.

You have data

quality issues,

whether you

know it or not. Thomas C.

Redman, Data

Quality: A Field

Guide

“The only people

who need not

worry about data

quality are those

who neither

create nor use

data.”

"Virtually everything in business today is

an undifferentiated commodity, except how

a company manages its information.

How you manage information determines

whether you win or lose.“

Bill Gates

Data Value Map - Start with why!

Your Data – What does everyone

want?

The happiness Equation as illustrated in the book “Solve for Happy” - Mo Gawdat

Why bother – Identify Business Value?

Because you want to

trust your data?

To improve the

maturity of your DQ

processes?

Because you want to

fuel your CRM & ERP

with high octane fuel?

Why bother? Some business value ideas…

Increase

• Trust (Happiness)

• Profitability

• Agility

• Visibility

Decrease

• Risk (Compliance)

• Waste

• Re-Work

• Friction

Two main barriers inhibiting “Data

Management” improvement initiatives

"Management does not see this as an imperative" and...

• This is interesting though, given that the majority (63 percent) have not

attempted to calculate the cost of data errors, and 42 percent make no

effort to measure data quality.

"It's very difficult to present a business case.“

Data - Ball of Grief

SCV

Value

Valid

Value

Parent

ChildOrphan

Fie

ld

Colu

mn

?

Who?

What?

Why?

Where?

When?

How?

Unravelling the Data Management Problem

Inputs Outputs

Key Questions

Where are

you now?

How did you

get here?

Where are

you going?

What are the

obstacles?

What actions

are you

going to

take?

How will you

know when

you have

arrived?

Some Secrets

Secret 1 - IYKODWYDYGWYAG

If You Keep On Doing

What You Do You’ll Keep

On Getting What You Got

Secret 2 - Use a SYSTEM

S

Saving

Y

Your

S

Self

T

Time

E

Energy

M

Money

Secret 3 - METRICS

M

Measure

E

Everything

T

That

R

Results

I

In

C

Customer

S

Satisfaction

"Quality is not an act, it is a habit." - Aristotle

Prevent rather

than cure

evaluate the

causes of

potential errors

in the concept

phase (Data

Value Map)

Understand the

flow of data

(work)

The later an

error is

detected, the

more difficult

and expensive is

to correct

(1:10:100

Principle)

Always seek to

increase data

flow (reduce

interfaces and

double

handling)

Never pass a

known defect

downstream

Never allow

local

optimization to

cause global

degradation

Achieve a

profound

understanding

of the data eco

system

Quality control

is no longer an

area of

expertise, but

should be of

common

knowledge for

each everyone

A possible solution?

Thankfully, there’s now a simple,

paper based, tech free, silo-

busting way to better

understand your data supply

chain…

The Data Value Map…21

Data Value Map

Source: T. Nagle; D Sammon, “The Data Value Map: A Framework for developing shared understanding on data initiatives”, Twenty-Fifth European Conference on Information Systems (ECIS), Guimarães, Portugal, June 2017

A discursive template for building shared understanding around data initiatives

Closing the communication gap between data creators and data users.

Based on “The six honest men”

The DVM creates a shared understanding

Clear language

• Its makes it very easy to engage with other departments in the company as the

language is direct, clean and focused

Data discussion support

• Triggers discussion on potential for integration of other data around customer’s

interaction with us, e.g. non-digital channels

Linking different stakeholders

• This tool and the approach facilitated a better conversation between the business

and the technical areas

Shared vision

• We used the DVM to go from a discussion of “this is how we generally do

things…” to one of how can we create a “single common view”

Data Value Map - Overview

Where to start?

25

Kick off the conversation among the

data stakeholders using “Data Value

Map Question Cards”.

Together, the stakeholders develop an

agreed understanding of their data

initiative, the business value it delivers,

and pain points along the way.

The stakeholders also agree “data

behaviours” (aka “Data Governance”) to

ensure data is fit for purpose

Add people (data stakeholders)

An A0 printout of the Data Value Map,

Lots of Post-its,

A facilitator and…

The “Data Value Map Question Cards”…

Post-ItsPost-ItsPost-ItsPost-ItsPost-Its

3

5

1

Stakeholders

2

Data Value Map

4

Facilitator

Step 1 – Set the scene

It’s a collaborative

tool or discursive

template that

facilitates shared

understanding

through productive

conversations

around data

initiatives

As with all good

conversations, they

are best had with

more people than

just yourself and to

build a shared

understanding, the

more people you

have the better

No matter where

you start, if you

don’t engage and

converse with others

you will never get

anywhere near the

potential value that

the DVM has to

offer

Also, it goes without

saying, getting the

right people in the

room will provide a

much more

productive

conversation

Try and include

people from

different areas with

different roles to

facilitate a rich

discussion with

multiple

perspectives

Once you have your

room and people,

sprinkle with sticky

notes and as large a

printout of the DVM

as possible

Having an A0 works

very well as it allows

people to engage in

conversations and

discussions by

placing and moving

the sticky notes on

the DVM

Step 2 – Define the objective

The next step is to decide what

your collaborative conversation

will focus on?

These come from the time-

frames you can base

conversations around: (the to-be

or as-is state) and the depth of

conversation you want to have

(high level summary or detailed)

For the most part summary

implementations are presented

on a single DVM, tend to

synthesise differing perspectives

and highlight common trends

and priorities.

They also can act as a nice

stepping stone into identifying

more specific areas for further

(detailed) DVM investigations.

On the other hand, detailed

implementations aim to lift the

lid on a predefined focus.

Challenging all assumptions (not

taking anything for granted),

discussions can generate

multiple DVM’s to detail the

depth of the conversations.

Putting these two together, you

end up with a 2×2 matrix and

four main starting points:

• (i) new opportunities

• (ii) pain points

• (iii) operational realities

• (iv) roadmaps

Step 3 – Facilitate conversation and understanding

The next step is about

facilitating and reflecting

on constructive

discussion

Provide the opportunity

for everybody to

contribute, generate

consensus, challenge

group-think, build a

shared vision and

understanding

This step is easy for

some and difficult for

others

Use the six questions

(what, why, when….) to

highlight similarities and

differences, or classify

benefits by category

(stop, start, improve)

Do not try and impose

your own understanding

on the group but

instead let the

understanding evolve

Step 4 – Record and distribute for

feedback and reiterate

Once the conversations

are finished, record the

output

This may be in the form of

a photo or a digital

version (e.g. a slide deck),

which can be distributed

to the participants to

encourage further

feedback

The output will only

record a snapshot of the

level of understanding

created and while it may

not record all the richness

of discussion, it can be a

springboard for future

conversations and actions

Business Value – leveraging the

data assets

The business value component outlines a focus on the

business value created from data initiatives

While a focus on value is always assumed, it is not always

applied

For example, 83% of the organisations in the survey

infrequently, if ever, calculate the cost of bad data. This is

quite a big oversight when you take into account:

• (i) the estimated $3.1 trillion lost to bad data in the US per year (IBM 2016,

Redman 2016), and…

• (ii) Redman’s (2008) rule of thumb which notes, to do a task with bad data

will cost you 10 times more than to do the same task with good data.

The DVM aims to continually emphasise a focus on value

It provides a shared understanding of how data is an

organisational asset and it can benefit the organisation

through:

• Cost reduction

• Revenue generation or

• Risk mitigation

Acquisition - gathering data from

business activity…

The acquisition phase of the

DVM details the gathering of

data on business activities and

entities

A rigorous examination of the

data sources is especially

important as it uncovers

potential data issues

Many of these issues arise as

many organizations do not know:

•What data they have

•How critical it is

•Where the data is stored or…

•The degree of data redundancy

The aim is to rigorously examine

why the data needs to be

acquired.

As a result, there is a better

understanding of the value of

each the data sources

Integration – Combining data sets from

numerous sources…

The integration component of the DVM

describes how the acquired data sources

are combined?

This is an important aspect in the process

In a survey, 45% of companies stated that

because their data was being stored in

silos, it was their biggest challenge in

driving analytics investments

This visualisation of the integration aspect

of an organisations data drives an

awakening of how integrated, or indeed,

disintegrated their data actually is. And

the reality of achieving a single version of

the truth (SVOTT)

Analysis – providing analytics on

subsets of data

The data analysis component describes the

implementation of analytics on subsets of data.

By exploring the types of analytics required to generate

real business value, the aim is to achieve a more grounded

understanding of the role of analytics and reduce the risk

of mindless behaviour

Delivery - Supplying analytical results

in a suitable format

How the analytical

results are delivered in

a suitable form to its

users

The way a data user

will receive their

business data to

answer the business

questions they have

posed

For example, a data

user may want to know

the top performing

product in terms of

sales revenue

The most important

thing to the data user

is the delivery of high-

quality data in a timely

manner

Data should be

presented with a

certain amount of

visual impact so that

the story being told by

the data is easily

interpreted

The objective to gain

an understanding of

the needs of the data

users and how they

would like the data to

be delivered

Governance – Promoting mindful

routines to ensure reliability and

success…

Data Governance Underpins the four

components:

• Acquisition,

• Integration

• Analysis

• Delivery

In the DVM framework, governance is

defined as “the promotion of behaviours

for good data practice”

DVM the research has shown that data

governance is a problem area for the

majority of organizations.

DG behaviours can be categorized below:

• (i) data principles – clarifying the role of data as an

asset

• (ii) data quality – establishing the requirements of

intended data use

• (iii) metadata – establishing the semantics or

‘content’ of data so it is interpretable by users,

• (iv) data access – specifying access requirements of

data, and (v) data lifecycle – determining the

definition, production, retention and retirement of

data

The governance component aims to build

an understanding around what type of

data governance organisations have in

place, what their maturity level is, and

ensure it becomes an integral part of data

initiatives

Questions underlying the application of the Data Value Map

Data:

• Why is the data acquired/integrated/analysed/delivered?

• This ensures the underlying objectives and motivations for

doing the data task are clearly detailed.

• What data is acquired/integrated/analysed/delivered?

• This ensures the actual data involved in the operation is

defined.

• When is the data acquired/integrated/analysed/delivered?

• This ensures the temporal aspect of data is not overlooked.

Very important in keeping the quality of data high in terms of

timeliness.

People:

• Who acquires/integrates/analyses/delivers the data?

• This ensures all stakeholders are identified for all data tasks

outlined. If possible these stakeholders should be included in

discussions.

Technology:

• Where is data acquired/integrated/analysed/delivered?

• Details the data initiative technologies that are needed for

each of the data tasks.

Process:

• How is the data acquired/integrated/analysed/delivered?

• Outlines the actual processes needed to move the data from

one phase to the next.

Output Card - Value

Benefit Measure Owner

Card # Reduced BI Development Costs 60% Reduction in man hours Data Manager

# 61 Better CRM and AX adoption 20% increase in productivity Data Manager

# 62 Reduced data defects 30% reduction in scrap and

re-work

Data Manager, CRM Dev Team

# 65 Increased customer retention 10% retention improvement Sales Manager

# 64 Greater Customer Satisfaction 15% uplift in satisfaction

questionnaires

Marketing, I.T., Data Manager

# 65

Output Card - Acquisition

Data Acquisition People Technology Process

Card

#

What data is

acquired?

Why is it

acquired?

When is it

acquired?

Who is it

acquired

from?

Who

acquires it?

Where is it

acquired

from?

Where is it

stored?

How is it

acquired?

# 10 Sales Leads Records

leads

Daily Sales Dept Sales Team Sales Team Excel Manual

process

# 11 Web Orders Distributor

Orders

Daily Distributors I.T Team Website SQL Server Automated

Process

# 12 Customer Names &

Addresses

Contact

records

Ad-

Hoc/Daily

Sales Team Business CRM

Application

CRM

Database

Automated

in CRM

# 13

Output Card - Integration

Category People Technology Process

Card # What data is integrated? Why is it

integrated?

When is it

integrated?

Who integrates

it?

Where is it

integrated?

How is it

integrated?

# 10 Sales Leads Records leads in

CRM

Daily Sales Dept Sales Team ETL

# 20 Web Orders Distributor

Orders in

CRM/AX

Daily Distributors Website ETL

# 30 Customer Names &

Addresses

Contact records Ad-Hoc/Daily Sales Team CRM Application Automated in

CRM

# 40

Output Card - Analysis

Category People Technology Process

Card # What data is Analysed? Why is it

Analysed?

When is it

Analysed?

Who Analyses

it?

Where is it

Analysed?

How is it

Analysed?

# 20 Sales Leads To count and

score leads

Daily BI Team Power BI Graphically

# 21 Web Orders Distributor

Ranking

Daily BI Team Power BI Graphically

# 22 Customer Names &

Addresses

Contact RFM Ad-Hoc/Daily BI Team Power BI Graphically

# 23

Output Card - Delivery

Category People Technology Process

Card # What data is Delivered? Why is it

Delivered?

When is it

Delivered?

Who Delivers

it?

Where is it

Delivered?

How is it

Delivered?

# 31 Sales Leads To count and

score leads

Daily BI Team Power BI Drill down

dashboard

# 32 Web Orders Distributor

Ranking

Daily BI Team Power BI Drill down

dashboard

# 33 Customer Names &

Addresses

Contact RFM Ad-Hoc/Daily BI Team Power BI Drill down

dashboard

# 34

Output Card - Governance

Card # Category Behaviour Benchmark Owner

# 41 Data Principles Awareness Building Communication Plan Senior Management

Team

# 42 Data Quality Automated checks, best practices DQ Scorecard Data Manager

# 43 Metadata Develop Data Dictionaries Data Dictionaries

developed for each

category

Data Manager

# 44 Data Access Client Data defined under a set of

security policies and procedures

0 security breaches IT, Operations,

Infrastructure, Data

Manager

# 45 Data Lifecycle Formalised plan for lifecycle management Signed off plan IT, Operations,

Infrastructure, Data

Manager

# 46

DVM Success - Consider this multi-step approach

Recognise there is a

problem?

Identify

Quantify what’s specifically

doing the damage?

Quantify

Gather evidence, what,

when, where and how

large is the problem?

Qualify

Acknowledge the scale of the

task?

Accept

Define the goals and what will

be measured?

Define

Carry out the tasks agreed in

the order or significance

Perform

Final parting bits of DVM advice.

It is always worth remembering that the

value is not in the tool itself but in the

conversations it creates

So make sure you listen as much as you

talk and try and make sense of other

perspectives

Most importantly, the guide is only a

guide, so don’t make it a straight jacket

There are multiple ways to use the DVM

and once you get familiar with it, you

will find out what works for you

Data Quality Improvement

Without DQ, BI is BS

Up to 1/3 of the database could be wrong?

Duplicates, inaccurate incomplete and invalid?

Even if it contains just 50,000 records, there could be

15,000 records impacting business success?

Fighting the war on error...

Real-World Data Quality Challenges

The database where one man had been pregnant three times and

one woman had 97 children

The database where 80 percent of customers had no children

because entering 0 meant ‘Zero’ and also meant ‘don’t know’

The database where hundreds of customers had the same “unique”

Social Security Number

The database where many babies and children had mortgages

The database where 30,000 people were born on 1/1/00

The database where some customers were born in the future

How can you possibly create a successful operational or analytical CRM or

ERP environment or create Business metrics if you’re building on sand?

Some DQ Defect Examples...

Retail company

found over 1m

records contained

home telephone

number of

“000000000”

Airline found

addresses containing

flight numbers

Insurance company

found customer

records with

99/99/99 in creation

date field of policy

Car rental company

discovered duplicate

agreement numbers

in their European

data warehouse

Healthcare company

found 9 different

values in gender

field

Food/Beverage retail

chain found the

same product was

their No 1 and No 2

best sellers across

their business

The Dirty Data Diseases

Corporate Alzheimer’s

• A disease where no one can remember who's responsible for data quality or why?

Obese data

• More than you should have - duplicates, deads and gone-away records bloating the file

Corporate Anxiety and Stress

• A nervous complaint - where you are unable to trust your data

Chronic copying

• An unnecessary desire to copy and store the same records in multiple locations

Data Haemophilia

• Just can't stop the loss of data and money that's pouring out of the business

Halitosis

• The data just stinks

The Great Data Quality Delusion

Everyone

understands the

importance of data

quality

Everyone agrees

data quality is

important

Everyone cares

about data quality

Everyone knows

what actions to take

to improve data

quality

Simplifying and Unravelling Complex DQ Processes

Suppress

Parse

Format

Validate

Verify

Authenticate

Enhance

Match

SRV

MRM

Decisions

Rules

Creation the Golden Record

DQ Innovation - Traffic light indicator

Objective Green Amber Red

How many people understand the DQ goals of the organisation? Everyone Most Few or None

How often do we hold idea DQ improvement generation meetings? More than once a

month

A few times a year Never

Do we set DQ goals for ideas initiatives and innovation? Yes – everyone has

them

A few No – none

What is our attitude to data risk management? Aware - We are risk

averse

We try risks

sometimes

Risks are fine – we

manage them

Do we look for DQ improvement outside our organisation? We often look

outside for ideas

We sometimes look

outside

We rarely look outside

for ideas

Do we encourage, recognise and reward internal DQ improvement ideas? Yes Somewhat No

Can anyone question or challenge anything DQ improvement and IQ

improvement related?

People can question

and they do

People sometimes

challenge

The boss knows best

Do we try prototypes and pilots to test our DQ ideas We’re always trying

new things

Sometimes we try

new things

We rarely try new things

Actions you can take to reduce duplication

User training

Set guidelines

Define picklists

(Option Sets)

Avoid free form

Values

Use duplicate detection

rules

Clean from the top down

Use Dynamics duplicate detection

Use DQ Perfect &

Merge

Data is your Differentiator

Better Data

Better

Information

Better

Business

“IF YOU REALLY WANT

TO DO SOMETHING,

YOU WILL FIND A WAY.

IF YOU DON’T YOU’LL

FIND AN EXCUSE”

Jim Rohn

Questions…

Build a better business based on trusted data…

Talk to a consultant

• www.DQGlobal.com

• sales@DQGlobal.com

• +44 2392 988303 (Europe)

• +1 314-253-7873 (North America)

Disclaimer

Solutions to improve

the quality of data in

your CRM (CE) and

ERP (F&O)

How Perfect & Merge

deals with duplicates

for Dynamics 365Master Record Management

Considerations - Matching & Mastering of CRM Records?

Problems:

Advanced

Matching?

Which

record is

Best?

Which

data

survives?

•Field

Groups?

•Field Values?

Merging

of Notes?

Suppress

false

positives?

Solutions:

Automate

processesHumanise

exceptions

Define the

master

record

Define the

field merge

rules

Perfect & Merge Overview

Single Entity

deduplication

X-Entity

Matching

Real time review Multiple record

review

Winning (best)

record selection

Merge data

from duplicates

Automate

processes

Suppress false

positivesJump to CRM

record

Secure Scheduled Cloud Service

Perfect & Merge – Session Results

Perfect & Merge Duplicate Review Screen

Multiple Duplicate Review

Promote a Record to Master

Promote Fields to the Master

Defer to another user

Suppress Non-Duplicates

Auto-promote, Auto-Fill & Auto-Merge

Post Merge Rules for related items

Rules for… “Best Record” Detection

Rules for…“Best Field Group” and “Best Field”

Match Session Definition

International language capabilities

Account, Contact or Lead

X-Entity Matching

Personal or System Views

Templated from base definitions

Field Groups - Weightings and Phonetics

Fields group members

Priority Weighting

Phonetic Algorithms

Consider for Matching/Scoring

Transformation Rules

Multiple Transformation Rules

Processing Sequences

Review in test window

Modify from templated rules

Data Transformation Rules

Visual Rules Testing

13 Transformation Categories

5 spoken Languages

6 Phonetic Algorithms

Custom Transformations

Match Scoring and Rules

% Match thresholds

Contacts at an Account true/false

Scoring NULL comparisons

Scoring Value to NULL

What the reviewer sees…

Perfect & Merge for Dynamics 365

Simplicity

Managed Plug-In which is quick

and easy to set up, and works

with any deployment:

• On-Premise

• On-Line

• Internet-Facing (IFD)

Application Integration Technology

A powerful all-in-one solution that lets you tackle your Data

Management projects with ease.

It’s user friendly design seamlessly integrates a powerful business

rules engine, data migration and integration tool, built in process

automation, data cleansing and other capabilities to provide a

world class solution to your data management challenges.

appRules Portal™ has over 100 connectors to a host of business

applications, including: CRM, ERP and Accounting solutions as

well as connectors to social media, big data, spreadsheets, files

and both on-premise and cloud hosted databases.

Integrate | Automate | Consolidate | Cleanse

100+ Connectors

including:

• CRM

• ERP

• Marketing

• Azure

• SharePoint

• SQLServer

• Access

• Excel

Hundreds of

activities for easy

assembly of

workflows

Configuration not

deep customisation

or code

Integrated Data

Quality

Master Record

Management

Integrate | Automate | Consolidate | Cleanse

100+ Connectors for Integration

Your System – You Configure the Workflows

Validate Phone – Workflow example

Rules – Decision Tables and Rule Sets

Best Record & Best Field Determination

DQ Capabilities – in 5 spoken languages

ClassifyCompare

SimilarityFormat

Generate

•Phonetic Tokens

•Patterns)Case Transform

Validate

Verify

•Addresses

•People

•Companies

Parse

Master Record

Merge

• Best Record

• Best Fields

Fuzzy Search

String

Management

Congruence

Checks

Data Value Mapping

Promoting Behaviours which ensure

Higher Quality Data

Presenter

Martin Doyle

Data is your Differentiator

Better Data

Better

Information

Better

Business

“IF YOU REALLY WANT

TO DO SOMETHING,

YOU WILL FIND A WAY.

IF YOU DON’T YOU’LL

FIND AN EXCUSE”

Jim Rohn

Questions…

Build a better business based on trusted data…

Talk to a consultant

• www.DQGlobal.com

• sales@DQGlobal.com

• +44 2392 988303 (Europe)

• +1 314-253-7873 (North America)