agile data governance tutorial

Agile Data Governance

Tami FlowersDirector, Governance SolutionsMetaGovernance Solutions, LLC

DGIQ June 2016

Who am I?

Tami Flowers Director, Governance Solutions at MetaGovernance PMI-PMP, PMI-ACP, and Certified Scrum Master Agile Coach

Developer Project Manager

Enterprise Architect

Data Governance Consultant

MetaGovernance Solutions LLC. All Rights Reserved. 2016

Today

Data Governance Agile How to meld them together

A few minutes on the topic An “experience” Debrief/Reflection


Experience: Ice Breaker1. Write your answer to the following questions on

sticky notes. One answer per note. As many answers as you wish. What are you hoping to learn today? What specific questions are you hoping get answered

today?2. Discuss amongst your team. Group like answers

together.3. Have each person mark their level of experience

with the following:


No experience

Some experience

Quite a bit of experience

AgileData Governance

Data Governance

Two viewpoints Establishing a Data Governance Organization

Framework Implementing Data Governance actionable

Data Governance


Data Management Functions

Data Governance

Data Architecture

Data Development

Data Operations

Data Security

Reference and Master

DataEDW and BI

Document & Content

Metadata

Data Quality

Data Governance is at the heart of all Data Management activities. As defined by the Data Management Association – Data Management Body of Knowledge (DAMA-DMBOK)


Data Governance Framework

Data Governance OrganizationData Management FunctionsProjects

Agile Data Governance


Data Governance Organization Framework

Executives

Business & Technical Members

Stewards/analysts/architects/ administrators/engineers

Consumers

Data Governance Council (DGC)

Data Governance Steering Committee

(DGSC)

Data Stewardship Teams (virtual)

• Establish & train DGC members• Agree on Data Governance roadmap• Determine members & empower DGSC• Define Data Governance policy• Meet on recurring basis

• Receive training and education• Execute DG roadmap• Define DG communication plan• Select & empower stewardship

teams• Define roles and responsibilities• Define charter• Meet on recurring basis

• Technical and Business Members• Receive training and education• Execute DG roadmap

• Business Members • Receive training and

educationEnd Users


Data Governance Organization Framework Actionable Data Governance

Executives

Business & Technical Members

Stewards/analysts/architects/ administrators/engineers

Consumers

Data Governance Council (DGC)

Data Governance Steering Committee

(DGSC)

Data Stewardship Teams (virtual)

• Lead and promote program• Empower DGSC• Arbitrate data related issues, as needed

• Execute DG communication plan• Define data management policies,

procedures and standards• Define stakeholders, and sources and use of

data per subject area• Approve and oversee corrective actions

• Technical and Business Members• Implement policies, procedures,

standards and controls• Identify and report data issues• Corrective action input & execution• Corrective action status reporting

• Business Members that consume and use data

• Identify and report data issues

End Users


Registered Data Governance Stakeholders

Data Subject Area ID

Major Subject Group 1st Sub Grouping 2nd Sub Grouping A

ccou

ntin

g O

pera

tions

Col

late

ral

Cre

dit R

isk

Fina

nce

Fina

ncia

l Rep

ortin

g

Inte

rnal

Aud

it

Mar

ket R

isk

Sale

s

Loan

Ope

ratio

ns

Com

plia

nce

Info

rmat

ion

Tech

nolo

gy

1 Trade Loan Adjustable-Rate Loan

2 Trade Loan Fixed-Rate Loan

3 Trade Loan Structured Loan

4 Trade Loan Amortizing Loan

5 Trade Loan Letters of Credit

6 Trade Loan Prepayment Fee

11 Financial Accounting Accrued Interest

12 Financial Accounting Outstanding Principle

LEGEND

CUSTODIAN

CONSUMERSTEWARDDELEGATE


The Agile Manifesto

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over Processes and toolsWorking software over Comprehensive documentationCustomer collaboration over Contract negotiationResponding to change over Following a plan

That is, while there is value in the items on the right, we value the items on the left more.

In February 2001, 17 software developers met at the Snowbird resort in Utah to discuss lightweight development methods. They published the Manifesto for Agile Software Development.


Agile Principles1. Our highest priority is to satisfy the customer

through early and continuous deliveryof valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.


Agile Principles6. The most efficient and effective method of

conveying information to and within a development team is face-to-face conversation.

7. Working software is the primary measure of progress.8. Agile processes promote sustainable development.

The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10.Simplicity--the art of maximizing the amount of work not done--is essential.

11.The best architectures, requirements, and designs emerge from self-organizing teams.

12.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.


Experience: Agile Principles

How can we apply these principles to establishing a data governance framework

How can we apply these principles to actionable data governance (day-to-day governance)?


The many flavors


Scrum


Lean Software Development

Eliminate Waste

Empower the Team

Deliver Fast

Optimize the

Whole

Build Quality

In

Defer Decision

s

Amplify Learning


Experience: Agile Practices to Lean Principles

Agile Practice/Lean Principle

Eliminate Waste

Empower the Team

Deliver Fast

Optimize the Whole

Build Quality in

Defer Decisions

Amplify Learning

Teams make own decisionsJust in time planning

Team retrospectives

Two week iterations

Unit test as we go

Shadow the business to learn what they doMetaGovernance Solutions LLC. All Rights Reserved. 2016

Experience: Data Governance to Lean Principles

Agile Practice/Lean Principle

Eliminate Waste

Empower the Team

Deliver Fast

Optimize the Whole

Build Quality in

Defer Decisions

Amplify Learning

Teams make decisions

Just in time planning

Retrospectives

Frequent iterations

Validate/approve as we goInvolve the business stakeholders MetaGovernance Solutions LLC. All Rights Reserved. 2016

Kanban Capture the work Make it visible Each story is about the same size Limit the work in progress (WIP)


Experience: WIP

8-10 people per team 4 workers – will do the work 4 managers – will watch over their worker 1-2 timers – will time how long each round

takes

This is called the Penny Game and is available on www.tastycupcakes.org


Commonalities of all Agile/Lean flavors

Capture the work Make it visible Minimally Viable Product

(MVP) Engage the owner/customer/

stakeholders Have owner prioritize the

work Work on the highest priority

work first Iterate Continually retrospect and

improve


Roles Stakeholders

Product owner is one All have a stake in the project DGSC members

Product Owner Chairperson of DGSC Has the vision and can convey it to the team and stakeholders Prioritizes backlog Provides direction to and is engaged with the team

Scrum Master – facilitator Bulldozer: Remove roadblocks Servant leader: Do anything needed to help team perform at their

highest level Stewardship Team Roles

Stewards Developers/Engineers/Architects/Administrators Business Analysts/Quality Assurance Analysts


CeremoniesCeremonies Traditional

Agile/Lean Project

Data Governance Framework

Actionable Data Governance

Planning Meetings

Start of every sprint/iteration

DGSC or subgroup meetings

Start of every sprint/iteration

Scrums/Stand Ups

Daily, 15 minutes

1-3 times a week

Daily, 15 minutes

Review meetings

End of every scrum/iteration

DGSC or subgroup meetings

End of every sprint/iteration

Retrospectives



End of every sprint/ iteration

Backlog Grooming

In the week between sprints/iterations

In between sprints/ iterations

In the week between sprints/iterationsMetaGovernance Solutions LLC. All Rights Reserved. 2016

4 quadrants to help with backlog grooming

Broccoli Organic Beef

Twinkies Éclair

Effort

Valu

e

One of our classmates shared this quadrant with us during a discussion. They use this in their organization to help discuss the value of a story in the backlog and it helps them select stories for the sprint/iteration.


ArtifactsArtifacts Traditional

Agile/Lean Data Governance Framework

Actionable Data Governance

Product Backlog

Created from conversations with stakeholders

Created from Data Governance Roadmap

Created from DG policies and procedures

Sprint/Iteration Backlog

Prioritized by product owner during planning meeting

Prioritized by DGSC chairperson during planning meeting

Prioritized by product owner during planning meeting

Stories Created from conversations with stakeholders

Created from Data Governance Roadmap

Created from DG policies and procedures


Data Governance artifacts within a project Stories

As a <role> I want <what> so that <business value> If not a story, include data governance as

acceptance criteria in stories As an accountant, I want current balances for loans, so

that I can complete daily reporting All data elements are defined in the business dictionary Data profiling of the balances occurs when loading into

the EDW An automated control is added so the current balance is

automatically reconciled between the G/L and Loan System


Reality Check

Agile isn’t a silver bullet that will magically fix everything New, unexpected challenges will occur

Teamwork Soft skills Growing pains


Reality Check

Data Governance isn’t a silver bullet that will be easy to quickly implement Must include the business Must be added to existing processes to be

successful Lots of training and education is needed Executive support is critical Will uncover data issues that some don’t want

uncovered


Experience: Adding Governance to a Project Identify what is needed to add governance to a

project. What 2-3 items are the highest priority?

Project: bring new data source in Needs: Ensure all registered stakeholders and their

data governance roles are known. Identify and define business attributes. Profile data for potential issues.


Group experience: What stories are needed if we are bringing in a new data source? Identify data quality dimensions and thresholds Identify archiving and data retention needs Classify the data for visibility and security Identify data stewards Establish profiling and cleansing rules Establish SLAs with the provider Identify legal aspects, regulatory impacts Identify restrictions on the data, permissible use Evaluate risk involved with data Determine requirements to match to master or reference data; identity

attributes Determine upwards and downward impacts to business processes Configuration management; hardware/software needed, documentation Define metadata, definitions, associations.


Registered Data Governance Stakeholders (we also want to identify and record this metadata each iteration/sprint)

Data Subject Area ID

Major Subject Group 1st Sub Grouping 2nd Sub Grouping A

ccou

ntin

g O

pera

tions

Col

late

ral

Cre

dit R

isk

Fina

nce

Fina

ncia

l Rep

ortin

g

Inte

rnal

Aud

it

Mar

ket R

isk

Sale

s

Loan

Ope

ratio

ns

Com

plia

nce

Info

rmat

ion

Tech

nolo

gy

1 Trade Loan Adjustable-Rate Loan

2 Trade Loan Fixed-Rate Loan

3 Trade Loan Structured Loan

4 Trade Loan Amortizing Loan

5 Trade Loan Letters of Credit

6 Trade Loan Prepayment Fee

11 Financial Accounting Accrued Interest

12 Financial Accounting Outstanding Principle

LEGEND

CUSTODIAN

CONSUMERSTEWARDDELEGATE


Data Governance (Overall Framework)

Provide data governance common vocabularyDefine key governance rolesDefine roles & responsibilities of key gov rolesDefine data governance policyDefine concept of data governance controls

Project Specific Data Governance –Risk Project

Confirm data subject areas within project scopeConfirm system of record for each subject areaDefine data quality issues and standardsDefine data quality testing & remediation strategyConfirm data gov roles for each subject areaDetermine existing data controls for each sub areaDetermine recommended data gov controlsDefine recommended data gov control & quality implementation priorityObtain DGWG approval

FEB MAR APR MAY JUN JULJAN AUG SEP

Risk Project Timeline (sample timeline)


Experience: Setting up a Governance Framework Identify what is needed to setup a governance

framework. Generally this is done by looking at the data governance roadmap (see next slide for example)

Have the product owner prioritize the first 3 items.


Data Governance (DG) Framework

Establish Data Governance Council (DGC)- Identify and appoint members- Train and educate DGC- Agree on common data governance vocabulary- Agree on common DG stakeholder roles- Agree on roles and responsibilities for

stakeholders- Agree on scope, authority and structure of DGC- Agree on initial DGSC members- Define DGC charter and guiding principles- Agree on key metrics to evaluate governance- Participate in recurring DGC meetings- Champion and promote DGEstablish Data Governance Steering Committee- Identify and appoint members- Train and educate DGSC- Define common data governance vocabulary- Define key DG stakeholder roles- Define roles and responsibilities for DG roles- Define scope, authority and structure of DGSC- Identify and appoint data stewards- Agree on DGSC charter and guiding principles- Define objectives of DGSC- Identify Governance Coordinator for DG

activities- Participate in recurring DGSC meetings - Champion, communicate and promote DG

Data Governance Framework Sample TimelineQ1 Q2 Q3 Q4 Q5 Q6 Q7 Q8


Ways Agile can help

Increased communication and visibility Increased teamwork More frequent delivery of business value Implement governance on high priority items first,

show success, move on…continuous improvement


Continuous improvement

Use retrospectives to make your process better It’s ok to add just 1 or 2 new things at a time Get help if you need it, lots of resources and experts

are out there Mentoring and coaching will be critical


Experience: Temperature Reading Appreciations Puzzles Complaints with recommendations Hopes and wishes New Information


Appreciations

Puzzles

Complaints w/Recommendations

Hopes &Wishes

New Information

Retrospectives

At end of each sprint/iteration What went well – keep doing What didn’t go well – stop doing What could go better – change it

Entire team attends


Experience: Retrospective

What went well What didn’t go well What could go better


Lessons learned

There isn’t one definition of Agile; all companies tend to do it differently. That’s ok. Make it what works best for your organization.

If you are setting up governance for the first time, get the framework in place first. Then determine low hanging fruit, such as data quality issues, and implement governance starting there.

Data Governance MUST involve the business; it can’t just be an IT focus.

A lot of training and education is needed for governance. Data Governance must be added to existing processes

and procedures in order to give it teeth. SDLC Architecture reviews


Good resources for Agile and Team Building Mike Cohn – MountainGoatSoftware.com PMI-ACP Exam Prep – Mike Griffiths Johanna Rothman books: http://www.jrothman.com/books/ The Human Side of Agile, Gil Broza Agile Retrospectives: Making Good Teams Great, Esther Derby and Diana

Larsen Crucial Conversations, Kerry Patterson Crucial Confrontations, Kerry Patterson Influencer The Deadline: A Novel about Project Management, Tom DeMarco Teamwork is an Individual Skill, Christopher Avery Behind Closed Doors: Secrets of Great Management, Johanna Rothman and

Esther Derby DGIQ presentations: Roberta, Maggie from GM


http://www.jrothman.com/books/

Me

Twitter: TamiLFlowers LinkedIn: Tami Flowers Email: [email protected] Slideshare: www.slideshare.net\tamiflowers www.metagovernance.com (blog) Blog for DataVersity

Thanks!


http://www.slideshare.net/tamiflowers

http://www.metagovernance.com/

agile data governance tutorial

Data & Analytics