lean data architecture: minimize investment, maximize value brandon byars bbyars@thoughtworks.com...

Lean Data Architecture:Minimize Investment,

Maximize Value

Brandon Byarsbbyars@thoughtworks.com@BrandonByars

Manvir Grewalmgrewal@thoughtworks.com@ManvirGrewal

Brave New World

Every 2 days we create as much informationas we did from the dawn of civilization to 2003

- Eric Schmidt, 2011

Big Data• Volume• Velocity• Variety

NoSQL•Schemaless•Non-relational

Future, Backwards

Data Platform

Support variety of analytics/users

Ubiquitous access

Logicwolf.com

Evolves as business changes

Sound good…So what's stopping you?

Companies lose $8.2 Million annually due to bad data – Gartner

Study estimated total cost to US economy of data quality problems at over US$600 billion per annum - Wayne Eckerson

SpreadMarts

Didn’t test well enough…

Upfront and Inflexible design

Vk.com

Bring everything, we may need it

Manual Testing is good for…

Not good for data or software

Spreadsheet comparisons…over and over!

Where are your bottlenecks?

The Data Focus Problem

"Data warehousing and BI is one of the largest expenses in CIO budgets, and to-date has had limited value to the business."

- Jeff Smith, CEO Shared Services, Suncorp

Gartner estimates that between 70-80% of BI projects are failures

How do we get to the future?

Wallchan.com

App Analysis Modeling DBA ETL Testing Report

2 weeks4 weeks

2 weeks4 weeks

2 weeks4 weeks

2 weeks4 weeks

2 weeks4 weeks

2 weeks4 weeks

2 weeks

Customer

Value Streams

App Analysis Modeling

Systems Thinking

How long would it take you to add a single column to a report?

From Idea to Prod…

Muda

Eight Wastes

Logismarket.de

BI/DW: Status Quo

Data

Lean BI: Value First

value

Cost & Time

100%

100%

MVP

Sometimes we accidentally build something that nobody wants, in which case it doesn’t matter if we do it on time and on budget

Prototype reportDashboard populated with static dataRudimentary scoring model

Reduce Batch Size

Dev2ops.org

Reduce WIP

Work In Progress

L = λW

Simplify Architecture – Message Based

ReportsListenerBus

DB

DB

DB DB

Automated Testing

How do we test?

Test data set based on profilingVolume is not necessaryVariety is necessaryVariability/Complexity is necessaryVelocity is necessary

System testingPlay transactions from sources in UAT/pre-prod environment

Developer Sandboxes

Automated Build

Continuous Integration

References

Data Explosion is coming!